2024-08-31T12:10:47.400 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-08-31T12:10:47.504 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-31T12:10:47.600 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/mchangir-2024-08-30_08:30:37-fs-wip-mchangir-testing-20240828.085847-main-debug-distro-default-smithi/7881510 branch: wip-mchangir-testing-20240828.085847-main-debug description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} email: mchangir@redhat.com first_in_suite: false job_id: '7881510' kernel: kdb: 1 sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: mchangir-2024-08-30_08:30:37-fs-wip-mchangir-testing-20240828.085847-main-debug-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-mchangir-testing-20240828.085847-main-debug ceph: cephfs: max_mds: 3 session_timeout: 300 cluster-conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 2 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_dir_prefetch: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mon: mon op complaint time: 120 mon warn on pool no app: false osd: osd op complaint time: 180 conf: client: client_shutdown_timeout: 120 mds: mds oft prefetch dirfrags: true mds_sleep_rank_change: 5000000.0 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 filestore flush min: 0 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - fs.*is degraded - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - insufficient standby MDS daemons available - MDS_UP_LESS_THAN_MAX - online, but wants - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - SLOW_OPS - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - noscrub - nodeep-scrub - overall HEALTH_ - \(MON_DOWN\) - SLOW_OPS - slow request sha1: d19e6118f7015f0047264006cb0c873528eb3ae6 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs check-counter: counters: mds: - mds.exported - mds.imported install: ceph: flavor: default sha1: d19e6118f7015f0047264006cb0c873528eb3ae6 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-mchangir-testing-20240828.085847-main-debug sha1: d19e6118f7015f0047264006cb0c873528eb3ae6 owner: scheduled_mchangir@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - mds.e - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - mds.f - osd.4 - osd.5 - osd.6 - osd.7 seed: 5413 sha1: d19e6118f7015f0047264006cb0c873528eb3ae6 sleep_before_teardown: 0 subset: 347/512 suite: fs suite_branch: wip-mchangir-testing-20240828.085847-main-debug suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_d19e6118f7015f0047264006cb0c873528eb3ae6/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: d19e6118f7015f0047264006cb0c873528eb3ae6 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi019.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB/HQNvJy60UtVNzln9ZKUAdsLVpf8jpDF6Rxh2Wh23CL/UTLoV32SrzlNCVqrzMxhndMNyIAOUOz33w/YNfubs= smithi043.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDheWTOfQHsyDOKQ8feiqxC7FBApnpQOewzW8s5nX1GD1XY6B8MYuEynuH3Z3A0czLPADAaOh0nF44SrB9+8GbM= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - ceph-fuse: null - mon_thrash: check_mds_failover: true revive_delay: 20 thrash_delay: 10 - workunit: clients: all: - suites/ffsb.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-08-30_08:30:37 tube: smithi user: mchangir verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-08-31T12:10:47.600 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_d19e6118f7015f0047264006cb0c873528eb3ae6/qa; will attempt to use it 2024-08-31T12:10:47.601 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_d19e6118f7015f0047264006cb0c873528eb3ae6/qa/tasks 2024-08-31T12:10:47.601 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-08-31T12:10:47.603 INFO:teuthology.task.internal:Checking packages... 2024-08-31T12:10:47.624 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd19e6118f7015f0047264006cb0c873528eb3ae6' 2024-08-31T12:10:47.624 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-31T12:10:47.624 INFO:teuthology.packaging:ref: None 2024-08-31T12:10:47.624 INFO:teuthology.packaging:tag: None 2024-08-31T12:10:47.624 INFO:teuthology.packaging:branch: wip-mchangir-testing-20240828.085847-main-debug 2024-08-31T12:10:47.624 INFO:teuthology.packaging:sha1: d19e6118f7015f0047264006cb0c873528eb3ae6 2024-08-31T12:10:47.625 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-mchangir-testing-20240828.085847-main-debug 2024-08-31T12:10:47.763 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4560.gd19e6118 2024-08-31T12:10:47.765 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-08-31T12:10:47.772 INFO:teuthology.task.internal:no buildpackages task found 2024-08-31T12:10:47.772 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-08-31T12:10:47.816 INFO:teuthology.task.internal:Saving configuration 2024-08-31T12:10:47.834 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-08-31T12:10:47.852 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-08-31T12:10:47.874 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi019.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-08-30_08:30:37-fs-wip-mchangir-testing-20240828.085847-main-debug-distro-default-smithi/7881510', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-31 12:06:01.479031', 'locked_by': 'scheduled_mchangir@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB/HQNvJy60UtVNzln9ZKUAdsLVpf8jpDF6Rxh2Wh23CL/UTLoV32SrzlNCVqrzMxhndMNyIAOUOz33w/YNfubs='} 2024-08-31T12:10:47.897 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi043.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-08-30_08:30:37-fs-wip-mchangir-testing-20240828.085847-main-debug-distro-default-smithi/7881510', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-31 12:06:01.480970', 'locked_by': 'scheduled_mchangir@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDheWTOfQHsyDOKQ8feiqxC7FBApnpQOewzW8s5nX1GD1XY6B8MYuEynuH3Z3A0czLPADAaOh0nF44SrB9+8GbM='} 2024-08-31T12:10:47.897 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-08-31T12:10:47.905 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-08-31T12:10:47.905 INFO:teuthology.task.internal:roles: ubuntu@smithi043.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-08-31T12:10:47.905 INFO:teuthology.run_tasks:Running task console_log... 2024-08-31T12:10:47.972 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4984c8aef0>, signals=[15]) 2024-08-31T12:10:47.972 INFO:teuthology.run_tasks:Running task internal.connect... 2024-08-31T12:10:47.988 INFO:teuthology.task.internal:Opening connections... 2024-08-31T12:10:47.988 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi019.front.sepia.ceph.com 2024-08-31T12:10:47.990 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-31T12:10:48.075 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi043.front.sepia.ceph.com 2024-08-31T12:10:48.076 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-31T12:10:48.151 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-08-31T12:10:48.160 DEBUG:teuthology.orchestra.run.smithi019:> uname -m 2024-08-31T12:10:48.179 INFO:teuthology.orchestra.run.smithi019.stdout:x86_64 2024-08-31T12:10:48.179 DEBUG:teuthology.orchestra.run.smithi019:> cat /etc/os-release 2024-08-31T12:10:48.236 INFO:teuthology.orchestra.run.smithi019.stdout:NAME="CentOS Stream" 2024-08-31T12:10:48.236 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION="9" 2024-08-31T12:10:48.236 INFO:teuthology.orchestra.run.smithi019.stdout:ID="centos" 2024-08-31T12:10:48.236 INFO:teuthology.orchestra.run.smithi019.stdout:ID_LIKE="rhel fedora" 2024-08-31T12:10:48.236 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION_ID="9" 2024-08-31T12:10:48.236 INFO:teuthology.orchestra.run.smithi019.stdout:PLATFORM_ID="platform:el9" 2024-08-31T12:10:48.236 INFO:teuthology.orchestra.run.smithi019.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-31T12:10:48.236 INFO:teuthology.orchestra.run.smithi019.stdout:ANSI_COLOR="0;31" 2024-08-31T12:10:48.236 INFO:teuthology.orchestra.run.smithi019.stdout:LOGO="fedora-logo-icon" 2024-08-31T12:10:48.236 INFO:teuthology.orchestra.run.smithi019.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-31T12:10:48.237 INFO:teuthology.orchestra.run.smithi019.stdout:HOME_URL="https://centos.org/" 2024-08-31T12:10:48.237 INFO:teuthology.orchestra.run.smithi019.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-08-31T12:10:48.237 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-31T12:10:48.237 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-31T12:10:48.237 INFO:teuthology.lock.ops:Updating smithi019.front.sepia.ceph.com on lock server 2024-08-31T12:10:48.254 DEBUG:teuthology.orchestra.run.smithi043:> uname -m 2024-08-31T12:10:48.272 INFO:teuthology.orchestra.run.smithi043.stdout:x86_64 2024-08-31T12:10:48.272 DEBUG:teuthology.orchestra.run.smithi043:> cat /etc/os-release 2024-08-31T12:10:48.330 INFO:teuthology.orchestra.run.smithi043.stdout:NAME="CentOS Stream" 2024-08-31T12:10:48.330 INFO:teuthology.orchestra.run.smithi043.stdout:VERSION="9" 2024-08-31T12:10:48.330 INFO:teuthology.orchestra.run.smithi043.stdout:ID="centos" 2024-08-31T12:10:48.330 INFO:teuthology.orchestra.run.smithi043.stdout:ID_LIKE="rhel fedora" 2024-08-31T12:10:48.330 INFO:teuthology.orchestra.run.smithi043.stdout:VERSION_ID="9" 2024-08-31T12:10:48.330 INFO:teuthology.orchestra.run.smithi043.stdout:PLATFORM_ID="platform:el9" 2024-08-31T12:10:48.330 INFO:teuthology.orchestra.run.smithi043.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-31T12:10:48.330 INFO:teuthology.orchestra.run.smithi043.stdout:ANSI_COLOR="0;31" 2024-08-31T12:10:48.330 INFO:teuthology.orchestra.run.smithi043.stdout:LOGO="fedora-logo-icon" 2024-08-31T12:10:48.331 INFO:teuthology.orchestra.run.smithi043.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-31T12:10:48.331 INFO:teuthology.orchestra.run.smithi043.stdout:HOME_URL="https://centos.org/" 2024-08-31T12:10:48.331 INFO:teuthology.orchestra.run.smithi043.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-08-31T12:10:48.331 INFO:teuthology.orchestra.run.smithi043.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-31T12:10:48.331 INFO:teuthology.orchestra.run.smithi043.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-31T12:10:48.331 INFO:teuthology.lock.ops:Updating smithi043.front.sepia.ceph.com on lock server 2024-08-31T12:10:48.354 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-08-31T12:10:48.363 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-08-31T12:10:48.463 INFO:teuthology.task.internal:Checking for old test directory... 2024-08-31T12:10:48.463 DEBUG:teuthology.orchestra.run.smithi019:> test '!' -e /home/ubuntu/cephtest 2024-08-31T12:10:48.466 DEBUG:teuthology.orchestra.run.smithi043:> test '!' -e /home/ubuntu/cephtest 2024-08-31T12:10:48.483 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-08-31T12:10:48.493 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-08-31T12:10:48.494 DEBUG:teuthology.orchestra.run.smithi019:> test -z $(ls -A /var/lib/ceph) 2024-08-31T12:10:48.523 DEBUG:teuthology.orchestra.run.smithi043:> test -z $(ls -A /var/lib/ceph) 2024-08-31T12:10:48.550 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-08-31T12:10:48.612 INFO:teuthology.run_tasks:Running task kernel... 2024-08-31T12:10:48.642 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-08-31T12:10:48.642 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-08-31T12:10:48.643 DEBUG:teuthology.orchestra.run.smithi019:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-31T12:10:48.643 DEBUG:teuthology.orchestra.run.smithi043:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-31T12:10:48.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T12:10:48.660 DEBUG:teuthology.orchestra.run.smithi019:> uname -r 2024-08-31T12:10:48.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T12:10:48.661 DEBUG:teuthology.orchestra.run.smithi043:> uname -r 2024-08-31T12:10:48.718 INFO:teuthology.orchestra.run.smithi019.stdout:5.14.0-503.el9.x86_64 2024-08-31T12:10:48.719 INFO:teuthology.task.kernel:Running kernel on smithi019: 5.14.0-503.el9.x86_64 2024-08-31T12:10:48.719 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum install -y kernel 2024-08-31T12:10:48.720 INFO:teuthology.orchestra.run.smithi043.stdout:5.14.0-503.el9.x86_64 2024-08-31T12:10:48.720 INFO:teuthology.task.kernel:Running kernel on smithi043: 5.14.0-503.el9.x86_64 2024-08-31T12:10:48.720 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum install -y kernel 2024-08-31T12:10:51.669 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - BaseOS 6.3 MB/s | 8.2 MB 00:01 2024-08-31T12:10:52.364 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - BaseOS 4.3 MB/s | 8.2 MB 00:01 2024-08-31T12:10:54.557 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - AppStream 21 MB/s | 20 MB 00:00 2024-08-31T12:10:54.862 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - AppStream 10 MB/s | 20 MB 00:02 2024-08-31T12:10:59.191 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.6 MB 00:00 2024-08-31T12:11:00.195 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - CRB 4.9 MB/s | 6.6 MB 00:01 2024-08-31T12:11:00.850 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - Extras packages 46 kB/s | 18 kB 00:00 2024-08-31T12:11:01.596 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - Extras packages 128 kB/s | 18 kB 00:00 2024-08-31T12:11:02.020 INFO:teuthology.orchestra.run.smithi043.stdout:Extra Packages for Enterprise Linux 22 MB/s | 22 MB 00:01 2024-08-31T12:11:02.392 INFO:teuthology.orchestra.run.smithi019.stdout:Extra Packages for Enterprise Linux 34 MB/s | 22 MB 00:00 2024-08-31T12:11:07.651 INFO:teuthology.orchestra.run.smithi043.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-08-31T12:11:08.073 INFO:teuthology.orchestra.run.smithi019.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-08-31T12:11:09.733 INFO:teuthology.orchestra.run.smithi043.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-08-31T12:11:09.733 INFO:teuthology.orchestra.run.smithi043.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-08-31T12:11:09.818 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-08-31T12:11:09.819 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2024-08-31T12:11:09.819 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-08-31T12:11:09.923 DEBUG:teuthology.orchestra.run.smithi043:> echo no | sudo yum reinstall kernel || true 2024-08-31T12:11:10.229 INFO:teuthology.orchestra.run.smithi019.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-08-31T12:11:10.229 INFO:teuthology.orchestra.run.smithi019.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-08-31T12:11:10.319 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-08-31T12:11:10.320 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-08-31T12:11:10.320 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-08-31T12:11:10.427 DEBUG:teuthology.orchestra.run.smithi019:> echo no | sudo yum reinstall kernel || true 2024-08-31T12:11:10.473 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:00:03 ago on Sat 31 Aug 2024 12:11:07 PM UTC. 2024-08-31T12:11:10.630 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-08-31T12:11:10.635 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-08-31T12:11:10.635 INFO:teuthology.orchestra.run.smithi043.stdout: Package Architecture Version Repository Size 2024-08-31T12:11:10.635 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-08-31T12:11:10.635 INFO:teuthology.orchestra.run.smithi043.stdout:Reinstalling: 2024-08-31T12:11:10.635 INFO:teuthology.orchestra.run.smithi043.stdout: kernel x86_64 5.14.0-503.el9 baseos 1.9 M 2024-08-31T12:11:10.635 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:11:10.635 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-08-31T12:11:10.635 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-08-31T12:11:10.635 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:11:10.636 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 1.9 M 2024-08-31T12:11:10.636 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 0 2024-08-31T12:11:10.636 INFO:teuthology.orchestra.run.smithi043.stderr:Operation aborted. 2024-08-31T12:11:10.693 INFO:teuthology.orchestra.run.smithi043.stdout:Is this ok [y/N]: 2024-08-31T12:11:10.694 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum reinstall -y kernel || true 2024-08-31T12:11:11.019 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:00:03 ago on Sat 31 Aug 2024 12:11:08 PM UTC. 2024-08-31T12:11:11.174 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-08-31T12:11:11.182 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-08-31T12:11:11.182 INFO:teuthology.orchestra.run.smithi019.stdout: Package Architecture Version Repository Size 2024-08-31T12:11:11.183 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-08-31T12:11:11.183 INFO:teuthology.orchestra.run.smithi019.stdout:Reinstalling: 2024-08-31T12:11:11.183 INFO:teuthology.orchestra.run.smithi019.stdout: kernel x86_64 5.14.0-503.el9 baseos 1.9 M 2024-08-31T12:11:11.183 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:11:11.183 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-08-31T12:11:11.183 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-08-31T12:11:11.183 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:11:11.183 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 1.9 M 2024-08-31T12:11:11.183 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 0 2024-08-31T12:11:11.183 INFO:teuthology.orchestra.run.smithi019.stderr:Operation aborted. 2024-08-31T12:11:11.239 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:00:04 ago on Sat 31 Aug 2024 12:11:07 PM UTC. 2024-08-31T12:11:11.240 INFO:teuthology.orchestra.run.smithi019.stdout:Is this ok [y/N]: 2024-08-31T12:11:11.240 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum reinstall -y kernel || true 2024-08-31T12:11:11.370 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-08-31T12:11:11.371 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-08-31T12:11:11.371 INFO:teuthology.orchestra.run.smithi043.stdout: Package Architecture Version Repository Size 2024-08-31T12:11:11.371 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-08-31T12:11:11.371 INFO:teuthology.orchestra.run.smithi043.stdout:Reinstalling: 2024-08-31T12:11:11.371 INFO:teuthology.orchestra.run.smithi043.stdout: kernel x86_64 5.14.0-503.el9 baseos 1.9 M 2024-08-31T12:11:11.371 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:11:11.371 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-08-31T12:11:11.371 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-08-31T12:11:11.372 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:11:11.372 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 1.9 M 2024-08-31T12:11:11.372 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 0 2024-08-31T12:11:11.372 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-08-31T12:11:11.787 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:00:03 ago on Sat 31 Aug 2024 12:11:08 PM UTC. 2024-08-31T12:11:11.920 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-08-31T12:11:11.921 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-08-31T12:11:11.921 INFO:teuthology.orchestra.run.smithi019.stdout: Package Architecture Version Repository Size 2024-08-31T12:11:11.921 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-08-31T12:11:11.921 INFO:teuthology.orchestra.run.smithi019.stdout:Reinstalling: 2024-08-31T12:11:11.921 INFO:teuthology.orchestra.run.smithi019.stdout: kernel x86_64 5.14.0-503.el9 baseos 1.9 M 2024-08-31T12:11:11.921 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:11:11.921 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-08-31T12:11:11.921 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-08-31T12:11:11.921 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:11:11.921 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 1.9 M 2024-08-31T12:11:11.921 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 0 2024-08-31T12:11:11.922 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-08-31T12:11:12.104 INFO:teuthology.orchestra.run.smithi043.stdout:kernel-5.14.0-503.el9.x86_64.rpm 4.0 MB/s | 1.9 MB 00:00 2024-08-31T12:11:12.104 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-08-31T12:11:12.104 INFO:teuthology.orchestra.run.smithi043.stdout:Total 2.6 MB/s | 1.9 MB 00:00 2024-08-31T12:11:12.142 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-08-31T12:11:12.284 INFO:teuthology.orchestra.run.smithi019.stdout:kernel-5.14.0-503.el9.x86_64.rpm 5.5 MB/s | 1.9 MB 00:00 2024-08-31T12:11:12.284 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-08-31T12:11:12.284 INFO:teuthology.orchestra.run.smithi019.stdout:Total 5.2 MB/s | 1.9 MB 00:00 2024-08-31T12:11:12.322 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-08-31T12:11:12.373 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-08-31T12:11:12.373 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-08-31T12:11:12.405 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-08-31T12:11:12.405 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-08-31T12:11:12.556 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-08-31T12:11:12.556 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-08-31T12:11:12.588 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-08-31T12:11:12.589 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-08-31T12:11:12.596 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-08-31T12:11:12.680 INFO:teuthology.orchestra.run.smithi043.stdout: Reinstalling : kernel-5.14.0-503.el9.x86_64 1/2 2024-08-31T12:11:12.681 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : kernel-5.14.0-503.el9.x86_64 2/2 2024-08-31T12:11:12.751 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-08-31T12:11:12.894 INFO:teuthology.orchestra.run.smithi019.stdout: Reinstalling : kernel-5.14.0-503.el9.x86_64 1/2 2024-08-31T12:11:12.894 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : kernel-5.14.0-503.el9.x86_64 2/2 2024-08-31T12:11:13.507 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: kernel-5.14.0-503.el9.x86_64 2/2 2024-08-31T12:11:13.508 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : kernel-5.14.0-503.el9.x86_64 1/2 2024-08-31T12:11:13.569 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-5.14.0-503.el9.x86_64 2/2 2024-08-31T12:11:13.569 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : kernel-5.14.0-503.el9.x86_64 1/2 2024-08-31T12:11:14.241 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : kernel-5.14.0-503.el9.x86_64 2/2 2024-08-31T12:11:14.242 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:11:14.242 INFO:teuthology.orchestra.run.smithi043.stdout:Reinstalled: 2024-08-31T12:11:14.242 INFO:teuthology.orchestra.run.smithi043.stdout: kernel-5.14.0-503.el9.x86_64 2024-08-31T12:11:14.242 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:11:14.242 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-08-31T12:11:14.267 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : kernel-5.14.0-503.el9.x86_64 2/2 2024-08-31T12:11:14.268 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:11:14.268 INFO:teuthology.orchestra.run.smithi019.stdout:Reinstalled: 2024-08-31T12:11:14.268 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-5.14.0-503.el9.x86_64 2024-08-31T12:11:14.268 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:11:14.268 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-08-31T12:11:14.428 DEBUG:teuthology.orchestra.run.smithi043:> rpm -q kernel | sort -rV | head -n 1 2024-08-31T12:11:14.457 DEBUG:teuthology.orchestra.run.smithi019:> rpm -q kernel | sort -rV | head -n 1 2024-08-31T12:11:14.486 INFO:teuthology.orchestra.run.smithi043.stdout:kernel-5.14.0-503.el9.x86_64 2024-08-31T12:11:14.487 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-503.el9.x86_64 2024-08-31T12:11:14.487 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-08-31T12:11:14.487 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-08-31T12:11:14.487 DEBUG:teuthology.orchestra.run.smithi043:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-08-31T12:11:14.516 INFO:teuthology.orchestra.run.smithi019.stdout:kernel-5.14.0-503.el9.x86_64 2024-08-31T12:11:14.516 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-503.el9.x86_64 2024-08-31T12:11:14.516 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-08-31T12:11:14.516 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-08-31T12:11:14.516 DEBUG:teuthology.orchestra.run.smithi019:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-08-31T12:11:14.517 INFO:teuthology.orchestra.run.smithi043.stdout:ttyS1 2024-08-31T12:11:14.538 DEBUG:teuthology.parallel:result is None 2024-08-31T12:11:14.547 INFO:teuthology.orchestra.run.smithi019.stdout:ttyS1 2024-08-31T12:11:14.565 DEBUG:teuthology.parallel:result is None 2024-08-31T12:11:14.565 INFO:teuthology.run_tasks:Running task internal.base... 2024-08-31T12:11:14.572 INFO:teuthology.task.internal:Creating test directory... 2024-08-31T12:11:14.572 DEBUG:teuthology.orchestra.run.smithi019:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-31T12:11:14.608 DEBUG:teuthology.orchestra.run.smithi043:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-31T12:11:14.623 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-08-31T12:11:14.641 INFO:teuthology.run_tasks:Running task internal.archive... 2024-08-31T12:11:14.683 INFO:teuthology.task.internal:Creating archive directory... 2024-08-31T12:11:14.683 DEBUG:teuthology.orchestra.run.smithi019:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-31T12:11:14.686 DEBUG:teuthology.orchestra.run.smithi043:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-31T12:11:14.714 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-08-31T12:11:14.725 INFO:teuthology.task.internal:Enabling coredump saving... 2024-08-31T12:11:14.726 DEBUG:teuthology.orchestra.run.smithi019:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-31T12:11:14.750 DEBUG:teuthology.orchestra.run.smithi043:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-31T12:11:14.805 INFO:teuthology.orchestra.run.smithi019.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-31T12:11:14.808 INFO:teuthology.orchestra.run.smithi043.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-31T12:11:14.815 INFO:teuthology.orchestra.run.smithi019.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-31T12:11:14.818 INFO:teuthology.orchestra.run.smithi043.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-31T12:11:14.820 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-08-31T12:11:14.864 INFO:teuthology.task.internal:Configuring sudo... 2024-08-31T12:11:14.864 DEBUG:teuthology.orchestra.run.smithi019:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-31T12:11:14.867 DEBUG:teuthology.orchestra.run.smithi043:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-31T12:11:14.891 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-08-31T12:11:14.917 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-08-31T12:11:14.917 DEBUG:teuthology.orchestra.run.smithi019:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-31T12:11:14.931 DEBUG:teuthology.orchestra.run.smithi043:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-31T12:11:14.946 DEBUG:teuthology.orchestra.run.smithi019:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-31T12:11:15.020 DEBUG:teuthology.orchestra.run.smithi019:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-31T12:11:15.081 DEBUG:teuthology.orchestra.run.smithi019:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-31T12:11:15.158 DEBUG:teuthology.orchestra.run.smithi019:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-31T12:11:15.203 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-08-31T12:11:15.203 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-31T12:11:15.267 DEBUG:teuthology.orchestra.run.smithi043:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-31T12:11:15.309 DEBUG:teuthology.orchestra.run.smithi043:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-31T12:11:15.359 DEBUG:teuthology.orchestra.run.smithi043:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-31T12:11:15.441 DEBUG:teuthology.orchestra.run.smithi043:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-31T12:11:15.481 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-08-31T12:11:15.481 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-31T12:11:15.546 DEBUG:teuthology.orchestra.run.smithi019:> sudo service rsyslog restart 2024-08-31T12:11:15.548 DEBUG:teuthology.orchestra.run.smithi043:> sudo service rsyslog restart 2024-08-31T12:11:15.578 INFO:teuthology.orchestra.run.smithi019.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-31T12:11:15.623 INFO:teuthology.orchestra.run.smithi043.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-31T12:11:16.125 INFO:teuthology.run_tasks:Running task internal.timer... 2024-08-31T12:11:16.133 INFO:teuthology.task.internal:Starting timer... 2024-08-31T12:11:16.133 INFO:teuthology.run_tasks:Running task pcp... 2024-08-31T12:11:16.160 INFO:teuthology.run_tasks:Running task selinux... 2024-08-31T12:11:16.185 DEBUG:teuthology.orchestra.run.smithi019:> sudo service auditd rotate 2024-08-31T12:11:16.237 INFO:teuthology.orchestra.run.smithi019.stdout:Rotating logs: 2024-08-31T12:11:16.239 DEBUG:teuthology.orchestra.run.smithi043:> sudo service auditd rotate 2024-08-31T12:11:16.282 INFO:teuthology.orchestra.run.smithi043.stdout:Rotating logs: 2024-08-31T12:11:16.284 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-08-31T12:11:16.284 DEBUG:teuthology.orchestra.run.smithi019:> /usr/sbin/getenforce 2024-08-31T12:11:16.312 INFO:teuthology.orchestra.run.smithi019.stdout:Permissive 2024-08-31T12:11:16.313 DEBUG:teuthology.orchestra.run.smithi043:> /usr/sbin/getenforce 2024-08-31T12:11:16.343 INFO:teuthology.orchestra.run.smithi043.stdout:Permissive 2024-08-31T12:11:16.344 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi019.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi043.front.sepia.ceph.com': 'permissive'} 2024-08-31T12:11:16.344 DEBUG:teuthology.orchestra.run.smithi019:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-08-31T12:11:16.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T12:11:16.378 DEBUG:teuthology.orchestra.run.smithi043:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-08-31T12:11:16.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T12:11:16.408 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-08-31T12:11:16.408 DEBUG:teuthology.orchestra.run.smithi019:> sudo /usr/sbin/setenforce permissive 2024-08-31T12:11:16.445 DEBUG:teuthology.orchestra.run.smithi043:> sudo /usr/sbin/setenforce permissive 2024-08-31T12:11:16.484 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-08-31T12:11:16.493 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-08-31T12:11:16.504 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-08-31T12:11:16.505 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi019.front.sepia.ceph.com,smithi043.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-08-31T12:13:00.711 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi019.front.sepia.ceph.com'), Remote(name='ubuntu@smithi043.front.sepia.ceph.com')] 2024-08-31T12:13:00.712 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-08-31T12:13:00.713 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-31T12:13:00.791 DEBUG:teuthology.orchestra.run.smithi019:> true 2024-08-31T12:13:00.872 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-08-31T12:13:00.873 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi043.front.sepia.ceph.com' 2024-08-31T12:13:00.873 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-31T12:13:00.947 DEBUG:teuthology.orchestra.run.smithi043:> true 2024-08-31T12:13:01.033 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi043.front.sepia.ceph.com' 2024-08-31T12:13:01.033 INFO:teuthology.run_tasks:Running task clock... 2024-08-31T12:13:01.044 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-08-31T12:13:01.045 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-31T12:13:01.048 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-31T12:13:01.051 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-31T12:13:01.051 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-31T12:13:01.077 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-31T12:13:01.092 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-31T12:13:01.105 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-31T12:13:01.117 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: ntpd: command not found 2024-08-31T12:13:01.119 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-31T12:13:01.128 INFO:teuthology.orchestra.run.smithi019.stdout:506 Cannot talk to daemon 2024-08-31T12:13:01.143 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-31T12:13:01.145 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: ntpd: command not found 2024-08-31T12:13:01.157 INFO:teuthology.orchestra.run.smithi043.stdout:506 Cannot talk to daemon 2024-08-31T12:13:01.157 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-31T12:13:01.171 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-31T12:13:01.185 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-31T12:13:01.214 INFO:teuthology.orchestra.run.smithi019.stderr:bash: line 1: ntpq: command not found 2024-08-31T12:13:01.217 INFO:teuthology.orchestra.run.smithi019.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-31T12:13:01.217 INFO:teuthology.orchestra.run.smithi019.stdout:=============================================================================== 2024-08-31T12:13:01.217 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-31T12:13:01.218 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-31T12:13:01.218 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-31T12:13:01.218 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-31T12:13:01.247 INFO:teuthology.orchestra.run.smithi043.stderr:bash: line 1: ntpq: command not found 2024-08-31T12:13:01.253 INFO:teuthology.orchestra.run.smithi043.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-31T12:13:01.253 INFO:teuthology.orchestra.run.smithi043.stdout:=============================================================================== 2024-08-31T12:13:01.253 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-31T12:13:01.253 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-31T12:13:01.253 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-31T12:13:01.253 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-31T12:13:01.253 INFO:teuthology.run_tasks:Running task install... 2024-08-31T12:13:01.261 DEBUG:teuthology.task.install:project ceph 2024-08-31T12:13:01.262 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd19e6118f7015f0047264006cb0c873528eb3ae6'}} 2024-08-31T12:13:01.262 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd19e6118f7015f0047264006cb0c873528eb3ae6'} 2024-08-31T12:13:01.262 INFO:teuthology.task.install:Using flavor: default 2024-08-31T12:13:01.267 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-08-31T12:13:01.267 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-08-31T12:13:01.268 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd19e6118f7015f0047264006cb0c873528eb3ae6', 'tag': None, 'wait_for_package': False} 2024-08-31T12:13:01.268 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d19e6118f7015f0047264006cb0c873528eb3ae6 2024-08-31T12:13:01.270 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd19e6118f7015f0047264006cb0c873528eb3ae6', 'tag': None, 'wait_for_package': False} 2024-08-31T12:13:01.270 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d19e6118f7015f0047264006cb0c873528eb3ae6 2024-08-31T12:13:01.410 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-20240828.085847-main-debug/d19e6118f7015f0047264006cb0c873528eb3ae6/centos/9/flavors/default/ 2024-08-31T12:13:01.410 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4560.gd19e6118 2024-08-31T12:13:01.424 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-20240828.085847-main-debug/d19e6118f7015f0047264006cb0c873528eb3ae6/centos/9/flavors/default/ 2024-08-31T12:13:01.424 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4560.gd19e6118 2024-08-31T12:13:01.518 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-20240828.085847-main-debug/d19e6118f7015f0047264006cb0c873528eb3ae6/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-20240828.085847-main-debug/d19e6118f7015f0047264006cb0c873528eb3ae6/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-20240828.085847-main-debug/d19e6118f7015f0047264006cb0c873528eb3ae6/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-31T12:13:01.518 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-08-31T12:13:01.518 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-31T12:13:01.529 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-20240828.085847-main-debug/d19e6118f7015f0047264006cb0c873528eb3ae6/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-20240828.085847-main-debug/d19e6118f7015f0047264006cb0c873528eb3ae6/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-20240828.085847-main-debug/d19e6118f7015f0047264006cb0c873528eb3ae6/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-31T12:13:01.529 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-08-31T12:13:01.530 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-31T12:13:01.543 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-08-31T12:13:01.544 DEBUG:teuthology.orchestra.run.smithi019:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d19e6118f7015f0047264006cb0c873528eb3ae6/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-31T12:13:01.556 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-08-31T12:13:01.556 DEBUG:teuthology.orchestra.run.smithi043:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d19e6118f7015f0047264006cb0c873528eb3ae6/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-31T12:13:01.610 DEBUG:teuthology.orchestra.run.smithi019:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-31T12:13:01.622 DEBUG:teuthology.orchestra.run.smithi043:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-31T12:13:01.685 DEBUG:teuthology.orchestra.run.smithi019:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-31T12:13:01.699 DEBUG:teuthology.orchestra.run.smithi043:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-31T12:13:01.748 INFO:teuthology.orchestra.run.smithi019.stdout:check_obsoletes = 1 2024-08-31T12:13:01.750 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum clean all 2024-08-31T12:13:01.763 INFO:teuthology.orchestra.run.smithi043.stdout:check_obsoletes = 1 2024-08-31T12:13:01.765 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum clean all 2024-08-31T12:13:01.981 INFO:teuthology.orchestra.run.smithi019.stdout:45 files removed 2024-08-31T12:13:01.996 INFO:teuthology.orchestra.run.smithi043.stdout:45 files removed 2024-08-31T12:13:02.009 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-08-31T12:13:02.023 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-08-31T12:13:02.559 INFO:teuthology.orchestra.run.smithi019.stdout:ceph packages for x86_64 313 kB/s | 84 kB 00:00 2024-08-31T12:13:02.571 INFO:teuthology.orchestra.run.smithi043.stdout:ceph packages for x86_64 316 kB/s | 84 kB 00:00 2024-08-31T12:13:02.851 INFO:teuthology.orchestra.run.smithi019.stdout:ceph noarch packages 97 kB/s | 24 kB 00:00 2024-08-31T12:13:02.855 INFO:teuthology.orchestra.run.smithi043.stdout:ceph noarch packages 101 kB/s | 24 kB 00:00 2024-08-31T12:13:03.127 INFO:teuthology.orchestra.run.smithi019.stdout:ceph source packages 8.1 kB/s | 1.9 kB 00:00 2024-08-31T12:13:03.139 INFO:teuthology.orchestra.run.smithi043.stdout:ceph source packages 7.8 kB/s | 1.9 kB 00:00 2024-08-31T12:13:03.735 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.2 MB 00:00 2024-08-31T12:13:04.897 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - BaseOS 4.8 MB/s | 8.2 MB 00:01 2024-08-31T12:13:06.699 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - AppStream 30 MB/s | 20 MB 00:00 2024-08-31T12:13:11.100 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.6 MB 00:00 2024-08-31T12:13:12.634 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - Extras packages 71 kB/s | 18 kB 00:00 2024-08-31T12:13:13.404 INFO:teuthology.orchestra.run.smithi043.stdout:Extra Packages for Enterprise Linux 35 MB/s | 22 MB 00:00 2024-08-31T12:13:19.044 INFO:teuthology.orchestra.run.smithi043.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-08-31T12:13:21.432 INFO:teuthology.orchestra.run.smithi043.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-08-31T12:13:21.433 INFO:teuthology.orchestra.run.smithi043.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-08-31T12:13:21.447 INFO:teuthology.orchestra.run.smithi043.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-08-31T12:13:21.448 INFO:teuthology.orchestra.run.smithi043.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-08-31T12:13:21.449 INFO:teuthology.orchestra.run.smithi043.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-08-31T12:13:21.450 INFO:teuthology.orchestra.run.smithi043.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-08-31T12:13:21.452 INFO:teuthology.orchestra.run.smithi043.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-08-31T12:13:21.455 INFO:teuthology.orchestra.run.smithi043.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-08-31T12:13:21.456 INFO:teuthology.orchestra.run.smithi043.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-08-31T12:13:21.521 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-08-31T12:13:21.530 INFO:teuthology.orchestra.run.smithi043.stdout:======================================================================================= 2024-08-31T12:13:21.530 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2024-08-31T12:13:21.530 INFO:teuthology.orchestra.run.smithi043.stdout:======================================================================================= 2024-08-31T12:13:21.531 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-08-31T12:13:21.531 INFO:teuthology.orchestra.run.smithi043.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-08-31T12:13:21.531 INFO:teuthology.orchestra.run.smithi043.stdout: ceph x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 6.6 k 2024-08-31T12:13:21.531 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-base x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 5.5 M 2024-08-31T12:13:21.531 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-fuse x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 874 k 2024-08-31T12:13:21.531 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 148 k 2024-08-31T12:13:21.531 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 1.5 M 2024-08-31T12:13:21.531 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 154 k 2024-08-31T12:13:21.531 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 14 M 2024-08-31T12:13:21.531 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 7.4 M 2024-08-31T12:13:21.531 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-rook noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 49 k 2024-08-31T12:13:21.531 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-radosgw x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 10 M 2024-08-31T12:13:21.531 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-test x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 44 M 2024-08-31T12:13:21.532 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-volume noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 285 k 2024-08-31T12:13:21.532 INFO:teuthology.orchestra.run.smithi043.stdout: cephadm noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 781 k 2024-08-31T12:13:21.532 INFO:teuthology.orchestra.run.smithi043.stdout: cephfs-mirror x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 226 k 2024-08-31T12:13:21.532 INFO:teuthology.orchestra.run.smithi043.stdout: cephfs-shell x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 27 k 2024-08-31T12:13:21.532 INFO:teuthology.orchestra.run.smithi043.stdout: cephfs-top noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 18 k 2024-08-31T12:13:21.532 INFO:teuthology.orchestra.run.smithi043.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-08-31T12:13:21.532 INFO:teuthology.orchestra.run.smithi043.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-08-31T12:13:21.532 INFO:teuthology.orchestra.run.smithi043.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-08-31T12:13:21.532 INFO:teuthology.orchestra.run.smithi043.stdout: libcephfs-devel x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 32 k 2024-08-31T12:13:21.532 INFO:teuthology.orchestra.run.smithi043.stdout: libcephfs2 x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 766 k 2024-08-31T12:13:21.532 INFO:teuthology.orchestra.run.smithi043.stdout: librados-devel x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 127 k 2024-08-31T12:13:21.532 INFO:teuthology.orchestra.run.smithi043.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-08-31T12:13:21.532 INFO:teuthology.orchestra.run.smithi043.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-08-31T12:13:21.532 INFO:teuthology.orchestra.run.smithi043.stdout: openssl-devel x86_64 1:3.2.2-4.el9 appstream 4.4 M 2024-08-31T12:13:21.533 INFO:teuthology.orchestra.run.smithi043.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-08-31T12:13:21.533 INFO:teuthology.orchestra.run.smithi043.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-08-31T12:13:21.533 INFO:teuthology.orchestra.run.smithi043.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-08-31T12:13:21.533 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cephfs x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 161 k 2024-08-31T12:13:21.533 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rados x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 321 k 2024-08-31T12:13:21.533 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rbd x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 305 k 2024-08-31T12:13:21.533 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rgw x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 99 k 2024-08-31T12:13:21.533 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-fuse x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 86 k 2024-08-31T12:13:21.533 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-mirror x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 3.1 M 2024-08-31T12:13:21.533 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-nbd x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 170 k 2024-08-31T12:13:21.533 INFO:teuthology.orchestra.run.smithi043.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-08-31T12:13:21.533 INFO:teuthology.orchestra.run.smithi043.stdout:Upgrading: 2024-08-31T12:13:21.534 INFO:teuthology.orchestra.run.smithi043.stdout: librados2 x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 3.4 M 2024-08-31T12:13:21.534 INFO:teuthology.orchestra.run.smithi043.stdout: librbd1 x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 3.1 M 2024-08-31T12:13:21.534 INFO:teuthology.orchestra.run.smithi043.stdout:Installing dependencies: 2024-08-31T12:13:21.534 INFO:teuthology.orchestra.run.smithi043.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-08-31T12:13:21.534 INFO:teuthology.orchestra.run.smithi043.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-08-31T12:13:21.534 INFO:teuthology.orchestra.run.smithi043.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-08-31T12:13:21.534 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-common x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 20 M 2024-08-31T12:13:21.534 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 34 k 2024-08-31T12:13:21.534 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mds x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 2.3 M 2024-08-31T12:13:21.534 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 290 k 2024-08-31T12:13:21.534 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mon x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 4.9 M 2024-08-31T12:13:21.534 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-osd x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 18 M 2024-08-31T12:13:21.534 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 16 k 2024-08-31T12:13:21.535 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-selinux x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 25 k 2024-08-31T12:13:21.535 INFO:teuthology.orchestra.run.smithi043.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-08-31T12:13:21.535 INFO:teuthology.orchestra.run.smithi043.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-08-31T12:13:21.535 INFO:teuthology.orchestra.run.smithi043.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-08-31T12:13:21.535 INFO:teuthology.orchestra.run.smithi043.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-31T12:13:21.535 INFO:teuthology.orchestra.run.smithi043.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-31T12:13:21.535 INFO:teuthology.orchestra.run.smithi043.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-08-31T12:13:21.535 INFO:teuthology.orchestra.run.smithi043.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-08-31T12:13:21.535 INFO:teuthology.orchestra.run.smithi043.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-08-31T12:13:21.535 INFO:teuthology.orchestra.run.smithi043.stdout: libcephsqlite x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 159 k 2024-08-31T12:13:21.535 INFO:teuthology.orchestra.run.smithi043.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-08-31T12:13:21.535 INFO:teuthology.orchestra.run.smithi043.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-31T12:13:21.535 INFO:teuthology.orchestra.run.smithi043.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-31T12:13:21.536 INFO:teuthology.orchestra.run.smithi043.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-08-31T12:13:21.536 INFO:teuthology.orchestra.run.smithi043.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-31T12:13:21.536 INFO:teuthology.orchestra.run.smithi043.stdout: libradosstriper1 x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 506 k 2024-08-31T12:13:21.536 INFO:teuthology.orchestra.run.smithi043.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-31T12:13:21.536 INFO:teuthology.orchestra.run.smithi043.stdout: librgw2 x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 5.3 M 2024-08-31T12:13:21.536 INFO:teuthology.orchestra.run.smithi043.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-31T12:13:21.536 INFO:teuthology.orchestra.run.smithi043.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-08-31T12:13:21.536 INFO:teuthology.orchestra.run.smithi043.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-31T12:13:21.536 INFO:teuthology.orchestra.run.smithi043.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-08-31T12:13:21.536 INFO:teuthology.orchestra.run.smithi043.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-08-31T12:13:21.536 INFO:teuthology.orchestra.run.smithi043.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-08-31T12:13:21.536 INFO:teuthology.orchestra.run.smithi043.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-08-31T12:13:21.536 INFO:teuthology.orchestra.run.smithi043.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-08-31T12:13:21.536 INFO:teuthology.orchestra.run.smithi043.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-08-31T12:13:21.537 INFO:teuthology.orchestra.run.smithi043.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-08-31T12:13:21.537 INFO:teuthology.orchestra.run.smithi043.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-08-31T12:13:21.537 INFO:teuthology.orchestra.run.smithi043.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-08-31T12:13:21.537 INFO:teuthology.orchestra.run.smithi043.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-08-31T12:13:21.537 INFO:teuthology.orchestra.run.smithi043.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-08-31T12:13:21.537 INFO:teuthology.orchestra.run.smithi043.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-08-31T12:13:21.537 INFO:teuthology.orchestra.run.smithi043.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-31T12:13:21.537 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-08-31T12:13:21.537 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ceph-argparse x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 45 k 2024-08-31T12:13:21.537 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ceph-common x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 141 k 2024-08-31T12:13:21.537 INFO:teuthology.orchestra.run.smithi043.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-08-31T12:13:21.537 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-31T12:13:21.538 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-31T12:13:21.538 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-31T12:13:21.538 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-08-31T12:13:21.538 INFO:teuthology.orchestra.run.smithi043.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-08-31T12:13:21.538 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-31T12:13:21.538 INFO:teuthology.orchestra.run.smithi043.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-08-31T12:13:21.538 INFO:teuthology.orchestra.run.smithi043.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-08-31T12:13:21.538 INFO:teuthology.orchestra.run.smithi043.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-08-31T12:13:21.538 INFO:teuthology.orchestra.run.smithi043.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-08-31T12:13:21.538 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-31T12:13:21.538 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-31T12:13:21.538 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-08-31T12:13:21.538 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-31T12:13:21.539 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-31T12:13:21.539 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-08-31T12:13:21.539 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-08-31T12:13:21.539 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-08-31T12:13:21.539 INFO:teuthology.orchestra.run.smithi043.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-08-31T12:13:21.539 INFO:teuthology.orchestra.run.smithi043.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-31T12:13:21.539 INFO:teuthology.orchestra.run.smithi043.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-31T12:13:21.539 INFO:teuthology.orchestra.run.smithi043.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-31T12:13:21.539 INFO:teuthology.orchestra.run.smithi043.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-31T12:13:21.539 INFO:teuthology.orchestra.run.smithi043.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-08-31T12:13:21.539 INFO:teuthology.orchestra.run.smithi043.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-08-31T12:13:21.539 INFO:teuthology.orchestra.run.smithi043.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-08-31T12:13:21.539 INFO:teuthology.orchestra.run.smithi043.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-08-31T12:13:21.540 INFO:teuthology.orchestra.run.smithi043.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-08-31T12:13:21.540 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-31T12:13:21.540 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-31T12:13:21.540 INFO:teuthology.orchestra.run.smithi043.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-31T12:13:21.540 INFO:teuthology.orchestra.run.smithi043.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-31T12:13:21.540 INFO:teuthology.orchestra.run.smithi043.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-08-31T12:13:21.540 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-31T12:13:21.540 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-08-31T12:13:21.540 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-08-31T12:13:21.540 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-31T12:13:21.540 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-08-31T12:13:21.540 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-31T12:13:21.540 INFO:teuthology.orchestra.run.smithi043.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-08-31T12:13:21.541 INFO:teuthology.orchestra.run.smithi043.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-31T12:13:21.541 INFO:teuthology.orchestra.run.smithi043.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-08-31T12:13:21.541 INFO:teuthology.orchestra.run.smithi043.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-08-31T12:13:21.541 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-08-31T12:13:21.541 INFO:teuthology.orchestra.run.smithi043.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-08-31T12:13:21.541 INFO:teuthology.orchestra.run.smithi043.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-31T12:13:21.541 INFO:teuthology.orchestra.run.smithi043.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-31T12:13:21.541 INFO:teuthology.orchestra.run.smithi043.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-08-31T12:13:21.541 INFO:teuthology.orchestra.run.smithi043.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-31T12:13:21.541 INFO:teuthology.orchestra.run.smithi043.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-08-31T12:13:21.541 INFO:teuthology.orchestra.run.smithi043.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-08-31T12:13:21.541 INFO:teuthology.orchestra.run.smithi043.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-08-31T12:13:21.541 INFO:teuthology.orchestra.run.smithi043.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-31T12:13:21.542 INFO:teuthology.orchestra.run.smithi043.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-31T12:13:21.542 INFO:teuthology.orchestra.run.smithi043.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-08-31T12:13:21.542 INFO:teuthology.orchestra.run.smithi043.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-08-31T12:13:21.542 INFO:teuthology.orchestra.run.smithi043.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-08-31T12:13:21.542 INFO:teuthology.orchestra.run.smithi043.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-31T12:13:21.542 INFO:teuthology.orchestra.run.smithi043.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-31T12:13:21.542 INFO:teuthology.orchestra.run.smithi043.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-08-31T12:13:21.542 INFO:teuthology.orchestra.run.smithi043.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-31T12:13:21.542 INFO:teuthology.orchestra.run.smithi043.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-08-31T12:13:21.542 INFO:teuthology.orchestra.run.smithi043.stdout:Installing weak dependencies: 2024-08-31T12:13:21.542 INFO:teuthology.orchestra.run.smithi043.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-08-31T12:13:21.542 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:13:21.542 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-08-31T12:13:21.542 INFO:teuthology.orchestra.run.smithi043.stdout:======================================================================================= 2024-08-31T12:13:21.543 INFO:teuthology.orchestra.run.smithi043.stdout:Install 148 Packages 2024-08-31T12:13:21.543 INFO:teuthology.orchestra.run.smithi043.stdout:Upgrade 2 Packages 2024-08-31T12:13:21.543 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:13:21.543 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 224 M 2024-08-31T12:13:21.543 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-08-31T12:13:22.011 INFO:teuthology.orchestra.run.smithi043.stdout:(1/150): ceph-19.3.0-4560.gd19e6118.el9.x86_64. 49 kB/s | 6.6 kB 00:00 2024-08-31T12:13:22.160 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 20 MB 00:17 2024-08-31T12:13:22.245 INFO:teuthology.orchestra.run.smithi043.stdout:(2/150): ceph-fuse-19.3.0-4560.gd19e6118.el9.x8 3.7 MB/s | 874 kB 00:00 2024-08-31T12:13:22.362 INFO:teuthology.orchestra.run.smithi043.stdout:(3/150): ceph-base-19.3.0-4560.gd19e6118.el9.x8 11 MB/s | 5.5 MB 00:00 2024-08-31T12:13:22.396 INFO:teuthology.orchestra.run.smithi043.stdout:(4/150): ceph-immutable-object-cache-19.3.0-456 984 kB/s | 148 kB 00:00 2024-08-31T12:13:22.479 INFO:teuthology.orchestra.run.smithi043.stdout:(5/150): ceph-mds-19.3.0-4560.gd19e6118.el9.x86 19 MB/s | 2.3 MB 00:00 2024-08-31T12:13:22.555 INFO:teuthology.orchestra.run.smithi043.stdout:(6/150): ceph-mgr-19.3.0-4560.gd19e6118.el9.x86 9.6 MB/s | 1.5 MB 00:00 2024-08-31T12:13:22.805 INFO:teuthology.orchestra.run.smithi043.stdout:(7/150): ceph-mon-19.3.0-4560.gd19e6118.el9.x86 15 MB/s | 4.9 MB 00:00 2024-08-31T12:13:23.006 INFO:teuthology.orchestra.run.smithi043.stdout:(8/150): ceph-common-19.3.0-4560.gd19e6118.el9. 18 MB/s | 20 MB 00:01 2024-08-31T12:13:23.047 INFO:teuthology.orchestra.run.smithi043.stdout:(9/150): ceph-selinux-19.3.0-4560.gd19e6118.el9 609 kB/s | 25 kB 00:00 2024-08-31T12:13:23.731 INFO:teuthology.orchestra.run.smithi043.stdout:(10/150): ceph-osd-19.3.0-4560.gd19e6118.el9.x8 15 MB/s | 18 MB 00:01 2024-08-31T12:13:23.781 INFO:teuthology.orchestra.run.smithi043.stdout:(11/150): cephfs-mirror-19.3.0-4560.gd19e6118.e 4.4 MB/s | 226 kB 00:00 2024-08-31T12:13:23.898 INFO:teuthology.orchestra.run.smithi043.stdout:(12/150): ceph-radosgw-19.3.0-4560.gd19e6118.el 9.6 MB/s | 10 MB 00:01 2024-08-31T12:13:23.914 INFO:teuthology.orchestra.run.smithi043.stdout:(13/150): cephfs-shell-19.3.0-4560.gd19e6118.el 202 kB/s | 27 kB 00:00 2024-08-31T12:13:23.932 INFO:teuthology.orchestra.run.smithi043.stdout:(14/150): libcephfs-devel-19.3.0-4560.gd19e6118 951 kB/s | 32 kB 00:00 2024-08-31T12:13:24.049 INFO:teuthology.orchestra.run.smithi043.stdout:(15/150): libcephfs2-19.3.0-4560.gd19e6118.el9. 5.6 MB/s | 766 kB 00:00 2024-08-31T12:13:24.074 INFO:teuthology.orchestra.run.smithi043.stdout:(16/150): libcephsqlite-19.3.0-4560.gd19e6118.e 1.1 MB/s | 159 kB 00:00 2024-08-31T12:13:24.107 INFO:teuthology.orchestra.run.smithi043.stdout:(17/150): librados-devel-19.3.0-4560.gd19e6118. 2.1 MB/s | 127 kB 00:00 2024-08-31T12:13:24.141 INFO:teuthology.orchestra.run.smithi043.stdout:(18/150): libradosstriper1-19.3.0-4560.gd19e611 7.4 MB/s | 506 kB 00:00 2024-08-31T12:13:24.208 INFO:teuthology.orchestra.run.smithi043.stdout:(19/150): python3-ceph-argparse-19.3.0-4560.gd1 676 kB/s | 45 kB 00:00 2024-08-31T12:13:24.275 INFO:teuthology.orchestra.run.smithi043.stdout:(20/150): python3-ceph-common-19.3.0-4560.gd19e 2.1 MB/s | 141 kB 00:00 2024-08-31T12:13:24.359 INFO:teuthology.orchestra.run.smithi043.stdout:(21/150): python3-cephfs-19.3.0-4560.gd19e6118. 1.9 MB/s | 161 kB 00:00 2024-08-31T12:13:24.426 INFO:teuthology.orchestra.run.smithi043.stdout:(22/150): librgw2-19.3.0-4560.gd19e6118.el9.x86 17 MB/s | 5.3 MB 00:00 2024-08-31T12:13:24.451 INFO:teuthology.orchestra.run.smithi043.stdout:(23/150): python3-rados-19.3.0-4560.gd19e6118.e 3.4 MB/s | 321 kB 00:00 2024-08-31T12:13:24.476 INFO:teuthology.orchestra.run.smithi043.stdout:(24/150): python3-rbd-19.3.0-4560.gd19e6118.el9 6.0 MB/s | 305 kB 00:00 2024-08-31T12:13:24.501 INFO:teuthology.orchestra.run.smithi043.stdout:(25/150): python3-rgw-19.3.0-4560.gd19e6118.el9 1.9 MB/s | 99 kB 00:00 2024-08-31T12:13:24.527 INFO:teuthology.orchestra.run.smithi043.stdout:(26/150): rbd-fuse-19.3.0-4560.gd19e6118.el9.x8 1.7 MB/s | 86 kB 00:00 2024-08-31T12:13:24.577 INFO:teuthology.orchestra.run.smithi043.stdout:(27/150): rbd-nbd-19.3.0-4560.gd19e6118.el9.x86 3.3 MB/s | 170 kB 00:00 2024-08-31T12:13:24.652 INFO:teuthology.orchestra.run.smithi043.stdout:(28/150): ceph-grafana-dashboards-19.3.0-4560.g 458 kB/s | 34 kB 00:00 2024-08-31T12:13:25.061 INFO:teuthology.orchestra.run.smithi043.stdout:(29/150): ceph-test-19.3.0-4560.gd19e6118.el9.x 22 MB/s | 44 MB 00:02 2024-08-31T12:13:25.086 INFO:teuthology.orchestra.run.smithi043.stdout:(30/150): ceph-mgr-cephadm-19.3.0-4560.gd19e611 355 kB/s | 154 kB 00:00 2024-08-31T12:13:25.144 INFO:teuthology.orchestra.run.smithi043.stdout:(31/150): rbd-mirror-19.3.0-4560.gd19e6118.el9. 4.8 MB/s | 3.1 MB 00:00 2024-08-31T12:13:25.270 INFO:teuthology.orchestra.run.smithi043.stdout:(32/150): ceph-mgr-modules-core-19.3.0-4560.gd1 2.3 MB/s | 290 kB 00:00 2024-08-31T12:13:25.353 INFO:teuthology.orchestra.run.smithi043.stdout:(33/150): ceph-mgr-rook-19.3.0-4560.gd19e6118.e 593 kB/s | 49 kB 00:00 2024-08-31T12:13:25.429 INFO:teuthology.orchestra.run.smithi043.stdout:(34/150): ceph-prometheus-alerts-19.3.0-4560.gd 219 kB/s | 16 kB 00:00 2024-08-31T12:13:25.579 INFO:teuthology.orchestra.run.smithi043.stdout:(35/150): ceph-mgr-dashboard-19.3.0-4560.gd19e6 27 MB/s | 14 MB 00:00 2024-08-31T12:13:25.604 INFO:teuthology.orchestra.run.smithi043.stdout:(36/150): ceph-volume-19.3.0-4560.gd19e6118.el9 1.6 MB/s | 285 kB 00:00 2024-08-31T12:13:25.647 INFO:teuthology.orchestra.run.smithi043.stdout:(37/150): cephadm-19.3.0-4560.gd19e6118.el9.noa 11 MB/s | 781 kB 00:00 2024-08-31T12:13:25.671 INFO:teuthology.orchestra.run.smithi043.stdout:(38/150): cephfs-top-19.3.0-4560.gd19e6118.el9. 273 kB/s | 18 kB 00:00 2024-08-31T12:13:25.764 INFO:teuthology.orchestra.run.smithi043.stdout:(39/150): ceph-mgr-diskprediction-local-19.3.0- 11 MB/s | 7.4 MB 00:00 2024-08-31T12:13:25.906 INFO:teuthology.orchestra.run.smithi043.stdout:(40/150): fuse-2.9.9-16.el9.x86_64.rpm 309 kB/s | 80 kB 00:00 2024-08-31T12:13:25.964 INFO:teuthology.orchestra.run.smithi043.stdout:(41/150): libquadmath-11.5.0-2.el9.x86_64.rpm 956 kB/s | 191 kB 00:00 2024-08-31T12:13:26.023 INFO:teuthology.orchestra.run.smithi043.stdout:(42/150): libgfortran-11.5.0-2.el9.x86_64.rpm 2.2 MB/s | 801 kB 00:00 2024-08-31T12:13:26.048 INFO:teuthology.orchestra.run.smithi043.stdout:(43/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.7 MB/s | 253 kB 00:00 2024-08-31T12:13:26.082 INFO:teuthology.orchestra.run.smithi043.stdout:(44/150): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-08-31T12:13:26.107 INFO:teuthology.orchestra.run.smithi043.stdout:(45/150): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 135 kB 00:00 2024-08-31T12:13:26.132 INFO:teuthology.orchestra.run.smithi043.stdout:(46/150): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-08-31T12:13:26.174 INFO:teuthology.orchestra.run.smithi043.stdout:(47/150): python3-cryptography-36.0.1-4.el9.x86 5.8 MB/s | 1.2 MB 00:00 2024-08-31T12:13:26.200 INFO:teuthology.orchestra.run.smithi043.stdout:(48/150): python3-urllib3-1.26.5-6.el9.noarch.r 2.3 MB/s | 215 kB 00:00 2024-08-31T12:13:26.316 INFO:teuthology.orchestra.run.smithi043.stdout:(49/150): boost-program-options-1.75.0-8.el9.x8 918 kB/s | 107 kB 00:00 2024-08-31T12:13:26.341 INFO:teuthology.orchestra.run.smithi043.stdout:(50/150): automake-1.16.2-8.el9.noarch.rpm 3.2 MB/s | 689 kB 00:00 2024-08-31T12:13:26.367 INFO:teuthology.orchestra.run.smithi043.stdout:(51/150): elfutils-libelf-devel-0.191-4.el9.x86 432 kB/s | 22 kB 00:00 2024-08-31T12:13:26.392 INFO:teuthology.orchestra.run.smithi043.stdout:(52/150): flex-2.6.4-9.el9.x86_64.rpm 6.1 MB/s | 313 kB 00:00 2024-08-31T12:13:26.417 INFO:teuthology.orchestra.run.smithi043.stdout:(53/150): flexiblas-3.0.4-8.el9.x86_64.rpm 655 kB/s | 33 kB 00:00 2024-08-31T12:13:26.459 INFO:teuthology.orchestra.run.smithi043.stdout:(54/150): bison-3.7.4-5.el9.x86_64.rpm 3.2 MB/s | 943 kB 00:00 2024-08-31T12:13:26.484 INFO:teuthology.orchestra.run.smithi043.stdout:(55/150): flexiblas-openblas-openmp-3.0.4-8.el9 272 kB/s | 18 kB 00:00 2024-08-31T12:13:26.510 INFO:teuthology.orchestra.run.smithi043.stdout:(56/150): libaio-devel-0.3.111-13.el9.x86_64.rp 222 kB/s | 11 kB 00:00 2024-08-31T12:13:26.535 INFO:teuthology.orchestra.run.smithi043.stdout:(57/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-08-31T12:13:26.544 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.6 MB 00:00 2024-08-31T12:13:26.560 INFO:teuthology.orchestra.run.smithi043.stdout:(58/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 903 kB/s | 45 kB 00:00 2024-08-31T12:13:26.594 INFO:teuthology.orchestra.run.smithi043.stdout:(59/150): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 662 kB 00:00 2024-08-31T12:13:26.635 INFO:teuthology.orchestra.run.smithi043.stdout:(60/150): libuuid-devel-2.37.4-20.el9.x86_64.rp 588 kB/s | 24 kB 00:00 2024-08-31T12:13:26.686 INFO:teuthology.orchestra.run.smithi043.stdout:(61/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 10 MB/s | 3.0 MB 00:00 2024-08-31T12:13:26.711 INFO:teuthology.orchestra.run.smithi043.stdout:(62/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 630 kB/s | 47 kB 00:00 2024-08-31T12:13:26.745 INFO:teuthology.orchestra.run.smithi043.stdout:(63/150): libtool-2.4.6-46.el9.x86_64.rpm 3.1 MB/s | 581 kB 00:00 2024-08-31T12:13:26.770 INFO:teuthology.orchestra.run.smithi043.stdout:(64/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.4 MB/s | 292 kB 00:00 2024-08-31T12:13:26.796 INFO:teuthology.orchestra.run.smithi043.stdout:(65/150): openblas-0.3.26-2.el9.x86_64.rpm 765 kB/s | 38 kB 00:00 2024-08-31T12:13:26.821 INFO:teuthology.orchestra.run.smithi043.stdout:(66/150): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-08-31T12:13:26.905 INFO:teuthology.orchestra.run.smithi043.stdout:(67/150): openblas-openmp-0.3.26-2.el9.x86_64.r 37 MB/s | 4.9 MB 00:00 2024-08-31T12:13:26.946 INFO:teuthology.orchestra.run.smithi043.stdout:(68/150): postgresql-13.16-1.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-08-31T12:13:26.980 INFO:teuthology.orchestra.run.smithi043.stdout:(69/150): postgresql-contrib-13.16-1.el9.x86_64 12 MB/s | 910 kB 00:00 2024-08-31T12:13:27.005 INFO:teuthology.orchestra.run.smithi043.stdout:(70/150): postgresql-private-libs-13.16-1.el9.x 2.3 MB/s | 137 kB 00:00 2024-08-31T12:13:27.081 INFO:teuthology.orchestra.run.smithi043.stdout:(71/150): openssl-devel-3.2.2-4.el9.x86_64.rpm 16 MB/s | 4.4 MB 00:00 2024-08-31T12:13:27.123 INFO:teuthology.orchestra.run.smithi043.stdout:(72/150): protobuf-3.14.0-13.el9.x86_64.rpm 8.6 MB/s | 1.0 MB 00:00 2024-08-31T12:13:27.147 INFO:teuthology.orchestra.run.smithi043.stdout:(73/150): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-08-31T12:13:27.190 INFO:teuthology.orchestra.run.smithi043.stdout:(74/150): python3-colorama-0.4.6-3.el9.noarch.r 1.4 MB/s | 61 kB 00:00 2024-08-31T12:13:27.257 INFO:teuthology.orchestra.run.smithi043.stdout:(75/150): postgresql-server-13.16-1.el9.x86_64. 21 MB/s | 5.8 MB 00:00 2024-08-31T12:13:27.282 INFO:teuthology.orchestra.run.smithi043.stdout:(76/150): python3-devel-3.9.19-8.el9.x86_64.rpm 2.6 MB/s | 245 kB 00:00 2024-08-31T12:13:27.324 INFO:teuthology.orchestra.run.smithi043.stdout:(77/150): python3-jinja2-2.11.3-6.el9.noarch.rp 3.6 MB/s | 248 kB 00:00 2024-08-31T12:13:27.349 INFO:teuthology.orchestra.run.smithi043.stdout:(78/150): python3-jsonpatch-1.21-16.el9.noarch. 383 kB/s | 26 kB 00:00 2024-08-31T12:13:27.374 INFO:teuthology.orchestra.run.smithi043.stdout:(79/150): python3-jsonpointer-2.0-4.el9.noarch. 381 kB/s | 19 kB 00:00 2024-08-31T12:13:27.399 INFO:teuthology.orchestra.run.smithi043.stdout:(80/150): python3-mako-1.1.4-6.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-08-31T12:13:27.425 INFO:teuthology.orchestra.run.smithi043.stdout:(81/150): python3-markupsafe-1.1.1-12.el9.x86_6 696 kB/s | 35 kB 00:00 2024-08-31T12:13:27.517 INFO:teuthology.orchestra.run.smithi043.stdout:(82/150): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 6.0 MB 00:00 2024-08-31T12:13:27.542 INFO:teuthology.orchestra.run.smithi043.stdout:(83/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.1 MB/s | 250 kB 00:00 2024-08-31T12:13:27.567 INFO:teuthology.orchestra.run.smithi043.stdout:(84/150): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-08-31T12:13:27.592 INFO:teuthology.orchestra.run.smithi043.stdout:(85/150): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-08-31T12:13:27.618 INFO:teuthology.orchestra.run.smithi043.stdout:(86/150): python3-prettytable-0.7.2-27.el9.noar 840 kB/s | 42 kB 00:00 2024-08-31T12:13:27.643 INFO:teuthology.orchestra.run.smithi043.stdout:(87/150): python3-protobuf-3.14.0-13.el9.noarch 5.2 MB/s | 269 kB 00:00 2024-08-31T12:13:27.668 INFO:teuthology.orchestra.run.smithi043.stdout:(88/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-08-31T12:13:27.694 INFO:teuthology.orchestra.run.smithi043.stdout:(89/150): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-08-31T12:13:27.719 INFO:teuthology.orchestra.run.smithi043.stdout:(90/150): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-08-31T12:13:27.744 INFO:teuthology.orchestra.run.smithi043.stdout:(91/150): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-08-31T12:13:27.820 INFO:teuthology.orchestra.run.smithi043.stdout:(92/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 12 MB/s | 5.1 MB 00:00 2024-08-31T12:13:27.845 INFO:teuthology.orchestra.run.smithi043.stdout:(93/150): python3-toml-0.10.2-6.el9.noarch.rpm 416 kB/s | 42 kB 00:00 2024-08-31T12:13:27.872 INFO:teuthology.orchestra.run.smithi043.stdout:(94/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 886 kB/s | 44 kB 00:00 2024-08-31T12:13:27.895 INFO:teuthology.orchestra.run.smithi043.stdout:(95/150): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-08-31T12:13:27.921 INFO:teuthology.orchestra.run.smithi043.stdout:(96/150): qatlib-service-24.02.0-1.el9.x86_64.r 710 kB/s | 36 kB 00:00 2024-08-31T12:13:27.946 INFO:teuthology.orchestra.run.smithi043.stdout:(97/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 940 kB/s | 47 kB 00:00 2024-08-31T12:13:27.971 INFO:teuthology.orchestra.run.smithi043.stdout:(98/150): socat-1.7.4.1-6.el9.x86_64.rpm 5.9 MB/s | 304 kB 00:00 2024-08-31T12:13:27.997 INFO:teuthology.orchestra.run.smithi043.stdout:(99/150): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-08-31T12:13:28.022 INFO:teuthology.orchestra.run.smithi043.stdout:(100/150): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 1.9 MB/s | 96 kB 00:00 2024-08-31T12:13:28.047 INFO:teuthology.orchestra.run.smithi043.stdout:(101/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-08-31T12:13:28.072 INFO:teuthology.orchestra.run.smithi043.stdout:(102/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 894 kB/s | 45 kB 00:00 2024-08-31T12:13:28.131 INFO:teuthology.orchestra.run.smithi043.stdout:(103/150): lua-devel-5.4.4-4.el9.x86_64.rpm 267 kB/s | 22 kB 00:00 2024-08-31T12:13:28.181 INFO:teuthology.orchestra.run.smithi043.stdout:(104/150): abseil-cpp-20211102.0-3.el9.x86_64.r 11 MB/s | 548 kB 00:00 2024-08-31T12:13:28.215 INFO:teuthology.orchestra.run.smithi043.stdout:(105/150): gperftools-libs-2.9.1-2.el9.x86_64.r 9.1 MB/s | 309 kB 00:00 2024-08-31T12:13:28.240 INFO:teuthology.orchestra.run.smithi043.stdout:(106/150): grpc-data-1.46.7-10.el9.noarch.rpm 775 kB/s | 19 kB 00:00 2024-08-31T12:13:28.303 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 18 kB 00:00 2024-08-31T12:13:28.440 INFO:teuthology.orchestra.run.smithi043.stdout:(107/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 23 MB/s | 16 MB 00:00 2024-08-31T12:13:28.474 INFO:teuthology.orchestra.run.smithi043.stdout:(108/150): protobuf-compiler-3.14.0-13.el9.x86_ 2.1 MB/s | 863 kB 00:00 2024-08-31T12:13:28.516 INFO:teuthology.orchestra.run.smithi043.stdout:(109/150): libarrow-doc-9.0.0-11.el9.noarch.rpm 347 kB/s | 26 kB 00:00 2024-08-31T12:13:28.541 INFO:teuthology.orchestra.run.smithi043.stdout:(110/150): libunwind-1.6.2-1.el9.x86_64.rpm 2.9 MB/s | 67 kB 00:00 2024-08-31T12:13:28.566 INFO:teuthology.orchestra.run.smithi043.stdout:(111/150): liboath-2.6.7-2.el9.x86_64.rpm 577 kB/s | 49 kB 00:00 2024-08-31T12:13:28.591 INFO:teuthology.orchestra.run.smithi043.stdout:(112/150): luarocks-3.9.2-5.el9.noarch.rpm 3.0 MB/s | 151 kB 00:00 2024-08-31T12:13:28.633 INFO:teuthology.orchestra.run.smithi043.stdout:(113/150): python3-asyncssh-2.13.2-5.el9.noarch 13 MB/s | 548 kB 00:00 2024-08-31T12:13:28.650 INFO:teuthology.orchestra.run.smithi043.stdout:(114/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 2.6 MB/s | 43 kB 00:00 2024-08-31T12:13:28.675 INFO:teuthology.orchestra.run.smithi043.stdout:(115/150): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-08-31T12:13:28.717 INFO:teuthology.orchestra.run.smithi043.stdout:(116/150): parquet-libs-9.0.0-11.el9.x86_64.rpm 5.5 MB/s | 839 kB 00:00 2024-08-31T12:13:28.742 INFO:teuthology.orchestra.run.smithi043.stdout:(117/150): python3-certifi-2023.05.07-4.el9.noa 214 kB/s | 14 kB 00:00 2024-08-31T12:13:28.767 INFO:teuthology.orchestra.run.smithi043.stdout:(118/150): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-08-31T12:13:28.809 INFO:teuthology.orchestra.run.smithi043.stdout:(119/150): python3-cherrypy-18.6.1-2.el9.noarch 5.2 MB/s | 358 kB 00:00 2024-08-31T12:13:28.835 INFO:teuthology.orchestra.run.smithi043.stdout:(120/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.4 MB/s | 303 kB 00:00 2024-08-31T12:13:28.860 INFO:teuthology.orchestra.run.smithi043.stdout:(121/150): python3-google-auth-2.34.0-1.el9.noa 4.3 MB/s | 220 kB 00:00 2024-08-31T12:13:28.927 INFO:teuthology.orchestra.run.smithi043.stdout:(122/150): libarrow-9.0.0-11.el9.x86_64.rpm 6.4 MB/s | 4.4 MB 00:00 2024-08-31T12:13:28.961 INFO:teuthology.orchestra.run.smithi043.stdout:(123/150): python3-jaraco-8.2.1-3.el9.noarch.rp 314 kB/s | 11 kB 00:00 2024-08-31T12:13:28.986 INFO:teuthology.orchestra.run.smithi043.stdout:(124/150): python3-jaraco-classes-3.2.1-5.el9.n 710 kB/s | 18 kB 00:00 2024-08-31T12:13:29.012 INFO:teuthology.orchestra.run.smithi043.stdout:(125/150): python3-grpcio-tools-1.46.7-10.el9.x 953 kB/s | 144 kB 00:00 2024-08-31T12:13:29.037 INFO:teuthology.orchestra.run.smithi043.stdout:(126/150): python3-jaraco-collections-3.0.0-8.e 461 kB/s | 23 kB 00:00 2024-08-31T12:13:29.062 INFO:teuthology.orchestra.run.smithi043.stdout:(127/150): python3-jaraco-functools-3.5.0-2.el9 387 kB/s | 19 kB 00:00 2024-08-31T12:13:29.129 INFO:teuthology.orchestra.run.smithi043.stdout:(128/150): python3-jaraco-text-3.2.0-6.el9.noar 214 kB/s | 20 kB 00:00 2024-08-31T12:13:29.154 INFO:teuthology.orchestra.run.smithi043.stdout:(129/150): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-08-31T12:13:29.180 INFO:teuthology.orchestra.run.smithi043.stdout:(130/150): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-08-31T12:13:29.205 INFO:teuthology.orchestra.run.smithi043.stdout:(131/150): python3-natsort-7.1.1-5.el9.noarch.r 2.3 MB/s | 58 kB 00:00 2024-08-31T12:13:29.247 INFO:teuthology.orchestra.run.smithi043.stdout:(132/150): python3-kubernetes-26.1.0-2.el9.noar 5.5 MB/s | 1.0 MB 00:00 2024-08-31T12:13:29.272 INFO:teuthology.orchestra.run.smithi043.stdout:(133/150): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-08-31T12:13:29.289 INFO:teuthology.orchestra.run.smithi043.stdout:(134/150): python3-portend-3.1.0-2.el9.noarch.r 393 kB/s | 16 kB 00:00 2024-08-31T12:13:29.314 INFO:teuthology.orchestra.run.smithi043.stdout:(135/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 2.1 MB/s | 90 kB 00:00 2024-08-31T12:13:29.331 INFO:teuthology.orchestra.run.smithi043.stdout:(136/150): python3-pyperclip-1.8.2-2.el9.noarch 673 kB/s | 28 kB 00:00 2024-08-31T12:13:29.348 INFO:teuthology.orchestra.run.smithi043.stdout:(137/150): python3-repoze-lru-0.7-16.el9.noarch 918 kB/s | 31 kB 00:00 2024-08-31T12:13:29.390 INFO:teuthology.orchestra.run.smithi043.stdout:(138/150): python3-grpcio-1.46.7-10.el9.x86_64. 3.7 MB/s | 2.0 MB 00:00 2024-08-31T12:13:29.416 INFO:teuthology.orchestra.run.smithi043.stdout:(139/150): python3-rsa-4.9-2.el9.noarch.rpm 883 kB/s | 59 kB 00:00 2024-08-31T12:13:29.441 INFO:teuthology.orchestra.run.smithi043.stdout:(140/150): python3-tempora-5.0.0-2.el9.noarch.r 710 kB/s | 36 kB 00:00 2024-08-31T12:13:29.466 INFO:teuthology.orchestra.run.smithi019.stdout:Extra Packages for Enterprise Linux 22 MB/s | 22 MB 00:01 2024-08-31T12:13:29.466 INFO:teuthology.orchestra.run.smithi043.stdout:(141/150): python3-typing-extensions-4.12.2-2.e 1.4 MB/s | 74 kB 00:00 2024-08-31T12:13:29.492 INFO:teuthology.orchestra.run.smithi043.stdout:(142/150): python3-routes-2.5.1-5.el9.noarch.rp 1.2 MB/s | 188 kB 00:00 2024-08-31T12:13:29.517 INFO:teuthology.orchestra.run.smithi043.stdout:(143/150): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-08-31T12:13:29.542 INFO:teuthology.orchestra.run.smithi043.stdout:(144/150): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-08-31T12:13:29.568 INFO:teuthology.orchestra.run.smithi043.stdout:(145/150): python3-zc-lockfile-2.0-10.el9.noarc 398 kB/s | 20 kB 00:00 2024-08-31T12:13:29.593 INFO:teuthology.orchestra.run.smithi043.stdout:(146/150): re2-20211101-20.el9.x86_64.rpm 3.7 MB/s | 191 kB 00:00 2024-08-31T12:13:29.618 INFO:teuthology.orchestra.run.smithi043.stdout:(147/150): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-08-31T12:13:29.668 INFO:teuthology.orchestra.run.smithi043.stdout:(148/150): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-08-31T12:13:30.319 INFO:teuthology.orchestra.run.smithi043.stdout:(149/150): librbd1-19.3.0-4560.gd19e6118.el9.x8 4.5 MB/s | 3.1 MB 00:00 2024-08-31T12:13:30.385 INFO:teuthology.orchestra.run.smithi043.stdout:(150/150): librados2-19.3.0-4560.gd19e6118.el9. 4.3 MB/s | 3.4 MB 00:00 2024-08-31T12:13:30.391 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-08-31T12:13:30.392 INFO:teuthology.orchestra.run.smithi043.stdout:Total 25 MB/s | 224 MB 00:08 2024-08-31T12:13:31.519 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-08-31T12:13:31.659 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-08-31T12:13:31.659 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-08-31T12:13:33.504 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-08-31T12:13:33.504 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-08-31T12:13:35.080 INFO:teuthology.orchestra.run.smithi019.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-08-31T12:13:37.477 INFO:teuthology.orchestra.run.smithi019.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-08-31T12:13:37.478 INFO:teuthology.orchestra.run.smithi019.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-08-31T12:13:37.491 INFO:teuthology.orchestra.run.smithi019.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-08-31T12:13:37.492 INFO:teuthology.orchestra.run.smithi019.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-08-31T12:13:37.493 INFO:teuthology.orchestra.run.smithi019.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-08-31T12:13:37.494 INFO:teuthology.orchestra.run.smithi019.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-08-31T12:13:37.496 INFO:teuthology.orchestra.run.smithi019.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-08-31T12:13:37.499 INFO:teuthology.orchestra.run.smithi019.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-08-31T12:13:37.500 INFO:teuthology.orchestra.run.smithi019.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-08-31T12:13:37.565 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-08-31T12:13:37.574 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================= 2024-08-31T12:13:37.574 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-08-31T12:13:37.574 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================= 2024-08-31T12:13:37.574 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-08-31T12:13:37.574 INFO:teuthology.orchestra.run.smithi019.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-08-31T12:13:37.574 INFO:teuthology.orchestra.run.smithi019.stdout: ceph x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 6.6 k 2024-08-31T12:13:37.574 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-base x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 5.5 M 2024-08-31T12:13:37.574 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-fuse x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 874 k 2024-08-31T12:13:37.574 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 148 k 2024-08-31T12:13:37.574 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 1.5 M 2024-08-31T12:13:37.574 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 154 k 2024-08-31T12:13:37.575 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 14 M 2024-08-31T12:13:37.575 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 7.4 M 2024-08-31T12:13:37.575 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-rook noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 49 k 2024-08-31T12:13:37.575 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-radosgw x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 10 M 2024-08-31T12:13:37.575 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-test x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 44 M 2024-08-31T12:13:37.575 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-volume noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 285 k 2024-08-31T12:13:37.575 INFO:teuthology.orchestra.run.smithi019.stdout: cephadm noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 781 k 2024-08-31T12:13:37.575 INFO:teuthology.orchestra.run.smithi019.stdout: cephfs-mirror x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 226 k 2024-08-31T12:13:37.575 INFO:teuthology.orchestra.run.smithi019.stdout: cephfs-shell x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 27 k 2024-08-31T12:13:37.575 INFO:teuthology.orchestra.run.smithi019.stdout: cephfs-top noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 18 k 2024-08-31T12:13:37.575 INFO:teuthology.orchestra.run.smithi019.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-08-31T12:13:37.575 INFO:teuthology.orchestra.run.smithi019.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-08-31T12:13:37.576 INFO:teuthology.orchestra.run.smithi019.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-08-31T12:13:37.576 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs-devel x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 32 k 2024-08-31T12:13:37.576 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs2 x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 766 k 2024-08-31T12:13:37.576 INFO:teuthology.orchestra.run.smithi019.stdout: librados-devel x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 127 k 2024-08-31T12:13:37.576 INFO:teuthology.orchestra.run.smithi019.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-08-31T12:13:37.576 INFO:teuthology.orchestra.run.smithi019.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-08-31T12:13:37.576 INFO:teuthology.orchestra.run.smithi019.stdout: openssl-devel x86_64 1:3.2.2-4.el9 appstream 4.4 M 2024-08-31T12:13:37.576 INFO:teuthology.orchestra.run.smithi019.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-08-31T12:13:37.576 INFO:teuthology.orchestra.run.smithi019.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-08-31T12:13:37.576 INFO:teuthology.orchestra.run.smithi019.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-08-31T12:13:37.576 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cephfs x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 161 k 2024-08-31T12:13:37.576 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rados x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 321 k 2024-08-31T12:13:37.577 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rbd x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 305 k 2024-08-31T12:13:37.577 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rgw x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 99 k 2024-08-31T12:13:37.577 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-fuse x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 86 k 2024-08-31T12:13:37.577 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-mirror x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 3.1 M 2024-08-31T12:13:37.577 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-nbd x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 170 k 2024-08-31T12:13:37.577 INFO:teuthology.orchestra.run.smithi019.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-08-31T12:13:37.577 INFO:teuthology.orchestra.run.smithi019.stdout:Upgrading: 2024-08-31T12:13:37.577 INFO:teuthology.orchestra.run.smithi019.stdout: librados2 x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 3.4 M 2024-08-31T12:13:37.577 INFO:teuthology.orchestra.run.smithi019.stdout: librbd1 x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 3.1 M 2024-08-31T12:13:37.577 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-08-31T12:13:37.577 INFO:teuthology.orchestra.run.smithi019.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-08-31T12:13:37.578 INFO:teuthology.orchestra.run.smithi019.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-08-31T12:13:37.578 INFO:teuthology.orchestra.run.smithi019.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-08-31T12:13:37.578 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-common x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 20 M 2024-08-31T12:13:37.578 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 34 k 2024-08-31T12:13:37.578 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mds x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 2.3 M 2024-08-31T12:13:37.578 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 290 k 2024-08-31T12:13:37.578 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mon x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 4.9 M 2024-08-31T12:13:37.578 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-osd x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 18 M 2024-08-31T12:13:37.578 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4560.gd19e6118.el9 ceph-noarch 16 k 2024-08-31T12:13:37.578 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-selinux x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 25 k 2024-08-31T12:13:37.578 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-08-31T12:13:37.579 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-08-31T12:13:37.579 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-08-31T12:13:37.579 INFO:teuthology.orchestra.run.smithi019.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-31T12:13:37.579 INFO:teuthology.orchestra.run.smithi019.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-31T12:13:37.579 INFO:teuthology.orchestra.run.smithi019.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-08-31T12:13:37.579 INFO:teuthology.orchestra.run.smithi019.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-08-31T12:13:37.579 INFO:teuthology.orchestra.run.smithi019.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-08-31T12:13:37.579 INFO:teuthology.orchestra.run.smithi019.stdout: libcephsqlite x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 159 k 2024-08-31T12:13:37.579 INFO:teuthology.orchestra.run.smithi019.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-08-31T12:13:37.579 INFO:teuthology.orchestra.run.smithi019.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-31T12:13:37.579 INFO:teuthology.orchestra.run.smithi019.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-31T12:13:37.579 INFO:teuthology.orchestra.run.smithi019.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-08-31T12:13:37.579 INFO:teuthology.orchestra.run.smithi019.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-31T12:13:37.580 INFO:teuthology.orchestra.run.smithi019.stdout: libradosstriper1 x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 506 k 2024-08-31T12:13:37.580 INFO:teuthology.orchestra.run.smithi019.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-31T12:13:37.580 INFO:teuthology.orchestra.run.smithi019.stdout: librgw2 x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 5.3 M 2024-08-31T12:13:37.580 INFO:teuthology.orchestra.run.smithi019.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-31T12:13:37.580 INFO:teuthology.orchestra.run.smithi019.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-08-31T12:13:37.580 INFO:teuthology.orchestra.run.smithi019.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-31T12:13:37.580 INFO:teuthology.orchestra.run.smithi019.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-08-31T12:13:37.580 INFO:teuthology.orchestra.run.smithi019.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-08-31T12:13:37.580 INFO:teuthology.orchestra.run.smithi019.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-08-31T12:13:37.580 INFO:teuthology.orchestra.run.smithi019.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-08-31T12:13:37.580 INFO:teuthology.orchestra.run.smithi019.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-08-31T12:13:37.580 INFO:teuthology.orchestra.run.smithi019.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-08-31T12:13:37.581 INFO:teuthology.orchestra.run.smithi019.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-08-31T12:13:37.581 INFO:teuthology.orchestra.run.smithi019.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-08-31T12:13:37.581 INFO:teuthology.orchestra.run.smithi019.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-08-31T12:13:37.581 INFO:teuthology.orchestra.run.smithi019.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-08-31T12:13:37.581 INFO:teuthology.orchestra.run.smithi019.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-08-31T12:13:37.581 INFO:teuthology.orchestra.run.smithi019.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-08-31T12:13:37.581 INFO:teuthology.orchestra.run.smithi019.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-31T12:13:37.581 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-08-31T12:13:37.581 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-argparse x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 45 k 2024-08-31T12:13:37.581 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-common x86_64 2:19.3.0-4560.gd19e6118.el9 ceph 141 k 2024-08-31T12:13:37.581 INFO:teuthology.orchestra.run.smithi019.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-08-31T12:13:37.581 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-31T12:13:37.581 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-31T12:13:37.582 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-31T12:13:37.582 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-08-31T12:13:37.582 INFO:teuthology.orchestra.run.smithi019.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-08-31T12:13:37.582 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-31T12:13:37.582 INFO:teuthology.orchestra.run.smithi019.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-08-31T12:13:37.582 INFO:teuthology.orchestra.run.smithi019.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-08-31T12:13:37.582 INFO:teuthology.orchestra.run.smithi019.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-08-31T12:13:37.582 INFO:teuthology.orchestra.run.smithi019.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-08-31T12:13:37.582 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-31T12:13:37.582 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-31T12:13:37.582 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-08-31T12:13:37.582 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-31T12:13:37.582 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-31T12:13:37.583 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-08-31T12:13:37.583 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-08-31T12:13:37.583 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-08-31T12:13:37.583 INFO:teuthology.orchestra.run.smithi019.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-08-31T12:13:37.583 INFO:teuthology.orchestra.run.smithi019.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-31T12:13:37.583 INFO:teuthology.orchestra.run.smithi019.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-31T12:13:37.583 INFO:teuthology.orchestra.run.smithi019.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-31T12:13:37.583 INFO:teuthology.orchestra.run.smithi019.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-31T12:13:37.583 INFO:teuthology.orchestra.run.smithi019.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-08-31T12:13:37.583 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-08-31T12:13:37.583 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-08-31T12:13:37.583 INFO:teuthology.orchestra.run.smithi019.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-08-31T12:13:37.583 INFO:teuthology.orchestra.run.smithi019.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-08-31T12:13:37.584 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-31T12:13:37.584 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-31T12:13:37.584 INFO:teuthology.orchestra.run.smithi019.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-31T12:13:37.584 INFO:teuthology.orchestra.run.smithi019.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-31T12:13:37.584 INFO:teuthology.orchestra.run.smithi019.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-08-31T12:13:37.584 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-31T12:13:37.584 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-08-31T12:13:37.584 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-08-31T12:13:37.584 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-31T12:13:37.584 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-08-31T12:13:37.584 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-31T12:13:37.584 INFO:teuthology.orchestra.run.smithi019.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-08-31T12:13:37.585 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-31T12:13:37.585 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-08-31T12:13:37.585 INFO:teuthology.orchestra.run.smithi019.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-08-31T12:13:37.585 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-08-31T12:13:37.585 INFO:teuthology.orchestra.run.smithi019.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-08-31T12:13:37.585 INFO:teuthology.orchestra.run.smithi019.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-31T12:13:37.585 INFO:teuthology.orchestra.run.smithi019.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-31T12:13:37.585 INFO:teuthology.orchestra.run.smithi019.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-08-31T12:13:37.585 INFO:teuthology.orchestra.run.smithi019.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-31T12:13:37.585 INFO:teuthology.orchestra.run.smithi019.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-08-31T12:13:37.585 INFO:teuthology.orchestra.run.smithi019.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-08-31T12:13:37.585 INFO:teuthology.orchestra.run.smithi019.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-08-31T12:13:37.585 INFO:teuthology.orchestra.run.smithi019.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-31T12:13:37.586 INFO:teuthology.orchestra.run.smithi019.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-31T12:13:37.586 INFO:teuthology.orchestra.run.smithi019.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-08-31T12:13:37.586 INFO:teuthology.orchestra.run.smithi019.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-08-31T12:13:37.586 INFO:teuthology.orchestra.run.smithi019.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-08-31T12:13:37.586 INFO:teuthology.orchestra.run.smithi019.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-31T12:13:37.586 INFO:teuthology.orchestra.run.smithi019.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-31T12:13:37.586 INFO:teuthology.orchestra.run.smithi019.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-08-31T12:13:37.586 INFO:teuthology.orchestra.run.smithi019.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-31T12:13:37.586 INFO:teuthology.orchestra.run.smithi019.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-08-31T12:13:37.586 INFO:teuthology.orchestra.run.smithi019.stdout:Installing weak dependencies: 2024-08-31T12:13:37.586 INFO:teuthology.orchestra.run.smithi019.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-08-31T12:13:37.586 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:13:37.586 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-08-31T12:13:37.586 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================= 2024-08-31T12:13:37.587 INFO:teuthology.orchestra.run.smithi019.stdout:Install 148 Packages 2024-08-31T12:13:37.587 INFO:teuthology.orchestra.run.smithi019.stdout:Upgrade 2 Packages 2024-08-31T12:13:37.587 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:13:37.587 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 224 M 2024-08-31T12:13:37.587 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-08-31T12:13:38.256 INFO:teuthology.orchestra.run.smithi019.stdout:(1/150): ceph-19.3.0-4560.gd19e6118.el9.x86_64. 53 kB/s | 6.6 kB 00:00 2024-08-31T12:13:38.448 INFO:teuthology.orchestra.run.smithi019.stdout:(2/150): ceph-fuse-19.3.0-4560.gd19e6118.el9.x8 4.5 MB/s | 874 kB 00:00 2024-08-31T12:13:38.507 INFO:teuthology.orchestra.run.smithi019.stdout:(3/150): ceph-immutable-object-cache-19.3.0-456 2.5 MB/s | 148 kB 00:00 2024-08-31T12:13:38.799 INFO:teuthology.orchestra.run.smithi019.stdout:(4/150): ceph-mds-19.3.0-4560.gd19e6118.el9.x86 7.8 MB/s | 2.3 MB 00:00 2024-08-31T12:13:38.874 INFO:teuthology.orchestra.run.smithi019.stdout:(5/150): ceph-base-19.3.0-4560.gd19e6118.el9.x8 7.4 MB/s | 5.5 MB 00:00 2024-08-31T12:13:39.033 INFO:teuthology.orchestra.run.smithi019.stdout:(6/150): ceph-mgr-19.3.0-4560.gd19e6118.el9.x86 6.5 MB/s | 1.5 MB 00:00 2024-08-31T12:13:39.433 INFO:teuthology.orchestra.run.smithi019.stdout:(7/150): ceph-mon-19.3.0-4560.gd19e6118.el9.x86 8.8 MB/s | 4.9 MB 00:00 2024-08-31T12:13:39.434 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-08-31T12:13:39.513 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-08-31T12:13:39.546 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-08-31T12:13:39.590 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-08-31T12:13:40.010 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-08-31T12:13:40.080 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : librados2-2:19.3.0-4560.gd19e6118.el9.x86_64 5/152 2024-08-31T12:13:40.208 INFO:teuthology.orchestra.run.smithi019.stdout:(8/150): ceph-common-19.3.0-4560.gd19e6118.el9. 9.6 MB/s | 20 MB 00:02 2024-08-31T12:13:40.209 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librados2-2:19.3.0-4560.gd19e6118.el9.x86_64 5/152 2024-08-31T12:13:40.239 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libcephfs2-2:19.3.0-4560.gd19e6118.el9.x86_64 6/152 2024-08-31T12:13:40.300 INFO:teuthology.orchestra.run.smithi019.stdout:(9/150): ceph-selinux-19.3.0-4560.gd19e6118.el9 275 kB/s | 25 kB 00:00 2024-08-31T12:13:40.334 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: libcephfs2-2:19.3.0-4560.gd19e6118.el9.x86_64 6/152 2024-08-31T12:13:40.376 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-rados-2:19.3.0-4560.gd19e6118.el9.x86_64 7/152 2024-08-31T12:13:40.404 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-08-31T12:13:40.430 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-08-31T12:13:40.591 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 10/152 2024-08-31T12:13:40.600 INFO:teuthology.orchestra.run.smithi019.stdout:(10/150): ceph-radosgw-19.3.0-4560.gd19e6118.el 8.9 MB/s | 10 MB 00:01 2024-08-31T12:13:40.653 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : postgresql-13.16-1.el9.x86_64 11/152 2024-08-31T12:13:40.680 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libcephsqlite-2:19.3.0-4560.gd19e6118.el9.x86_64 12/152 2024-08-31T12:13:40.693 INFO:teuthology.orchestra.run.smithi019.stdout:(11/150): cephfs-mirror-19.3.0-4560.gd19e6118.e 2.4 MB/s | 226 kB 00:00 2024-08-31T12:13:40.771 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4560.gd19e6118.el9.x86_64 12/152 2024-08-31T12:13:40.793 INFO:teuthology.orchestra.run.smithi019.stdout:(12/150): cephfs-shell-19.3.0-4560.gd19e6118.el 270 kB/s | 27 kB 00:00 2024-08-31T12:13:40.806 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libradosstriper1-2:19.3.0-4560.gd19e6118.el9.x86 13/152 2024-08-31T12:13:40.852 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4560.gd19e6118.el9.x86 13/152 2024-08-31T12:13:40.874 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-08-31T12:13:40.885 INFO:teuthology.orchestra.run.smithi019.stdout:(13/150): libcephfs-devel-19.3.0-4560.gd19e6118 345 kB/s | 32 kB 00:00 2024-08-31T12:13:40.899 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-08-31T12:13:41.018 INFO:teuthology.orchestra.run.smithi019.stdout:(14/150): libcephfs2-19.3.0-4560.gd19e6118.el9. 5.6 MB/s | 766 kB 00:00 2024-08-31T12:13:41.094 INFO:teuthology.orchestra.run.smithi019.stdout:(15/150): libcephsqlite-19.3.0-4560.gd19e6118.e 2.1 MB/s | 159 kB 00:00 2024-08-31T12:13:41.188 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-08-31T12:13:41.224 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-08-31T12:13:41.334 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : re2-1:20211101-20.el9.x86_64 16/152 2024-08-31T12:13:41.361 INFO:teuthology.orchestra.run.smithi019.stdout:(16/150): ceph-osd-19.3.0-4560.gd19e6118.el9.x8 7.7 MB/s | 18 MB 00:02 2024-08-31T12:13:41.374 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 17/152 2024-08-31T12:13:41.416 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : liboath-2.6.7-2.el9.x86_64 18/152 2024-08-31T12:13:41.493 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 19/152 2024-08-31T12:13:41.494 INFO:teuthology.orchestra.run.smithi019.stdout:(17/150): librados-devel-19.3.0-4560.gd19e6118. 316 kB/s | 127 kB 00:00 2024-08-31T12:13:41.536 INFO:teuthology.orchestra.run.smithi019.stdout:(18/150): libradosstriper1-19.3.0-4560.gd19e611 2.8 MB/s | 506 kB 00:00 2024-08-31T12:13:41.543 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 20/152 2024-08-31T12:13:41.582 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-packaging-20.9-5.el9.noarch 21/152 2024-08-31T12:13:41.625 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 22/152 2024-08-31T12:13:41.637 INFO:teuthology.orchestra.run.smithi019.stdout:(19/150): python3-ceph-argparse-19.3.0-4560.gd1 451 kB/s | 45 kB 00:00 2024-08-31T12:13:41.658 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 23/152 2024-08-31T12:13:41.685 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : lua-5.4.4-4.el9.x86_64 24/152 2024-08-31T12:13:41.737 INFO:teuthology.orchestra.run.smithi019.stdout:(20/150): python3-ceph-common-19.3.0-4560.gd19e 1.4 MB/s | 141 kB 00:00 2024-08-31T12:13:41.758 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 25/152 2024-08-31T12:13:41.818 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 26/152 2024-08-31T12:13:41.820 INFO:teuthology.orchestra.run.smithi019.stdout:(21/150): python3-cephfs-19.3.0-4560.gd19e6118. 1.9 MB/s | 161 kB 00:00 2024-08-31T12:13:41.846 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 27/152 2024-08-31T12:13:41.891 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 28/152 2024-08-31T12:13:41.896 INFO:teuthology.orchestra.run.smithi019.stdout:(22/150): python3-rados-19.3.0-4560.gd19e6118.e 4.2 MB/s | 321 kB 00:00 2024-08-31T12:13:41.970 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 29/152 2024-08-31T12:13:42.001 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-ceph-common-2:19.3.0-4560.gd19e6118.el9. 30/152 2024-08-31T12:13:42.013 INFO:teuthology.orchestra.run.smithi019.stdout:(23/150): python3-rbd-19.3.0-4560.gd19e6118.el9 2.6 MB/s | 305 kB 00:00 2024-08-31T12:13:42.045 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-ceph-argparse-2:19.3.0-4560.gd19e6118.el 31/152 2024-08-31T12:13:42.113 INFO:teuthology.orchestra.run.smithi019.stdout:(24/150): python3-rgw-19.3.0-4560.gd19e6118.el9 995 kB/s | 99 kB 00:00 2024-08-31T12:13:42.115 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cephfs-2:19.3.0-4560.gd19e6118.el9.x86_6 32/152 2024-08-31T12:13:42.191 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/152 2024-08-31T12:13:42.222 INFO:teuthology.orchestra.run.smithi019.stdout:(25/150): rbd-fuse-19.3.0-4560.gd19e6118.el9.x8 796 kB/s | 86 kB 00:00 2024-08-31T12:13:42.238 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : luarocks-3.9.2-5.el9.noarch 34/152 2024-08-31T12:13:42.317 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 35/152 2024-08-31T12:13:42.463 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/152 2024-08-31T12:13:42.517 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/152 2024-08-31T12:13:42.805 INFO:teuthology.orchestra.run.smithi019.stdout:(26/150): librgw2-19.3.0-4560.gd19e6118.el9.x86 4.0 MB/s | 5.3 MB 00:01 2024-08-31T12:13:42.897 INFO:teuthology.orchestra.run.smithi019.stdout:(27/150): rbd-nbd-19.3.0-4560.gd19e6118.el9.x86 1.8 MB/s | 170 kB 00:00 2024-08-31T12:13:42.981 INFO:teuthology.orchestra.run.smithi019.stdout:(28/150): ceph-grafana-dashboards-19.3.0-4560.g 410 kB/s | 34 kB 00:00 2024-08-31T12:13:43.073 INFO:teuthology.orchestra.run.smithi019.stdout:(29/150): ceph-mgr-cephadm-19.3.0-4560.gd19e611 1.6 MB/s | 154 kB 00:00 2024-08-31T12:13:43.092 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/152 2024-08-31T12:13:43.173 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/152 2024-08-31T12:13:43.198 INFO:teuthology.orchestra.run.smithi019.stdout:(30/150): rbd-mirror-19.3.0-4560.gd19e6118.el9. 3.2 MB/s | 3.1 MB 00:00 2024-08-31T12:13:43.207 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 40/152 2024-08-31T12:13:43.250 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 41/152 2024-08-31T12:13:43.288 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : librados-devel-2:19.3.0-4560.gd19e6118.el9.x86_6 42/152 2024-08-31T12:13:43.321 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/152 2024-08-31T12:13:43.418 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/152 2024-08-31T12:13:43.480 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 45/152 2024-08-31T12:13:43.554 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 46/152 2024-08-31T12:13:43.590 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 47/152 2024-08-31T12:13:43.626 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 48/152 2024-08-31T12:13:43.674 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 49/152 2024-08-31T12:13:43.708 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 50/152 2024-08-31T12:13:43.753 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 51/152 2024-08-31T12:13:43.794 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 52/152 2024-08-31T12:13:43.949 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 53/152 2024-08-31T12:13:43.987 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 54/152 2024-08-31T12:13:44.029 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 55/152 2024-08-31T12:13:44.140 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-08-31T12:13:44.170 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 57/152 2024-08-31T12:13:44.202 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-08-31T12:13:44.227 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-08-31T12:13:44.252 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 60/152 2024-08-31T12:13:44.330 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-08-31T12:13:44.461 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-08-31T12:13:44.493 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-08-31T12:13:44.521 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-08-31T12:13:44.555 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-08-31T12:13:44.600 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-08-31T12:13:44.645 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 67/152 2024-08-31T12:13:44.685 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-08-31T12:13:44.715 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-08-31T12:13:44.750 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-08-31T12:13:44.783 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-08-31T12:13:44.825 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-08-31T12:13:44.859 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-08-31T12:13:44.950 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-08-31T12:13:44.965 INFO:teuthology.orchestra.run.smithi019.stdout:(31/150): ceph-test-19.3.0-4560.gd19e6118.el9.x 9.4 MB/s | 44 MB 00:04 2024-08-31T12:13:45.057 INFO:teuthology.orchestra.run.smithi019.stdout:(32/150): ceph-mgr-diskprediction-local-19.3.0- 4.0 MB/s | 7.4 MB 00:01 2024-08-31T12:13:45.082 INFO:teuthology.orchestra.run.smithi019.stdout:(33/150): ceph-mgr-modules-core-19.3.0-4560.gd1 2.4 MB/s | 290 kB 00:00 2024-08-31T12:13:45.108 INFO:teuthology.orchestra.run.smithi019.stdout:(34/150): ceph-mgr-rook-19.3.0-4560.gd19e6118.e 986 kB/s | 49 kB 00:00 2024-08-31T12:13:45.133 INFO:teuthology.orchestra.run.smithi019.stdout:(35/150): ceph-prometheus-alerts-19.3.0-4560.gd 328 kB/s | 16 kB 00:00 2024-08-31T12:13:45.208 INFO:teuthology.orchestra.run.smithi019.stdout:(36/150): ceph-volume-19.3.0-4560.gd19e6118.el9 2.8 MB/s | 285 kB 00:00 2024-08-31T12:13:45.209 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-08-31T12:13:45.238 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-08-31T12:13:45.350 INFO:teuthology.orchestra.run.smithi019.stdout:(37/150): ceph-mgr-dashboard-19.3.0-4560.gd19e6 6.2 MB/s | 14 MB 00:02 2024-08-31T12:13:45.351 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-08-31T12:13:45.376 INFO:teuthology.orchestra.run.smithi019.stdout:(38/150): cephfs-top-19.3.0-4560.gd19e6118.el9. 109 kB/s | 18 kB 00:00 2024-08-31T12:13:45.401 INFO:teuthology.orchestra.run.smithi019.stdout:(39/150): cephadm-19.3.0-4560.gd19e6118.el9.noa 2.9 MB/s | 781 kB 00:00 2024-08-31T12:13:45.407 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 78/152 2024-08-31T12:13:45.810 INFO:teuthology.orchestra.run.smithi019.stdout:(40/150): fuse-2.9.9-16.el9.x86_64.rpm 174 kB/s | 80 kB 00:00 2024-08-31T12:13:45.935 INFO:teuthology.orchestra.run.smithi019.stdout:(41/150): libquadmath-11.5.0-2.el9.x86_64.rpm 358 kB/s | 191 kB 00:00 2024-08-31T12:13:46.093 INFO:teuthology.orchestra.run.smithi019.stdout:(42/150): libgfortran-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 801 kB 00:00 2024-08-31T12:13:46.188 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 79/152 2024-08-31T12:13:46.244 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 80/152 2024-08-31T12:13:46.277 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 81/152 2024-08-31T12:13:46.319 INFO:teuthology.orchestra.run.smithi019.stdout:(43/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 498 kB/s | 253 kB 00:00 2024-08-31T12:13:46.382 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 82/152 2024-08-31T12:13:46.441 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 83/152 2024-08-31T12:13:46.486 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 84/152 2024-08-31T12:13:46.519 INFO:teuthology.orchestra.run.smithi019.stdout:(44/150): python3-ply-3.11-14.el9.noarch.rpm 250 kB/s | 106 kB 00:00 2024-08-31T12:13:46.586 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-08-31T12:13:46.603 INFO:teuthology.orchestra.run.smithi019.stdout:(45/150): python3-cryptography-36.0.1-4.el9.x86 1.8 MB/s | 1.2 MB 00:00 2024-08-31T12:13:46.621 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-08-31T12:13:46.753 INFO:teuthology.orchestra.run.smithi019.stdout:(46/150): python3-pycparser-2.20-6.el9.noarch.r 311 kB/s | 135 kB 00:00 2024-08-31T12:13:46.811 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : openblas-0.3.26-2.el9.x86_64 87/152 2024-08-31T12:13:46.844 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 88/152 2024-08-31T12:13:46.915 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-08-31T12:13:47.045 INFO:teuthology.orchestra.run.smithi019.stdout:(47/150): automake-1.16.2-8.el9.noarch.rpm 2.3 MB/s | 689 kB 00:00 2024-08-31T12:13:47.070 INFO:teuthology.orchestra.run.smithi019.stdout:(48/150): python3-requests-2.25.1-8.el9.noarch. 227 kB/s | 125 kB 00:00 2024-08-31T12:13:47.112 INFO:teuthology.orchestra.run.smithi019.stdout:(49/150): bison-3.7.4-5.el9.x86_64.rpm 14 MB/s | 943 kB 00:00 2024-08-31T12:13:47.154 INFO:teuthology.orchestra.run.smithi019.stdout:(50/150): elfutils-libelf-devel-0.191-4.el9.x86 523 kB/s | 22 kB 00:00 2024-08-31T12:13:47.196 INFO:teuthology.orchestra.run.smithi019.stdout:(51/150): flex-2.6.4-9.el9.x86_64.rpm 7.3 MB/s | 313 kB 00:00 2024-08-31T12:13:47.221 INFO:teuthology.orchestra.run.smithi019.stdout:(52/150): boost-program-options-1.75.0-8.el9.x8 709 kB/s | 107 kB 00:00 2024-08-31T12:13:47.247 INFO:teuthology.orchestra.run.smithi019.stdout:(53/150): flexiblas-3.0.4-8.el9.x86_64.rpm 652 kB/s | 33 kB 00:00 2024-08-31T12:13:47.272 INFO:teuthology.orchestra.run.smithi019.stdout:(54/150): python3-urllib3-1.26.5-6.el9.noarch.r 321 kB/s | 215 kB 00:00 2024-08-31T12:13:47.297 INFO:teuthology.orchestra.run.smithi019.stdout:(55/150): flexiblas-openblas-openmp-3.0.4-8.el9 362 kB/s | 18 kB 00:00 2024-08-31T12:13:47.339 INFO:teuthology.orchestra.run.smithi019.stdout:(56/150): libaio-devel-0.3.111-13.el9.x86_64.rp 167 kB/s | 11 kB 00:00 2024-08-31T12:13:47.364 INFO:teuthology.orchestra.run.smithi019.stdout:(57/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.3 MB/s | 160 kB 00:00 2024-08-31T12:13:47.473 INFO:teuthology.orchestra.run.smithi019.stdout:(58/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 12 MB/s | 3.0 MB 00:00 2024-08-31T12:13:47.498 INFO:teuthology.orchestra.run.smithi019.stdout:(59/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 284 kB/s | 45 kB 00:00 2024-08-31T12:13:47.532 INFO:teuthology.orchestra.run.smithi019.stdout:(60/150): libtool-2.4.6-46.el9.x86_64.rpm 9.7 MB/s | 581 kB 00:00 2024-08-31T12:13:47.557 INFO:teuthology.orchestra.run.smithi019.stdout:(61/150): libuuid-devel-2.37.4-20.el9.x86_64.rp 418 kB/s | 24 kB 00:00 2024-08-31T12:13:47.582 INFO:teuthology.orchestra.run.smithi019.stdout:(62/150): librdkafka-1.6.1-102.el9.x86_64.rpm 3.0 MB/s | 662 kB 00:00 2024-08-31T12:13:47.608 INFO:teuthology.orchestra.run.smithi019.stdout:(63/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 627 kB/s | 47 kB 00:00 2024-08-31T12:13:47.615 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-08-31T12:13:47.633 INFO:teuthology.orchestra.run.smithi019.stdout:(64/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.8 MB/s | 292 kB 00:00 2024-08-31T12:13:47.658 INFO:teuthology.orchestra.run.smithi019.stdout:(65/150): openblas-0.3.26-2.el9.x86_64.rpm 767 kB/s | 38 kB 00:00 2024-08-31T12:13:47.684 INFO:teuthology.orchestra.run.smithi019.stdout:(66/150): lua-5.4.4-4.el9.x86_64.rpm 1.8 MB/s | 188 kB 00:00 2024-08-31T12:13:47.748 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-08-31T12:13:47.776 INFO:teuthology.orchestra.run.smithi019.stdout:(67/150): openblas-openmp-0.3.26-2.el9.x86_64.r 35 MB/s | 4.9 MB 00:00 2024-08-31T12:13:47.835 INFO:teuthology.orchestra.run.smithi019.stdout:(68/150): openssl-devel-3.2.2-4.el9.x86_64.rpm 25 MB/s | 4.4 MB 00:00 2024-08-31T12:13:47.876 INFO:teuthology.orchestra.run.smithi019.stdout:(69/150): postgresql-contrib-13.16-1.el9.x86_64 8.9 MB/s | 910 kB 00:00 2024-08-31T12:13:47.902 INFO:teuthology.orchestra.run.smithi019.stdout:(70/150): postgresql-private-libs-13.16-1.el9.x 2.0 MB/s | 137 kB 00:00 2024-08-31T12:13:47.977 INFO:teuthology.orchestra.run.smithi019.stdout:(71/150): postgresql-13.16-1.el9.x86_64.rpm 5.4 MB/s | 1.6 MB 00:00 2024-08-31T12:13:48.019 INFO:teuthology.orchestra.run.smithi019.stdout:(72/150): protobuf-3.14.0-13.el9.x86_64.rpm 8.6 MB/s | 1.0 MB 00:00 2024-08-31T12:13:48.044 INFO:teuthology.orchestra.run.smithi019.stdout:(73/150): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-08-31T12:13:48.078 INFO:teuthology.orchestra.run.smithi019.stdout:(74/150): python3-colorama-0.4.6-3.el9.noarch.r 1.8 MB/s | 61 kB 00:00 2024-08-31T12:13:48.153 INFO:teuthology.orchestra.run.smithi019.stdout:(75/150): postgresql-server-13.16-1.el9.x86_64. 21 MB/s | 5.8 MB 00:00 2024-08-31T12:13:48.179 INFO:teuthology.orchestra.run.smithi019.stdout:(76/150): python3-devel-3.9.19-8.el9.x86_64.rpm 2.4 MB/s | 245 kB 00:00 2024-08-31T12:13:48.204 INFO:teuthology.orchestra.run.smithi019.stdout:(77/150): python3-jinja2-2.11.3-6.el9.noarch.rp 4.8 MB/s | 248 kB 00:00 2024-08-31T12:13:48.229 INFO:teuthology.orchestra.run.smithi019.stdout:(78/150): python3-jsonpatch-1.21-16.el9.noarch. 511 kB/s | 26 kB 00:00 2024-08-31T12:13:48.254 INFO:teuthology.orchestra.run.smithi019.stdout:(79/150): python3-jsonpointer-2.0-4.el9.noarch. 381 kB/s | 19 kB 00:00 2024-08-31T12:13:48.280 INFO:teuthology.orchestra.run.smithi019.stdout:(80/150): python3-mako-1.1.4-6.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-08-31T12:13:48.305 INFO:teuthology.orchestra.run.smithi019.stdout:(81/150): python3-markupsafe-1.1.1-12.el9.x86_6 693 kB/s | 35 kB 00:00 2024-08-31T12:13:48.380 INFO:teuthology.orchestra.run.smithi019.stdout:(82/150): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 6.0 MB 00:00 2024-08-31T12:13:48.406 INFO:teuthology.orchestra.run.smithi019.stdout:(83/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.4 MB/s | 250 kB 00:00 2024-08-31T12:13:48.431 INFO:teuthology.orchestra.run.smithi019.stdout:(84/150): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-08-31T12:13:48.456 INFO:teuthology.orchestra.run.smithi019.stdout:(85/150): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-08-31T12:13:48.481 INFO:teuthology.orchestra.run.smithi019.stdout:(86/150): python3-prettytable-0.7.2-27.el9.noar 848 kB/s | 42 kB 00:00 2024-08-31T12:13:48.507 INFO:teuthology.orchestra.run.smithi019.stdout:(87/150): python3-protobuf-3.14.0-13.el9.noarch 5.3 MB/s | 269 kB 00:00 2024-08-31T12:13:48.532 INFO:teuthology.orchestra.run.smithi019.stdout:(88/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-08-31T12:13:48.557 INFO:teuthology.orchestra.run.smithi019.stdout:(89/150): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-08-31T12:13:48.583 INFO:teuthology.orchestra.run.smithi019.stdout:(90/150): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-08-31T12:13:48.608 INFO:teuthology.orchestra.run.smithi019.stdout:(91/150): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-08-31T12:13:48.683 INFO:teuthology.orchestra.run.smithi019.stdout:(92/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 13 MB/s | 5.1 MB 00:00 2024-08-31T12:13:48.709 INFO:teuthology.orchestra.run.smithi019.stdout:(93/150): python3-toml-0.10.2-6.el9.noarch.rpm 415 kB/s | 42 kB 00:00 2024-08-31T12:13:48.734 INFO:teuthology.orchestra.run.smithi019.stdout:(94/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 881 kB/s | 44 kB 00:00 2024-08-31T12:13:48.770 INFO:teuthology.orchestra.run.smithi019.stdout:(95/150): qatlib-24.02.0-1.el9.x86_64.rpm 3.5 MB/s | 221 kB 00:00 2024-08-31T12:13:48.796 INFO:teuthology.orchestra.run.smithi019.stdout:(96/150): qatlib-service-24.02.0-1.el9.x86_64.r 585 kB/s | 36 kB 00:00 2024-08-31T12:13:48.821 INFO:teuthology.orchestra.run.smithi019.stdout:(97/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 943 kB/s | 47 kB 00:00 2024-08-31T12:13:48.846 INFO:teuthology.orchestra.run.smithi019.stdout:(98/150): socat-1.7.4.1-6.el9.x86_64.rpm 5.9 MB/s | 304 kB 00:00 2024-08-31T12:13:48.863 INFO:teuthology.orchestra.run.smithi019.stdout:(99/150): uuid-1.6.2-55.el9.x86_64.rpm 1.4 MB/s | 58 kB 00:00 2024-08-31T12:13:48.889 INFO:teuthology.orchestra.run.smithi019.stdout:(100/150): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 2.2 MB/s | 96 kB 00:00 2024-08-31T12:13:48.905 INFO:teuthology.orchestra.run.smithi019.stdout:(101/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-08-31T12:13:48.934 INFO:teuthology.orchestra.run.smithi019.stdout:(102/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 988 kB/s | 45 kB 00:00 2024-08-31T12:13:49.057 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-08-31T12:13:49.123 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-08-31T12:13:49.150 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-08-31T12:13:49.159 INFO:teuthology.orchestra.run.smithi019.stdout:(103/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 28 MB/s | 16 MB 00:00 2024-08-31T12:13:49.184 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 95/152 2024-08-31T12:13:49.184 INFO:teuthology.orchestra.run.smithi019.stdout:(104/150): lua-devel-5.4.4-4.el9.x86_64.rpm 80 kB/s | 22 kB 00:00 2024-08-31T12:13:49.243 INFO:teuthology.orchestra.run.smithi019.stdout:(105/150): abseil-cpp-20211102.0-3.el9.x86_64.r 6.4 MB/s | 548 kB 00:00 2024-08-31T12:13:49.269 INFO:teuthology.orchestra.run.smithi019.stdout:(106/150): grpc-data-1.46.7-10.el9.noarch.rpm 783 kB/s | 19 kB 00:00 2024-08-31T12:13:49.294 INFO:teuthology.orchestra.run.smithi019.stdout:(107/150): gperftools-libs-2.9.1-2.el9.x86_64.r 2.8 MB/s | 309 kB 00:00 2024-08-31T12:13:49.327 INFO:teuthology.orchestra.run.smithi019.stdout:(108/150): libarrow-doc-9.0.0-11.el9.noarch.rpm 785 kB/s | 26 kB 00:00 2024-08-31T12:13:49.361 INFO:teuthology.orchestra.run.smithi019.stdout:(109/150): protobuf-compiler-3.14.0-13.el9.x86_ 2.0 MB/s | 863 kB 00:00 2024-08-31T12:13:49.386 INFO:teuthology.orchestra.run.smithi019.stdout:(110/150): liboath-2.6.7-2.el9.x86_64.rpm 829 kB/s | 49 kB 00:00 2024-08-31T12:13:49.428 INFO:teuthology.orchestra.run.smithi019.stdout:(111/150): luarocks-3.9.2-5.el9.noarch.rpm 4.4 MB/s | 151 kB 00:00 2024-08-31T12:13:49.462 INFO:teuthology.orchestra.run.smithi019.stdout:(112/150): libunwind-1.6.2-1.el9.x86_64.rpm 672 kB/s | 67 kB 00:00 2024-08-31T12:13:49.520 INFO:teuthology.orchestra.run.smithi019.stdout:(113/150): parquet-libs-9.0.0-11.el9.x86_64.rpm 9.1 MB/s | 839 kB 00:00 2024-08-31T12:13:49.527 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-08-31T12:13:49.545 INFO:teuthology.orchestra.run.smithi019.stdout:(114/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-08-31T12:13:49.571 INFO:teuthology.orchestra.run.smithi019.stdout:(115/150): python3-asyncssh-2.13.2-5.el9.noarch 4.9 MB/s | 548 kB 00:00 2024-08-31T12:13:49.571 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : librbd1-2:19.3.0-4560.gd19e6118.el9.x86_64 97/152 2024-08-31T12:13:49.596 INFO:teuthology.orchestra.run.smithi019.stdout:(116/150): python3-cachetools-4.2.4-1.el9.noarc 640 kB/s | 32 kB 00:00 2024-08-31T12:13:49.621 INFO:teuthology.orchestra.run.smithi019.stdout:(117/150): python3-certifi-2023.05.07-4.el9.noa 281 kB/s | 14 kB 00:00 2024-08-31T12:13:49.646 INFO:teuthology.orchestra.run.smithi019.stdout:(118/150): python3-cheroot-8.6.0-4.el9.noarch.r 3.4 MB/s | 172 kB 00:00 2024-08-31T12:13:49.663 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librbd1-2:19.3.0-4560.gd19e6118.el9.x86_64 97/152 2024-08-31T12:13:49.680 INFO:teuthology.orchestra.run.smithi019.stdout:(119/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 8.9 MB/s | 303 kB 00:00 2024-08-31T12:13:49.693 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-rbd-2:19.3.0-4560.gd19e6118.el9.x86_64 98/152 2024-08-31T12:13:49.705 INFO:teuthology.orchestra.run.smithi019.stdout:(120/150): python3-cherrypy-18.6.1-2.el9.noarch 4.2 MB/s | 358 kB 00:00 2024-08-31T12:13:49.729 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-08-31T12:13:49.731 INFO:teuthology.orchestra.run.smithi019.stdout:(121/150): python3-google-auth-2.34.0-1.el9.noa 4.3 MB/s | 220 kB 00:00 2024-08-31T12:13:49.756 INFO:teuthology.orchestra.run.smithi019.stdout:(122/150): python3-grpcio-tools-1.46.7-10.el9.x 5.6 MB/s | 144 kB 00:00 2024-08-31T12:13:49.773 INFO:teuthology.orchestra.run.smithi019.stdout:(123/150): python3-jaraco-8.2.1-3.el9.noarch.rp 636 kB/s | 11 kB 00:00 2024-08-31T12:13:49.840 INFO:teuthology.orchestra.run.smithi019.stdout:(124/150): libarrow-9.0.0-11.el9.x86_64.rpm 7.7 MB/s | 4.4 MB 00:00 2024-08-31T12:13:49.857 INFO:teuthology.orchestra.run.smithi019.stdout:(125/150): python3-jaraco-classes-3.2.1-5.el9.n 211 kB/s | 18 kB 00:00 2024-08-31T12:13:49.874 INFO:teuthology.orchestra.run.smithi019.stdout:(126/150): python3-jaraco-collections-3.0.0-8.e 691 kB/s | 23 kB 00:00 2024-08-31T12:13:49.891 INFO:teuthology.orchestra.run.smithi019.stdout:(127/150): python3-jaraco-functools-3.5.0-2.el9 581 kB/s | 19 kB 00:00 2024-08-31T12:13:49.916 INFO:teuthology.orchestra.run.smithi019.stdout:(128/150): python3-jaraco-text-3.2.0-6.el9.noar 473 kB/s | 20 kB 00:00 2024-08-31T12:13:49.942 INFO:teuthology.orchestra.run.smithi019.stdout:(129/150): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-08-31T12:13:49.984 INFO:teuthology.orchestra.run.smithi019.stdout:(130/150): python3-more-itertools-8.12.0-2.el9. 1.9 MB/s | 79 kB 00:00 2024-08-31T12:13:50.009 INFO:teuthology.orchestra.run.smithi019.stdout:(131/150): python3-natsort-7.1.1-5.el9.noarch.r 2.3 MB/s | 58 kB 00:00 2024-08-31T12:13:50.051 INFO:teuthology.orchestra.run.smithi019.stdout:(132/150): python3-kubernetes-26.1.0-2.el9.noar 6.4 MB/s | 1.0 MB 00:00 2024-08-31T12:13:50.076 INFO:teuthology.orchestra.run.smithi019.stdout:(133/150): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-08-31T12:13:50.101 INFO:teuthology.orchestra.run.smithi019.stdout:(134/150): python3-portend-3.1.0-2.el9.noarch.r 663 kB/s | 16 kB 00:00 2024-08-31T12:13:50.127 INFO:teuthology.orchestra.run.smithi019.stdout:(135/150): python3-pyperclip-1.8.2-2.el9.noarch 1.1 MB/s | 28 kB 00:00 2024-08-31T12:13:50.152 INFO:teuthology.orchestra.run.smithi019.stdout:(136/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-08-31T12:13:50.177 INFO:teuthology.orchestra.run.smithi019.stdout:(137/150): python3-repoze-lru-0.7-16.el9.noarch 611 kB/s | 31 kB 00:00 2024-08-31T12:13:50.194 INFO:teuthology.orchestra.run.smithi019.stdout:(138/150): python3-rsa-4.9-2.el9.noarch.rpm 3.5 MB/s | 59 kB 00:00 2024-08-31T12:13:50.244 INFO:teuthology.orchestra.run.smithi019.stdout:(139/150): python3-grpcio-1.46.7-10.el9.x86_64. 3.8 MB/s | 2.0 MB 00:00 2024-08-31T12:13:50.255 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 100/152 2024-08-31T12:13:50.261 INFO:teuthology.orchestra.run.smithi019.stdout:(140/150): python3-tempora-5.0.0-2.el9.noarch.r 534 kB/s | 36 kB 00:00 2024-08-31T12:13:50.287 INFO:teuthology.orchestra.run.smithi019.stdout:(141/150): python3-routes-2.5.1-5.el9.noarch.rp 1.4 MB/s | 188 kB 00:00 2024-08-31T12:13:50.312 INFO:teuthology.orchestra.run.smithi019.stdout:(142/150): python3-typing-extensions-4.12.2-2.e 1.1 MB/s | 74 kB 00:00 2024-08-31T12:13:50.315 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : librgw2-2:19.3.0-4560.gd19e6118.el9.x86_64 101/152 2024-08-31T12:13:50.338 INFO:teuthology.orchestra.run.smithi019.stdout:(143/150): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-08-31T12:13:50.363 INFO:teuthology.orchestra.run.smithi019.stdout:(144/150): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-08-31T12:13:50.387 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librgw2-2:19.3.0-4560.gd19e6118.el9.x86_64 101/152 2024-08-31T12:13:50.388 INFO:teuthology.orchestra.run.smithi019.stdout:(145/150): python3-zc-lockfile-2.0-10.el9.noarc 399 kB/s | 20 kB 00:00 2024-08-31T12:13:50.413 INFO:teuthology.orchestra.run.smithi019.stdout:(146/150): re2-20211101-20.el9.x86_64.rpm 3.7 MB/s | 191 kB 00:00 2024-08-31T12:13:50.417 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-rgw-2:19.3.0-4560.gd19e6118.el9.x86_64 102/152 2024-08-31T12:13:50.438 INFO:teuthology.orchestra.run.smithi019.stdout:(147/150): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-08-31T12:13:50.507 INFO:teuthology.orchestra.run.smithi019.stdout:(148/150): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-08-31T12:13:50.756 INFO:teuthology.orchestra.run.smithi019.stdout:(149/150): librbd1-19.3.0-4560.gd19e6118.el9.x8 9.9 MB/s | 3.1 MB 00:00 2024-08-31T12:13:50.831 INFO:teuthology.orchestra.run.smithi019.stdout:(150/150): librados2-19.3.0-4560.gd19e6118.el9. 8.1 MB/s | 3.4 MB 00:00 2024-08-31T12:13:50.837 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-08-31T12:13:50.837 INFO:teuthology.orchestra.run.smithi019.stdout:Total 17 MB/s | 224 MB 00:13 2024-08-31T12:13:51.943 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-08-31T12:13:52.082 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-08-31T12:13:52.082 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-08-31T12:13:52.983 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-common-2:19.3.0-4560.gd19e6118.el9.x86_64 103/152 2024-08-31T12:13:53.013 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-common-2:19.3.0-4560.gd19e6118.el9.x86_64 103/152 2024-08-31T12:13:53.615 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-common-2:19.3.0-4560.gd19e6118.el9.x86_64 103/152 2024-08-31T12:13:53.658 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-base-2:19.3.0-4560.gd19e6118.el9.x86_64 104/152 2024-08-31T12:13:53.741 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-base-2:19.3.0-4560.gd19e6118.el9.x86_64 104/152 2024-08-31T12:13:53.741 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-31T12:13:53.741 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-31T12:13:53.741 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:13:53.776 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-selinux-2:19.3.0-4560.gd19e6118.el9.x86_64 105/152 2024-08-31T12:13:53.933 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-08-31T12:13:53.934 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-08-31T12:14:00.082 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-08-31T12:14:00.150 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-08-31T12:14:00.176 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-08-31T12:14:00.219 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-08-31T12:14:00.668 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-08-31T12:14:00.709 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : librados2-2:19.3.0-4560.gd19e6118.el9.x86_64 5/152 2024-08-31T12:14:00.831 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librados2-2:19.3.0-4560.gd19e6118.el9.x86_64 5/152 2024-08-31T12:14:00.860 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libcephfs2-2:19.3.0-4560.gd19e6118.el9.x86_64 6/152 2024-08-31T12:14:00.954 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libcephfs2-2:19.3.0-4560.gd19e6118.el9.x86_64 6/152 2024-08-31T12:14:00.996 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rados-2:19.3.0-4560.gd19e6118.el9.x86_64 7/152 2024-08-31T12:14:01.032 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-08-31T12:14:01.057 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-08-31T12:14:01.220 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 10/152 2024-08-31T12:14:01.273 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : postgresql-13.16-1.el9.x86_64 11/152 2024-08-31T12:14:01.299 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libcephsqlite-2:19.3.0-4560.gd19e6118.el9.x86_64 12/152 2024-08-31T12:14:01.408 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4560.gd19e6118.el9.x86_64 12/152 2024-08-31T12:14:01.434 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libradosstriper1-2:19.3.0-4560.gd19e6118.el9.x86 13/152 2024-08-31T12:14:01.496 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4560.gd19e6118.el9.x86 13/152 2024-08-31T12:14:01.518 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-08-31T12:14:01.544 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-08-31T12:14:01.848 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-08-31T12:14:01.884 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-08-31T12:14:01.996 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : re2-1:20211101-20.el9.x86_64 16/152 2024-08-31T12:14:02.024 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 17/152 2024-08-31T12:14:02.059 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : liboath-2.6.7-2.el9.x86_64 18/152 2024-08-31T12:14:02.135 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 19/152 2024-08-31T12:14:02.195 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 20/152 2024-08-31T12:14:02.234 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-packaging-20.9-5.el9.noarch 21/152 2024-08-31T12:14:02.278 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 22/152 2024-08-31T12:14:02.310 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 23/152 2024-08-31T12:14:02.338 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : lua-5.4.4-4.el9.x86_64 24/152 2024-08-31T12:14:02.410 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 25/152 2024-08-31T12:14:02.462 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 26/152 2024-08-31T12:14:02.499 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 27/152 2024-08-31T12:14:02.535 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 28/152 2024-08-31T12:14:02.614 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 29/152 2024-08-31T12:14:02.653 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-ceph-common-2:19.3.0-4560.gd19e6118.el9. 30/152 2024-08-31T12:14:02.697 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-ceph-argparse-2:19.3.0-4560.gd19e6118.el 31/152 2024-08-31T12:14:02.725 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cephfs-2:19.3.0-4560.gd19e6118.el9.x86_6 32/152 2024-08-31T12:14:02.799 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/152 2024-08-31T12:14:02.849 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : luarocks-3.9.2-5.el9.noarch 34/152 2024-08-31T12:14:02.935 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 35/152 2024-08-31T12:14:03.089 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/152 2024-08-31T12:14:03.152 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/152 2024-08-31T12:14:03.718 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/152 2024-08-31T12:14:03.808 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/152 2024-08-31T12:14:03.842 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 40/152 2024-08-31T12:14:03.885 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 41/152 2024-08-31T12:14:03.924 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librados-devel-2:19.3.0-4560.gd19e6118.el9.x86_6 42/152 2024-08-31T12:14:03.956 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/152 2024-08-31T12:14:04.052 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/152 2024-08-31T12:14:04.114 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 45/152 2024-08-31T12:14:04.197 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 46/152 2024-08-31T12:14:04.233 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 47/152 2024-08-31T12:14:04.269 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 48/152 2024-08-31T12:14:04.317 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 49/152 2024-08-31T12:14:04.352 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 50/152 2024-08-31T12:14:04.395 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 51/152 2024-08-31T12:14:04.438 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 52/152 2024-08-31T12:14:04.582 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 53/152 2024-08-31T12:14:04.622 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 54/152 2024-08-31T12:14:04.664 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 55/152 2024-08-31T12:14:04.771 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-08-31T12:14:04.805 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 57/152 2024-08-31T12:14:04.854 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-08-31T12:14:04.878 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-08-31T12:14:04.903 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 60/152 2024-08-31T12:14:04.981 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-08-31T12:14:05.119 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-08-31T12:14:05.161 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-08-31T12:14:05.189 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-08-31T12:14:05.223 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-08-31T12:14:05.268 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-08-31T12:14:05.305 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 67/152 2024-08-31T12:14:05.403 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-08-31T12:14:05.676 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-08-31T12:14:05.744 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-08-31T12:14:05.777 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-08-31T12:14:05.829 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-08-31T12:14:05.886 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-08-31T12:14:05.985 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-08-31T12:14:06.035 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-08-31T12:14:06.072 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-08-31T12:14:06.082 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4560.gd19e6118.el9.x86_64 105/152 2024-08-31T12:14:06.082 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /sys 2024-08-31T12:14:06.082 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /proc 2024-08-31T12:14:06.082 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /mnt 2024-08-31T12:14:06.082 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /var/tmp 2024-08-31T12:14:06.082 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /home 2024-08-31T12:14:06.082 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /root 2024-08-31T12:14:06.082 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /tmp 2024-08-31T12:14:06.082 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:14:06.192 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-08-31T12:14:06.249 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 78/152 2024-08-31T12:14:06.273 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-volume-2:19.3.0-4560.gd19e6118.el9.noarch 106/152 2024-08-31T12:14:06.290 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-volume-2:19.3.0-4560.gd19e6118.el9.noarch 106/152 2024-08-31T12:14:06.290 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-31T12:14:06.290 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-08-31T12:14:06.290 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:14:07.023 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 79/152 2024-08-31T12:14:07.070 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 80/152 2024-08-31T12:14:07.104 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 81/152 2024-08-31T12:14:07.208 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 82/152 2024-08-31T12:14:07.260 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 83/152 2024-08-31T12:14:07.305 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 84/152 2024-08-31T12:14:07.405 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-08-31T12:14:07.449 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-08-31T12:14:07.641 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : openblas-0.3.26-2.el9.x86_64 87/152 2024-08-31T12:14:07.671 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 88/152 2024-08-31T12:14:07.751 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-08-31T12:14:07.943 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-osd-2:19.3.0-4560.gd19e6118.el9.x86_64 107/152 2024-08-31T12:14:07.970 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-osd-2:19.3.0-4560.gd19e6118.el9.x86_64 107/152 2024-08-31T12:14:07.970 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-31T12:14:07.970 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-31T12:14:07.971 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-31T12:14:07.971 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-31T12:14:07.971 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:14:08.210 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mds-2:19.3.0-4560.gd19e6118.el9.x86_64 108/152 2024-08-31T12:14:08.234 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mds-2:19.3.0-4560.gd19e6118.el9.x86_64 108/152 2024-08-31T12:14:08.234 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-31T12:14:08.234 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-31T12:14:08.234 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-31T12:14:08.235 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-31T12:14:08.235 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:14:08.442 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-08-31T12:14:08.558 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-08-31T12:14:08.719 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mon-2:19.3.0-4560.gd19e6118.el9.x86_64 109/152 2024-08-31T12:14:08.743 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mon-2:19.3.0-4560.gd19e6118.el9.x86_64 109/152 2024-08-31T12:14:08.743 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-31T12:14:08.743 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-31T12:14:08.743 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-31T12:14:08.743 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-31T12:14:08.743 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:14:08.850 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-08-31T12:14:08.908 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-08-31T12:14:08.971 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-08-31T12:14:09.165 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-08-31T12:14:09.212 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-08-31T12:14:09.292 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-08-31T12:14:09.390 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-08-31T12:14:09.540 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-08-31T12:14:09.579 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-08-31T12:14:09.611 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-08-31T12:14:09.638 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : fuse-2.9.9-16.el9.x86_64 120/152 2024-08-31T12:14:09.877 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-08-31T12:14:09.958 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-08-31T12:14:09.987 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-08-31T12:14:10.020 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 95/152 2024-08-31T12:14:10.085 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: cephadm-2:19.3.0-4560.gd19e6118.el9.noarch 121/152 2024-08-31T12:14:10.112 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : cephadm-2:19.3.0-4560.gd19e6118.el9.noarch 121/152 2024-08-31T12:14:10.144 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4560.gd19e6118.e 122/152 2024-08-31T12:14:10.213 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4560.gd19e6118. 123/152 2024-08-31T12:14:10.243 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4560.gd19e6118.el9.noa 124/152 2024-08-31T12:14:10.366 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-08-31T12:14:10.407 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : librbd1-2:19.3.0-4560.gd19e6118.el9.x86_64 97/152 2024-08-31T12:14:10.499 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librbd1-2:19.3.0-4560.gd19e6118.el9.x86_64 97/152 2024-08-31T12:14:10.537 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rbd-2:19.3.0-4560.gd19e6118.el9.x86_64 98/152 2024-08-31T12:14:10.574 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-08-31T12:14:11.102 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 100/152 2024-08-31T12:14:11.135 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librgw2-2:19.3.0-4560.gd19e6118.el9.x86_64 101/152 2024-08-31T12:14:11.206 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librgw2-2:19.3.0-4560.gd19e6118.el9.x86_64 101/152 2024-08-31T12:14:11.237 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rgw-2:19.3.0-4560.gd19e6118.el9.x86_64 102/152 2024-08-31T12:14:13.590 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4560.gd19e6118.el9.noa 124/152 2024-08-31T12:14:13.649 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4560.gd19e6118.el9.n 125/152 2024-08-31T12:14:13.660 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-common-2:19.3.0-4560.gd19e6118.el9.x86_64 103/152 2024-08-31T12:14:13.699 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-common-2:19.3.0-4560.gd19e6118.el9.x86_64 103/152 2024-08-31T12:14:14.303 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-common-2:19.3.0-4560.gd19e6118.el9.x86_64 103/152 2024-08-31T12:14:14.344 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-base-2:19.3.0-4560.gd19e6118.el9.x86_64 104/152 2024-08-31T12:14:14.427 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-base-2:19.3.0-4560.gd19e6118.el9.x86_64 104/152 2024-08-31T12:14:14.427 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-31T12:14:14.427 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-31T12:14:14.427 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:14:14.454 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-selinux-2:19.3.0-4560.gd19e6118.el9.x86_64 105/152 2024-08-31T12:14:14.661 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4560.gd19e6118.el9.n 125/152 2024-08-31T12:14:14.690 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4560.gd19 126/152 2024-08-31T12:14:14.809 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4560.gd19 126/152 2024-08-31T12:14:14.982 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4560.gd19e6118.el 127/152 2024-08-31T12:14:15.009 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-2:19.3.0-4560.gd19e6118.el9.x86_64 128/152 2024-08-31T12:14:15.033 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4560.gd19e6118.el9.x86_64 128/152 2024-08-31T12:14:15.033 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-31T12:14:15.033 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-31T12:14:15.033 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-31T12:14:15.033 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-31T12:14:15.033 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:14:15.085 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-rook-2:19.3.0-4560.gd19e6118.el9.noarch 129/152 2024-08-31T12:14:15.096 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4560.gd19e6118.el9.noarch 129/152 2024-08-31T12:14:15.199 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-2:19.3.0-4560.gd19e6118.el9.x86_64 130/152 2024-08-31T12:14:15.259 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-fuse-2:19.3.0-4560.gd19e6118.el9.x86_64 131/152 2024-08-31T12:14:16.270 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libtool-2.4.6-46.el9.x86_64 132/152 2024-08-31T12:14:16.296 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-radosgw-2:19.3.0-4560.gd19e6118.el9.x86_64 133/152 2024-08-31T12:14:16.319 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4560.gd19e6118.el9.x86_64 133/152 2024-08-31T12:14:16.319 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-31T12:14:16.320 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-31T12:14:16.320 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-31T12:14:16.320 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-31T12:14:16.320 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:14:16.363 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4560.gd19e6 134/152 2024-08-31T12:14:16.385 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4560.gd19e6 134/152 2024-08-31T12:14:16.386 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-31T12:14:16.386 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-31T12:14:16.386 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:14:16.438 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : cephfs-mirror-2:19.3.0-4560.gd19e6118.el9.x86_64 135/152 2024-08-31T12:14:16.461 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: cephfs-mirror-2:19.3.0-4560.gd19e6118.el9.x86_64 135/152 2024-08-31T12:14:16.462 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-31T12:14:16.462 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-08-31T12:14:16.462 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-08-31T12:14:16.462 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-08-31T12:14:16.462 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:14:16.797 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : rbd-mirror-2:19.3.0-4560.gd19e6118.el9.x86_64 136/152 2024-08-31T12:14:16.821 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4560.gd19e6118.el9.x86_64 136/152 2024-08-31T12:14:16.821 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-31T12:14:16.821 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-31T12:14:16.821 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-31T12:14:16.822 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-31T12:14:16.822 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:14:21.410 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-test-2:19.3.0-4560.gd19e6118.el9.x86_64 137/152 2024-08-31T12:14:21.445 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : rbd-fuse-2:19.3.0-4560.gd19e6118.el9.x86_64 138/152 2024-08-31T12:14:21.485 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : rbd-nbd-2:19.3.0-4560.gd19e6118.el9.x86_64 139/152 2024-08-31T12:14:21.517 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 140/152 2024-08-31T12:14:21.544 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 141/152 2024-08-31T12:14:21.735 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : cephfs-shell-2:19.3.0-4560.gd19e6118.el9.x86_64 142/152 2024-08-31T12:14:21.782 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 143/152 2024-08-31T12:14:21.817 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libcephfs-devel-2:19.3.0-4560.gd19e6118.el9.x86_ 144/152 2024-08-31T12:14:22.416 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 145/152 2024-08-31T12:14:22.459 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 145/152 2024-08-31T12:14:22.479 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 145/152 2024-08-31T12:14:23.361 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : cephfs-top-2:19.3.0-4560.gd19e6118.el9.noarch 146/152 2024-08-31T12:14:23.439 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : openssl-devel-1:3.2.2-4.el9.x86_64 147/152 2024-08-31T12:14:23.480 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-08-31T12:14:23.573 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-08-31T12:14:23.601 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-08-31T12:14:23.601 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-08-31T12:14:23.693 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-08-31T12:14:23.693 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-08-31T12:14:26.556 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4560.gd19e6118.el9.x86_64 105/152 2024-08-31T12:14:26.556 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /sys 2024-08-31T12:14:26.556 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /proc 2024-08-31T12:14:26.556 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /mnt 2024-08-31T12:14:26.556 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /var/tmp 2024-08-31T12:14:26.556 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /home 2024-08-31T12:14:26.556 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /root 2024-08-31T12:14:26.556 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /tmp 2024-08-31T12:14:26.556 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:14:26.742 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-volume-2:19.3.0-4560.gd19e6118.el9.noarch 106/152 2024-08-31T12:14:26.758 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-volume-2:19.3.0-4560.gd19e6118.el9.noarch 106/152 2024-08-31T12:14:26.758 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-31T12:14:26.758 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-08-31T12:14:26.758 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:14:28.403 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-osd-2:19.3.0-4560.gd19e6118.el9.x86_64 107/152 2024-08-31T12:14:28.430 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-osd-2:19.3.0-4560.gd19e6118.el9.x86_64 107/152 2024-08-31T12:14:28.430 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-31T12:14:28.430 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-31T12:14:28.430 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-31T12:14:28.431 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-31T12:14:28.431 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:14:28.672 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mds-2:19.3.0-4560.gd19e6118.el9.x86_64 108/152 2024-08-31T12:14:28.696 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mds-2:19.3.0-4560.gd19e6118.el9.x86_64 108/152 2024-08-31T12:14:28.696 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-31T12:14:28.696 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-31T12:14:28.697 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-31T12:14:28.697 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-31T12:14:28.697 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:14:29.181 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mon-2:19.3.0-4560.gd19e6118.el9.x86_64 109/152 2024-08-31T12:14:29.205 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mon-2:19.3.0-4560.gd19e6118.el9.x86_64 109/152 2024-08-31T12:14:29.205 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-31T12:14:29.205 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-31T12:14:29.205 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-31T12:14:29.205 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-31T12:14:29.205 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:14:29.314 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-08-31T12:14:29.372 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-08-31T12:14:29.436 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-08-31T12:14:29.631 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-08-31T12:14:29.676 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-08-31T12:14:29.757 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-08-31T12:14:29.847 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-08-31T12:14:29.997 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-08-31T12:14:30.035 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-08-31T12:14:30.066 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-08-31T12:14:30.094 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : fuse-2.9.9-16.el9.x86_64 120/152 2024-08-31T12:14:30.442 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-08-31T12:14:30.443 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-2:19.3.0-4560.gd19e6118.el9.x86_64 1/152 2024-08-31T12:14:30.443 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-base-2:19.3.0-4560.gd19e6118.el9.x86_64 2/152 2024-08-31T12:14:30.443 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-common-2:19.3.0-4560.gd19e6118.el9.x86_64 3/152 2024-08-31T12:14:30.443 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-fuse-2:19.3.0-4560.gd19e6118.el9.x86_64 4/152 2024-08-31T12:14:30.443 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4560.gd19e6 5/152 2024-08-31T12:14:30.443 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mds-2:19.3.0-4560.gd19e6118.el9.x86_64 6/152 2024-08-31T12:14:30.443 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-2:19.3.0-4560.gd19e6118.el9.x86_64 7/152 2024-08-31T12:14:30.443 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mon-2:19.3.0-4560.gd19e6118.el9.x86_64 8/152 2024-08-31T12:14:30.443 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-osd-2:19.3.0-4560.gd19e6118.el9.x86_64 9/152 2024-08-31T12:14:30.444 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-radosgw-2:19.3.0-4560.gd19e6118.el9.x86_64 10/152 2024-08-31T12:14:30.444 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-selinux-2:19.3.0-4560.gd19e6118.el9.x86_64 11/152 2024-08-31T12:14:30.444 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-test-2:19.3.0-4560.gd19e6118.el9.x86_64 12/152 2024-08-31T12:14:30.444 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : cephfs-mirror-2:19.3.0-4560.gd19e6118.el9.x86_64 13/152 2024-08-31T12:14:30.444 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : cephfs-shell-2:19.3.0-4560.gd19e6118.el9.x86_64 14/152 2024-08-31T12:14:30.444 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libcephfs-devel-2:19.3.0-4560.gd19e6118.el9.x86_ 15/152 2024-08-31T12:14:30.444 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libcephfs2-2:19.3.0-4560.gd19e6118.el9.x86_64 16/152 2024-08-31T12:14:30.444 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libcephsqlite-2:19.3.0-4560.gd19e6118.el9.x86_64 17/152 2024-08-31T12:14:30.444 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librados-devel-2:19.3.0-4560.gd19e6118.el9.x86_6 18/152 2024-08-31T12:14:30.444 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libradosstriper1-2:19.3.0-4560.gd19e6118.el9.x86 19/152 2024-08-31T12:14:30.444 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librgw2-2:19.3.0-4560.gd19e6118.el9.x86_64 20/152 2024-08-31T12:14:30.444 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4560.gd19e6118.el 21/152 2024-08-31T12:14:30.445 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-ceph-common-2:19.3.0-4560.gd19e6118.el9. 22/152 2024-08-31T12:14:30.445 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cephfs-2:19.3.0-4560.gd19e6118.el9.x86_6 23/152 2024-08-31T12:14:30.445 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-rados-2:19.3.0-4560.gd19e6118.el9.x86_64 24/152 2024-08-31T12:14:30.445 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-rbd-2:19.3.0-4560.gd19e6118.el9.x86_64 25/152 2024-08-31T12:14:30.445 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-rgw-2:19.3.0-4560.gd19e6118.el9.x86_64 26/152 2024-08-31T12:14:30.445 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : rbd-fuse-2:19.3.0-4560.gd19e6118.el9.x86_64 27/152 2024-08-31T12:14:30.445 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : rbd-mirror-2:19.3.0-4560.gd19e6118.el9.x86_64 28/152 2024-08-31T12:14:30.445 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : rbd-nbd-2:19.3.0-4560.gd19e6118.el9.x86_64 29/152 2024-08-31T12:14:30.445 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4560.gd19e6118. 30/152 2024-08-31T12:14:30.445 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4560.gd19e6118.el9.noa 31/152 2024-08-31T12:14:30.445 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4560.gd19e6118.el9.n 32/152 2024-08-31T12:14:30.445 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4560.gd19 33/152 2024-08-31T12:14:30.445 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4560.gd19e6118.el 34/152 2024-08-31T12:14:30.446 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4560.gd19e6118.el9.noarch 35/152 2024-08-31T12:14:30.446 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4560.gd19e6118.e 36/152 2024-08-31T12:14:30.446 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-volume-2:19.3.0-4560.gd19e6118.el9.noarch 37/152 2024-08-31T12:14:30.446 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : cephadm-2:19.3.0-4560.gd19e6118.el9.noarch 38/152 2024-08-31T12:14:30.446 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : cephfs-top-2:19.3.0-4560.gd19e6118.el9.noarch 39/152 2024-08-31T12:14:30.446 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/152 2024-08-31T12:14:30.446 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/152 2024-08-31T12:14:30.446 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/152 2024-08-31T12:14:30.446 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-08-31T12:14:30.446 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-08-31T12:14:30.446 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-08-31T12:14:30.446 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-08-31T12:14:30.446 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-08-31T12:14:30.447 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/152 2024-08-31T12:14:30.447 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-08-31T12:14:30.447 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-08-31T12:14:30.447 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-08-31T12:14:30.447 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/152 2024-08-31T12:14:30.447 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-08-31T12:14:30.447 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-08-31T12:14:30.447 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-08-31T12:14:30.447 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-08-31T12:14:30.448 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-08-31T12:14:30.448 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-08-31T12:14:30.448 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-08-31T12:14:30.448 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-08-31T12:14:30.448 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/152 2024-08-31T12:14:30.448 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 62/152 2024-08-31T12:14:30.448 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-08-31T12:14:30.448 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-08-31T12:14:30.448 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-08-31T12:14:30.448 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 66/152 2024-08-31T12:14:30.449 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 67/152 2024-08-31T12:14:30.449 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : openssl-devel-1:3.2.2-4.el9.x86_64 68/152 2024-08-31T12:14:30.449 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : postgresql-13.16-1.el9.x86_64 69/152 2024-08-31T12:14:30.449 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 70/152 2024-08-31T12:14:30.449 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 71/152 2024-08-31T12:14:30.449 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 72/152 2024-08-31T12:14:30.449 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-08-31T12:14:30.449 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-08-31T12:14:30.449 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-08-31T12:14:30.450 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/152 2024-08-31T12:14:30.450 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 77/152 2024-08-31T12:14:30.450 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/152 2024-08-31T12:14:30.450 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/152 2024-08-31T12:14:30.450 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/152 2024-08-31T12:14:30.450 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/152 2024-08-31T12:14:30.450 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/152 2024-08-31T12:14:30.450 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/152 2024-08-31T12:14:30.450 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/152 2024-08-31T12:14:30.450 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/152 2024-08-31T12:14:30.450 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/152 2024-08-31T12:14:30.450 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/152 2024-08-31T12:14:30.450 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/152 2024-08-31T12:14:30.450 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/152 2024-08-31T12:14:30.451 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/152 2024-08-31T12:14:30.451 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/152 2024-08-31T12:14:30.451 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/152 2024-08-31T12:14:30.451 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-08-31T12:14:30.451 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/152 2024-08-31T12:14:30.451 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/152 2024-08-31T12:14:30.452 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/152 2024-08-31T12:14:30.452 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/152 2024-08-31T12:14:30.452 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/152 2024-08-31T12:14:30.452 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/152 2024-08-31T12:14:30.452 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/152 2024-08-31T12:14:30.452 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 101/152 2024-08-31T12:14:30.452 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/152 2024-08-31T12:14:30.452 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/152 2024-08-31T12:14:30.452 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/152 2024-08-31T12:14:30.453 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/152 2024-08-31T12:14:30.453 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/152 2024-08-31T12:14:30.453 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/152 2024-08-31T12:14:30.453 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/152 2024-08-31T12:14:30.453 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/152 2024-08-31T12:14:30.453 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/152 2024-08-31T12:14:30.453 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/152 2024-08-31T12:14:30.453 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/152 2024-08-31T12:14:30.453 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/152 2024-08-31T12:14:30.453 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/152 2024-08-31T12:14:30.453 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/152 2024-08-31T12:14:30.453 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/152 2024-08-31T12:14:30.454 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/152 2024-08-31T12:14:30.454 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/152 2024-08-31T12:14:30.454 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/152 2024-08-31T12:14:30.454 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/152 2024-08-31T12:14:30.454 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/152 2024-08-31T12:14:30.454 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 122/152 2024-08-31T12:14:30.454 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-08-31T12:14:30.455 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-08-31T12:14:30.455 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-08-31T12:14:30.455 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-08-31T12:14:30.455 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-08-31T12:14:30.455 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-08-31T12:14:30.455 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-08-31T12:14:30.455 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-08-31T12:14:30.455 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-08-31T12:14:30.455 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-08-31T12:14:30.455 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-08-31T12:14:30.455 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-08-31T12:14:30.455 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-08-31T12:14:30.456 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-08-31T12:14:30.456 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-08-31T12:14:30.456 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-08-31T12:14:30.456 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-08-31T12:14:30.456 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-08-31T12:14:30.456 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-08-31T12:14:30.456 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 142/152 2024-08-31T12:14:30.456 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 143/152 2024-08-31T12:14:30.456 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-08-31T12:14:30.456 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-08-31T12:14:30.456 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-08-31T12:14:30.456 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : re2-1:20211101-20.el9.x86_64 147/152 2024-08-31T12:14:30.456 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-08-31T12:14:30.456 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librados2-2:19.3.0-4560.gd19e6118.el9.x86_64 149/152 2024-08-31T12:14:30.457 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-08-31T12:14:30.457 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librbd1-2:19.3.0-4560.gd19e6118.el9.x86_64 151/152 2024-08-31T12:14:30.583 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: cephadm-2:19.3.0-4560.gd19e6118.el9.noarch 121/152 2024-08-31T12:14:30.609 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : cephadm-2:19.3.0-4560.gd19e6118.el9.noarch 121/152 2024-08-31T12:14:30.642 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4560.gd19e6118.e 122/152 2024-08-31T12:14:30.711 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4560.gd19e6118. 123/152 2024-08-31T12:14:30.740 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4560.gd19e6118.el9.noa 124/152 2024-08-31T12:14:32.726 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-08-31T12:14:32.726 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:14:32.726 INFO:teuthology.orchestra.run.smithi043.stdout:Upgraded: 2024-08-31T12:14:32.726 INFO:teuthology.orchestra.run.smithi043.stdout: librados2-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:32.726 INFO:teuthology.orchestra.run.smithi043.stdout: librbd1-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:32.726 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-08-31T12:14:32.726 INFO:teuthology.orchestra.run.smithi043.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-08-31T12:14:32.726 INFO:teuthology.orchestra.run.smithi043.stdout: automake-1.16.2-8.el9.noarch 2024-08-31T12:14:32.726 INFO:teuthology.orchestra.run.smithi043.stdout: bison-3.7.4-5.el9.x86_64 2024-08-31T12:14:32.726 INFO:teuthology.orchestra.run.smithi043.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-08-31T12:14:32.727 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:32.727 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-base-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:32.727 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-common-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:32.727 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-fuse-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:32.727 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-grafana-dashboards-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-31T12:14:32.727 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-immutable-object-cache-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:32.727 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mds-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:32.727 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:32.727 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-cephadm-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-31T12:14:32.727 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-dashboard-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-31T12:14:32.728 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-31T12:14:32.728 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-modules-core-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-31T12:14:32.728 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-rook-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-31T12:14:32.728 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mon-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:32.728 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-osd-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:32.728 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-prometheus-alerts-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-31T12:14:32.728 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-radosgw-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:32.728 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-selinux-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:32.728 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-test-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:32.729 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-volume-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-31T12:14:32.729 INFO:teuthology.orchestra.run.smithi043.stdout: cephadm-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-31T12:14:32.729 INFO:teuthology.orchestra.run.smithi043.stdout: cephfs-mirror-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:32.729 INFO:teuthology.orchestra.run.smithi043.stdout: cephfs-shell-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:32.729 INFO:teuthology.orchestra.run.smithi043.stdout: cephfs-top-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-31T12:14:32.729 INFO:teuthology.orchestra.run.smithi043.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-08-31T12:14:32.729 INFO:teuthology.orchestra.run.smithi043.stdout: flex-2.6.4-9.el9.x86_64 2024-08-31T12:14:32.729 INFO:teuthology.orchestra.run.smithi043.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-08-31T12:14:32.729 INFO:teuthology.orchestra.run.smithi043.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-08-31T12:14:32.729 INFO:teuthology.orchestra.run.smithi043.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-08-31T12:14:32.730 INFO:teuthology.orchestra.run.smithi043.stdout: fuse-2.9.9-16.el9.x86_64 2024-08-31T12:14:32.730 INFO:teuthology.orchestra.run.smithi043.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-31T12:14:32.730 INFO:teuthology.orchestra.run.smithi043.stdout: grpc-data-1.46.7-10.el9.noarch 2024-08-31T12:14:32.730 INFO:teuthology.orchestra.run.smithi043.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-08-31T12:14:32.730 INFO:teuthology.orchestra.run.smithi043.stdout: libarrow-9.0.0-11.el9.x86_64 2024-08-31T12:14:32.730 INFO:teuthology.orchestra.run.smithi043.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-08-31T12:14:32.730 INFO:teuthology.orchestra.run.smithi043.stdout: libcephfs-devel-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:32.730 INFO:teuthology.orchestra.run.smithi043.stdout: libcephfs2-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:32.730 INFO:teuthology.orchestra.run.smithi043.stdout: libcephsqlite-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:32.730 INFO:teuthology.orchestra.run.smithi043.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-08-31T12:14:32.730 INFO:teuthology.orchestra.run.smithi043.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-31T12:14:32.730 INFO:teuthology.orchestra.run.smithi043.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-31T12:14:32.731 INFO:teuthology.orchestra.run.smithi043.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-08-31T12:14:32.731 INFO:teuthology.orchestra.run.smithi043.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-31T12:14:32.731 INFO:teuthology.orchestra.run.smithi043.stdout: librados-devel-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:32.731 INFO:teuthology.orchestra.run.smithi043.stdout: libradosstriper1-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:32.731 INFO:teuthology.orchestra.run.smithi043.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-31T12:14:32.731 INFO:teuthology.orchestra.run.smithi043.stdout: librgw2-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:32.731 INFO:teuthology.orchestra.run.smithi043.stdout: libtool-2.4.6-46.el9.x86_64 2024-08-31T12:14:32.731 INFO:teuthology.orchestra.run.smithi043.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-31T12:14:32.731 INFO:teuthology.orchestra.run.smithi043.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-08-31T12:14:32.731 INFO:teuthology.orchestra.run.smithi043.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-08-31T12:14:32.731 INFO:teuthology.orchestra.run.smithi043.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-31T12:14:32.732 INFO:teuthology.orchestra.run.smithi043.stdout: lua-5.4.4-4.el9.x86_64 2024-08-31T12:14:32.732 INFO:teuthology.orchestra.run.smithi043.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-08-31T12:14:32.732 INFO:teuthology.orchestra.run.smithi043.stdout: luarocks-3.9.2-5.el9.noarch 2024-08-31T12:14:32.732 INFO:teuthology.orchestra.run.smithi043.stdout: openblas-0.3.26-2.el9.x86_64 2024-08-31T12:14:32.732 INFO:teuthology.orchestra.run.smithi043.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-08-31T12:14:32.732 INFO:teuthology.orchestra.run.smithi043.stdout: openssl-devel-1:3.2.2-4.el9.x86_64 2024-08-31T12:14:32.732 INFO:teuthology.orchestra.run.smithi043.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-08-31T12:14:32.732 INFO:teuthology.orchestra.run.smithi043.stdout: postgresql-13.16-1.el9.x86_64 2024-08-31T12:14:32.732 INFO:teuthology.orchestra.run.smithi043.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-08-31T12:14:32.732 INFO:teuthology.orchestra.run.smithi043.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-08-31T12:14:32.732 INFO:teuthology.orchestra.run.smithi043.stdout: postgresql-server-13.16-1.el9.x86_64 2024-08-31T12:14:32.732 INFO:teuthology.orchestra.run.smithi043.stdout: protobuf-3.14.0-13.el9.x86_64 2024-08-31T12:14:32.732 INFO:teuthology.orchestra.run.smithi043.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-08-31T12:14:32.733 INFO:teuthology.orchestra.run.smithi043.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-08-31T12:14:32.733 INFO:teuthology.orchestra.run.smithi043.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-08-31T12:14:32.733 INFO:teuthology.orchestra.run.smithi043.stdout: python3-babel-2.9.1-2.el9.noarch 2024-08-31T12:14:32.733 INFO:teuthology.orchestra.run.smithi043.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-31T12:14:32.733 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-08-31T12:14:32.733 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ceph-argparse-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:32.733 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ceph-common-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:32.733 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cephfs-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:32.733 INFO:teuthology.orchestra.run.smithi043.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-08-31T12:14:32.733 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-31T12:14:32.733 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-31T12:14:32.734 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-31T12:14:32.734 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-08-31T12:14:32.734 INFO:teuthology.orchestra.run.smithi043.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-08-31T12:14:32.734 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-31T12:14:32.734 INFO:teuthology.orchestra.run.smithi043.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-08-31T12:14:32.734 INFO:teuthology.orchestra.run.smithi043.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-08-31T12:14:32.734 INFO:teuthology.orchestra.run.smithi043.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-08-31T12:14:32.735 INFO:teuthology.orchestra.run.smithi043.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-08-31T12:14:32.735 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-31T12:14:32.735 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-31T12:14:32.735 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-31T12:14:32.735 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-31T12:14:32.735 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-31T12:14:32.735 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-08-31T12:14:32.735 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-08-31T12:14:32.736 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-08-31T12:14:32.736 INFO:teuthology.orchestra.run.smithi043.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-08-31T12:14:32.736 INFO:teuthology.orchestra.run.smithi043.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-31T12:14:32.736 INFO:teuthology.orchestra.run.smithi043.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-31T12:14:32.736 INFO:teuthology.orchestra.run.smithi043.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-31T12:14:32.736 INFO:teuthology.orchestra.run.smithi043.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-31T12:14:32.736 INFO:teuthology.orchestra.run.smithi043.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-08-31T12:14:32.737 INFO:teuthology.orchestra.run.smithi043.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-08-31T12:14:32.737 INFO:teuthology.orchestra.run.smithi043.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-08-31T12:14:32.737 INFO:teuthology.orchestra.run.smithi043.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-08-31T12:14:32.737 INFO:teuthology.orchestra.run.smithi043.stdout: python3-packaging-20.9-5.el9.noarch 2024-08-31T12:14:32.737 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-31T12:14:32.737 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ply-3.11-14.el9.noarch 2024-08-31T12:14:32.737 INFO:teuthology.orchestra.run.smithi043.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-31T12:14:32.738 INFO:teuthology.orchestra.run.smithi043.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-31T12:14:32.738 INFO:teuthology.orchestra.run.smithi043.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-08-31T12:14:32.738 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-31T12:14:32.738 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-08-31T12:14:32.738 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-08-31T12:14:32.738 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-31T12:14:32.738 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-08-31T12:14:32.738 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-31T12:14:32.738 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rados-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:32.738 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rbd-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:32.739 INFO:teuthology.orchestra.run.smithi043.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-08-31T12:14:32.739 INFO:teuthology.orchestra.run.smithi043.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-31T12:14:32.739 INFO:teuthology.orchestra.run.smithi043.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-08-31T12:14:32.739 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rgw-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:32.739 INFO:teuthology.orchestra.run.smithi043.stdout: python3-routes-2.5.1-5.el9.noarch 2024-08-31T12:14:32.739 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rsa-4.9-2.el9.noarch 2024-08-31T12:14:32.739 INFO:teuthology.orchestra.run.smithi043.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-08-31T12:14:32.739 INFO:teuthology.orchestra.run.smithi043.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-31T12:14:32.739 INFO:teuthology.orchestra.run.smithi043.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-31T12:14:32.739 INFO:teuthology.orchestra.run.smithi043.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-08-31T12:14:32.739 INFO:teuthology.orchestra.run.smithi043.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-31T12:14:32.739 INFO:teuthology.orchestra.run.smithi043.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-08-31T12:14:32.739 INFO:teuthology.orchestra.run.smithi043.stdout: python3-webob-1.8.8-2.el9.noarch 2024-08-31T12:14:32.740 INFO:teuthology.orchestra.run.smithi043.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-08-31T12:14:32.740 INFO:teuthology.orchestra.run.smithi043.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-31T12:14:32.740 INFO:teuthology.orchestra.run.smithi043.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-31T12:14:32.740 INFO:teuthology.orchestra.run.smithi043.stdout: qatlib-24.02.0-1.el9.x86_64 2024-08-31T12:14:32.740 INFO:teuthology.orchestra.run.smithi043.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-08-31T12:14:32.740 INFO:teuthology.orchestra.run.smithi043.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-08-31T12:14:32.740 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-fuse-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:32.740 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-mirror-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:32.740 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-nbd-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:32.740 INFO:teuthology.orchestra.run.smithi043.stdout: re2-1:20211101-20.el9.x86_64 2024-08-31T12:14:32.740 INFO:teuthology.orchestra.run.smithi043.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-31T12:14:32.740 INFO:teuthology.orchestra.run.smithi043.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-31T12:14:32.741 INFO:teuthology.orchestra.run.smithi043.stdout: uuid-1.6.2-55.el9.x86_64 2024-08-31T12:14:32.741 INFO:teuthology.orchestra.run.smithi043.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-08-31T12:14:32.741 INFO:teuthology.orchestra.run.smithi043.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-31T12:14:32.741 INFO:teuthology.orchestra.run.smithi043.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-08-31T12:14:32.741 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:14:32.741 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-08-31T12:14:33.106 DEBUG:teuthology.parallel:result is None 2024-08-31T12:14:34.110 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4560.gd19e6118.el9.noa 124/152 2024-08-31T12:14:34.172 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4560.gd19e6118.el9.n 125/152 2024-08-31T12:14:35.196 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4560.gd19e6118.el9.n 125/152 2024-08-31T12:14:35.222 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4560.gd19 126/152 2024-08-31T12:14:35.343 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4560.gd19 126/152 2024-08-31T12:14:35.514 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4560.gd19e6118.el 127/152 2024-08-31T12:14:35.541 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-2:19.3.0-4560.gd19e6118.el9.x86_64 128/152 2024-08-31T12:14:35.564 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4560.gd19e6118.el9.x86_64 128/152 2024-08-31T12:14:35.564 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-31T12:14:35.564 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-31T12:14:35.564 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-31T12:14:35.564 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-31T12:14:35.565 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:14:35.616 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-rook-2:19.3.0-4560.gd19e6118.el9.noarch 129/152 2024-08-31T12:14:35.628 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4560.gd19e6118.el9.noarch 129/152 2024-08-31T12:14:35.732 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-2:19.3.0-4560.gd19e6118.el9.x86_64 130/152 2024-08-31T12:14:35.782 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-fuse-2:19.3.0-4560.gd19e6118.el9.x86_64 131/152 2024-08-31T12:14:36.800 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libtool-2.4.6-46.el9.x86_64 132/152 2024-08-31T12:14:36.827 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-radosgw-2:19.3.0-4560.gd19e6118.el9.x86_64 133/152 2024-08-31T12:14:36.850 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4560.gd19e6118.el9.x86_64 133/152 2024-08-31T12:14:36.850 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-31T12:14:36.851 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-31T12:14:36.851 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-31T12:14:36.851 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-31T12:14:36.851 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:14:36.895 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4560.gd19e6 134/152 2024-08-31T12:14:36.917 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4560.gd19e6 134/152 2024-08-31T12:14:36.918 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-31T12:14:36.918 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-31T12:14:36.918 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:14:36.970 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : cephfs-mirror-2:19.3.0-4560.gd19e6118.el9.x86_64 135/152 2024-08-31T12:14:36.993 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: cephfs-mirror-2:19.3.0-4560.gd19e6118.el9.x86_64 135/152 2024-08-31T12:14:36.993 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-31T12:14:36.994 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-08-31T12:14:36.994 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-08-31T12:14:36.994 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-08-31T12:14:36.994 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:14:37.329 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : rbd-mirror-2:19.3.0-4560.gd19e6118.el9.x86_64 136/152 2024-08-31T12:14:37.353 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4560.gd19e6118.el9.x86_64 136/152 2024-08-31T12:14:37.353 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-31T12:14:37.353 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-31T12:14:37.353 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-31T12:14:37.353 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-31T12:14:37.353 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:14:41.975 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-test-2:19.3.0-4560.gd19e6118.el9.x86_64 137/152 2024-08-31T12:14:42.017 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : rbd-fuse-2:19.3.0-4560.gd19e6118.el9.x86_64 138/152 2024-08-31T12:14:42.057 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : rbd-nbd-2:19.3.0-4560.gd19e6118.el9.x86_64 139/152 2024-08-31T12:14:42.088 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 140/152 2024-08-31T12:14:42.116 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 141/152 2024-08-31T12:14:42.309 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : cephfs-shell-2:19.3.0-4560.gd19e6118.el9.x86_64 142/152 2024-08-31T12:14:42.353 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 143/152 2024-08-31T12:14:42.383 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libcephfs-devel-2:19.3.0-4560.gd19e6118.el9.x86_ 144/152 2024-08-31T12:14:42.958 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 145/152 2024-08-31T12:14:42.997 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 145/152 2024-08-31T12:14:43.017 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 145/152 2024-08-31T12:14:43.928 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : cephfs-top-2:19.3.0-4560.gd19e6118.el9.noarch 146/152 2024-08-31T12:14:44.019 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : openssl-devel-1:3.2.2-4.el9.x86_64 147/152 2024-08-31T12:14:44.061 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-08-31T12:14:44.145 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-08-31T12:14:44.181 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-08-31T12:14:44.181 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-08-31T12:14:44.257 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-08-31T12:14:44.257 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-08-31T12:14:51.286 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-08-31T12:14:51.287 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-2:19.3.0-4560.gd19e6118.el9.x86_64 1/152 2024-08-31T12:14:51.287 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-base-2:19.3.0-4560.gd19e6118.el9.x86_64 2/152 2024-08-31T12:14:51.287 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-common-2:19.3.0-4560.gd19e6118.el9.x86_64 3/152 2024-08-31T12:14:51.287 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-fuse-2:19.3.0-4560.gd19e6118.el9.x86_64 4/152 2024-08-31T12:14:51.287 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4560.gd19e6 5/152 2024-08-31T12:14:51.287 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mds-2:19.3.0-4560.gd19e6118.el9.x86_64 6/152 2024-08-31T12:14:51.287 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-2:19.3.0-4560.gd19e6118.el9.x86_64 7/152 2024-08-31T12:14:51.287 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mon-2:19.3.0-4560.gd19e6118.el9.x86_64 8/152 2024-08-31T12:14:51.287 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-osd-2:19.3.0-4560.gd19e6118.el9.x86_64 9/152 2024-08-31T12:14:51.287 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-radosgw-2:19.3.0-4560.gd19e6118.el9.x86_64 10/152 2024-08-31T12:14:51.287 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-selinux-2:19.3.0-4560.gd19e6118.el9.x86_64 11/152 2024-08-31T12:14:51.287 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-test-2:19.3.0-4560.gd19e6118.el9.x86_64 12/152 2024-08-31T12:14:51.288 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : cephfs-mirror-2:19.3.0-4560.gd19e6118.el9.x86_64 13/152 2024-08-31T12:14:51.288 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : cephfs-shell-2:19.3.0-4560.gd19e6118.el9.x86_64 14/152 2024-08-31T12:14:51.288 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libcephfs-devel-2:19.3.0-4560.gd19e6118.el9.x86_ 15/152 2024-08-31T12:14:51.288 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libcephfs2-2:19.3.0-4560.gd19e6118.el9.x86_64 16/152 2024-08-31T12:14:51.288 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libcephsqlite-2:19.3.0-4560.gd19e6118.el9.x86_64 17/152 2024-08-31T12:14:51.288 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librados-devel-2:19.3.0-4560.gd19e6118.el9.x86_6 18/152 2024-08-31T12:14:51.288 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libradosstriper1-2:19.3.0-4560.gd19e6118.el9.x86 19/152 2024-08-31T12:14:51.288 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librgw2-2:19.3.0-4560.gd19e6118.el9.x86_64 20/152 2024-08-31T12:14:51.288 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4560.gd19e6118.el 21/152 2024-08-31T12:14:51.288 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-ceph-common-2:19.3.0-4560.gd19e6118.el9. 22/152 2024-08-31T12:14:51.288 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cephfs-2:19.3.0-4560.gd19e6118.el9.x86_6 23/152 2024-08-31T12:14:51.288 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rados-2:19.3.0-4560.gd19e6118.el9.x86_64 24/152 2024-08-31T12:14:51.289 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rbd-2:19.3.0-4560.gd19e6118.el9.x86_64 25/152 2024-08-31T12:14:51.289 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rgw-2:19.3.0-4560.gd19e6118.el9.x86_64 26/152 2024-08-31T12:14:51.289 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : rbd-fuse-2:19.3.0-4560.gd19e6118.el9.x86_64 27/152 2024-08-31T12:14:51.289 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : rbd-mirror-2:19.3.0-4560.gd19e6118.el9.x86_64 28/152 2024-08-31T12:14:51.289 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : rbd-nbd-2:19.3.0-4560.gd19e6118.el9.x86_64 29/152 2024-08-31T12:14:51.289 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4560.gd19e6118. 30/152 2024-08-31T12:14:51.289 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4560.gd19e6118.el9.noa 31/152 2024-08-31T12:14:51.289 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4560.gd19e6118.el9.n 32/152 2024-08-31T12:14:51.289 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4560.gd19 33/152 2024-08-31T12:14:51.289 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4560.gd19e6118.el 34/152 2024-08-31T12:14:51.289 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4560.gd19e6118.el9.noarch 35/152 2024-08-31T12:14:51.289 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4560.gd19e6118.e 36/152 2024-08-31T12:14:51.289 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-volume-2:19.3.0-4560.gd19e6118.el9.noarch 37/152 2024-08-31T12:14:51.290 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : cephadm-2:19.3.0-4560.gd19e6118.el9.noarch 38/152 2024-08-31T12:14:51.290 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : cephfs-top-2:19.3.0-4560.gd19e6118.el9.noarch 39/152 2024-08-31T12:14:51.290 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/152 2024-08-31T12:14:51.290 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/152 2024-08-31T12:14:51.290 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/152 2024-08-31T12:14:51.290 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-08-31T12:14:51.290 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-08-31T12:14:51.290 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-08-31T12:14:51.290 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-08-31T12:14:51.290 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-08-31T12:14:51.290 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/152 2024-08-31T12:14:51.290 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-08-31T12:14:51.291 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-08-31T12:14:51.291 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-08-31T12:14:51.291 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/152 2024-08-31T12:14:51.291 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-08-31T12:14:51.291 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-08-31T12:14:51.291 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-08-31T12:14:51.291 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-08-31T12:14:51.291 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-08-31T12:14:51.291 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-08-31T12:14:51.291 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-08-31T12:14:51.291 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-08-31T12:14:51.291 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/152 2024-08-31T12:14:51.291 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 62/152 2024-08-31T12:14:51.292 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-08-31T12:14:51.292 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-08-31T12:14:51.292 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-08-31T12:14:51.292 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 66/152 2024-08-31T12:14:51.292 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 67/152 2024-08-31T12:14:51.292 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : openssl-devel-1:3.2.2-4.el9.x86_64 68/152 2024-08-31T12:14:51.292 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : postgresql-13.16-1.el9.x86_64 69/152 2024-08-31T12:14:51.292 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 70/152 2024-08-31T12:14:51.292 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 71/152 2024-08-31T12:14:51.292 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 72/152 2024-08-31T12:14:51.292 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-08-31T12:14:51.292 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-08-31T12:14:51.293 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-08-31T12:14:51.293 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/152 2024-08-31T12:14:51.293 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 77/152 2024-08-31T12:14:51.293 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/152 2024-08-31T12:14:51.293 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/152 2024-08-31T12:14:51.293 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/152 2024-08-31T12:14:51.293 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/152 2024-08-31T12:14:51.293 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/152 2024-08-31T12:14:51.293 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/152 2024-08-31T12:14:51.293 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/152 2024-08-31T12:14:51.293 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/152 2024-08-31T12:14:51.293 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/152 2024-08-31T12:14:51.293 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/152 2024-08-31T12:14:51.294 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/152 2024-08-31T12:14:51.294 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/152 2024-08-31T12:14:51.294 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/152 2024-08-31T12:14:51.294 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/152 2024-08-31T12:14:51.294 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/152 2024-08-31T12:14:51.294 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-08-31T12:14:51.294 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/152 2024-08-31T12:14:51.294 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/152 2024-08-31T12:14:51.294 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/152 2024-08-31T12:14:51.294 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/152 2024-08-31T12:14:51.294 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/152 2024-08-31T12:14:51.294 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/152 2024-08-31T12:14:51.294 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/152 2024-08-31T12:14:51.295 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 101/152 2024-08-31T12:14:51.295 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/152 2024-08-31T12:14:51.295 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/152 2024-08-31T12:14:51.295 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/152 2024-08-31T12:14:51.295 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/152 2024-08-31T12:14:51.295 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/152 2024-08-31T12:14:51.295 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/152 2024-08-31T12:14:51.295 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/152 2024-08-31T12:14:51.295 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/152 2024-08-31T12:14:51.295 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/152 2024-08-31T12:14:51.295 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/152 2024-08-31T12:14:51.295 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/152 2024-08-31T12:14:51.296 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/152 2024-08-31T12:14:51.296 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/152 2024-08-31T12:14:51.296 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/152 2024-08-31T12:14:51.296 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/152 2024-08-31T12:14:51.296 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/152 2024-08-31T12:14:51.296 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/152 2024-08-31T12:14:51.296 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/152 2024-08-31T12:14:51.296 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/152 2024-08-31T12:14:51.296 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/152 2024-08-31T12:14:51.296 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 122/152 2024-08-31T12:14:51.296 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-08-31T12:14:51.296 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-08-31T12:14:51.296 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-08-31T12:14:51.297 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-08-31T12:14:51.297 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-08-31T12:14:51.297 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-08-31T12:14:51.297 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-08-31T12:14:51.297 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-08-31T12:14:51.297 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-08-31T12:14:51.297 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-08-31T12:14:51.297 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-08-31T12:14:51.297 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-08-31T12:14:51.297 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-08-31T12:14:51.297 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-08-31T12:14:51.297 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-08-31T12:14:51.297 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-08-31T12:14:51.298 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-08-31T12:14:51.298 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-08-31T12:14:51.298 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-08-31T12:14:51.298 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 142/152 2024-08-31T12:14:51.298 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 143/152 2024-08-31T12:14:51.298 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-08-31T12:14:51.298 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-08-31T12:14:51.298 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-08-31T12:14:51.298 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : re2-1:20211101-20.el9.x86_64 147/152 2024-08-31T12:14:51.298 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-08-31T12:14:51.298 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librados2-2:19.3.0-4560.gd19e6118.el9.x86_64 149/152 2024-08-31T12:14:51.298 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-08-31T12:14:51.298 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librbd1-2:19.3.0-4560.gd19e6118.el9.x86_64 151/152 2024-08-31T12:14:53.656 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-08-31T12:14:53.656 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:14:53.656 INFO:teuthology.orchestra.run.smithi019.stdout:Upgraded: 2024-08-31T12:14:53.656 INFO:teuthology.orchestra.run.smithi019.stdout: librados2-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:53.656 INFO:teuthology.orchestra.run.smithi019.stdout: librbd1-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:53.656 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-08-31T12:14:53.656 INFO:teuthology.orchestra.run.smithi019.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-08-31T12:14:53.656 INFO:teuthology.orchestra.run.smithi019.stdout: automake-1.16.2-8.el9.noarch 2024-08-31T12:14:53.657 INFO:teuthology.orchestra.run.smithi019.stdout: bison-3.7.4-5.el9.x86_64 2024-08-31T12:14:53.657 INFO:teuthology.orchestra.run.smithi019.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-08-31T12:14:53.657 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:53.657 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-base-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:53.657 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-common-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:53.657 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-fuse-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:53.657 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-grafana-dashboards-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-31T12:14:53.657 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-immutable-object-cache-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:53.657 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mds-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:53.657 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:53.657 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-cephadm-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-31T12:14:53.657 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-dashboard-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-31T12:14:53.658 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-31T12:14:53.658 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-modules-core-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-31T12:14:53.658 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-rook-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-31T12:14:53.658 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mon-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:53.658 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-osd-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:53.658 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-prometheus-alerts-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-31T12:14:53.658 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-radosgw-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:53.658 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-selinux-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:53.658 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-test-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:53.658 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-volume-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-31T12:14:53.658 INFO:teuthology.orchestra.run.smithi019.stdout: cephadm-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-31T12:14:53.658 INFO:teuthology.orchestra.run.smithi019.stdout: cephfs-mirror-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:53.658 INFO:teuthology.orchestra.run.smithi019.stdout: cephfs-shell-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:53.659 INFO:teuthology.orchestra.run.smithi019.stdout: cephfs-top-2:19.3.0-4560.gd19e6118.el9.noarch 2024-08-31T12:14:53.659 INFO:teuthology.orchestra.run.smithi019.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-08-31T12:14:53.659 INFO:teuthology.orchestra.run.smithi019.stdout: flex-2.6.4-9.el9.x86_64 2024-08-31T12:14:53.659 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-08-31T12:14:53.659 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-08-31T12:14:53.659 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-08-31T12:14:53.659 INFO:teuthology.orchestra.run.smithi019.stdout: fuse-2.9.9-16.el9.x86_64 2024-08-31T12:14:53.659 INFO:teuthology.orchestra.run.smithi019.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-31T12:14:53.659 INFO:teuthology.orchestra.run.smithi019.stdout: grpc-data-1.46.7-10.el9.noarch 2024-08-31T12:14:53.659 INFO:teuthology.orchestra.run.smithi019.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-08-31T12:14:53.659 INFO:teuthology.orchestra.run.smithi019.stdout: libarrow-9.0.0-11.el9.x86_64 2024-08-31T12:14:53.659 INFO:teuthology.orchestra.run.smithi019.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-08-31T12:14:53.660 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs-devel-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:53.660 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs2-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:53.660 INFO:teuthology.orchestra.run.smithi019.stdout: libcephsqlite-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:53.660 INFO:teuthology.orchestra.run.smithi019.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-08-31T12:14:53.660 INFO:teuthology.orchestra.run.smithi019.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-31T12:14:53.660 INFO:teuthology.orchestra.run.smithi019.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-31T12:14:53.660 INFO:teuthology.orchestra.run.smithi019.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-08-31T12:14:53.660 INFO:teuthology.orchestra.run.smithi019.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-31T12:14:53.660 INFO:teuthology.orchestra.run.smithi019.stdout: librados-devel-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:53.660 INFO:teuthology.orchestra.run.smithi019.stdout: libradosstriper1-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:53.660 INFO:teuthology.orchestra.run.smithi019.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-31T12:14:53.660 INFO:teuthology.orchestra.run.smithi019.stdout: librgw2-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:53.661 INFO:teuthology.orchestra.run.smithi019.stdout: libtool-2.4.6-46.el9.x86_64 2024-08-31T12:14:53.661 INFO:teuthology.orchestra.run.smithi019.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-31T12:14:53.661 INFO:teuthology.orchestra.run.smithi019.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-08-31T12:14:53.661 INFO:teuthology.orchestra.run.smithi019.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-08-31T12:14:53.661 INFO:teuthology.orchestra.run.smithi019.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-31T12:14:53.661 INFO:teuthology.orchestra.run.smithi019.stdout: lua-5.4.4-4.el9.x86_64 2024-08-31T12:14:53.661 INFO:teuthology.orchestra.run.smithi019.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-08-31T12:14:53.661 INFO:teuthology.orchestra.run.smithi019.stdout: luarocks-3.9.2-5.el9.noarch 2024-08-31T12:14:53.661 INFO:teuthology.orchestra.run.smithi019.stdout: openblas-0.3.26-2.el9.x86_64 2024-08-31T12:14:53.661 INFO:teuthology.orchestra.run.smithi019.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-08-31T12:14:53.661 INFO:teuthology.orchestra.run.smithi019.stdout: openssl-devel-1:3.2.2-4.el9.x86_64 2024-08-31T12:14:53.661 INFO:teuthology.orchestra.run.smithi019.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-08-31T12:14:53.661 INFO:teuthology.orchestra.run.smithi019.stdout: postgresql-13.16-1.el9.x86_64 2024-08-31T12:14:53.662 INFO:teuthology.orchestra.run.smithi019.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-08-31T12:14:53.662 INFO:teuthology.orchestra.run.smithi019.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-08-31T12:14:53.662 INFO:teuthology.orchestra.run.smithi019.stdout: postgresql-server-13.16-1.el9.x86_64 2024-08-31T12:14:53.662 INFO:teuthology.orchestra.run.smithi019.stdout: protobuf-3.14.0-13.el9.x86_64 2024-08-31T12:14:53.662 INFO:teuthology.orchestra.run.smithi019.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-08-31T12:14:53.662 INFO:teuthology.orchestra.run.smithi019.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-08-31T12:14:53.662 INFO:teuthology.orchestra.run.smithi019.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-08-31T12:14:53.662 INFO:teuthology.orchestra.run.smithi019.stdout: python3-babel-2.9.1-2.el9.noarch 2024-08-31T12:14:53.662 INFO:teuthology.orchestra.run.smithi019.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-31T12:14:53.662 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-08-31T12:14:53.662 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-argparse-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:53.662 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-common-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:53.662 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cephfs-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:53.662 INFO:teuthology.orchestra.run.smithi019.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-08-31T12:14:53.663 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-31T12:14:53.663 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-31T12:14:53.663 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-31T12:14:53.663 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-08-31T12:14:53.663 INFO:teuthology.orchestra.run.smithi019.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-08-31T12:14:53.663 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-31T12:14:53.663 INFO:teuthology.orchestra.run.smithi019.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-08-31T12:14:53.663 INFO:teuthology.orchestra.run.smithi019.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-08-31T12:14:53.663 INFO:teuthology.orchestra.run.smithi019.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-08-31T12:14:53.663 INFO:teuthology.orchestra.run.smithi019.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-08-31T12:14:53.663 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-31T12:14:53.663 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-31T12:14:53.663 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-31T12:14:53.663 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-31T12:14:53.664 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-31T12:14:53.664 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-08-31T12:14:53.664 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-08-31T12:14:53.664 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-08-31T12:14:53.664 INFO:teuthology.orchestra.run.smithi019.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-08-31T12:14:53.664 INFO:teuthology.orchestra.run.smithi019.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-31T12:14:53.664 INFO:teuthology.orchestra.run.smithi019.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-31T12:14:53.664 INFO:teuthology.orchestra.run.smithi019.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-31T12:14:53.664 INFO:teuthology.orchestra.run.smithi019.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-31T12:14:53.664 INFO:teuthology.orchestra.run.smithi019.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-08-31T12:14:53.664 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-08-31T12:14:53.664 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-08-31T12:14:53.664 INFO:teuthology.orchestra.run.smithi019.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-08-31T12:14:53.665 INFO:teuthology.orchestra.run.smithi019.stdout: python3-packaging-20.9-5.el9.noarch 2024-08-31T12:14:53.665 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-31T12:14:53.665 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ply-3.11-14.el9.noarch 2024-08-31T12:14:53.665 INFO:teuthology.orchestra.run.smithi019.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-31T12:14:53.665 INFO:teuthology.orchestra.run.smithi019.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-31T12:14:53.665 INFO:teuthology.orchestra.run.smithi019.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-08-31T12:14:53.665 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-31T12:14:53.665 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-08-31T12:14:53.665 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-08-31T12:14:53.665 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-31T12:14:53.665 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-08-31T12:14:53.665 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-31T12:14:53.665 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rados-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:53.666 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rbd-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:53.666 INFO:teuthology.orchestra.run.smithi019.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-08-31T12:14:53.666 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-31T12:14:53.666 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-08-31T12:14:53.666 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rgw-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:53.666 INFO:teuthology.orchestra.run.smithi019.stdout: python3-routes-2.5.1-5.el9.noarch 2024-08-31T12:14:53.666 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rsa-4.9-2.el9.noarch 2024-08-31T12:14:53.666 INFO:teuthology.orchestra.run.smithi019.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-08-31T12:14:53.666 INFO:teuthology.orchestra.run.smithi019.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-31T12:14:53.666 INFO:teuthology.orchestra.run.smithi019.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-31T12:14:53.666 INFO:teuthology.orchestra.run.smithi019.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-08-31T12:14:53.666 INFO:teuthology.orchestra.run.smithi019.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-31T12:14:53.666 INFO:teuthology.orchestra.run.smithi019.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-08-31T12:14:53.666 INFO:teuthology.orchestra.run.smithi019.stdout: python3-webob-1.8.8-2.el9.noarch 2024-08-31T12:14:53.667 INFO:teuthology.orchestra.run.smithi019.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-08-31T12:14:53.667 INFO:teuthology.orchestra.run.smithi019.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-31T12:14:53.667 INFO:teuthology.orchestra.run.smithi019.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-31T12:14:53.667 INFO:teuthology.orchestra.run.smithi019.stdout: qatlib-24.02.0-1.el9.x86_64 2024-08-31T12:14:53.668 INFO:teuthology.orchestra.run.smithi019.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-08-31T12:14:53.668 INFO:teuthology.orchestra.run.smithi019.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-08-31T12:14:53.668 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-fuse-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:53.668 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-mirror-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:53.668 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-nbd-2:19.3.0-4560.gd19e6118.el9.x86_64 2024-08-31T12:14:53.668 INFO:teuthology.orchestra.run.smithi019.stdout: re2-1:20211101-20.el9.x86_64 2024-08-31T12:14:53.668 INFO:teuthology.orchestra.run.smithi019.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-31T12:14:53.669 INFO:teuthology.orchestra.run.smithi019.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-31T12:14:53.669 INFO:teuthology.orchestra.run.smithi019.stdout: uuid-1.6.2-55.el9.x86_64 2024-08-31T12:14:53.669 INFO:teuthology.orchestra.run.smithi019.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-08-31T12:14:53.669 INFO:teuthology.orchestra.run.smithi019.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-31T12:14:53.669 INFO:teuthology.orchestra.run.smithi019.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-08-31T12:14:53.669 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:14:53.669 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-08-31T12:14:53.985 DEBUG:teuthology.parallel:result is None 2024-08-31T12:14:53.985 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d19e6118f7015f0047264006cb0c873528eb3ae6 2024-08-31T12:14:54.128 DEBUG:teuthology.orchestra.run.smithi019:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-31T12:14:54.150 INFO:teuthology.orchestra.run.smithi019.stdout:19.3.0-4560.gd19e6118.el9 2024-08-31T12:14:54.150 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4560.gd19e6118.el9 2024-08-31T12:14:54.150 INFO:teuthology.task.install:The correct ceph version 19.3.0-4560.gd19e6118 is installed. 2024-08-31T12:14:54.152 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d19e6118f7015f0047264006cb0c873528eb3ae6 2024-08-31T12:14:54.293 DEBUG:teuthology.orchestra.run.smithi043:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-31T12:14:54.320 INFO:teuthology.orchestra.run.smithi043.stdout:19.3.0-4560.gd19e6118.el9 2024-08-31T12:14:54.320 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4560.gd19e6118.el9 2024-08-31T12:14:54.320 INFO:teuthology.task.install:The correct ceph version 19.3.0-4560.gd19e6118 is installed. 2024-08-31T12:14:54.321 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-08-31T12:14:54.322 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-08-31T12:14:54.322 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-31T12:14:54.348 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-08-31T12:14:54.348 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-31T12:14:54.393 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-08-31T12:14:54.393 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-08-31T12:14:54.393 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/usr/bin/daemon-helper 2024-08-31T12:14:54.417 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-31T12:14:54.482 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-08-31T12:14:54.482 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/usr/bin/daemon-helper 2024-08-31T12:14:54.511 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-31T12:14:54.580 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-08-31T12:14:54.580 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-08-31T12:14:54.581 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-31T12:14:54.605 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-31T12:14:54.670 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-08-31T12:14:54.670 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-31T12:14:54.698 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-31T12:14:54.765 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-08-31T12:14:54.766 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-08-31T12:14:54.766 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/usr/bin/stdin-killer 2024-08-31T12:14:54.790 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-31T12:14:54.855 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-08-31T12:14:54.856 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/usr/bin/stdin-killer 2024-08-31T12:14:54.880 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-31T12:14:54.945 INFO:teuthology.run_tasks:Running task ceph... 2024-08-31T12:14:55.019 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-08-31T12:14:55.019 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 777 /var/log/ceph 2024-08-31T12:14:55.022 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 777 /var/log/ceph 2024-08-31T12:14:55.046 INFO:tasks.ceph:Disabling ceph logrotate... 2024-08-31T12:14:55.046 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f -- /etc/logrotate.d/ceph 2024-08-31T12:14:55.087 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f -- /etc/logrotate.d/ceph 2024-08-31T12:14:55.111 INFO:tasks.ceph:Creating extra log directories... 2024-08-31T12:14:55.111 DEBUG:teuthology.orchestra.run.smithi019:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-08-31T12:14:55.151 DEBUG:teuthology.orchestra.run.smithi043:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-08-31T12:14:55.182 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-08-31T12:14:55.183 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-08-31T12:14:55.183 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-08-31T12:14:55.183 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-08-31T12:14:55.183 DEBUG:teuthology.orchestra.run.smithi019:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-08-31T12:14:55.228 DEBUG:teuthology.orchestra.run.smithi019:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-08-31T12:14:55.343 DEBUG:teuthology.orchestra.run.smithi019:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-08-31T12:14:55.368 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-08-31T12:14:55.368 DEBUG:teuthology.orchestra.run.smithi043:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-08-31T12:14:55.383 DEBUG:teuthology.orchestra.run.smithi043:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-08-31T12:14:55.489 DEBUG:teuthology.orchestra.run.smithi043:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-08-31T12:14:55.518 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-08-31T12:14:55.518 INFO:tasks.ceph:config {'conf': {'client': {'client_shutdown_timeout': 120}, 'mds': {'mds oft prefetch dirfrags': True, 'mds_sleep_rank_change': 5000000.0}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'fs.*is degraded', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'insufficient standby MDS daemons available', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'overall HEALTH_', '\\(MON_DOWN\\)', 'SLOW_OPS', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-08-31T12:14:55.518 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/mchangir-2024-08-30_08:30:37-fs-wip-mchangir-testing-20240828.085847-main-debug-distro-default-smithi/7881510', 'branch': 'wip-mchangir-testing-20240828.085847-main-debug', 'description': 'fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}', 'email': 'mchangir@redhat.com', 'first_in_suite': False, 'job_id': '7881510', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'mchangir-2024-08-30_08:30:37-fs-wip-mchangir-testing-20240828.085847-main-debug-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-mchangir-testing-20240828.085847-main-debug'}, 'ceph': {'cephfs': {'max_mds': 3, 'session_timeout': 300}, 'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 2, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon': {'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'osd op complaint time': 180}}, 'conf': {'client': {'client_shutdown_timeout': 120}, 'mds': {'mds oft prefetch dirfrags': True, 'mds_sleep_rank_change': 5000000.0}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'fs.*is degraded', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'insufficient standby MDS daemons available', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'overall HEALTH_', '\\(MON_DOWN\\)', 'SLOW_OPS', 'slow request'], 'sha1': 'd19e6118f7015f0047264006cb0c873528eb3ae6'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'check-counter': {'counters': {'mds': ['mds.exported', 'mds.imported']}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd19e6118f7015f0047264006cb0c873528eb3ae6'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-mchangir-testing-20240828.085847-main-debug', 'sha1': 'd19e6118f7015f0047264006cb0c873528eb3ae6'}}, 'owner': 'scheduled_mchangir@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 5413, 'sha1': 'd19e6118f7015f0047264006cb0c873528eb3ae6', 'sleep_before_teardown': 0, 'subset': '347/512', 'suite': 'fs', 'suite_branch': 'wip-mchangir-testing-20240828.085847-main-debug', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_d19e6118f7015f0047264006cb0c873528eb3ae6/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'd19e6118f7015f0047264006cb0c873528eb3ae6', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi019.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB/HQNvJy60UtVNzln9ZKUAdsLVpf8jpDF6Rxh2Wh23CL/UTLoV32SrzlNCVqrzMxhndMNyIAOUOz33w/YNfubs=', 'smithi043.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDheWTOfQHsyDOKQ8feiqxC7FBApnpQOewzW8s5nX1GD1XY6B8MYuEynuH3Z3A0czLPADAaOh0nF44SrB9+8GbM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd19e6118f7015f0047264006cb0c873528eb3ae6'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300}, 'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 2, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon': {'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'osd op complaint time': 180}}, 'conf': {'client': {'client_shutdown_timeout': 120}, 'mds': {'mds oft prefetch dirfrags': True, 'mds_sleep_rank_change': 5000000.0}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'fs.*is degraded', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'insufficient standby MDS daemons available', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'overall HEALTH_', '\\(MON_DOWN\\)', 'SLOW_OPS', 'slow request'], 'sha1': 'd19e6118f7015f0047264006cb0c873528eb3ae6', 'cluster': 'ceph'}}, {'ceph-fuse': None}, {'mon_thrash': {'check_mds_failover': True, 'revive_delay': 20, 'thrash_delay': 10}}, {'workunit': {'clients': {'all': ['suites/ffsb.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-08-30_08:30:37', 'tube': 'smithi', 'user': 'mchangir', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-08-31T12:14:55.518 DEBUG:teuthology.orchestra.run.smithi019:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-08-31T12:14:55.521 DEBUG:teuthology.orchestra.run.smithi043:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-08-31T12:14:55.568 DEBUG:teuthology.orchestra.run.smithi019:> sudo install -d -m0777 -- /var/run/ceph 2024-08-31T12:14:55.577 DEBUG:teuthology.orchestra.run.smithi043:> sudo install -d -m0777 -- /var/run/ceph 2024-08-31T12:14:55.632 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-08-31T12:14:55.632 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/scratch_devs of=/dev/stdout 2024-08-31T12:14:55.655 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-31T12:14:55.655 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_1 2024-08-31T12:14:55.710 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-31T12:14:55.710 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-31T12:14:55.710 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-08-31T12:14:55.710 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-31T12:14:55.710 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-08-31T12:14:55.711 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-08-31 12:14:44.735025930 +0000 2024-08-31T12:14:55.711 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-08-31 12:12:41.091290921 +0000 2024-08-31T12:14:55.711 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-08-31 12:12:41.091290921 +0000 2024-08-31T12:14:55.711 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2024-08-31 12:12:41.091290921 +0000 2024-08-31T12:14:55.711 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-31T12:14:55.773 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-08-31T12:14:55.774 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-08-31T12:14:55.774 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000209694 s, 2.4 MB/s 2024-08-31T12:14:55.775 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-31T12:14:55.832 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_2 2024-08-31T12:14:55.888 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-31T12:14:55.888 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-31T12:14:55.888 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-08-31T12:14:55.888 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-31T12:14:55.888 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-08-31T12:14:55.888 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-08-31 12:14:44.736025920 +0000 2024-08-31T12:14:55.888 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-08-31 12:12:41.426290161 +0000 2024-08-31T12:14:55.888 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-08-31 12:12:41.426290161 +0000 2024-08-31T12:14:55.888 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2024-08-31 12:12:41.426290161 +0000 2024-08-31T12:14:55.888 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-31T12:14:55.951 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-08-31T12:14:55.951 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-08-31T12:14:55.951 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000236447 s, 2.2 MB/s 2024-08-31T12:14:55.952 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-31T12:14:56.008 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_3 2024-08-31T12:14:56.063 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-31T12:14:56.063 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-31T12:14:56.063 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-08-31T12:14:56.064 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-31T12:14:56.064 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-08-31T12:14:56.064 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-08-31 12:14:44.736025920 +0000 2024-08-31T12:14:56.064 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-08-31 12:12:41.761289400 +0000 2024-08-31T12:14:56.064 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-08-31 12:12:41.761289400 +0000 2024-08-31T12:14:56.064 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2024-08-31 12:12:41.761289400 +0000 2024-08-31T12:14:56.064 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-31T12:14:56.126 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-08-31T12:14:56.126 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-08-31T12:14:56.127 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000194909 s, 2.6 MB/s 2024-08-31T12:14:56.128 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-31T12:14:56.183 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_4 2024-08-31T12:14:56.238 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-31T12:14:56.238 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-31T12:14:56.238 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-08-31T12:14:56.239 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-31T12:14:56.239 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-08-31T12:14:56.239 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-08-31 12:14:44.737025910 +0000 2024-08-31T12:14:56.239 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-08-31 12:12:42.092288649 +0000 2024-08-31T12:14:56.239 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-08-31 12:12:42.092288649 +0000 2024-08-31T12:14:56.239 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2024-08-31 12:12:42.092288649 +0000 2024-08-31T12:14:56.239 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-31T12:14:56.301 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-08-31T12:14:56.301 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-08-31T12:14:56.301 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000199039 s, 2.6 MB/s 2024-08-31T12:14:56.302 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-31T12:14:56.358 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-08-31T12:14:56.358 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-08-31T12:14:56.358 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/scratch_devs of=/dev/stdout 2024-08-31T12:14:56.375 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-31T12:14:56.375 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_1 2024-08-31T12:14:56.431 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-31T12:14:56.431 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-31T12:14:56.431 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-08-31T12:14:56.431 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-31T12:14:56.431 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2024-08-31T12:14:56.432 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2024-08-31 12:14:24.219766762 +0000 2024-08-31T12:14:56.432 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2024-08-31 12:12:41.069874044 +0000 2024-08-31T12:14:56.432 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2024-08-31 12:12:41.069874044 +0000 2024-08-31T12:14:56.432 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: 2024-08-31 12:12:41.069874044 +0000 2024-08-31T12:14:56.432 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-31T12:14:56.495 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2024-08-31T12:14:56.495 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2024-08-31T12:14:56.495 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.000101429 s, 5.0 MB/s 2024-08-31T12:14:56.496 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-31T12:14:56.552 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_2 2024-08-31T12:14:56.607 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-31T12:14:56.607 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-31T12:14:56.607 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-08-31T12:14:56.607 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-31T12:14:56.607 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2024-08-31T12:14:56.607 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2024-08-31 12:14:24.219766762 +0000 2024-08-31T12:14:56.608 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2024-08-31 12:12:41.394873239 +0000 2024-08-31T12:14:56.608 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2024-08-31 12:12:41.394873239 +0000 2024-08-31T12:14:56.608 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: 2024-08-31 12:12:41.394873239 +0000 2024-08-31T12:14:56.608 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-31T12:14:56.670 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2024-08-31T12:14:56.670 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2024-08-31T12:14:56.670 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 8.3431e-05 s, 6.1 MB/s 2024-08-31T12:14:56.671 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-31T12:14:56.729 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_3 2024-08-31T12:14:56.784 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-31T12:14:56.784 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-31T12:14:56.784 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-08-31T12:14:56.784 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-31T12:14:56.784 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2024-08-31T12:14:56.785 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2024-08-31 12:14:24.219766762 +0000 2024-08-31T12:14:56.785 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2024-08-31 12:12:41.709872459 +0000 2024-08-31T12:14:56.785 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2024-08-31 12:12:41.709872459 +0000 2024-08-31T12:14:56.785 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: 2024-08-31 12:12:41.709872459 +0000 2024-08-31T12:14:56.785 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-31T12:14:56.847 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2024-08-31T12:14:56.847 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2024-08-31T12:14:56.847 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 8.3826e-05 s, 6.1 MB/s 2024-08-31T12:14:56.848 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-31T12:14:56.904 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_4 2024-08-31T12:14:56.959 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-31T12:14:56.959 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-31T12:14:56.959 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-08-31T12:14:56.959 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-31T12:14:56.959 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2024-08-31T12:14:56.960 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2024-08-31 12:14:24.220766749 +0000 2024-08-31T12:14:56.960 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2024-08-31 12:12:42.034871654 +0000 2024-08-31T12:14:56.960 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2024-08-31 12:12:42.034871654 +0000 2024-08-31T12:14:56.960 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: 2024-08-31 12:12:42.034871654 +0000 2024-08-31T12:14:56.960 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-31T12:14:57.021 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2024-08-31T12:14:57.021 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2024-08-31T12:14:57.021 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 9.1283e-05 s, 5.6 MB/s 2024-08-31T12:14:57.023 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-31T12:14:57.078 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-08-31T12:14:57.079 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi019.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi043.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-08-31T12:14:57.079 INFO:tasks.ceph:Generating config... 2024-08-31T12:14:57.080 INFO:tasks.ceph:[client] client_shutdown_timeout = 120 2024-08-31T12:14:57.080 INFO:tasks.ceph:[mds] mds oft prefetch dirfrags = True 2024-08-31T12:14:57.080 INFO:tasks.ceph:[mds] mds_sleep_rank_change = 5000000.0 2024-08-31T12:14:57.080 INFO:tasks.ceph:[mgr] debug client = 20 2024-08-31T12:14:57.080 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-08-31T12:14:57.080 INFO:tasks.ceph:[mgr] debug ms = 1 2024-08-31T12:14:57.081 INFO:tasks.ceph:[mon] debug mon = 20 2024-08-31T12:14:57.081 INFO:tasks.ceph:[mon] debug ms = 1 2024-08-31T12:14:57.081 INFO:tasks.ceph:[mon] debug paxos = 20 2024-08-31T12:14:57.081 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-08-31T12:14:57.081 INFO:tasks.ceph:[osd] bdev async discard = True 2024-08-31T12:14:57.081 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-08-31T12:14:57.081 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-08-31T12:14:57.081 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-08-31T12:14:57.081 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-08-31T12:14:57.081 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-08-31T12:14:57.082 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-08-31T12:14:57.082 INFO:tasks.ceph:[osd] debug ms = 1 2024-08-31T12:14:57.082 INFO:tasks.ceph:[osd] debug osd = 20 2024-08-31T12:14:57.082 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-08-31T12:14:57.082 INFO:tasks.ceph:[osd] filestore flush min = 0 2024-08-31T12:14:57.082 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-08-31T12:14:57.082 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-08-31T12:14:57.082 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-08-31T12:14:57.082 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-08-31T12:14:57.082 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-08-31T12:14:57.082 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-08-31T12:14:57.082 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-08-31T12:14:57.082 INFO:tasks.ceph:Setting up mon.a... 2024-08-31T12:14:57.083 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-08-31T12:14:57.143 INFO:teuthology.orchestra.run.smithi019.stdout:creating /etc/ceph/ceph.keyring 2024-08-31T12:14:57.146 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-08-31T12:14:57.198 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-08-31T12:14:57.222 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.19'), ('mon.b', '172.21.15.43'), ('mon.c', '[v2:172.21.15.43:3301,v1:172.21.15.43:6790]')] 2024-08-31T12:14:57.222 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.19,172.21.15.43,[v2:172.21.15.43:3301,v1:172.21.15.43:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client_shutdown_timeout': 120}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.e': {}, 'mds.b': {}, 'mds.d': {}, 'mds.f': {}, 'mds': {'mds oft prefetch dirfrags': True, 'mds_sleep_rank_change': 5000000.0}} 2024-08-31T12:14:57.223 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-08-31T12:14:57.223 DEBUG:teuthology.orchestra.run.smithi019:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-08-31T12:14:57.277 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.19 --add b 172.21.15.43 --addv c '[v2:172.21.15.43:3301,v1:172.21.15.43:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-08-31T12:14:57.374 INFO:teuthology.orchestra.run.smithi019.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-08-31T12:14:57.374 INFO:teuthology.orchestra.run.smithi019.stdout:monmaptool: generated fsid 65c40ef5-9e6a-4782-8f93-4032c6800123 2024-08-31T12:14:57.374 INFO:teuthology.orchestra.run.smithi019.stdout:setting min_mon_release = quincy 2024-08-31T12:14:57.374 INFO:teuthology.orchestra.run.smithi019.stdout:epoch 0 2024-08-31T12:14:57.375 INFO:teuthology.orchestra.run.smithi019.stdout:fsid 65c40ef5-9e6a-4782-8f93-4032c6800123 2024-08-31T12:14:57.375 INFO:teuthology.orchestra.run.smithi019.stdout:last_changed 2024-08-31T12:14:57.374518+0000 2024-08-31T12:14:57.375 INFO:teuthology.orchestra.run.smithi019.stdout:created 2024-08-31T12:14:57.374518+0000 2024-08-31T12:14:57.375 INFO:teuthology.orchestra.run.smithi019.stdout:min_mon_release 17 (quincy) 2024-08-31T12:14:57.375 INFO:teuthology.orchestra.run.smithi019.stdout:election_strategy: 1 2024-08-31T12:14:57.375 INFO:teuthology.orchestra.run.smithi019.stdout:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-08-31T12:14:57.375 INFO:teuthology.orchestra.run.smithi019.stdout:1: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.b 2024-08-31T12:14:57.375 INFO:teuthology.orchestra.run.smithi019.stdout:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.c 2024-08-31T12:14:57.375 INFO:teuthology.orchestra.run.smithi019.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-08-31T12:14:57.377 DEBUG:teuthology.orchestra.run.smithi019:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-08-31T12:14:57.391 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 65c40ef5-9e6a-4782-8f93-4032c6800123... 2024-08-31T12:14:57.392 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-08-31T12:14:57.434 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-08-31T12:14:57.474 INFO:teuthology.orchestra.run.smithi019.stdout:[global] 2024-08-31T12:14:57.474 INFO:teuthology.orchestra.run.smithi019.stdout: chdir = "" 2024-08-31T12:14:57.474 INFO:teuthology.orchestra.run.smithi019.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-08-31T12:14:57.474 INFO:teuthology.orchestra.run.smithi019.stdout: auth supported = cephx 2024-08-31T12:14:57.474 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:14:57.474 INFO:teuthology.orchestra.run.smithi019.stdout: filestore xattr use omap = true 2024-08-31T12:14:57.474 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:14:57.475 INFO:teuthology.orchestra.run.smithi019.stdout: mon clock drift allowed = 1.000 2024-08-31T12:14:57.475 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:14:57.475 INFO:teuthology.orchestra.run.smithi019.stdout: osd crush chooseleaf type = 0 2024-08-31T12:14:57.475 INFO:teuthology.orchestra.run.smithi019.stdout: auth debug = true 2024-08-31T12:14:57.475 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:14:57.475 INFO:teuthology.orchestra.run.smithi019.stdout: ms die on old message = true 2024-08-31T12:14:57.475 INFO:teuthology.orchestra.run.smithi019.stdout: ms die on bug = true 2024-08-31T12:14:57.475 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:14:57.475 INFO:teuthology.orchestra.run.smithi019.stdout: mon max pg per osd = 10000 # >= luminous 2024-08-31T12:14:57.475 INFO:teuthology.orchestra.run.smithi019.stdout: mon pg warn max object skew = 0 2024-08-31T12:14:57.475 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:14:57.475 INFO:teuthology.orchestra.run.smithi019.stdout: # disable pg_autoscaler by default for new pools 2024-08-31T12:14:57.475 INFO:teuthology.orchestra.run.smithi019.stdout: osd_pool_default_pg_autoscale_mode = off 2024-08-31T12:14:57.476 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:14:57.476 INFO:teuthology.orchestra.run.smithi019.stdout: osd pool default size = 2 2024-08-31T12:14:57.476 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:14:57.476 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd allow primary affinity = true 2024-08-31T12:14:57.476 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd allow pg remap = true 2024-08-31T12:14:57.476 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on legacy crush tunables = false 2024-08-31T12:14:57.476 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on crush straw calc version zero = false 2024-08-31T12:14:57.476 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on no sortbitwise = false 2024-08-31T12:14:57.476 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on osd down out interval zero = false 2024-08-31T12:14:57.476 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on too few osds = false 2024-08-31T12:14:57.476 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-08-31T12:14:57.476 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_pool_no_redundancy = false 2024-08-31T12:14:57.476 INFO:teuthology.orchestra.run.smithi019.stdout: mon_allow_pool_size_one = true 2024-08-31T12:14:57.476 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:14:57.476 INFO:teuthology.orchestra.run.smithi019.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-08-31T12:14:57.477 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:14:57.477 INFO:teuthology.orchestra.run.smithi019.stdout: osd default data pool replay window = 5 2024-08-31T12:14:57.477 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:14:57.477 INFO:teuthology.orchestra.run.smithi019.stdout: mon allow pool delete = true 2024-08-31T12:14:57.477 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:14:57.477 INFO:teuthology.orchestra.run.smithi019.stdout: mon cluster log file level = debug 2024-08-31T12:14:57.477 INFO:teuthology.orchestra.run.smithi019.stdout: debug asserts on shutdown = true 2024-08-31T12:14:57.477 INFO:teuthology.orchestra.run.smithi019.stdout: mon health detail to clog = false 2024-08-31T12:14:57.477 INFO:teuthology.orchestra.run.smithi019.stdout: mon host = "172.21.15.19,172.21.15.43,[v2:172.21.15.43:3301,v1:172.21.15.43:6790]" 2024-08-31T12:14:57.477 INFO:teuthology.orchestra.run.smithi019.stdout: fsid = 65c40ef5-9e6a-4782-8f93-4032c6800123 2024-08-31T12:14:57.477 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:14:57.477 INFO:teuthology.orchestra.run.smithi019.stdout:[osd] 2024-08-31T12:14:57.477 INFO:teuthology.orchestra.run.smithi019.stdout: osd journal size = 100 2024-08-31T12:14:57.478 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:14:57.478 INFO:teuthology.orchestra.run.smithi019.stdout: osd scrub load threshold = 5.0 2024-08-31T12:14:57.478 INFO:teuthology.orchestra.run.smithi019.stdout: osd scrub max interval = 600 2024-08-31T12:14:57.478 INFO:teuthology.orchestra.run.smithi019.stdout: osd mclock profile = high_recovery_ops 2024-08-31T12:14:57.478 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:14:57.478 INFO:teuthology.orchestra.run.smithi019.stdout: osd recover clone overlap = true 2024-08-31T12:14:57.478 INFO:teuthology.orchestra.run.smithi019.stdout: osd recovery max chunk = 1048576 2024-08-31T12:14:57.478 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:14:57.478 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug shutdown = true 2024-08-31T12:14:57.478 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug op order = true 2024-08-31T12:14:57.478 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug verify stray on activate = true 2024-08-31T12:14:57.478 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug trim objects = true 2024-08-31T12:14:57.478 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:14:57.479 INFO:teuthology.orchestra.run.smithi019.stdout: osd open classes on start = true 2024-08-31T12:14:57.479 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug pg log writeout = true 2024-08-31T12:14:57.479 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:14:57.479 INFO:teuthology.orchestra.run.smithi019.stdout: osd deep scrub update digest min age = 30 2024-08-31T12:14:57.479 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:14:57.479 INFO:teuthology.orchestra.run.smithi019.stdout: osd map max advance = 10 2024-08-31T12:14:57.479 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:14:57.479 INFO:teuthology.orchestra.run.smithi019.stdout: journal zero on create = true 2024-08-31T12:14:57.479 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:14:57.479 INFO:teuthology.orchestra.run.smithi019.stdout: filestore ondisk finisher threads = 3 2024-08-31T12:14:57.479 INFO:teuthology.orchestra.run.smithi019.stdout: filestore apply finisher threads = 3 2024-08-31T12:14:57.479 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:14:57.479 INFO:teuthology.orchestra.run.smithi019.stdout: bdev debug aio = true 2024-08-31T12:14:57.480 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug misdirected ops = true 2024-08-31T12:14:57.480 INFO:teuthology.orchestra.run.smithi019.stdout: bdev async discard = True 2024-08-31T12:14:57.480 INFO:teuthology.orchestra.run.smithi019.stdout: bdev enable discard = True 2024-08-31T12:14:57.480 INFO:teuthology.orchestra.run.smithi019.stdout: bluestore allocator = bitmap 2024-08-31T12:14:57.480 INFO:teuthology.orchestra.run.smithi019.stdout: bluestore block size = 96636764160 2024-08-31T12:14:57.480 INFO:teuthology.orchestra.run.smithi019.stdout: bluestore fsck on mount = True 2024-08-31T12:14:57.480 INFO:teuthology.orchestra.run.smithi019.stdout: debug bluefs = 1/20 2024-08-31T12:14:57.480 INFO:teuthology.orchestra.run.smithi019.stdout: debug bluestore = 1/20 2024-08-31T12:14:57.480 INFO:teuthology.orchestra.run.smithi019.stdout: debug ms = 1 2024-08-31T12:14:57.480 INFO:teuthology.orchestra.run.smithi019.stdout: debug osd = 20 2024-08-31T12:14:57.480 INFO:teuthology.orchestra.run.smithi019.stdout: debug rocksdb = 4/10 2024-08-31T12:14:57.480 INFO:teuthology.orchestra.run.smithi019.stdout: filestore flush min = 0 2024-08-31T12:14:57.480 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd backfillfull_ratio = 0.85 2024-08-31T12:14:57.480 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd full ratio = 0.9 2024-08-31T12:14:57.481 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd nearfull ratio = 0.8 2024-08-31T12:14:57.481 INFO:teuthology.orchestra.run.smithi019.stdout: osd blocked scrub grace period = 3600 2024-08-31T12:14:57.481 INFO:teuthology.orchestra.run.smithi019.stdout: osd failsafe full ratio = 0.95 2024-08-31T12:14:57.481 INFO:teuthology.orchestra.run.smithi019.stdout: osd max markdown count = 1000 2024-08-31T12:14:57.481 INFO:teuthology.orchestra.run.smithi019.stdout: osd objectstore = bluestore 2024-08-31T12:14:57.481 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:14:57.481 INFO:teuthology.orchestra.run.smithi019.stdout:[mgr] 2024-08-31T12:14:57.481 INFO:teuthology.orchestra.run.smithi019.stdout: debug ms = 1 2024-08-31T12:14:57.481 INFO:teuthology.orchestra.run.smithi019.stdout: debug mgr = 20 2024-08-31T12:14:57.481 INFO:teuthology.orchestra.run.smithi019.stdout: debug mon = 20 2024-08-31T12:14:57.481 INFO:teuthology.orchestra.run.smithi019.stdout: debug auth = 20 2024-08-31T12:14:57.481 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min pgs per osd = 4 2024-08-31T12:14:57.481 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min bytes per osd = 10 2024-08-31T12:14:57.482 INFO:teuthology.orchestra.run.smithi019.stdout: mgr/telemetry/nag = false 2024-08-31T12:14:57.482 INFO:teuthology.orchestra.run.smithi019.stdout: debug client = 20 2024-08-31T12:14:57.482 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:14:57.482 INFO:teuthology.orchestra.run.smithi019.stdout:[mon] 2024-08-31T12:14:57.482 INFO:teuthology.orchestra.run.smithi019.stdout: debug ms = 1 2024-08-31T12:14:57.482 INFO:teuthology.orchestra.run.smithi019.stdout: debug mon = 20 2024-08-31T12:14:57.482 INFO:teuthology.orchestra.run.smithi019.stdout: debug paxos = 20 2024-08-31T12:14:57.482 INFO:teuthology.orchestra.run.smithi019.stdout: debug auth = 20 2024-08-31T12:14:57.482 INFO:teuthology.orchestra.run.smithi019.stdout: mon data avail warn = 5 2024-08-31T12:14:57.482 INFO:teuthology.orchestra.run.smithi019.stdout: mon mgr mkfs grace = 240 2024-08-31T12:14:57.482 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min pgs per osd = 4 2024-08-31T12:14:57.482 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd reporter subtree level = osd 2024-08-31T12:14:57.482 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd prime pg temp = true 2024-08-31T12:14:57.482 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min bytes per osd = 10 2024-08-31T12:14:57.483 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:14:57.483 INFO:teuthology.orchestra.run.smithi019.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-08-31T12:14:57.483 INFO:teuthology.orchestra.run.smithi019.stdout: auth mon ticket ttl = 660 # 11m 2024-08-31T12:14:57.483 INFO:teuthology.orchestra.run.smithi019.stdout: auth service ticket ttl = 240 # 4m 2024-08-31T12:14:57.483 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:14:57.483 INFO:teuthology.orchestra.run.smithi019.stdout: # don't complain about insecure global_id in the test suite 2024-08-31T12:14:57.483 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-08-31T12:14:57.483 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-08-31T12:14:57.483 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:14:57.483 INFO:teuthology.orchestra.run.smithi019.stdout: # 1m isn't quite enough 2024-08-31T12:14:57.483 INFO:teuthology.orchestra.run.smithi019.stdout: mon_down_mkfs_grace = 2m 2024-08-31T12:14:57.483 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:14:57.483 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_filestore_osds = false 2024-08-31T12:14:57.483 INFO:teuthology.orchestra.run.smithi019.stdout: mon down mkfs grace = 300 2024-08-31T12:14:57.484 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:14:57.484 INFO:teuthology.orchestra.run.smithi019.stdout:[client] 2024-08-31T12:14:57.484 INFO:teuthology.orchestra.run.smithi019.stdout: rgw cache enabled = true 2024-08-31T12:14:57.484 INFO:teuthology.orchestra.run.smithi019.stdout: rgw enable ops log = true 2024-08-31T12:14:57.484 INFO:teuthology.orchestra.run.smithi019.stdout: rgw enable usage log = true 2024-08-31T12:14:57.484 INFO:teuthology.orchestra.run.smithi019.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-08-31T12:14:57.484 INFO:teuthology.orchestra.run.smithi019.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-08-31T12:14:57.484 INFO:teuthology.orchestra.run.smithi019.stdout: client_shutdown_timeout = 120 2024-08-31T12:14:57.484 INFO:teuthology.orchestra.run.smithi019.stdout:[mon.a] 2024-08-31T12:14:57.484 INFO:teuthology.orchestra.run.smithi019.stdout:[mon.b] 2024-08-31T12:14:57.484 INFO:teuthology.orchestra.run.smithi019.stdout:[mon.c] 2024-08-31T12:14:57.484 INFO:teuthology.orchestra.run.smithi019.stdout:[mds.a] 2024-08-31T12:14:57.484 INFO:teuthology.orchestra.run.smithi019.stdout:[mds.c] 2024-08-31T12:14:57.484 INFO:teuthology.orchestra.run.smithi019.stdout:[mds.e] 2024-08-31T12:14:57.485 INFO:teuthology.orchestra.run.smithi019.stdout:[mds.b] 2024-08-31T12:14:57.485 INFO:teuthology.orchestra.run.smithi019.stdout:[mds.d] 2024-08-31T12:14:57.485 INFO:teuthology.orchestra.run.smithi019.stdout:[mds.f] 2024-08-31T12:14:57.485 INFO:teuthology.orchestra.run.smithi019.stdout:[mds] 2024-08-31T12:14:57.485 INFO:teuthology.orchestra.run.smithi019.stdout: mds oft prefetch dirfrags = True 2024-08-31T12:14:57.485 INFO:teuthology.orchestra.run.smithi019.stdout: mds_sleep_rank_change = 5000000.0 2024-08-31T12:14:57.485 INFO:teuthology.orchestra.run.smithi043.stdout:[global] 2024-08-31T12:14:57.486 INFO:teuthology.orchestra.run.smithi043.stdout: chdir = "" 2024-08-31T12:14:57.486 INFO:teuthology.orchestra.run.smithi043.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-08-31T12:14:57.486 INFO:teuthology.orchestra.run.smithi043.stdout: auth supported = cephx 2024-08-31T12:14:57.486 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:14:57.486 INFO:teuthology.orchestra.run.smithi043.stdout: filestore xattr use omap = true 2024-08-31T12:14:57.486 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:14:57.486 INFO:teuthology.orchestra.run.smithi043.stdout: mon clock drift allowed = 1.000 2024-08-31T12:14:57.486 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:14:57.486 INFO:teuthology.orchestra.run.smithi043.stdout: osd crush chooseleaf type = 0 2024-08-31T12:14:57.486 INFO:teuthology.orchestra.run.smithi043.stdout: auth debug = true 2024-08-31T12:14:57.486 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:14:57.486 INFO:teuthology.orchestra.run.smithi043.stdout: ms die on old message = true 2024-08-31T12:14:57.486 INFO:teuthology.orchestra.run.smithi043.stdout: ms die on bug = true 2024-08-31T12:14:57.487 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:14:57.487 INFO:teuthology.orchestra.run.smithi043.stdout: mon max pg per osd = 10000 # >= luminous 2024-08-31T12:14:57.487 INFO:teuthology.orchestra.run.smithi043.stdout: mon pg warn max object skew = 0 2024-08-31T12:14:57.487 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:14:57.487 INFO:teuthology.orchestra.run.smithi043.stdout: # disable pg_autoscaler by default for new pools 2024-08-31T12:14:57.487 INFO:teuthology.orchestra.run.smithi043.stdout: osd_pool_default_pg_autoscale_mode = off 2024-08-31T12:14:57.487 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:14:57.487 INFO:teuthology.orchestra.run.smithi043.stdout: osd pool default size = 2 2024-08-31T12:14:57.487 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:14:57.487 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd allow primary affinity = true 2024-08-31T12:14:57.487 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd allow pg remap = true 2024-08-31T12:14:57.487 INFO:teuthology.orchestra.run.smithi043.stdout: mon warn on legacy crush tunables = false 2024-08-31T12:14:57.487 INFO:teuthology.orchestra.run.smithi043.stdout: mon warn on crush straw calc version zero = false 2024-08-31T12:14:57.488 INFO:teuthology.orchestra.run.smithi043.stdout: mon warn on no sortbitwise = false 2024-08-31T12:14:57.488 INFO:teuthology.orchestra.run.smithi043.stdout: mon warn on osd down out interval zero = false 2024-08-31T12:14:57.488 INFO:teuthology.orchestra.run.smithi043.stdout: mon warn on too few osds = false 2024-08-31T12:14:57.488 INFO:teuthology.orchestra.run.smithi043.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-08-31T12:14:57.488 INFO:teuthology.orchestra.run.smithi043.stdout: mon_warn_on_pool_no_redundancy = false 2024-08-31T12:14:57.488 INFO:teuthology.orchestra.run.smithi043.stdout: mon_allow_pool_size_one = true 2024-08-31T12:14:57.488 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:14:57.488 INFO:teuthology.orchestra.run.smithi043.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-08-31T12:14:57.488 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:14:57.488 INFO:teuthology.orchestra.run.smithi043.stdout: osd default data pool replay window = 5 2024-08-31T12:14:57.488 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:14:57.488 INFO:teuthology.orchestra.run.smithi043.stdout: mon allow pool delete = true 2024-08-31T12:14:57.489 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:14:57.489 INFO:teuthology.orchestra.run.smithi043.stdout: mon cluster log file level = debug 2024-08-31T12:14:57.489 INFO:teuthology.orchestra.run.smithi043.stdout: debug asserts on shutdown = true 2024-08-31T12:14:57.489 INFO:teuthology.orchestra.run.smithi043.stdout: mon health detail to clog = false 2024-08-31T12:14:57.489 INFO:teuthology.orchestra.run.smithi043.stdout: mon host = "172.21.15.19,172.21.15.43,[v2:172.21.15.43:3301,v1:172.21.15.43:6790]" 2024-08-31T12:14:57.489 INFO:teuthology.orchestra.run.smithi043.stdout: fsid = 65c40ef5-9e6a-4782-8f93-4032c6800123 2024-08-31T12:14:57.489 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:14:57.489 INFO:teuthology.orchestra.run.smithi043.stdout:[osd] 2024-08-31T12:14:57.489 INFO:teuthology.orchestra.run.smithi043.stdout: osd journal size = 100 2024-08-31T12:14:57.489 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:14:57.489 INFO:teuthology.orchestra.run.smithi043.stdout: osd scrub load threshold = 5.0 2024-08-31T12:14:57.489 INFO:teuthology.orchestra.run.smithi043.stdout: osd scrub max interval = 600 2024-08-31T12:14:57.489 INFO:teuthology.orchestra.run.smithi043.stdout: osd mclock profile = high_recovery_ops 2024-08-31T12:14:57.490 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:14:57.490 INFO:teuthology.orchestra.run.smithi043.stdout: osd recover clone overlap = true 2024-08-31T12:14:57.490 INFO:teuthology.orchestra.run.smithi043.stdout: osd recovery max chunk = 1048576 2024-08-31T12:14:57.490 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:14:57.490 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug shutdown = true 2024-08-31T12:14:57.490 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug op order = true 2024-08-31T12:14:57.490 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug verify stray on activate = true 2024-08-31T12:14:57.490 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug trim objects = true 2024-08-31T12:14:57.490 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:14:57.490 INFO:teuthology.orchestra.run.smithi043.stdout: osd open classes on start = true 2024-08-31T12:14:57.490 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug pg log writeout = true 2024-08-31T12:14:57.490 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:14:57.491 INFO:teuthology.orchestra.run.smithi043.stdout: osd deep scrub update digest min age = 30 2024-08-31T12:14:57.491 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:14:57.491 INFO:teuthology.orchestra.run.smithi043.stdout: osd map max advance = 10 2024-08-31T12:14:57.491 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:14:57.491 INFO:teuthology.orchestra.run.smithi043.stdout: journal zero on create = true 2024-08-31T12:14:57.491 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:14:57.491 INFO:teuthology.orchestra.run.smithi043.stdout: filestore ondisk finisher threads = 3 2024-08-31T12:14:57.491 INFO:teuthology.orchestra.run.smithi043.stdout: filestore apply finisher threads = 3 2024-08-31T12:14:57.491 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:14:57.491 INFO:teuthology.orchestra.run.smithi043.stdout: bdev debug aio = true 2024-08-31T12:14:57.491 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug misdirected ops = true 2024-08-31T12:14:57.491 INFO:teuthology.orchestra.run.smithi043.stdout: bdev async discard = True 2024-08-31T12:14:57.491 INFO:teuthology.orchestra.run.smithi043.stdout: bdev enable discard = True 2024-08-31T12:14:57.491 INFO:teuthology.orchestra.run.smithi043.stdout: bluestore allocator = bitmap 2024-08-31T12:14:57.492 INFO:teuthology.orchestra.run.smithi043.stdout: bluestore block size = 96636764160 2024-08-31T12:14:57.492 INFO:teuthology.orchestra.run.smithi043.stdout: bluestore fsck on mount = True 2024-08-31T12:14:57.492 INFO:teuthology.orchestra.run.smithi043.stdout: debug bluefs = 1/20 2024-08-31T12:14:57.492 INFO:teuthology.orchestra.run.smithi043.stdout: debug bluestore = 1/20 2024-08-31T12:14:57.492 INFO:teuthology.orchestra.run.smithi043.stdout: debug ms = 1 2024-08-31T12:14:57.492 INFO:teuthology.orchestra.run.smithi043.stdout: debug osd = 20 2024-08-31T12:14:57.492 INFO:teuthology.orchestra.run.smithi043.stdout: debug rocksdb = 4/10 2024-08-31T12:14:57.492 INFO:teuthology.orchestra.run.smithi043.stdout: filestore flush min = 0 2024-08-31T12:14:57.492 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd backfillfull_ratio = 0.85 2024-08-31T12:14:57.492 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd full ratio = 0.9 2024-08-31T12:14:57.492 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd nearfull ratio = 0.8 2024-08-31T12:14:57.492 INFO:teuthology.orchestra.run.smithi043.stdout: osd blocked scrub grace period = 3600 2024-08-31T12:14:57.492 INFO:teuthology.orchestra.run.smithi043.stdout: osd failsafe full ratio = 0.95 2024-08-31T12:14:57.493 INFO:teuthology.orchestra.run.smithi043.stdout: osd max markdown count = 1000 2024-08-31T12:14:57.493 INFO:teuthology.orchestra.run.smithi043.stdout: osd objectstore = bluestore 2024-08-31T12:14:57.493 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:14:57.493 INFO:teuthology.orchestra.run.smithi043.stdout:[mgr] 2024-08-31T12:14:57.493 INFO:teuthology.orchestra.run.smithi043.stdout: debug ms = 1 2024-08-31T12:14:57.493 INFO:teuthology.orchestra.run.smithi043.stdout: debug mgr = 20 2024-08-31T12:14:57.493 INFO:teuthology.orchestra.run.smithi043.stdout: debug mon = 20 2024-08-31T12:14:57.493 INFO:teuthology.orchestra.run.smithi043.stdout: debug auth = 20 2024-08-31T12:14:57.493 INFO:teuthology.orchestra.run.smithi043.stdout: mon reweight min pgs per osd = 4 2024-08-31T12:14:57.493 INFO:teuthology.orchestra.run.smithi043.stdout: mon reweight min bytes per osd = 10 2024-08-31T12:14:57.493 INFO:teuthology.orchestra.run.smithi043.stdout: mgr/telemetry/nag = false 2024-08-31T12:14:57.493 INFO:teuthology.orchestra.run.smithi043.stdout: debug client = 20 2024-08-31T12:14:57.493 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:14:57.494 INFO:teuthology.orchestra.run.smithi043.stdout:[mon] 2024-08-31T12:14:57.494 INFO:teuthology.orchestra.run.smithi043.stdout: debug ms = 1 2024-08-31T12:14:57.494 INFO:teuthology.orchestra.run.smithi043.stdout: debug mon = 20 2024-08-31T12:14:57.494 INFO:teuthology.orchestra.run.smithi043.stdout: debug paxos = 20 2024-08-31T12:14:57.494 INFO:teuthology.orchestra.run.smithi043.stdout: debug auth = 20 2024-08-31T12:14:57.494 INFO:teuthology.orchestra.run.smithi043.stdout: mon data avail warn = 5 2024-08-31T12:14:57.494 INFO:teuthology.orchestra.run.smithi043.stdout: mon mgr mkfs grace = 240 2024-08-31T12:14:57.494 INFO:teuthology.orchestra.run.smithi043.stdout: mon reweight min pgs per osd = 4 2024-08-31T12:14:57.494 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd reporter subtree level = osd 2024-08-31T12:14:57.494 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd prime pg temp = true 2024-08-31T12:14:57.494 INFO:teuthology.orchestra.run.smithi043.stdout: mon reweight min bytes per osd = 10 2024-08-31T12:14:57.494 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:14:57.494 INFO:teuthology.orchestra.run.smithi043.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-08-31T12:14:57.495 INFO:teuthology.orchestra.run.smithi043.stdout: auth mon ticket ttl = 660 # 11m 2024-08-31T12:14:57.495 INFO:teuthology.orchestra.run.smithi043.stdout: auth service ticket ttl = 240 # 4m 2024-08-31T12:14:57.495 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:14:57.495 INFO:teuthology.orchestra.run.smithi043.stdout: # don't complain about insecure global_id in the test suite 2024-08-31T12:14:57.495 INFO:teuthology.orchestra.run.smithi043.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-08-31T12:14:57.495 INFO:teuthology.orchestra.run.smithi043.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-08-31T12:14:57.495 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:14:57.495 INFO:teuthology.orchestra.run.smithi043.stdout: # 1m isn't quite enough 2024-08-31T12:14:57.495 INFO:teuthology.orchestra.run.smithi043.stdout: mon_down_mkfs_grace = 2m 2024-08-31T12:14:57.495 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:14:57.495 INFO:teuthology.orchestra.run.smithi043.stdout: mon_warn_on_filestore_osds = false 2024-08-31T12:14:57.495 INFO:teuthology.orchestra.run.smithi043.stdout: mon down mkfs grace = 300 2024-08-31T12:14:57.495 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-08-31T12:14:57.496 INFO:teuthology.orchestra.run.smithi043.stdout:[client] 2024-08-31T12:14:57.496 INFO:teuthology.orchestra.run.smithi043.stdout: rgw cache enabled = true 2024-08-31T12:14:57.496 INFO:teuthology.orchestra.run.smithi043.stdout: rgw enable ops log = true 2024-08-31T12:14:57.496 INFO:teuthology.orchestra.run.smithi043.stdout: rgw enable usage log = true 2024-08-31T12:14:57.496 INFO:teuthology.orchestra.run.smithi043.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-08-31T12:14:57.496 INFO:teuthology.orchestra.run.smithi043.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-08-31T12:14:57.496 INFO:teuthology.orchestra.run.smithi043.stdout: client_shutdown_timeout = 120 2024-08-31T12:14:57.496 INFO:teuthology.orchestra.run.smithi043.stdout:[mon.a] 2024-08-31T12:14:57.496 INFO:teuthology.orchestra.run.smithi043.stdout:[mon.b] 2024-08-31T12:14:57.496 INFO:teuthology.orchestra.run.smithi043.stdout:[mon.c] 2024-08-31T12:14:57.496 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.a] 2024-08-31T12:14:57.496 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.c] 2024-08-31T12:14:57.497 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.e] 2024-08-31T12:14:57.497 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.b] 2024-08-31T12:14:57.497 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.d] 2024-08-31T12:14:57.497 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.f] 2024-08-31T12:14:57.497 INFO:teuthology.orchestra.run.smithi043.stdout:[mds] 2024-08-31T12:14:57.497 INFO:teuthology.orchestra.run.smithi043.stdout: mds oft prefetch dirfrags = True 2024-08-31T12:14:57.497 INFO:teuthology.orchestra.run.smithi043.stdout: mds_sleep_rank_change = 5000000.0 2024-08-31T12:14:57.498 INFO:tasks.ceph:Creating admin key on mon.a... 2024-08-31T12:14:57.498 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-08-31T12:14:57.602 INFO:tasks.ceph:Copying monmap to all nodes... 2024-08-31T12:14:57.602 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-08-31T12:14:57.602 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-08-31T12:14:57.617 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-08-31T12:14:57.618 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-08-31T12:14:57.672 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi019.front.sepia.ceph.com 2024-08-31T12:14:57.672 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-08-31T12:14:57.672 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/ceph/ceph.keyring 2024-08-31T12:14:57.672 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-08-31T12:14:57.746 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-08-31T12:14:57.746 DEBUG:teuthology.orchestra.run.smithi019:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-08-31T12:14:57.801 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi043.front.sepia.ceph.com 2024-08-31T12:14:57.801 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-08-31T12:14:57.801 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/ceph/ceph.keyring 2024-08-31T12:14:57.802 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-08-31T12:14:57.836 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-08-31T12:14:57.837 DEBUG:teuthology.orchestra.run.smithi043:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-08-31T12:14:57.892 INFO:tasks.ceph:Setting up mon nodes... 2024-08-31T12:14:57.893 INFO:tasks.ceph:Setting up mgr nodes... 2024-08-31T12:14:57.893 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-08-31T12:14:57.952 INFO:teuthology.orchestra.run.smithi019.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-08-31T12:14:57.957 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-08-31T12:14:58.024 INFO:teuthology.orchestra.run.smithi043.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-08-31T12:14:58.027 INFO:tasks.ceph:Setting up mds nodes... 2024-08-31T12:14:58.028 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-08-31T12:14:58.087 INFO:teuthology.orchestra.run.smithi019.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-08-31T12:14:58.090 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-08-31T12:14:58.114 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-08-31T12:14:58.214 INFO:teuthology.orchestra.run.smithi019.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-08-31T12:14:58.217 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-08-31T12:14:58.241 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/mds/ceph-e && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.e /var/lib/ceph/mds/ceph-e/keyring 2024-08-31T12:14:58.341 INFO:teuthology.orchestra.run.smithi019.stdout:creating /var/lib/ceph/mds/ceph-e/keyring 2024-08-31T12:14:58.345 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-e 2024-08-31T12:14:58.369 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-08-31T12:14:58.430 INFO:teuthology.orchestra.run.smithi043.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-08-31T12:14:58.433 DEBUG:teuthology.orchestra.run.smithi043:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-08-31T12:14:58.458 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-08-31T12:14:58.558 INFO:teuthology.orchestra.run.smithi043.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-08-31T12:14:58.561 DEBUG:teuthology.orchestra.run.smithi043:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-08-31T12:14:58.585 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /var/lib/ceph/mds/ceph-f && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.f /var/lib/ceph/mds/ceph-f/keyring 2024-08-31T12:14:58.685 INFO:teuthology.orchestra.run.smithi043.stdout:creating /var/lib/ceph/mds/ceph-f/keyring 2024-08-31T12:14:58.688 DEBUG:teuthology.orchestra.run.smithi043:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-f 2024-08-31T12:14:58.712 INFO:tasks.ceph_client:Setting up client nodes... 2024-08-31T12:14:58.712 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-08-31T12:14:58.764 INFO:teuthology.orchestra.run.smithi019.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-08-31T12:14:58.777 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-08-31T12:14:58.778 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi019.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi043.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-08-31T12:14:58.778 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-08-31T12:14:58.802 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-08-31T12:14:58.802 INFO:tasks.ceph:role: osd.0 2024-08-31T12:14:58.802 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi019.front.sepia.ceph.com 2024-08-31T12:14:58.802 DEBUG:teuthology.orchestra.run.smithi019:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-08-31T12:14:58.874 INFO:teuthology.orchestra.run.smithi019.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-08-31T12:14:58.874 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-31T12:14:58.874 INFO:teuthology.orchestra.run.smithi019.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-31T12:14:58.874 INFO:teuthology.orchestra.run.smithi019.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-31T12:14:58.874 INFO:teuthology.orchestra.run.smithi019.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-31T12:14:58.875 INFO:teuthology.orchestra.run.smithi019.stdout: = sunit=0 swidth=0 blks 2024-08-31T12:14:58.875 INFO:teuthology.orchestra.run.smithi019.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-31T12:14:58.875 INFO:teuthology.orchestra.run.smithi019.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-31T12:14:58.875 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-31T12:14:58.875 INFO:teuthology.orchestra.run.smithi019.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-31T12:14:59.369 INFO:teuthology.orchestra.run.smithi019.stdout:Discarding blocks...Done. 2024-08-31T12:14:59.370 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi019.front.sepia.ceph.com -o noatime 2024-08-31T12:14:59.371 DEBUG:teuthology.orchestra.run.smithi019:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-08-31T12:14:59.486 DEBUG:teuthology.orchestra.run.smithi019:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-08-31T12:14:59.517 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-08-31T12:14:59.583 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-08-31T12:14:59.583 INFO:tasks.ceph:role: osd.1 2024-08-31T12:14:59.583 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi019.front.sepia.ceph.com 2024-08-31T12:14:59.583 DEBUG:teuthology.orchestra.run.smithi019:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-08-31T12:14:59.656 INFO:teuthology.orchestra.run.smithi019.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-08-31T12:14:59.656 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-31T12:14:59.656 INFO:teuthology.orchestra.run.smithi019.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-31T12:14:59.656 INFO:teuthology.orchestra.run.smithi019.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-31T12:14:59.656 INFO:teuthology.orchestra.run.smithi019.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-31T12:14:59.656 INFO:teuthology.orchestra.run.smithi019.stdout: = sunit=0 swidth=0 blks 2024-08-31T12:14:59.656 INFO:teuthology.orchestra.run.smithi019.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-31T12:14:59.656 INFO:teuthology.orchestra.run.smithi019.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-31T12:14:59.656 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-31T12:14:59.656 INFO:teuthology.orchestra.run.smithi019.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-31T12:15:00.103 INFO:teuthology.orchestra.run.smithi019.stdout:Discarding blocks...Done. 2024-08-31T12:15:00.105 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi019.front.sepia.ceph.com -o noatime 2024-08-31T12:15:00.105 DEBUG:teuthology.orchestra.run.smithi019:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-08-31T12:15:00.178 DEBUG:teuthology.orchestra.run.smithi019:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-08-31T12:15:00.248 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-08-31T12:15:00.313 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-08-31T12:15:00.313 INFO:tasks.ceph:role: osd.2 2024-08-31T12:15:00.313 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi019.front.sepia.ceph.com 2024-08-31T12:15:00.313 DEBUG:teuthology.orchestra.run.smithi019:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-08-31T12:15:00.385 INFO:teuthology.orchestra.run.smithi019.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-08-31T12:15:00.385 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-31T12:15:00.385 INFO:teuthology.orchestra.run.smithi019.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-31T12:15:00.385 INFO:teuthology.orchestra.run.smithi019.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-31T12:15:00.385 INFO:teuthology.orchestra.run.smithi019.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-31T12:15:00.385 INFO:teuthology.orchestra.run.smithi019.stdout: = sunit=0 swidth=0 blks 2024-08-31T12:15:00.385 INFO:teuthology.orchestra.run.smithi019.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-31T12:15:00.386 INFO:teuthology.orchestra.run.smithi019.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-31T12:15:00.386 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-31T12:15:00.386 INFO:teuthology.orchestra.run.smithi019.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-31T12:15:00.877 INFO:teuthology.orchestra.run.smithi019.stdout:Discarding blocks...Done. 2024-08-31T12:15:00.879 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi019.front.sepia.ceph.com -o noatime 2024-08-31T12:15:00.879 DEBUG:teuthology.orchestra.run.smithi019:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-08-31T12:15:00.984 DEBUG:teuthology.orchestra.run.smithi019:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-08-31T12:15:01.012 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-08-31T12:15:01.077 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-08-31T12:15:01.077 INFO:tasks.ceph:role: osd.3 2024-08-31T12:15:01.077 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi019.front.sepia.ceph.com 2024-08-31T12:15:01.077 DEBUG:teuthology.orchestra.run.smithi019:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-08-31T12:15:01.149 INFO:teuthology.orchestra.run.smithi019.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-08-31T12:15:01.149 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-31T12:15:01.149 INFO:teuthology.orchestra.run.smithi019.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-31T12:15:01.149 INFO:teuthology.orchestra.run.smithi019.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-31T12:15:01.150 INFO:teuthology.orchestra.run.smithi019.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-31T12:15:01.150 INFO:teuthology.orchestra.run.smithi019.stdout: = sunit=0 swidth=0 blks 2024-08-31T12:15:01.150 INFO:teuthology.orchestra.run.smithi019.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-31T12:15:01.150 INFO:teuthology.orchestra.run.smithi019.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-31T12:15:01.150 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-31T12:15:01.150 INFO:teuthology.orchestra.run.smithi019.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-31T12:15:01.687 INFO:teuthology.orchestra.run.smithi019.stdout:Discarding blocks...Done. 2024-08-31T12:15:01.689 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi019.front.sepia.ceph.com -o noatime 2024-08-31T12:15:01.689 DEBUG:teuthology.orchestra.run.smithi019:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-08-31T12:15:01.812 DEBUG:teuthology.orchestra.run.smithi019:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-08-31T12:15:01.843 DEBUG:teuthology.orchestra.run.smithi019:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-31T12:15:01.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:01.954+0000 7fa13d3fd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-08-31T12:15:01.959 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:01.957+0000 7fa13d3fd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-08-31T12:15:01.959 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:01.958+0000 7fa13d3fd640 -1 bdev(0x560ad8503900 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-08-31T12:15:01.959 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:01.958+0000 7fa13d3fd640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-08-31T12:15:05.127 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-08-31T12:15:05.156 DEBUG:teuthology.orchestra.run.smithi019:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-31T12:15:05.258 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:05.256+0000 7f09c7c18640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-08-31T12:15:05.258 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:05.257+0000 7f09c7c18640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-08-31T12:15:05.259 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:05.257+0000 7f09c7c18640 -1 bdev(0x55f2e79d5900 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-08-31T12:15:05.259 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:05.257+0000 7f09c7c18640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-08-31T12:15:08.429 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-08-31T12:15:08.458 DEBUG:teuthology.orchestra.run.smithi019:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-31T12:15:08.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:08.559+0000 7f7f8af1d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-08-31T12:15:08.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:08.559+0000 7f7f8af1d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-08-31T12:15:08.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:08.559+0000 7f7f8af1d640 -1 bdev(0x5604fd3a9900 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-08-31T12:15:08.562 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:08.560+0000 7f7f8af1d640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-08-31T12:15:11.495 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-08-31T12:15:11.525 DEBUG:teuthology.orchestra.run.smithi019:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-31T12:15:11.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:11.629+0000 7f753fcef640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-08-31T12:15:11.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:11.629+0000 7f753fcef640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-08-31T12:15:11.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:11.629+0000 7f753fcef640 -1 bdev(0x55a2111ab900 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-08-31T12:15:11.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:11.629+0000 7f753fcef640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-08-31T12:15:14.794 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-08-31T12:15:14.823 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-08-31T12:15:14.853 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-08-31T12:15:14.853 INFO:tasks.ceph:role: osd.4 2024-08-31T12:15:14.853 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi043.front.sepia.ceph.com 2024-08-31T12:15:14.854 DEBUG:teuthology.orchestra.run.smithi043:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-08-31T12:15:14.921 INFO:teuthology.orchestra.run.smithi043.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-08-31T12:15:14.921 INFO:teuthology.orchestra.run.smithi043.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-31T12:15:14.921 INFO:teuthology.orchestra.run.smithi043.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-31T12:15:14.921 INFO:teuthology.orchestra.run.smithi043.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-31T12:15:14.921 INFO:teuthology.orchestra.run.smithi043.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-31T12:15:14.922 INFO:teuthology.orchestra.run.smithi043.stdout: = sunit=0 swidth=0 blks 2024-08-31T12:15:14.922 INFO:teuthology.orchestra.run.smithi043.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-31T12:15:14.922 INFO:teuthology.orchestra.run.smithi043.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-31T12:15:14.922 INFO:teuthology.orchestra.run.smithi043.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-31T12:15:14.922 INFO:teuthology.orchestra.run.smithi043.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-31T12:15:14.953 INFO:teuthology.orchestra.run.smithi043.stdout:Discarding blocks...Done. 2024-08-31T12:15:14.955 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi043.front.sepia.ceph.com -o noatime 2024-08-31T12:15:14.955 DEBUG:teuthology.orchestra.run.smithi043:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-08-31T12:15:15.026 DEBUG:teuthology.orchestra.run.smithi043:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-08-31T12:15:15.096 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-08-31T12:15:15.163 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-08-31T12:15:15.163 INFO:tasks.ceph:role: osd.5 2024-08-31T12:15:15.163 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi043.front.sepia.ceph.com 2024-08-31T12:15:15.163 DEBUG:teuthology.orchestra.run.smithi043:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-08-31T12:15:15.227 INFO:teuthology.orchestra.run.smithi043.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-08-31T12:15:15.227 INFO:teuthology.orchestra.run.smithi043.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-31T12:15:15.227 INFO:teuthology.orchestra.run.smithi043.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-31T12:15:15.227 INFO:teuthology.orchestra.run.smithi043.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-31T12:15:15.227 INFO:teuthology.orchestra.run.smithi043.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-31T12:15:15.228 INFO:teuthology.orchestra.run.smithi043.stdout: = sunit=0 swidth=0 blks 2024-08-31T12:15:15.228 INFO:teuthology.orchestra.run.smithi043.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-31T12:15:15.228 INFO:teuthology.orchestra.run.smithi043.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-31T12:15:15.228 INFO:teuthology.orchestra.run.smithi043.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-31T12:15:15.228 INFO:teuthology.orchestra.run.smithi043.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-31T12:15:15.260 INFO:teuthology.orchestra.run.smithi043.stdout:Discarding blocks...Done. 2024-08-31T12:15:15.262 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi043.front.sepia.ceph.com -o noatime 2024-08-31T12:15:15.262 DEBUG:teuthology.orchestra.run.smithi043:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-08-31T12:15:15.331 DEBUG:teuthology.orchestra.run.smithi043:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-08-31T12:15:15.402 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-08-31T12:15:15.467 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-08-31T12:15:15.467 INFO:tasks.ceph:role: osd.6 2024-08-31T12:15:15.467 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi043.front.sepia.ceph.com 2024-08-31T12:15:15.467 DEBUG:teuthology.orchestra.run.smithi043:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-08-31T12:15:15.531 INFO:teuthology.orchestra.run.smithi043.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-08-31T12:15:15.531 INFO:teuthology.orchestra.run.smithi043.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-31T12:15:15.531 INFO:teuthology.orchestra.run.smithi043.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-31T12:15:15.532 INFO:teuthology.orchestra.run.smithi043.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-31T12:15:15.532 INFO:teuthology.orchestra.run.smithi043.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-31T12:15:15.532 INFO:teuthology.orchestra.run.smithi043.stdout: = sunit=0 swidth=0 blks 2024-08-31T12:15:15.532 INFO:teuthology.orchestra.run.smithi043.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-31T12:15:15.532 INFO:teuthology.orchestra.run.smithi043.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-31T12:15:15.532 INFO:teuthology.orchestra.run.smithi043.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-31T12:15:15.532 INFO:teuthology.orchestra.run.smithi043.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-31T12:15:15.565 INFO:teuthology.orchestra.run.smithi043.stdout:Discarding blocks...Done. 2024-08-31T12:15:15.568 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi043.front.sepia.ceph.com -o noatime 2024-08-31T12:15:15.568 DEBUG:teuthology.orchestra.run.smithi043:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-08-31T12:15:15.647 DEBUG:teuthology.orchestra.run.smithi043:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-08-31T12:15:15.720 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-08-31T12:15:15.784 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-08-31T12:15:15.785 INFO:tasks.ceph:role: osd.7 2024-08-31T12:15:15.785 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi043.front.sepia.ceph.com 2024-08-31T12:15:15.785 DEBUG:teuthology.orchestra.run.smithi043:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-08-31T12:15:15.850 INFO:teuthology.orchestra.run.smithi043.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-08-31T12:15:15.850 INFO:teuthology.orchestra.run.smithi043.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-31T12:15:15.850 INFO:teuthology.orchestra.run.smithi043.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-31T12:15:15.850 INFO:teuthology.orchestra.run.smithi043.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-31T12:15:15.850 INFO:teuthology.orchestra.run.smithi043.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-31T12:15:15.850 INFO:teuthology.orchestra.run.smithi043.stdout: = sunit=0 swidth=0 blks 2024-08-31T12:15:15.851 INFO:teuthology.orchestra.run.smithi043.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-31T12:15:15.851 INFO:teuthology.orchestra.run.smithi043.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-31T12:15:15.851 INFO:teuthology.orchestra.run.smithi043.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-31T12:15:15.851 INFO:teuthology.orchestra.run.smithi043.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-31T12:15:15.882 INFO:teuthology.orchestra.run.smithi043.stdout:Discarding blocks...Done. 2024-08-31T12:15:15.884 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi043.front.sepia.ceph.com -o noatime 2024-08-31T12:15:15.884 DEBUG:teuthology.orchestra.run.smithi043:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-08-31T12:15:15.957 DEBUG:teuthology.orchestra.run.smithi043:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-08-31T12:15:16.027 DEBUG:teuthology.orchestra.run.smithi043:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-31T12:15:16.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:16.127+0000 7fb829358640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-08-31T12:15:16.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:16.127+0000 7fb829358640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-08-31T12:15:16.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:16.128+0000 7fb829358640 -1 bdev(0x561727df1900 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-08-31T12:15:16.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:16.128+0000 7fb829358640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-08-31T12:15:19.293 DEBUG:teuthology.orchestra.run.smithi043:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-08-31T12:15:19.323 DEBUG:teuthology.orchestra.run.smithi043:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-31T12:15:19.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:19.425+0000 7fb08dbbf640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-08-31T12:15:19.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:19.426+0000 7fb08dbbf640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-08-31T12:15:19.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:19.426+0000 7fb08dbbf640 -1 bdev(0x55c4de8c1900 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-08-31T12:15:19.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:19.426+0000 7fb08dbbf640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-08-31T12:15:22.347 DEBUG:teuthology.orchestra.run.smithi043:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-08-31T12:15:22.377 DEBUG:teuthology.orchestra.run.smithi043:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-31T12:15:22.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:22.481+0000 7f0489a15640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-08-31T12:15:22.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:22.481+0000 7f0489a15640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-08-31T12:15:22.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:22.481+0000 7f0489a15640 -1 bdev(0x55fd3792b900 /var/lib/ceph/osd/ceph-6/block) open stat got: (1) Operation not permitted 2024-08-31T12:15:22.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:22.481+0000 7f0489a15640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-08-31T12:15:25.520 DEBUG:teuthology.orchestra.run.smithi019:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:15:25.523 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:15:25.661 DEBUG:teuthology.orchestra.run.smithi043:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-08-31T12:15:25.691 DEBUG:teuthology.orchestra.run.smithi043:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-31T12:15:25.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:25.792+0000 7f07f6833640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-08-31T12:15:25.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:25.792+0000 7f07f6833640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-08-31T12:15:25.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:25.792+0000 7f07f6833640 -1 bdev(0x5603fd229900 /var/lib/ceph/osd/ceph-7/block) open stat got: (1) Operation not permitted 2024-08-31T12:15:25.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:25.792+0000 7f07f6833640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-08-31T12:15:28.952 DEBUG:teuthology.orchestra.run.smithi043:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-08-31T12:15:28.981 INFO:tasks.ceph:Reading keys from all nodes... 2024-08-31T12:15:28.981 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-08-31T12:15:28.981 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-08-31T12:15:29.012 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-08-31T12:15:29.012 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-08-31T12:15:29.082 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-08-31T12:15:29.082 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-08-31T12:15:29.150 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-08-31T12:15:29.150 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/mds/ceph-e/keyring of=/dev/stdout 2024-08-31T12:15:29.220 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-08-31T12:15:29.220 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-08-31T12:15:29.287 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-08-31T12:15:29.288 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-08-31T12:15:29.354 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-08-31T12:15:29.354 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-08-31T12:15:29.422 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-08-31T12:15:29.422 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-08-31T12:15:29.489 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-08-31T12:15:29.489 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-08-31T12:15:29.519 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-08-31T12:15:29.519 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-08-31T12:15:29.588 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-08-31T12:15:29.588 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-08-31T12:15:29.656 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-08-31T12:15:29.656 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/mds/ceph-f/keyring of=/dev/stdout 2024-08-31T12:15:29.723 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-08-31T12:15:29.723 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-08-31T12:15:29.789 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-08-31T12:15:29.789 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-08-31T12:15:29.855 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-08-31T12:15:29.855 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-08-31T12:15:29.921 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-08-31T12:15:29.921 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-08-31T12:15:29.986 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-08-31T12:15:29.986 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-08-31T12:15:30.005 INFO:tasks.ceph:Adding keys to all mons... 2024-08-31T12:15:30.005 DEBUG:teuthology.orchestra.run.smithi019:> sudo tee -a /etc/ceph/ceph.keyring 2024-08-31T12:15:30.048 DEBUG:teuthology.orchestra.run.smithi043:> sudo tee -a /etc/ceph/ceph.keyring 2024-08-31T12:15:30.072 INFO:teuthology.orchestra.run.smithi043.stdout:[mgr.y] 2024-08-31T12:15:30.072 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQBBCdNmE8nJOBAAt24uZYB5EAGCuojZQ+pTbQ== 2024-08-31T12:15:30.072 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.a] 2024-08-31T12:15:30.072 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQBCCdNmans9BRAA/4TGA3fRHgrKTbh3rd7CZg== 2024-08-31T12:15:30.072 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.c] 2024-08-31T12:15:30.072 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQBCCdNmDqDIDBAAyXwoCRiUdcAxVyQb0BVp3w== 2024-08-31T12:15:30.072 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.e] 2024-08-31T12:15:30.072 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQBCCdNmX05gFBAAGQhYCsRmSxrmOnGe5P8uWg== 2024-08-31T12:15:30.072 INFO:teuthology.orchestra.run.smithi043.stdout:[osd.0] 2024-08-31T12:15:30.073 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQBFCdNmo3L8OBAANt36LzLy+GoVxav+j/Hxeg== 2024-08-31T12:15:30.073 INFO:teuthology.orchestra.run.smithi043.stdout:[osd.1] 2024-08-31T12:15:30.073 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQBJCdNm0S5mDxAAzliABesqGWFztG/fnbbSLw== 2024-08-31T12:15:30.073 INFO:teuthology.orchestra.run.smithi043.stdout:[osd.2] 2024-08-31T12:15:30.073 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQBMCdNmz1ZzIRAA2FDUBKOxiGiNJ8x+mucudQ== 2024-08-31T12:15:30.073 INFO:teuthology.orchestra.run.smithi043.stdout:[osd.3] 2024-08-31T12:15:30.073 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQBPCdNmWiGXJRAAKNjdeHTNY2r0tyG5HEcnfw== 2024-08-31T12:15:30.073 INFO:teuthology.orchestra.run.smithi043.stdout:[mgr.x] 2024-08-31T12:15:30.073 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQBCCdNmRMJ0ARAAX5hBLl4CUUHgbiNKlGcxHg== 2024-08-31T12:15:30.073 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.b] 2024-08-31T12:15:30.073 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQBCCdNmFQ+tGRAAqyWbZZ8NPEpOMZiY/NKdVw== 2024-08-31T12:15:30.073 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.d] 2024-08-31T12:15:30.073 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQBCCdNmvQ1HIRAAAYfMn4vKLbljvv8EkvCDzw== 2024-08-31T12:15:30.073 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.f] 2024-08-31T12:15:30.074 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQBCCdNmStzWKBAAqsS7hh4m7rquAnkq2+dMTg== 2024-08-31T12:15:30.074 INFO:teuthology.orchestra.run.smithi043.stdout:[osd.4] 2024-08-31T12:15:30.074 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQBUCdNmFVmiBxAARXxGyGLWcvP8/Xsq1M+WzQ== 2024-08-31T12:15:30.074 INFO:teuthology.orchestra.run.smithi043.stdout:[osd.5] 2024-08-31T12:15:30.074 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQBXCdNmZAFoGRAAjTEHSPUVr5R4ZqbGmL5qsQ== 2024-08-31T12:15:30.074 INFO:teuthology.orchestra.run.smithi043.stdout:[osd.6] 2024-08-31T12:15:30.074 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQBaCdNmfp+3HBAA1aexdO8mqUhFRdY0i3pShw== 2024-08-31T12:15:30.074 INFO:teuthology.orchestra.run.smithi043.stdout:[osd.7] 2024-08-31T12:15:30.074 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQBdCdNmSXtQLxAA1BC1dUPWahkMWG+Aq84+9Q== 2024-08-31T12:15:30.074 INFO:teuthology.orchestra.run.smithi043.stdout:[client.0] 2024-08-31T12:15:30.074 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQBCCdNmCcqMLRAA5+DFH0Rd9nRfnZD7wt/ySg== 2024-08-31T12:15:30.075 INFO:teuthology.orchestra.run.smithi019.stdout:[mgr.y] 2024-08-31T12:15:30.076 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQBBCdNmE8nJOBAAt24uZYB5EAGCuojZQ+pTbQ== 2024-08-31T12:15:30.076 INFO:teuthology.orchestra.run.smithi019.stdout:[mds.a] 2024-08-31T12:15:30.076 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQBCCdNmans9BRAA/4TGA3fRHgrKTbh3rd7CZg== 2024-08-31T12:15:30.076 INFO:teuthology.orchestra.run.smithi019.stdout:[mds.c] 2024-08-31T12:15:30.076 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQBCCdNmDqDIDBAAyXwoCRiUdcAxVyQb0BVp3w== 2024-08-31T12:15:30.076 INFO:teuthology.orchestra.run.smithi019.stdout:[mds.e] 2024-08-31T12:15:30.076 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQBCCdNmX05gFBAAGQhYCsRmSxrmOnGe5P8uWg== 2024-08-31T12:15:30.076 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.0] 2024-08-31T12:15:30.076 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQBFCdNmo3L8OBAANt36LzLy+GoVxav+j/Hxeg== 2024-08-31T12:15:30.076 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.1] 2024-08-31T12:15:30.076 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQBJCdNm0S5mDxAAzliABesqGWFztG/fnbbSLw== 2024-08-31T12:15:30.077 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.2] 2024-08-31T12:15:30.077 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQBMCdNmz1ZzIRAA2FDUBKOxiGiNJ8x+mucudQ== 2024-08-31T12:15:30.077 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.3] 2024-08-31T12:15:30.077 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQBPCdNmWiGXJRAAKNjdeHTNY2r0tyG5HEcnfw== 2024-08-31T12:15:30.077 INFO:teuthology.orchestra.run.smithi019.stdout:[mgr.x] 2024-08-31T12:15:30.077 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQBCCdNmRMJ0ARAAX5hBLl4CUUHgbiNKlGcxHg== 2024-08-31T12:15:30.077 INFO:teuthology.orchestra.run.smithi019.stdout:[mds.b] 2024-08-31T12:15:30.077 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQBCCdNmFQ+tGRAAqyWbZZ8NPEpOMZiY/NKdVw== 2024-08-31T12:15:30.077 INFO:teuthology.orchestra.run.smithi019.stdout:[mds.d] 2024-08-31T12:15:30.077 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQBCCdNmvQ1HIRAAAYfMn4vKLbljvv8EkvCDzw== 2024-08-31T12:15:30.077 INFO:teuthology.orchestra.run.smithi019.stdout:[mds.f] 2024-08-31T12:15:30.077 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQBCCdNmStzWKBAAqsS7hh4m7rquAnkq2+dMTg== 2024-08-31T12:15:30.077 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.4] 2024-08-31T12:15:30.077 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQBUCdNmFVmiBxAARXxGyGLWcvP8/Xsq1M+WzQ== 2024-08-31T12:15:30.078 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.5] 2024-08-31T12:15:30.078 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQBXCdNmZAFoGRAAjTEHSPUVr5R4ZqbGmL5qsQ== 2024-08-31T12:15:30.078 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.6] 2024-08-31T12:15:30.078 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQBaCdNmfp+3HBAA1aexdO8mqUhFRdY0i3pShw== 2024-08-31T12:15:30.078 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.7] 2024-08-31T12:15:30.078 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQBdCdNmSXtQLxAA1BC1dUPWahkMWG+Aq84+9Q== 2024-08-31T12:15:30.078 INFO:teuthology.orchestra.run.smithi019.stdout:[client.0] 2024-08-31T12:15:30.078 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQBCCdNmCcqMLRAA5+DFH0Rd9nRfnZD7wt/ySg== 2024-08-31T12:15:30.078 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-08-31T12:15:30.118 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-08-31T12:15:30.183 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-08-31T12:15:30.186 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-08-31T12:15:30.250 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-08-31T12:15:30.253 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-08-31T12:15:30.316 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-08-31T12:15:30.318 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-08-31T12:15:30.382 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-31T12:15:30.384 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-31T12:15:30.445 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-31T12:15:30.447 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-31T12:15:30.507 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-31T12:15:30.509 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-31T12:15:30.569 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-31T12:15:30.571 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-31T12:15:30.633 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-08-31T12:15:30.636 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-08-31T12:15:30.696 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-08-31T12:15:30.698 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-08-31T12:15:30.762 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-08-31T12:15:30.765 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-08-31T12:15:30.828 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-08-31T12:15:30.830 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-08-31T12:15:30.894 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-31T12:15:30.897 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-31T12:15:30.957 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-31T12:15:30.960 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-31T12:15:31.021 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-31T12:15:31.023 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-31T12:15:31.084 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-31T12:15:31.086 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-31T12:15:31.147 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-31T12:15:31.149 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-31T12:15:31.212 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-08-31T12:15:31.213 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-08-31T12:15:31.237 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-08-31T12:15:31.342 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-08-31T12:15:31.367 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-08-31T12:15:31.392 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-08-31T12:15:31.498 DEBUG:teuthology.orchestra.run.smithi043:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-08-31T12:15:31.522 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-08-31T12:15:31.586 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-08-31T12:15:31.692 DEBUG:teuthology.orchestra.run.smithi043:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-08-31T12:15:31.716 DEBUG:teuthology.orchestra.run.smithi019:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-08-31T12:15:31.718 DEBUG:teuthology.orchestra.run.smithi043:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-08-31T12:15:31.771 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-08-31T12:15:31.771 INFO:tasks.ceph.mon.a:Restarting daemon 2024-08-31T12:15:31.772 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-08-31T12:15:31.774 INFO:tasks.ceph.mon.a:Started 2024-08-31T12:15:31.774 INFO:tasks.ceph.mon.b:Restarting daemon 2024-08-31T12:15:31.774 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-08-31T12:15:31.814 INFO:tasks.ceph.mon.b:Started 2024-08-31T12:15:31.814 INFO:tasks.ceph.mon.c:Restarting daemon 2024-08-31T12:15:31.814 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-08-31T12:15:31.816 INFO:tasks.ceph.mon.c:Started 2024-08-31T12:15:31.817 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-08-31T12:15:31.817 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-08-31T12:15:31.817 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-08-31T12:15:31.819 INFO:tasks.ceph.mgr.y:Started 2024-08-31T12:15:31.819 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-08-31T12:15:31.819 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-08-31T12:15:31.822 INFO:tasks.ceph.mgr.x:Started 2024-08-31T12:15:31.823 INFO:tasks.ceph:setting config [client] client_mount_timeout = 600 2024-08-31T12:15:31.823 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph config set client client_mount_timeout 600 2024-08-31T12:15:31.826 INFO:tasks.ceph:setting config [client] debug_client = 20 2024-08-31T12:15:31.827 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph config set client debug_client 20 2024-08-31T12:15:31.870 INFO:tasks.ceph:setting config [client] debug_ms = 1 2024-08-31T12:15:31.871 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph config set client debug_ms 1 2024-08-31T12:15:31.913 INFO:tasks.ceph:setting config [client] rados_mon_op_timeout = 900 2024-08-31T12:15:31.914 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph config set client rados_mon_op_timeout 900 2024-08-31T12:15:31.957 INFO:tasks.ceph:setting config [client] rados_osd_op_timeout = 900 2024-08-31T12:15:31.958 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph config set client rados_osd_op_timeout 900 2024-08-31T12:15:32.001 INFO:tasks.ceph:setting config [mds] debug_mds = 20 2024-08-31T12:15:32.001 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph config set mds debug_mds 20 2024-08-31T12:15:32.043 INFO:tasks.ceph:setting config [mds] debug_mds_balancer = 20 2024-08-31T12:15:32.044 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph config set mds debug_mds_balancer 20 2024-08-31T12:15:32.086 INFO:tasks.ceph:setting config [mds] debug_ms = 2 2024-08-31T12:15:32.087 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph config set mds debug_ms 2 2024-08-31T12:15:32.129 INFO:tasks.ceph:setting config [mds] mds_bal_fragment_size_max = 10000 2024-08-31T12:15:32.130 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph config set mds mds_bal_fragment_size_max 10000 2024-08-31T12:15:32.175 INFO:tasks.ceph:setting config [mds] mds_bal_merge_size = 5 2024-08-31T12:15:32.176 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph config set mds mds_bal_merge_size 5 2024-08-31T12:15:32.221 INFO:tasks.ceph:setting config [mds] mds_bal_split_bits = 3 2024-08-31T12:15:32.222 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph config set mds mds_bal_split_bits 3 2024-08-31T12:15:32.268 INFO:tasks.ceph:setting config [mds] mds_bal_split_size = 100 2024-08-31T12:15:32.269 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph config set mds mds_bal_split_size 100 2024-08-31T12:15:32.313 INFO:tasks.ceph:setting config [mds] mds_debug_frag = True 2024-08-31T12:15:32.313 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph config set mds mds_debug_frag True 2024-08-31T12:15:32.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.331+0000 7ff469ffb640 1 -- 172.21.15.19:0/1885395911 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff484154ed0 msgr2=0x7ff4841593c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.331+0000 7ff469ffb640 1 --2- 172.21.15.19:0/1885395911 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff484154ed0 0x7ff4841593c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff47800c5e0 tx=0x7ff47800caa0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.335 INFO:tasks.ceph:setting config [mds] mds_debug_scatterstat = True 2024-08-31T12:15:32.335 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph config set mds mds_debug_scatterstat True 2024-08-31T12:15:32.336 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.332+0000 7ff469ffb640 1 -- 172.21.15.19:0/1885395911 shutdown_connections 2024-08-31T12:15:32.336 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.332+0000 7ff469ffb640 1 --2- 172.21.15.19:0/1885395911 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff4841f4ac0 0x7ff48413fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.336 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.332+0000 7ff469ffb640 1 --2- 172.21.15.19:0/1885395911 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff484154ed0 0x7ff4841593c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.336 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.332+0000 7ff469ffb640 1 --2- 172.21.15.19:0/1885395911 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff46c004d40 0x7ff4841e4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.332+0000 7ff469ffb640 1 -- 172.21.15.19:0/1885395911 >> 172.21.15.19:0/1885395911 conn(0x7ff484071a00 msgr2=0x7ff484157cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:32.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.332+0000 7ff469ffb640 1 -- 172.21.15.19:0/1885395911 shutdown_connections 2024-08-31T12:15:32.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.332+0000 7ff469ffb640 1 -- 172.21.15.19:0/1885395911 wait complete. 2024-08-31T12:15:32.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.337+0000 7fe4c9ffb640 1 -- 172.21.15.19:0/1503302906 <== mon.1 v2:172.21.15.43:3300/0 10 ==== config(4 keys) ==== 110+0+0 (secure 0 0 0) 0x7fe4d00233a0 con 0x7fe4d414c870 2024-08-31T12:15:32.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.337+0000 7fe4c9ffb640 1 -- 172.21.15.19:0/1503302906 <== mon.1 v2:172.21.15.43:3300/0 11 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v5)=0 v5) ==== 123+0+0 (secure 0 0 0) 0x7fe4d0023590 con 0x7fe4d414c870 2024-08-31T12:15:32.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.339+0000 7ff81cff9640 1 -- 172.21.15.19:0/3412186018 <== mon.0 v2:172.21.15.19:3300/0 10 ==== config(4 keys) ==== 110+0+0 (secure 0 0 0) 0x7ff81401d3b0 con 0x7ff820073430 2024-08-31T12:15:32.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.337+0000 7f39faffd640 1 -- 172.21.15.19:0/3524382841 <== mon.1 v2:172.21.15.43:3300/0 8 ==== config(4 keys) ==== 110+0+0 (secure 0 0 0) 0x7f3a00031700 con 0x7f3a0c0c7600 2024-08-31T12:15:32.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.338+0000 7f1c80ff9640 1 -- 172.21.15.19:0/1181236189 <== mon.2 v2:172.21.15.43:3301/0 10 ==== config(4 keys) ==== 110+0+0 (secure 0 0 0) 0x7f1c7802a400 con 0x7f1c840cb2b0 2024-08-31T12:15:32.342 INFO:tasks.ceph:setting config [mds] mds_op_complaint_time = 180 2024-08-31T12:15:32.342 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph config set mds mds_op_complaint_time 180 2024-08-31T12:15:32.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.339+0000 7fe4db994640 1 -- 172.21.15.19:0/1503302906 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4d414c870 msgr2=0x7fe4d4154310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.339+0000 7fe4db994640 1 --2- 172.21.15.19:0/1503302906 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4d414c870 0x7fe4d4154310 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe4d0006c70 tx=0x7fe4d0003aa0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.339+0000 7fe4db994640 1 -- 172.21.15.19:0/1503302906 shutdown_connections 2024-08-31T12:15:32.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.339+0000 7fe4db994640 1 --2- 172.21.15.19:0/1503302906 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe4d41f0630 0x7fe4d41508b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.339+0000 7fe4db994640 1 --2- 172.21.15.19:0/1503302906 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe4d4154850 0x7fe4d41f00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.339+0000 7fe4db994640 1 --2- 172.21.15.19:0/1503302906 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4d414c870 0x7fe4d4154310 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.339+0000 7fe4db994640 1 -- 172.21.15.19:0/1503302906 >> 172.21.15.19:0/1503302906 conn(0x7fe4d40cdca0 msgr2=0x7fe4d4149df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:32.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.340+0000 7fe4db994640 1 -- 172.21.15.19:0/1503302906 shutdown_connections 2024-08-31T12:15:32.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.340+0000 7fe4db994640 1 -- 172.21.15.19:0/1503302906 wait complete. 2024-08-31T12:15:32.349 INFO:tasks.ceph:setting config [mds] mds_verify_scatter = True 2024-08-31T12:15:32.349 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph config set mds mds_verify_scatter True 2024-08-31T12:15:32.350 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.346+0000 7f1c80ff9640 1 -- 172.21.15.19:0/1181236189 <== mon.2 v2:172.21.15.43:3301/0 11 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1c7802a790 con 0x7f1c840cb2b0 2024-08-31T12:15:32.350 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.346+0000 7f1c80ff9640 1 -- 172.21.15.19:0/1181236189 <== mon.2 v2:172.21.15.43:3301/0 12 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v6)=0 v6) ==== 123+0+0 (secure 0 0 0) 0x7f1c7802ab00 con 0x7f1c840cb2b0 2024-08-31T12:15:32.351 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.347+0000 7ff81cff9640 1 -- 172.21.15.19:0/3412186018 <== mon.0 v2:172.21.15.19:3300/0 11 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff81401d840 con 0x7ff820073430 2024-08-31T12:15:32.351 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.347+0000 7f39faffd640 1 -- 172.21.15.19:0/3524382841 <== mon.1 v2:172.21.15.43:3300/0 9 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3a00005400 con 0x7f3a0c0c7600 2024-08-31T12:15:32.352 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.349+0000 7f1c6a7fc640 1 -- 172.21.15.19:0/1181236189 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1c840cb2b0 msgr2=0x7f1c84140c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.352 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.349+0000 7f1c6a7fc640 1 --2- 172.21.15.19:0/1181236189 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1c840cb2b0 0x7f1c84140c40 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1c7800e6d0 tx=0x7f1c7800eb90 comp rx=0 tx=0).stop 2024-08-31T12:15:32.352 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.350+0000 7f1c6a7fc640 1 -- 172.21.15.19:0/1181236189 shutdown_connections 2024-08-31T12:15:32.352 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.350+0000 7f1c6a7fc640 1 --2- 172.21.15.19:0/1181236189 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c8414e740 0x7f1c84144170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.352 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.350+0000 7f1c6a7fc640 1 --2- 172.21.15.19:0/1181236189 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1c840cb2b0 0x7f1c84140c40 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.352 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.350+0000 7f1c6a7fc640 1 --2- 172.21.15.19:0/1181236189 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1c8406e130 0x7f1c840c97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.352 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.350+0000 7f1c6a7fc640 1 -- 172.21.15.19:0/1181236189 >> 172.21.15.19:0/1181236189 conn(0x7f1c840ca620 msgr2=0x7f1c84074430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:32.352 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.351+0000 7f1c6a7fc640 1 -- 172.21.15.19:0/1181236189 shutdown_connections 2024-08-31T12:15:32.353 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.351+0000 7f1c6a7fc640 1 -- 172.21.15.19:0/1181236189 wait complete. 2024-08-31T12:15:32.355 INFO:tasks.ceph:setting config [mds] mds_dir_prefetch = True 2024-08-31T12:15:32.355 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph config set mds mds_dir_prefetch True 2024-08-31T12:15:32.364 INFO:tasks.ceph:setting config [mds] osd_op_complaint_time = 180 2024-08-31T12:15:32.364 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph config set mds osd_op_complaint_time 180 2024-08-31T12:15:32.385 INFO:tasks.ceph:setting config [mds] rados_mon_op_timeout = 900 2024-08-31T12:15:32.385 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph config set mds rados_mon_op_timeout 900 2024-08-31T12:15:32.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.413+0000 7fd85e34e640 1 -- 172.21.15.19:0/858625515 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd8440037a0 msgr2=0x7fd844003bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.413+0000 7fd85e34e640 1 --2- 172.21.15.19:0/858625515 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd8440037a0 0x7fd844003bb0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd85806fe50 tx=0x7fd844005830 comp rx=0 tx=0).stop 2024-08-31T12:15:32.417 INFO:tasks.ceph:setting config [mds] rados_osd_op_timeout = 900 2024-08-31T12:15:32.417 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph config set mds rados_osd_op_timeout 900 2024-08-31T12:15:32.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.417+0000 7fd85db4d640 1 -- 172.21.15.19:0/858625515 reap_dead start 2024-08-31T12:15:32.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.417+0000 7fd85e34e640 1 -- 172.21.15.19:0/858625515 shutdown_connections 2024-08-31T12:15:32.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.417+0000 7fd85e34e640 1 -- 172.21.15.19:0/858625515 >> 172.21.15.19:0/858625515 conn(0x7fd8580c9020 msgr2=0x7fd8580c9430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:32.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.419+0000 7fd85e34e640 1 -- 172.21.15.19:0/858625515 shutdown_connections 2024-08-31T12:15:32.422 INFO:tasks.ceph:setting config [mon] mon_op_complaint_time = 120 2024-08-31T12:15:32.422 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph config set mon mon_op_complaint_time 120 2024-08-31T12:15:32.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.423+0000 7fd85e34e640 1 -- 172.21.15.19:0/858625515 wait complete. 2024-08-31T12:15:32.427 INFO:tasks.ceph:setting config [mon] mon_warn_on_pool_no_app = False 2024-08-31T12:15:32.427 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph config set mon mon_warn_on_pool_no_app False 2024-08-31T12:15:32.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.446+0000 7f919feda640 1 -- 172.21.15.19:0/617602656 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f91980ca980 msgr2=0x7f919815afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.448 INFO:tasks.ceph:setting config [osd] osd_op_complaint_time = 180 2024-08-31T12:15:32.448 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph config set osd osd_op_complaint_time 180 2024-08-31T12:15:32.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.445+0000 7fd85e34e640 1 Processor -- start 2024-08-31T12:15:32.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.445+0000 7fd85e34e640 1 -- start start 2024-08-31T12:15:32.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.447+0000 7fd85e34e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd858150e30 0x7fd8580c53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.448+0000 7fd85e34e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd858151240 0x7fd8580c5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.448+0000 7fd85e34e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd8581fe180 0x7fd8581fe590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.448+0000 7fd85e34e640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fd85815e6f0 con 0x7fd858151240 2024-08-31T12:15:32.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.448+0000 7fd85e34e640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd8580c6f10 con 0x7fd8581fe180 2024-08-31T12:15:32.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.448+0000 7fd85e34e640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fd8580c8890 con 0x7fd858150e30 2024-08-31T12:15:32.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.449+0000 7fd85d34c640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd858150e30 0x7fd8580c53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:32.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.449+0000 7fd85db4d640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd8581fe180 0x7fd8581fe590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:32.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.449+0000 7f919cc74640 1 -- 172.21.15.19:0/617602656 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f919002c820 con 0x7f91980ca980 2024-08-31T12:15:32.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.449+0000 7fd85d34c640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd858150e30 0x7fd8580c53e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.19:43098/0 (socket says 172.21.15.19:43098) 2024-08-31T12:15:32.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.449+0000 7fd85d34c640 1 -- 172.21.15.19:0/1229158451 learned_addr learned my addr 172.21.15.19:0/1229158451 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:15:32.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.449+0000 7f919feda640 1 --2- 172.21.15.19:0/617602656 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f91980ca980 0x7f919815afa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9198074100 tx=0x7f919002aac0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.453 DEBUG:tasks.ceph:set 23 configs 2024-08-31T12:15:32.453 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'client_mount_timeout', '600'], check_status=True, hostname='smithi019') 2024-08-31T12:15:32.453 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'debug_client', '20'], check_status=True, hostname='smithi019') 2024-08-31T12:15:32.454 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'debug_ms', '1'], check_status=True, hostname='smithi019') 2024-08-31T12:15:32.454 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi019') 2024-08-31T12:15:32.454 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi019') 2024-08-31T12:15:32.454 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'debug_mds', '20'], check_status=True, hostname='smithi019') 2024-08-31T12:15:32.454 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.450+0000 7fd85d34c640 1 -- 172.21.15.19:0/1229158451 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd8581fe180 msgr2=0x7fd8581fe590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.454 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.450+0000 7fd85d34c640 1 --2- 172.21.15.19:0/1229158451 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd8581fe180 0x7fd8581fe590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.450+0000 7fd85d34c640 1 -- 172.21.15.19:0/1229158451 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd858151240 msgr2=0x7fd8580c5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.450+0000 7fd85d34c640 1 --2- 172.21.15.19:0/1229158451 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd858151240 0x7fd8580c5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.450+0000 7fd85d34c640 1 -- 172.21.15.19:0/1229158451 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd85814c790 con 0x7fd858150e30 2024-08-31T12:15:32.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.449+0000 7f919feda640 1 -- 172.21.15.19:0/617602656 shutdown_connections 2024-08-31T12:15:32.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.449+0000 7f919feda640 1 --2- 172.21.15.19:0/617602656 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f91980ca980 0x7f919815afa0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.449+0000 7f919feda640 1 --2- 172.21.15.19:0/617602656 >> v2:172.21.15.43:3300/0 conn(0x7f9198074130 0x7f91980ca440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.450+0000 7f919feda640 1 -- 172.21.15.19:0/617602656 >> 172.21.15.19:0/617602656 conn(0x7f9198071a00 msgr2=0x7f9198071e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:32.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.453+0000 7f919feda640 1 -- 172.21.15.19:0/617602656 shutdown_connections 2024-08-31T12:15:32.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.453+0000 7f919feda640 1 -- 172.21.15.19:0/617602656 wait complete. 2024-08-31T12:15:32.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.453+0000 7fd85d34c640 1 --2- 172.21.15.19:0/1229158451 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd858150e30 0x7fd8580c53e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd85400e600 tx=0x7fd85400eac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:32.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.455+0000 7fd84dffb640 1 -- 172.21.15.19:0/1229158451 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd854015070 con 0x7fd858150e30 2024-08-31T12:15:32.459 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.455+0000 7fd85e34e640 1 -- 172.21.15.19:0/1229158451 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd85814c4f0 con 0x7fd858150e30 2024-08-31T12:15:32.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.455+0000 7fd85e34e640 1 -- 172.21.15.19:0/1229158451 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd858159f00 con 0x7fd858150e30 2024-08-31T12:15:32.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.456+0000 7fd84dffb640 1 -- 172.21.15.19:0/1229158451 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd854004310 con 0x7fd858150e30 2024-08-31T12:15:32.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.456+0000 7fd84dffb640 1 -- 172.21.15.19:0/1229158451 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd85400ca00 con 0x7fd858150e30 2024-08-31T12:15:32.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.456+0000 7f919feda640 1 Processor -- start 2024-08-31T12:15:32.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.456+0000 7f919feda640 1 -- start start 2024-08-31T12:15:32.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.459+0000 7f919feda640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9198074130 0x7f9198157b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.459+0000 7f919feda640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f919815b4e0 0x7f91981f5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.459+0000 7f919feda640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9198158430 0x7f9198147fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.459+0000 7f919feda640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f91980c9530 con 0x7f919815b4e0 2024-08-31T12:15:32.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.459+0000 7f919feda640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f91980c9250 con 0x7f9198158430 2024-08-31T12:15:32.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.459+0000 7f919feda640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f91980c5420 con 0x7f9198074130 2024-08-31T12:15:32.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.459+0000 7f919d475640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f919815b4e0 0x7f91981f5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:32.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.459+0000 7fd84dffb640 1 -- 172.21.15.19:0/1229158451 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd85400cba0 con 0x7fd858150e30 2024-08-31T12:15:32.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.459+0000 7fd84dffb640 1 -- 172.21.15.19:0/1229158451 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd8540114b0 con 0x7fd858150e30 2024-08-31T12:15:32.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.459+0000 7f919d475640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f919815b4e0 0x7f91981f5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:55158/0 (socket says 172.21.15.19:55158) 2024-08-31T12:15:32.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.459+0000 7f919d475640 1 -- 172.21.15.19:0/3454382734 learned_addr learned my addr 172.21.15.19:0/3454382734 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:15:32.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.459+0000 7f919dc76640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9198074130 0x7f9198157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:32.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.460+0000 7f919d475640 1 -- 172.21.15.19:0/3454382734 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9198074130 msgr2=0x7f9198157b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.460+0000 7f919d475640 1 --2- 172.21.15.19:0/3454382734 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9198074130 0x7f9198157b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.460+0000 7f919d475640 1 -- 172.21.15.19:0/3454382734 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9198158430 msgr2=0x7f9198147fe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:15:32.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.460+0000 7f919d475640 1 --2- 172.21.15.19:0/3454382734 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9198158430 0x7f9198147fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.460+0000 7f919d475640 1 -- 172.21.15.19:0/3454382734 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91981f5aa0 con 0x7f919815b4e0 2024-08-31T12:15:32.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.460+0000 7f919d475640 1 --2- 172.21.15.19:0/3454382734 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f919815b4e0 0x7f91981f5560 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9190009d60 tx=0x7f9190004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:32.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.461+0000 7f918e7fc640 1 -- 172.21.15.19:0/3454382734 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9190044070 con 0x7f919815b4e0 2024-08-31T12:15:32.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.461+0000 7f919feda640 1 -- 172.21.15.19:0/3454382734 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91980c7250 con 0x7f919815b4e0 2024-08-31T12:15:32.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.461+0000 7f919feda640 1 -- 172.21.15.19:0/3454382734 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9198154780 con 0x7f919815b4e0 2024-08-31T12:15:32.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.463+0000 7f918e7fc640 1 -- 172.21.15.19:0/3454382734 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9190003e20 con 0x7f919815b4e0 2024-08-31T12:15:32.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.463+0000 7f918e7fc640 1 -- 172.21.15.19:0/3454382734 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91900046c0 con 0x7f919815b4e0 2024-08-31T12:15:32.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.463+0000 7f918e7fc640 1 -- 172.21.15.19:0/3454382734 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9190047070 con 0x7f919815b4e0 2024-08-31T12:15:32.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.463+0000 7f918e7fc640 1 -- 172.21.15.19:0/3454382734 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f919003f4d0 con 0x7f919815b4e0 2024-08-31T12:15:32.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.463+0000 7f9183fff640 1 -- 172.21.15.19:0/3454382734 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9198155550 con 0x7f919815b4e0 2024-08-31T12:15:32.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.461+0000 7fd8437fe640 1 -- 172.21.15.19:0/1229158451 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd838005290 con 0x7fd858150e30 2024-08-31T12:15:32.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.471+0000 7fd84dffb640 1 -- 172.21.15.19:0/1229158451 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fd8540035f0 con 0x7fd858150e30 2024-08-31T12:15:32.477 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.476+0000 7f918e7fc640 1 -- 172.21.15.19:0/3454382734 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f919004e070 con 0x7f919815b4e0 2024-08-31T12:15:32.539 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.539+0000 7ff8067fc640 1 -- 172.21.15.19:0/3412186018 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command([{prefix=config set, name=debug_mds}] v 0) -- 0x7ff7f80052f0 con 0x7ff820073430 2024-08-31T12:15:32.545 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.545+0000 7ff81cff9640 1 -- 172.21.15.19:0/3412186018 <== mon.0 v2:172.21.15.19:3300/0 12 ==== mon_command_ack([{prefix=config set, name=debug_mds}]=0 v7)=0 v7) ==== 108+0+0 (secure 0 0 0) 0x7ff81401dad0 con 0x7ff820073430 2024-08-31T12:15:32.547 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.547+0000 7ff826484640 1 -- 172.21.15.19:0/3412186018 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff820073430 msgr2=0x7ff82014d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.547 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.547+0000 7ff826484640 1 --2- 172.21.15.19:0/3412186018 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff820073430 0x7ff82014d460 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff81400b680 tx=0x7ff81400bb40 comp rx=0 tx=0).stop 2024-08-31T12:15:32.549 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.547+0000 7ff826484640 1 -- 172.21.15.19:0/3412186018 shutdown_connections 2024-08-31T12:15:32.549 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.547+0000 7ff826484640 1 --2- 172.21.15.19:0/3412186018 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff8200c7a70 0x7ff82013d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.549 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.547+0000 7ff826484640 1 --2- 172.21.15.19:0/3412186018 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff820073980 0x7ff820156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.549 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.547+0000 7ff826484640 1 --2- 172.21.15.19:0/3412186018 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff820073430 0x7ff82014d460 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.549 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.547+0000 7ff826484640 1 -- 172.21.15.19:0/3412186018 >> 172.21.15.19:0/3412186018 conn(0x7ff820070d90 msgr2=0x7ff82013e230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:32.549 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.548+0000 7ff826484640 1 -- 172.21.15.19:0/3412186018 shutdown_connections 2024-08-31T12:15:32.549 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.548+0000 7ff826484640 1 -- 172.21.15.19:0/3412186018 wait complete. 2024-08-31T12:15:32.583 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'debug_mds_balancer', '20'], check_status=True, hostname='smithi019') 2024-08-31T12:15:32.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.630+0000 7fc257fff640 1 -- 172.21.15.19:0/1338430828 reap_dead start 2024-08-31T12:15:32.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.630+0000 7fc25db6e640 1 -- 172.21.15.19:0/1338430828 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc244003a50 msgr2=0x7fc244003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.630+0000 7fc25db6e640 1 --2- 172.21.15.19:0/1338430828 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc244003a50 0x7fc244003ec0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc24800ad00 tx=0x7fc244005830 comp rx=0 tx=0).stop 2024-08-31T12:15:32.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.633+0000 7fc25db6e640 1 -- 172.21.15.19:0/1338430828 shutdown_connections 2024-08-31T12:15:32.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.633+0000 7fc25db6e640 1 --2- 172.21.15.19:0/1338430828 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc244003a50 0x7fc244003ec0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.633+0000 7fc25db6e640 1 -- 172.21.15.19:0/1338430828 >> 172.21.15.19:0/1338430828 conn(0x7fc258070d90 msgr2=0x7fc2580711a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:32.639 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.638+0000 7ff78568a640 1 -- 172.21.15.19:0/855616776 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff7780df610 msgr2=0x7ff7780454a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.639 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.638+0000 7ff78568a640 1 --2- 172.21.15.19:0/855616776 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff7780df610 0x7ff7780454a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff76c009830 tx=0x7ff76c02a5f0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.639 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.639+0000 7fc25db6e640 1 -- 172.21.15.19:0/1338430828 shutdown_connections 2024-08-31T12:15:32.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.641+0000 7fc25db6e640 1 -- 172.21.15.19:0/1338430828 wait complete. 2024-08-31T12:15:32.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.643+0000 7fc25db6e640 1 Processor -- start 2024-08-31T12:15:32.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.646+0000 7ff78568a640 1 -- 172.21.15.19:0/855616776 shutdown_connections 2024-08-31T12:15:32.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.646+0000 7ff78568a640 1 --2- 172.21.15.19:0/855616776 >> v2:172.21.15.43:3300/0 conn(0x7ff778005c40 0x7ff778006050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.646+0000 7ff78568a640 1 --2- 172.21.15.19:0/855616776 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff7780df610 0x7ff7780454a0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.646+0000 7ff78568a640 1 -- 172.21.15.19:0/855616776 >> 172.21.15.19:0/855616776 conn(0x7ff77804fe20 msgr2=0x7ff778050230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:32.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.644+0000 7fc25db6e640 1 -- start start 2024-08-31T12:15:32.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.646+0000 7fc25db6e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc244003a50 0x7fc25814caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.646+0000 7fc25db6e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc25815a500 0x7fc258143660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.646+0000 7fc25db6e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc258143ba0 0x7fc2581f4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.646+0000 7fc25db6e640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fc25815b3a0 con 0x7fc25815a500 2024-08-31T12:15:32.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.646+0000 7fc25db6e640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc25815b0c0 con 0x7fc244003a50 2024-08-31T12:15:32.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.646+0000 7fc25db6e640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fc25815b680 con 0x7fc258143ba0 2024-08-31T12:15:32.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.646+0000 7fc256ffd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc25815a500 0x7fc258143660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:32.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.646+0000 7fc256ffd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc25815a500 0x7fc258143660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:55192/0 (socket says 172.21.15.19:55192) 2024-08-31T12:15:32.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.646+0000 7fc256ffd640 1 -- 172.21.15.19:0/1631849490 learned_addr learned my addr 172.21.15.19:0/1631849490 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:15:32.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.647+0000 7fc257fff640 1 --2- 172.21.15.19:0/1631849490 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc258143ba0 0x7fc2581f4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:32.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.647+0000 7ff78568a640 1 -- 172.21.15.19:0/855616776 shutdown_connections 2024-08-31T12:15:32.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.647+0000 7fc256ffd640 1 -- 172.21.15.19:0/1631849490 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc258143ba0 msgr2=0x7fc2581f4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.647+0000 7fc256ffd640 1 --2- 172.21.15.19:0/1631849490 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc258143ba0 0x7fc2581f4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.647+0000 7fc256ffd640 1 -- 172.21.15.19:0/1631849490 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc244003a50 msgr2=0x7fc25814caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.647+0000 7fc256ffd640 1 --2- 172.21.15.19:0/1631849490 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc244003a50 0x7fc25814caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.647+0000 7fc256ffd640 1 -- 172.21.15.19:0/1631849490 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc258143fb0 con 0x7fc25815a500 2024-08-31T12:15:32.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.647+0000 7fc256ffd640 1 --2- 172.21.15.19:0/1631849490 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc25815a500 0x7fc258143660 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc24000d770 tx=0x7fc24000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:32.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.651+0000 7ff78568a640 1 -- 172.21.15.19:0/855616776 wait complete. 2024-08-31T12:15:32.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.653+0000 7ff78568a640 1 Processor -- start 2024-08-31T12:15:32.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.654+0000 7ff78568a640 1 -- start start 2024-08-31T12:15:32.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.656+0000 7fc24ffff640 1 -- 172.21.15.19:0/1631849490 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc240017070 con 0x7fc25815a500 2024-08-31T12:15:32.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.657+0000 7fc24ffff640 1 -- 172.21.15.19:0/1631849490 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc240009d20 con 0x7fc25815a500 2024-08-31T12:15:32.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.657+0000 7fc24ffff640 1 -- 172.21.15.19:0/1631849490 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2400106c0 con 0x7fc25815a500 2024-08-31T12:15:32.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.656+0000 7ff78568a640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff778005c40 0x7ff778181fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.656+0000 7ff78568a640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff778044d10 0x7ff7780d2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.656+0000 7ff78568a640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff7780df610 0x7ff7780cd620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.656+0000 7ff78568a640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7ff7780465c0 con 0x7ff7780df610 2024-08-31T12:15:32.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.656+0000 7ff78568a640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff778006590 con 0x7ff778005c40 2024-08-31T12:15:32.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.656+0000 7ff78568a640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7ff7780468a0 con 0x7ff778044d10 2024-08-31T12:15:32.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.656+0000 7ff784e89640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff7780df610 0x7ff7780cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:32.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.657+0000 7ff784e89640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff7780df610 0x7ff7780cd620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:55202/0 (socket says 172.21.15.19:55202) 2024-08-31T12:15:32.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.657+0000 7ff784e89640 1 -- 172.21.15.19:0/2515398365 learned_addr learned my addr 172.21.15.19:0/2515398365 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:15:32.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.657+0000 7ff784e89640 1 -- 172.21.15.19:0/2515398365 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff778044d10 msgr2=0x7ff7780d2a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:15:32.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.657+0000 7ff784e89640 1 --2- 172.21.15.19:0/2515398365 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff778044d10 0x7ff7780d2a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.657+0000 7ff784e89640 1 -- 172.21.15.19:0/2515398365 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff778005c40 msgr2=0x7ff778181fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-08-31T12:15:32.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.657+0000 7ff784e89640 1 --2- 172.21.15.19:0/2515398365 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff778005c40 0x7ff778181fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.657+0000 7ff784e89640 1 -- 172.21.15.19:0/2515398365 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7780d9520 con 0x7ff7780df610 2024-08-31T12:15:32.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.659+0000 7ff784e89640 1 --2- 172.21.15.19:0/2515398365 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff7780df610 0x7ff7780cd620 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff78006c3d0 tx=0x7ff7800513f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:32.662 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.658+0000 7fc25db6e640 1 -- 172.21.15.19:0/1631849490 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2580cc220 con 0x7fc25815a500 2024-08-31T12:15:32.662 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.658+0000 7fc25db6e640 1 -- 172.21.15.19:0/1631849490 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2580cac50 con 0x7fc25815a500 2024-08-31T12:15:32.662 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.660+0000 7fc24ffff640 1 -- 172.21.15.19:0/1631849490 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc24001a070 con 0x7fc25815a500 2024-08-31T12:15:32.662 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.660+0000 7fc24ffff640 1 -- 172.21.15.19:0/1631849490 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc24001cc70 con 0x7fc25815a500 2024-08-31T12:15:32.662 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.661+0000 7f3a10f90640 1 -- 172.21.15.19:0/3524382841 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=debug_mds_balancer}] v 0) -- 0x7f39e40052f0 con 0x7f3a0c0c7600 2024-08-31T12:15:32.664 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.663+0000 7fc25db6e640 1 -- 172.21.15.19:0/1631849490 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2580749b0 con 0x7fc25815a500 2024-08-31T12:15:32.664 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.664+0000 7ff77cff9640 1 -- 172.21.15.19:0/2515398365 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff780099070 con 0x7ff7780df610 2024-08-31T12:15:32.665 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.664+0000 7ff77cff9640 1 -- 172.21.15.19:0/2515398365 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff780094720 con 0x7ff7780df610 2024-08-31T12:15:32.665 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.664+0000 7ff77cff9640 1 -- 172.21.15.19:0/2515398365 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff780094b30 con 0x7ff7780df610 2024-08-31T12:15:32.665 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.664+0000 7ff78568a640 1 -- 172.21.15.19:0/2515398365 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7780d97c0 con 0x7ff7780df610 2024-08-31T12:15:32.665 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.664+0000 7ff78568a640 1 -- 172.21.15.19:0/2515398365 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7780d9d80 con 0x7ff7780df610 2024-08-31T12:15:32.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.666+0000 7ff78568a640 1 -- 172.21.15.19:0/2515398365 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff778006050 con 0x7ff7780df610 2024-08-31T12:15:32.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.670+0000 7f39faffd640 1 -- 172.21.15.19:0/3524382841 <== mon.1 v2:172.21.15.43:3300/0 10 ==== mon_command_ack([{prefix=config set, name=debug_mds_balancer}]=0 v8)=0 v8) ==== 117+0+0 (secure 0 0 0) 0x7f3a0003bb90 con 0x7f3a0c0c7600 2024-08-31T12:15:32.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.672+0000 7f39f8ff9640 1 -- 172.21.15.19:0/3524382841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a0c0c7600 msgr2=0x7f3a0c15a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.676 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.672+0000 7f39f8ff9640 1 --2- 172.21.15.19:0/3524382841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a0c0c7600 0x7f3a0c15a650 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c0c9660 tx=0x7f3a000046c0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.676 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.672+0000 7f39f8ff9640 1 -- 172.21.15.19:0/3524382841 shutdown_connections 2024-08-31T12:15:32.676 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.672+0000 7f39f8ff9640 1 --2- 172.21.15.19:0/3524382841 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3a0c15bae0 0x7f3a0c14f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.676 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.672+0000 7f39f8ff9640 1 --2- 172.21.15.19:0/3524382841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a0c0c7600 0x7f3a0c15a650 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.676 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.672+0000 7f39f8ff9640 1 --2- 172.21.15.19:0/3524382841 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3a0c073280 0x7f3a0c1f27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.676 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.672+0000 7f39f8ff9640 1 -- 172.21.15.19:0/3524382841 >> 172.21.15.19:0/3524382841 conn(0x7f3a0c070d90 msgr2=0x7f3a0c0c1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:32.676 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.673+0000 7f39f8ff9640 1 -- 172.21.15.19:0/3524382841 shutdown_connections 2024-08-31T12:15:32.677 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.673+0000 7f39f8ff9640 1 -- 172.21.15.19:0/3524382841 wait complete. 2024-08-31T12:15:32.677 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.673+0000 7ff77cff9640 1 -- 172.21.15.19:0/2515398365 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff78009c070 con 0x7ff7780df610 2024-08-31T12:15:32.677 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.673+0000 7ff77cff9640 1 -- 172.21.15.19:0/2515398365 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff78009d3b0 con 0x7ff7780df610 2024-08-31T12:15:32.681 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.673+0000 7fc24ffff640 1 -- 172.21.15.19:0/1631849490 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fc24001c450 con 0x7fc25815a500 2024-08-31T12:15:32.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.689+0000 7ff77cff9640 1 -- 172.21.15.19:0/2515398365 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7ff780069f40 con 0x7ff7780df610 2024-08-31T12:15:32.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.717+0000 7fd51d68d640 1 -- 172.21.15.19:0/3527553041 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd518167c80 msgr2=0x7fd518168090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.717+0000 7fd51d68d640 1 --2- 172.21.15.19:0/3527553041 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd518167c80 0x7fd518168090 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd50c007350 tx=0x7fd50c02b1c0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.717+0000 7fd51d68d640 1 -- 172.21.15.19:0/3527553041 shutdown_connections 2024-08-31T12:15:32.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.717+0000 7fd51d68d640 1 --2- 172.21.15.19:0/3527553041 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd518167c80 0x7fd518168090 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.717+0000 7fd51d68d640 1 --2- 172.21.15.19:0/3527553041 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd5180c8950 0x7fd5180c8d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.717+0000 7fd51d68d640 1 -- 172.21.15.19:0/3527553041 >> 172.21.15.19:0/3527553041 conn(0x7fd518070ba0 msgr2=0x7fd51806e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:32.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.721+0000 7fd51d68d640 1 -- 172.21.15.19:0/3527553041 shutdown_connections 2024-08-31T12:15:32.725 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'debug_ms', '2'], check_status=True, hostname='smithi019') 2024-08-31T12:15:32.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.721+0000 7fd51d68d640 1 -- 172.21.15.19:0/3527553041 wait complete. 2024-08-31T12:15:32.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.724+0000 7fd51d68d640 1 Processor -- start 2024-08-31T12:15:32.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.724+0000 7fd51d68d640 1 -- start start 2024-08-31T12:15:32.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.726+0000 7fd51d68d640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd51808c6b0 0x7fd5181f2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.727+0000 7fd51d68d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd5180c8950 0x7fd5181689c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.727+0000 7fd51d68d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd51820f190 0x7fd51820c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.727+0000 7fd51d68d640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fd5180c6100 con 0x7fd51820f190 2024-08-31T12:15:32.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.727+0000 7fd51d68d640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd5181685d0 con 0x7fd51808c6b0 2024-08-31T12:15:32.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.727+0000 7fd51d68d640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fd5180c6280 con 0x7fd5180c8950 2024-08-31T12:15:32.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.727+0000 7fd5177fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd51820f190 0x7fd51820c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:32.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.727+0000 7fd5177fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd51820f190 0x7fd51820c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:55216/0 (socket says 172.21.15.19:55216) 2024-08-31T12:15:32.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.727+0000 7fd5177fe640 1 -- 172.21.15.19:0/4196592595 learned_addr learned my addr 172.21.15.19:0/4196592595 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:15:32.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.728+0000 7fd5177fe640 1 -- 172.21.15.19:0/4196592595 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd5180c8950 msgr2=0x7fd5181689c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.728+0000 7fd5177fe640 1 --2- 172.21.15.19:0/4196592595 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd5180c8950 0x7fd5181689c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.728+0000 7fd5177fe640 1 -- 172.21.15.19:0/4196592595 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd51808c6b0 msgr2=0x7fd5181f2da0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-08-31T12:15:32.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.728+0000 7fd5177fe640 1 --2- 172.21.15.19:0/4196592595 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd51808c6b0 0x7fd5181f2da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.728+0000 7fd5177fe640 1 -- 172.21.15.19:0/4196592595 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd51815fe50 con 0x7fd51820f190 2024-08-31T12:15:32.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.729+0000 7fd5177fe640 1 --2- 172.21.15.19:0/4196592595 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd51820f190 0x7fd51820c440 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd50c007440 tx=0x7fd50c00a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:32.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.738+0000 7fd4ff7fe640 1 -- 172.21.15.19:0/4196592595 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd50c00a880 con 0x7fd51820f190 2024-08-31T12:15:32.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.738+0000 7fd4ff7fe640 1 -- 172.21.15.19:0/4196592595 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd50c02be40 con 0x7fd51820f190 2024-08-31T12:15:32.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.738+0000 7fd4ff7fe640 1 -- 172.21.15.19:0/4196592595 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd50c004280 con 0x7fd51820f190 2024-08-31T12:15:32.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.738+0000 7fd51d68d640 1 -- 172.21.15.19:0/4196592595 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5181f68a0 con 0x7fd51820f190 2024-08-31T12:15:32.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.738+0000 7fd51d68d640 1 -- 172.21.15.19:0/4196592595 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5181f3690 con 0x7fd51820f190 2024-08-31T12:15:32.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.742+0000 7fd4fd7fa640 1 -- 172.21.15.19:0/4196592595 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd518074390 con 0x7fd51820f190 2024-08-31T12:15:32.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.743+0000 7fd4ff7fe640 1 -- 172.21.15.19:0/4196592595 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd50c0035f0 con 0x7fd51820f190 2024-08-31T12:15:32.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.743+0000 7fd4ff7fe640 1 -- 172.21.15.19:0/4196592595 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd50c008410 con 0x7fd51820f190 2024-08-31T12:15:32.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.746+0000 7fd4ff7fe640 1 -- 172.21.15.19:0/4196592595 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fd518074390 con 0x7fd51820f190 2024-08-31T12:15:32.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.755+0000 7ffb7d98b640 1 -- 172.21.15.19:0/2012898351 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffb780c8950 msgr2=0x7ffb780c8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.755+0000 7ffb7d98b640 1 --2- 172.21.15.19:0/2012898351 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffb780c8950 0x7ffb780c8d60 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ffb6800c230 tx=0x7ffb6802c3c0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.755+0000 7ffb7d98b640 1 -- 172.21.15.19:0/2012898351 shutdown_connections 2024-08-31T12:15:32.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.755+0000 7ffb7d98b640 1 --2- 172.21.15.19:0/2012898351 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffb780c8950 0x7ffb780c8d60 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.755+0000 7ffb7d98b640 1 -- 172.21.15.19:0/2012898351 >> 172.21.15.19:0/2012898351 conn(0x7ffb78070ba0 msgr2=0x7ffb7806e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:32.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.758+0000 7ffb7d98b640 1 -- 172.21.15.19:0/2012898351 shutdown_connections 2024-08-31T12:15:32.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.758+0000 7ffb7d98b640 1 -- 172.21.15.19:0/2012898351 wait complete. 2024-08-31T12:15:32.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.761+0000 7ffb7d98b640 1 Processor -- start 2024-08-31T12:15:32.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.761+0000 7ffb7d98b640 1 -- start start 2024-08-31T12:15:32.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.763+0000 7ffb7d98b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffb7808c6b0 0x7ffb78164f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.763+0000 7ffb7d98b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffb78167c80 0x7ffb781630e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.764+0000 7ffb7d98b640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb7815e240 0x7ffb7820a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.764+0000 7ffb7d98b640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7ffb780c6280 con 0x7ffb7808c6b0 2024-08-31T12:15:32.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.764+0000 7ffb7d98b640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ffb780c6100 con 0x7ffb7815e240 2024-08-31T12:15:32.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.764+0000 7ffb7d98b640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7ffb7806eca0 con 0x7ffb78167c80 2024-08-31T12:15:32.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.765+0000 7ffb777fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb7815e240 0x7ffb7820a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:32.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.766+0000 7ffb777fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb7815e240 0x7ffb7820a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:41478/0 (socket says 172.21.15.19:41478) 2024-08-31T12:15:32.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.766+0000 7ffb777fe640 1 -- 172.21.15.19:0/3691795524 learned_addr learned my addr 172.21.15.19:0/3691795524 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:15:32.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.766+0000 7ffb767fc640 1 --2- 172.21.15.19:0/3691795524 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffb78167c80 0x7ffb781630e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:32.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.766+0000 7ffb777fe640 1 -- 172.21.15.19:0/3691795524 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffb78167c80 msgr2=0x7ffb781630e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.766+0000 7ffb777fe640 1 --2- 172.21.15.19:0/3691795524 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffb78167c80 0x7ffb781630e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.766+0000 7ffb777fe640 1 -- 172.21.15.19:0/3691795524 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffb7808c6b0 msgr2=0x7ffb78164f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.766+0000 7ffb777fe640 1 --2- 172.21.15.19:0/3691795524 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffb7808c6b0 0x7ffb78164f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.766+0000 7ffb777fe640 1 -- 172.21.15.19:0/3691795524 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb780ccad0 con 0x7ffb7815e240 2024-08-31T12:15:32.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.766+0000 7ffb777fe640 1 --2- 172.21.15.19:0/3691795524 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb7815e240 0x7ffb7820a010 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c00c390 tx=0x7ffb6c00c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:32.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.764+0000 7fd5cd962640 1 -- 172.21.15.19:0/3939464351 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd5c0005c40 msgr2=0x7fd5b0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.764+0000 7fd5cd962640 1 --2- 172.21.15.19:0/3939464351 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd5c0005c40 0x7fd5b0004800 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd5c806b430 tx=0x7fd5b0005830 comp rx=0 tx=0).stop 2024-08-31T12:15:32.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.768+0000 7ffb5ffff640 1 -- 172.21.15.19:0/3691795524 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb6c011790 con 0x7ffb7815e240 2024-08-31T12:15:32.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.768+0000 7ffb5ffff640 1 -- 172.21.15.19:0/3691795524 <== mon.1 v2:172.21.15.43:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffb6c017070 con 0x7ffb7815e240 2024-08-31T12:15:32.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.768+0000 7ffb7d98b640 1 -- 172.21.15.19:0/3691795524 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb780c2270 con 0x7ffb7815e240 2024-08-31T12:15:32.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.768+0000 7ffb7d98b640 1 -- 172.21.15.19:0/3691795524 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb780cc7f0 con 0x7ffb7815e240 2024-08-31T12:15:32.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.769+0000 7ffb5ffff640 1 -- 172.21.15.19:0/3691795524 <== mon.1 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb6c011cc0 con 0x7ffb7815e240 2024-08-31T12:15:32.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.767+0000 7fd5c77fe640 1 -- 172.21.15.19:0/3939464351 reap_dead start 2024-08-31T12:15:32.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.767+0000 7fd5cd962640 1 -- 172.21.15.19:0/3939464351 shutdown_connections 2024-08-31T12:15:32.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.767+0000 7fd5cd962640 1 -- 172.21.15.19:0/3939464351 >> 172.21.15.19:0/3939464351 conn(0x7fd5c004fe20 msgr2=0x7fd5c0050230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:32.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.767+0000 7fd5cd962640 1 -- 172.21.15.19:0/3939464351 shutdown_connections 2024-08-31T12:15:32.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.771+0000 7f2afe394640 1 -- 172.21.15.19:0/3306331068 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2af8150910 msgr2=0x7f2af8150d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.771+0000 7fd5cd962640 1 -- 172.21.15.19:0/3939464351 wait complete. 2024-08-31T12:15:32.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.774+0000 7ffb5dffb640 1 -- 172.21.15.19:0/3691795524 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb58005290 con 0x7ffb7815e240 2024-08-31T12:15:32.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.774+0000 7ffb5ffff640 1 -- 172.21.15.19:0/3691795524 <== mon.1 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ffb6c01c460 con 0x7ffb7815e240 2024-08-31T12:15:32.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.774+0000 7ffb5ffff640 1 -- 172.21.15.19:0/3691795524 <== mon.1 v2:172.21.15.43:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ffb6c012810 con 0x7ffb7815e240 2024-08-31T12:15:32.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.774+0000 7fd5cd962640 1 Processor -- start 2024-08-31T12:15:32.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.771+0000 7f2afe394640 1 --2- 172.21.15.19:0/3306331068 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2af8150910 0x7f2af8150d20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2af80c3e10 tx=0x7f2af002a9c0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.775+0000 7f2afe394640 1 -- 172.21.15.19:0/3306331068 shutdown_connections 2024-08-31T12:15:32.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.775+0000 7f2afe394640 1 --2- 172.21.15.19:0/3306331068 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2af8150910 0x7f2af8150d20 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.775+0000 7f2afe394640 1 --2- 172.21.15.19:0/3306331068 >> v2:172.21.15.19:3300/0 conn(0x7f2af80c7a80 0x7f2af8072db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.775+0000 7f2afe394640 1 -- 172.21.15.19:0/3306331068 >> 172.21.15.19:0/3306331068 conn(0x7f2af80cc9e0 msgr2=0x7f2af80c8d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:32.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.775+0000 7f2afe394640 1 -- 172.21.15.19:0/3306331068 shutdown_connections 2024-08-31T12:15:32.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.775+0000 7fd5cd962640 1 -- start start 2024-08-31T12:15:32.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.777+0000 7fd5cd962640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd5c00da1a0 0x7fd5c00c7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.777+0000 7fd5cd962640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd5c00d2d70 0x7fd5c00d3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.777+0000 7fd5cd962640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd5c00cbc50 0x7fd5c00cc060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.777+0000 7fd5cd962640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fd5c00468a0 con 0x7fd5c00cbc50 2024-08-31T12:15:32.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.777+0000 7fd5cd962640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd5c00465c0 con 0x7fd5c00da1a0 2024-08-31T12:15:32.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.777+0000 7fd5cd962640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fd5c0006050 con 0x7fd5c00d2d70 2024-08-31T12:15:32.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.777+0000 7fd5c77fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd5c00cbc50 0x7fd5c00cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:32.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.777+0000 7fd5c77fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd5c00cbc50 0x7fd5c00cc060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:55246/0 (socket says 172.21.15.19:55246) 2024-08-31T12:15:32.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.777+0000 7fd5c77fe640 1 -- 172.21.15.19:0/4035385145 learned_addr learned my addr 172.21.15.19:0/4035385145 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:15:32.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.778+0000 7fd5c67fc640 1 --2- 172.21.15.19:0/4035385145 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd5c00d2d70 0x7fd5c00d3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:32.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.778+0000 7fd5c6ffd640 1 --2- 172.21.15.19:0/4035385145 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd5c00da1a0 0x7fd5c00c7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:32.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.775+0000 7f2afe394640 1 -- 172.21.15.19:0/3306331068 wait complete. 2024-08-31T12:15:32.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.778+0000 7f2afe394640 1 Processor -- start 2024-08-31T12:15:32.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.778+0000 7f2afe394640 1 -- start start 2024-08-31T12:15:32.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.780+0000 7f2afe394640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2af80c7a80 0x7f2af814f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.780+0000 7f2afe394640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2af815b9d0 0x7f2af8074240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.780+0000 7f2afe394640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2af8146d80 0x7f2af8147190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.780+0000 7f2afe394640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f2af80c7280 con 0x7f2af815b9d0 2024-08-31T12:15:32.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.780+0000 7f2afe394640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2af80c6fa0 con 0x7f2af8146d80 2024-08-31T12:15:32.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.780+0000 7f2afe394640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f2af80c7560 con 0x7f2af80c7a80 2024-08-31T12:15:32.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.781+0000 7f2afc931640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2af8146d80 0x7f2af8147190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:32.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.781+0000 7f2afc931640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2af8146d80 0x7f2af8147190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:41500/0 (socket says 172.21.15.19:41500) 2024-08-31T12:15:32.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.781+0000 7f2afc931640 1 -- 172.21.15.19:0/1731406799 learned_addr learned my addr 172.21.15.19:0/1731406799 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:15:32.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.778+0000 7fd5c77fe640 1 -- 172.21.15.19:0/4035385145 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd5c00d2d70 msgr2=0x7fd5c00d3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.778+0000 7fd5c77fe640 1 --2- 172.21.15.19:0/4035385145 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd5c00d2d70 0x7fd5c00d3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.778+0000 7fd5c77fe640 1 -- 172.21.15.19:0/4035385145 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd5c00da1a0 msgr2=0x7fd5c00c7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.778+0000 7fd5c77fe640 1 --2- 172.21.15.19:0/4035385145 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd5c00da1a0 0x7fd5c00c7ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.778+0000 7fd5c77fe640 1 -- 172.21.15.19:0/4035385145 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5c00ccdf0 con 0x7fd5c00cbc50 2024-08-31T12:15:32.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.778+0000 7fd5c77fe640 1 --2- 172.21.15.19:0/4035385145 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd5c00cbc50 0x7fd5c00cc060 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd5c0007740 tx=0x7fd5b800ac40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:32.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.779+0000 7fd5affff640 1 -- 172.21.15.19:0/4035385145 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5b800ae30 con 0x7fd5c00cbc50 2024-08-31T12:15:32.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.779+0000 7fd5cd962640 1 -- 172.21.15.19:0/4035385145 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5c01717c0 con 0x7fd5c00cbc50 2024-08-31T12:15:32.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.779+0000 7fd5cd962640 1 -- 172.21.15.19:0/4035385145 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5c00cd390 con 0x7fd5c00cbc50 2024-08-31T12:15:32.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.781+0000 7fd5affff640 1 -- 172.21.15.19:0/4035385145 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd5b8004920 con 0x7fd5c00cbc50 2024-08-31T12:15:32.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.782+0000 7fd5affff640 1 -- 172.21.15.19:0/4035385145 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5b8004d50 con 0x7fd5c00cbc50 2024-08-31T12:15:32.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.782+0000 7fd5affff640 1 -- 172.21.15.19:0/4035385145 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd5b8019020 con 0x7fd5c00cbc50 2024-08-31T12:15:32.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.782+0000 7fd5affff640 1 -- 172.21.15.19:0/4035385145 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd5b800f450 con 0x7fd5c00cbc50 2024-08-31T12:15:32.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.782+0000 7fd5adffb640 1 -- 172.21.15.19:0/4035385145 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5a8005290 con 0x7fd5c00cbc50 2024-08-31T12:15:32.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.781+0000 7f2aef7fe640 1 --2- 172.21.15.19:0/1731406799 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2af815b9d0 0x7f2af8074240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:32.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.784+0000 7f2afc931640 1 -- 172.21.15.19:0/1731406799 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2af80c7a80 msgr2=0x7f2af814f6b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:15:32.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.784+0000 7f2afc931640 1 --2- 172.21.15.19:0/1731406799 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2af80c7a80 0x7f2af814f6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.784+0000 7f2afc931640 1 -- 172.21.15.19:0/1731406799 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2af815b9d0 msgr2=0x7f2af8074240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.784+0000 7f2afc931640 1 --2- 172.21.15.19:0/1731406799 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2af815b9d0 0x7f2af8074240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.784+0000 7f2afc931640 1 -- 172.21.15.19:0/1731406799 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2af8074780 con 0x7f2af8146d80 2024-08-31T12:15:32.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.784+0000 7f2aef7fe640 1 --2- 172.21.15.19:0/1731406799 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2af815b9d0 0x7f2af8074240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-31T12:15:32.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.784+0000 7f2afc931640 1 --2- 172.21.15.19:0/1731406799 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2af8146d80 0x7f2af8147190 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2af0009d60 tx=0x7f2af0039c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:32.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.784+0000 7ffb5ffff640 1 -- 172.21.15.19:0/3691795524 <== mon.1 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7ffb6c012bd0 con 0x7ffb7815e240 2024-08-31T12:15:32.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.788+0000 7f2af57fa640 1 -- 172.21.15.19:0/1731406799 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af0033040 con 0x7f2af8146d80 2024-08-31T12:15:32.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.788+0000 7f2afe394640 1 -- 172.21.15.19:0/1731406799 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2af8147930 con 0x7f2af8146d80 2024-08-31T12:15:32.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.788+0000 7f2afe394640 1 -- 172.21.15.19:0/1731406799 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2af8149090 con 0x7f2af8146d80 2024-08-31T12:15:32.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.789+0000 7f2af57fa640 1 -- 172.21.15.19:0/1731406799 <== mon.1 v2:172.21.15.43:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2af00378b0 con 0x7f2af8146d80 2024-08-31T12:15:32.792 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.789+0000 7f2af57fa640 1 -- 172.21.15.19:0/1731406799 <== mon.1 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af0037ca0 con 0x7f2af8146d80 2024-08-31T12:15:32.792 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.790+0000 7fd5affff640 1 -- 172.21.15.19:0/4035385145 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fd5b8007c90 con 0x7fd5c00cbc50 2024-08-31T12:15:32.792 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.791+0000 7f2af57fa640 1 -- 172.21.15.19:0/1731406799 <== mon.1 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2af0036070 con 0x7f2af8146d80 2024-08-31T12:15:32.792 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.791+0000 7f2af57fa640 1 -- 172.21.15.19:0/1731406799 <== mon.1 v2:172.21.15.43:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2af00433f0 con 0x7f2af8146d80 2024-08-31T12:15:32.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.794+0000 7f2aee7fc640 1 -- 172.21.15.19:0/1731406799 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2adc005290 con 0x7f2af8146d80 2024-08-31T12:15:32.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.798+0000 7f2af57fa640 1 -- 172.21.15.19:0/1731406799 <== mon.1 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f2af003e070 con 0x7f2af8146d80 2024-08-31T12:15:32.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.802+0000 7f9183fff640 1 -- 172.21.15.19:0/3454382734 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) -- 0x7f91981588c0 con 0x7f919815b4e0 2024-08-31T12:15:32.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.807+0000 7f716faa9640 1 -- 172.21.15.19:0/1132005076 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f71600df640 msgr2=0x7f71600454d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.807+0000 7f716faa9640 1 --2- 172.21.15.19:0/1132005076 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f71600df640 0x7f71600454d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7168075480 tx=0x7f71680a6f90 comp rx=0 tx=0).stop 2024-08-31T12:15:32.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.807+0000 7f716faa9640 1 -- 172.21.15.19:0/1132005076 shutdown_connections 2024-08-31T12:15:32.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.807+0000 7f716faa9640 1 --2- 172.21.15.19:0/1132005076 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f71600df640 0x7f71600454d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.807+0000 7f716faa9640 1 -- 172.21.15.19:0/1132005076 >> 172.21.15.19:0/1132005076 conn(0x7f716004fe50 msgr2=0x7f7160050260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:32.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.808+0000 7f716faa9640 1 -- 172.21.15.19:0/1132005076 shutdown_connections 2024-08-31T12:15:32.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.808+0000 7f716faa9640 1 -- 172.21.15.19:0/1132005076 wait complete. 2024-08-31T12:15:32.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.810+0000 7f716faa9640 1 Processor -- start 2024-08-31T12:15:32.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.811+0000 7f716faa9640 1 -- start start 2024-08-31T12:15:32.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.811+0000 7fc64160e640 1 -- 172.21.15.19:0/3361220915 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc63c0c8950 msgr2=0x7fc63c0c8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.811+0000 7fc64160e640 1 --2- 172.21.15.19:0/3361220915 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc63c0c8950 0x7fc63c0c8d60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc63c06dc00 tx=0x7fc63402cd10 comp rx=0 tx=0).stop 2024-08-31T12:15:32.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.811+0000 7fc64160e640 1 -- 172.21.15.19:0/3361220915 shutdown_connections 2024-08-31T12:15:32.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.811+0000 7fc64160e640 1 --2- 172.21.15.19:0/3361220915 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc63c167c80 0x7fc63c168090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.811+0000 7fc64160e640 1 --2- 172.21.15.19:0/3361220915 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc63c0c8950 0x7fc63c0c8d60 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.811+0000 7fc64160e640 1 --2- 172.21.15.19:0/3361220915 >> v2:172.21.15.43:3300/0 conn(0x7fc63c08c6b0 0x7fc63c073990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.811+0000 7fc64160e640 1 -- 172.21.15.19:0/3361220915 >> 172.21.15.19:0/3361220915 conn(0x7fc63c070ba0 msgr2=0x7fc63c06e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:32.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.811+0000 7fc64160e640 1 -- 172.21.15.19:0/3361220915 shutdown_connections 2024-08-31T12:15:32.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.811+0000 7fc64160e640 1 -- 172.21.15.19:0/3361220915 wait complete. 2024-08-31T12:15:32.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.814+0000 7fc64160e640 1 Processor -- start 2024-08-31T12:15:32.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.814+0000 7fc64160e640 1 -- start start 2024-08-31T12:15:32.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.816+0000 7fc64160e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc63c08c6b0 0x7fc63c169550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.816+0000 7fc64160e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc63c167c80 0x7fc63c169d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.816+0000 7fc64160e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc63c0c0790 0x7fc63c15ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.816+0000 7fc64160e640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fc63c073ed0 con 0x7fc63c0c0790 2024-08-31T12:15:32.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.816+0000 7fc64160e640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc63c1685d0 con 0x7fc63c08c6b0 2024-08-31T12:15:32.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.816+0000 7fc64160e640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fc63c074050 con 0x7fc63c167c80 2024-08-31T12:15:32.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.817+0000 7fc63affd640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc63c08c6b0 0x7fc63c169550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:32.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.817+0000 7fc63affd640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc63c08c6b0 0x7fc63c169550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:41524/0 (socket says 172.21.15.19:41524) 2024-08-31T12:15:32.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.817+0000 7fc63affd640 1 -- 172.21.15.19:0/3011667190 learned_addr learned my addr 172.21.15.19:0/3011667190 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:15:32.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.817+0000 7fc63a7fc640 1 --2- 172.21.15.19:0/3011667190 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc63c167c80 0x7fc63c169d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:32.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.817+0000 7fc63b7fe640 1 --2- 172.21.15.19:0/3011667190 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc63c0c0790 0x7fc63c15ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:32.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.819+0000 7fc63affd640 1 -- 172.21.15.19:0/3011667190 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc63c167c80 msgr2=0x7fc63c169d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.819+0000 7fc63affd640 1 --2- 172.21.15.19:0/3011667190 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc63c167c80 0x7fc63c169d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.819+0000 7fc63affd640 1 -- 172.21.15.19:0/3011667190 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc63c0c0790 msgr2=0x7fc63c15ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.819+0000 7fc63affd640 1 --2- 172.21.15.19:0/3011667190 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc63c0c0790 0x7fc63c15ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.819+0000 7fc63affd640 1 -- 172.21.15.19:0/3011667190 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc63c0ca330 con 0x7fc63c08c6b0 2024-08-31T12:15:32.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.819+0000 7fc63affd640 1 --2- 172.21.15.19:0/3011667190 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc63c08c6b0 0x7fc63c169550 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc62c00ea60 tx=0x7fc62c00ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:32.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.816+0000 7f918e7fc640 1 -- 172.21.15.19:0/3454382734 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v9)=0 v9) ==== 127+0+0 (secure 0 0 0) 0x7f9190032030 con 0x7f919815b4e0 2024-08-31T12:15:32.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.819+0000 7f9183fff640 1 -- 172.21.15.19:0/3454382734 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f919815b4e0 msgr2=0x7f91981f5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.819+0000 7f9183fff640 1 --2- 172.21.15.19:0/3454382734 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f919815b4e0 0x7f91981f5560 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9190009d60 tx=0x7f9190004270 comp rx=0 tx=0).stop 2024-08-31T12:15:32.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.824+0000 7f9183fff640 1 -- 172.21.15.19:0/3454382734 shutdown_connections 2024-08-31T12:15:32.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.824+0000 7f9183fff640 1 --2- 172.21.15.19:0/3454382734 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9198158430 0x7f9198147fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.824+0000 7f9183fff640 1 --2- 172.21.15.19:0/3454382734 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f919815b4e0 0x7f91981f5560 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.824+0000 7f9183fff640 1 --2- 172.21.15.19:0/3454382734 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9198074130 0x7f9198157b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.824+0000 7f9183fff640 1 -- 172.21.15.19:0/3454382734 >> 172.21.15.19:0/3454382734 conn(0x7f9198071a00 msgr2=0x7f91981e5060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:32.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.824+0000 7fc6237fe640 1 -- 172.21.15.19:0/3011667190 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62c00cdc0 con 0x7fc63c08c6b0 2024-08-31T12:15:32.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.824+0000 7fc64160e640 1 -- 172.21.15.19:0/3011667190 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc63c16a490 con 0x7fc63c08c6b0 2024-08-31T12:15:32.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.824+0000 7fc64160e640 1 -- 172.21.15.19:0/3011667190 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc63c16a750 con 0x7fc63c08c6b0 2024-08-31T12:15:32.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.825+0000 7f716faa9640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7160005c70 0x7f71600dbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.826+0000 7f716faa9640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7160044d40 0x7f716017c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.826+0000 7fc6237fe640 1 -- 172.21.15.19:0/3011667190 <== mon.1 v2:172.21.15.43:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc62c005020 con 0x7fc63c08c6b0 2024-08-31T12:15:32.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.826+0000 7fc6237fe640 1 -- 172.21.15.19:0/3011667190 <== mon.1 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62c005470 con 0x7fc63c08c6b0 2024-08-31T12:15:32.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.826+0000 7f9183fff640 1 -- 172.21.15.19:0/3454382734 shutdown_connections 2024-08-31T12:15:32.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.826+0000 7fc6237fe640 1 -- 172.21.15.19:0/3011667190 <== mon.1 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc62c0047a0 con 0x7fc63c08c6b0 2024-08-31T12:15:32.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.826+0000 7f716faa9640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f71600d0480 0x7f716004a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.831 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.826+0000 7fc6237fe640 1 -- 172.21.15.19:0/3011667190 <== mon.1 v2:172.21.15.43:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc62c017b70 con 0x7fc63c08c6b0 2024-08-31T12:15:32.831 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.826+0000 7f716faa9640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f71600468d0 con 0x7f71600d0480 2024-08-31T12:15:32.831 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.826+0000 7f716e046640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f71600d0480 0x7f716004a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:32.832 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.826+0000 7f7456825640 1 -- 172.21.15.19:0/2823695511 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7438004800 msgr2=0x7f7438004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.832 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.826+0000 7f7456825640 1 --2- 172.21.15.19:0/2823695511 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7438004800 0x7f7438004c10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7450069f50 tx=0x7f7438005830 comp rx=0 tx=0).stop 2024-08-31T12:15:32.832 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.827+0000 7fc6217fa640 1 -- 172.21.15.19:0/3011667190 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc61c005290 con 0x7fc63c08c6b0 2024-08-31T12:15:32.832 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.826+0000 7f716e046640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f71600d0480 0x7f716004a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:55288/0 (socket says 172.21.15.19:55288) 2024-08-31T12:15:32.833 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.827+0000 7f9183fff640 1 -- 172.21.15.19:0/3454382734 wait complete. 2024-08-31T12:15:32.833 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.826+0000 7f716e046640 1 -- 172.21.15.19:0/809483613 learned_addr learned my addr 172.21.15.19:0/809483613 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:15:32.833 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.827+0000 7f716d845640 1 --2- 172.21.15.19:0/809483613 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7160005c70 0x7f71600dbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:32.834 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.827+0000 7f716d044640 1 --2- 172.21.15.19:0/809483613 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7160044d40 0x7f716017c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:32.834 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.828+0000 7f716faa9640 1 -- 172.21.15.19:0/809483613 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f71600d9310 con 0x7f7160044d40 2024-08-31T12:15:32.834 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.828+0000 7f716faa9640 1 -- 172.21.15.19:0/809483613 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f71600072b0 con 0x7f7160005c70 2024-08-31T12:15:32.834 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.829+0000 7f7456825640 1 -- 172.21.15.19:0/2823695511 shutdown_connections 2024-08-31T12:15:32.834 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.829+0000 7f7456825640 1 --2- 172.21.15.19:0/2823695511 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7438004800 0x7f7438004c10 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.834 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.829+0000 7f7456825640 1 -- 172.21.15.19:0/2823695511 >> 172.21.15.19:0/2823695511 conn(0x7f7450070ba0 msgr2=0x7f745006e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:32.834 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.829+0000 7f7456825640 1 -- 172.21.15.19:0/2823695511 shutdown_connections 2024-08-31T12:15:32.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.830+0000 7f716e046640 1 -- 172.21.15.19:0/809483613 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7160005c70 msgr2=0x7f71600dbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.830+0000 7fc6237fe640 1 -- 172.21.15.19:0/3011667190 <== mon.1 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fc62c017e20 con 0x7fc63c08c6b0 2024-08-31T12:15:32.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.830+0000 7f716e046640 1 --2- 172.21.15.19:0/809483613 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7160005c70 0x7f71600dbee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.830+0000 7f716e046640 1 -- 172.21.15.19:0/809483613 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7160044d40 msgr2=0x7f716017c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.830+0000 7f716e046640 1 --2- 172.21.15.19:0/809483613 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7160044d40 0x7f716017c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.830+0000 7f716e046640 1 -- 172.21.15.19:0/809483613 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f716004f8b0 con 0x7f71600d0480 2024-08-31T12:15:32.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.830+0000 7f716e046640 1 --2- 172.21.15.19:0/809483613 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f71600d0480 0x7f716004a4a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f715800c3f0 tx=0x7f715800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:32.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.830+0000 7f716d845640 1 --2- 172.21.15.19:0/809483613 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7160005c70 0x7f71600dbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:15:32.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.830+0000 7fd8437fe640 1 -- 172.21.15.19:0/1229158451 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command([{prefix=config set, name=debug_ms}] v 0) -- 0x7fd838005580 con 0x7fd858150e30 2024-08-31T12:15:32.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.833+0000 7f715e7fc640 1 -- 172.21.15.19:0/809483613 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71580052d0 con 0x7f71600d0480 2024-08-31T12:15:32.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.833+0000 7f715e7fc640 1 -- 172.21.15.19:0/809483613 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7158006820 con 0x7f71600d0480 2024-08-31T12:15:32.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.833+0000 7f716faa9640 1 -- 172.21.15.19:0/809483613 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71600d0d50 con 0x7f71600d0480 2024-08-31T12:15:32.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.833+0000 7f716faa9640 1 -- 172.21.15.19:0/809483613 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f716017cec0 con 0x7f71600d0480 2024-08-31T12:15:32.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.834+0000 7febaf096640 1 -- 172.21.15.19:0/3610671166 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feba8157ab0 msgr2=0x7feba80c78a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.834+0000 7febaf096640 1 --2- 172.21.15.19:0/3610671166 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feba8157ab0 0x7feba80c78a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7feba0005fd0 tx=0x7feba002a900 comp rx=0 tx=0).stop 2024-08-31T12:15:32.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.829+0000 7f7456825640 1 -- 172.21.15.19:0/2823695511 wait complete. 2024-08-31T12:15:32.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.838+0000 7f715e7fc640 1 -- 172.21.15.19:0/809483613 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7158006c70 con 0x7f71600d0480 2024-08-31T12:15:32.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.840+0000 7fb062447640 1 -- 172.21.15.19:0/4035990667 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb05404dd30 msgr2=0x7fb054006130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.840+0000 7fb062447640 1 --2- 172.21.15.19:0/4035990667 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb05404dd30 0x7fb054006130 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb05c0757d0 tx=0x7fb05c0a6fe0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.841 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.840+0000 7fd84dffb640 1 -- 172.21.15.19:0/1229158451 <== mon.2 v2:172.21.15.43:3301/0 7 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v10)=0 v10) ==== 106+0+0 (secure 0 0 0) 0x7fd8540037d0 con 0x7fd858150e30 2024-08-31T12:15:32.841 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.841+0000 7faff8162640 1 -- 172.21.15.19:0/2935341543 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faff00b63a0 msgr2=0x7faff00b5c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.841 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.841+0000 7faff8162640 1 --2- 172.21.15.19:0/2935341543 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faff00b63a0 0x7faff00b5c30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fafe8009ca0 tx=0x7fafe802aa00 comp rx=0 tx=0).stop 2024-08-31T12:15:32.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.841+0000 7fb062447640 1 -- 172.21.15.19:0/4035990667 shutdown_connections 2024-08-31T12:15:32.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.841+0000 7fb062447640 1 --2- 172.21.15.19:0/4035990667 >> v2:172.21.15.19:3300/0 conn(0x7fb0540e7030 0x7fb0540e7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.841+0000 7fb062447640 1 --2- 172.21.15.19:0/4035990667 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb05404dd30 0x7fb054006130 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.841+0000 7fb062447640 1 --2- 172.21.15.19:0/4035990667 >> v2:172.21.15.43:3300/0 conn(0x7fb05404fb90 0x7fb05404e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.841+0000 7fb062447640 1 -- 172.21.15.19:0/4035990667 >> 172.21.15.19:0/4035990667 conn(0x7fb0540543a0 msgr2=0x7fb0540547b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:32.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.841+0000 7fb062447640 1 -- 172.21.15.19:0/4035990667 shutdown_connections 2024-08-31T12:15:32.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.842+0000 7f715e7fc640 1 -- 172.21.15.19:0/809483613 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7158007020 con 0x7f71600d0480 2024-08-31T12:15:32.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.842+0000 7f715e7fc640 1 -- 172.21.15.19:0/809483613 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7158003c80 con 0x7f71600d0480 2024-08-31T12:15:32.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.842+0000 7febaf096640 1 -- 172.21.15.19:0/3610671166 shutdown_connections 2024-08-31T12:15:32.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.842+0000 7febaf096640 1 --2- 172.21.15.19:0/3610671166 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feba8157ab0 0x7feba80c78a0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.844 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.842+0000 7febaf096640 1 -- 172.21.15.19:0/3610671166 >> 172.21.15.19:0/3610671166 conn(0x7feba80cc9e0 msgr2=0x7feba80c8d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:32.844 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.843+0000 7f7456825640 1 Processor -- start 2024-08-31T12:15:32.844 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.844+0000 7febaf096640 1 -- 172.21.15.19:0/3610671166 shutdown_connections 2024-08-31T12:15:32.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.841+0000 7fb062447640 1 -- 172.21.15.19:0/4035990667 wait complete. 2024-08-31T12:15:32.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.844+0000 7fb062447640 1 Processor -- start 2024-08-31T12:15:32.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.844+0000 7fb062447640 1 -- start start 2024-08-31T12:15:32.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.845+0000 7fd841d2b640 1 -- 172.21.15.19:0/422799422 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd83c168750 msgr2=0x7fd83c161f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.847 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.845+0000 7fd841d2b640 1 --2- 172.21.15.19:0/422799422 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd83c168750 0x7fd83c161f40 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd834005fd0 tx=0x7fd83402a8e0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.847 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.846+0000 7fd841d2b640 1 -- 172.21.15.19:0/422799422 shutdown_connections 2024-08-31T12:15:32.847 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.846+0000 7fd841d2b640 1 --2- 172.21.15.19:0/422799422 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd83c168750 0x7fd83c161f40 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.847 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.846+0000 7fd841d2b640 1 --2- 172.21.15.19:0/422799422 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd83c08c840 0x7fd83c0c7860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.847 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.846+0000 7fd841d2b640 1 -- 172.21.15.19:0/422799422 >> 172.21.15.19:0/422799422 conn(0x7fd83c0cc9e0 msgr2=0x7fd83c0c8d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:32.847 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.844+0000 7faff8162640 1 -- 172.21.15.19:0/2935341543 shutdown_connections 2024-08-31T12:15:32.847 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.844+0000 7faff8162640 1 --2- 172.21.15.19:0/2935341543 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faff00b63a0 0x7faff00b5c30 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.847 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.844+0000 7faff8162640 1 -- 172.21.15.19:0/2935341543 >> 172.21.15.19:0/2935341543 conn(0x7faff0069260 msgr2=0x7faff00cd230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:32.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.848+0000 7f716faa9640 1 -- 172.21.15.19:0/809483613 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7150005290 con 0x7f71600d0480 2024-08-31T12:15:32.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.844+0000 7febaf096640 1 -- 172.21.15.19:0/3610671166 wait complete. 2024-08-31T12:15:32.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.847+0000 7febaf096640 1 Processor -- start 2024-08-31T12:15:32.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.848+0000 7fb062447640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb05404dd30 0x7fb05400d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.848+0000 7fb062447640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb05404fb90 0x7fb05400db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.848+0000 7fb061445640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb05404dd30 0x7fb05400d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:32.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.848+0000 7fb061445640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb05404dd30 0x7fb05400d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.19:43280/0 (socket says 172.21.15.19:43280) 2024-08-31T12:15:32.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.846+0000 7faff8162640 1 -- 172.21.15.19:0/2935341543 shutdown_connections 2024-08-31T12:15:32.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.848+0000 7fb060c44640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb05404fb90 0x7fb05400db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:32.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.846+0000 7faff8162640 1 -- 172.21.15.19:0/2935341543 wait complete. 2024-08-31T12:15:32.852 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.848+0000 7fb060c44640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb05404fb90 0x7fb05400db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:55312/0 (socket says 172.21.15.19:55312) 2024-08-31T12:15:32.852 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.849+0000 7fb061445640 1 -- 172.21.15.19:0/1042992775 learned_addr learned my addr 172.21.15.19:0/1042992775 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:15:32.852 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.849+0000 7fb062447640 1 --2- 172.21.15.19:0/1042992775 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb0540e7030 0x7fb0540e3000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.852 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.849+0000 7fb062447640 1 -- 172.21.15.19:0/1042992775 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fb0540e6b10 con 0x7fb05404fb90 2024-08-31T12:15:32.852 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.849+0000 7fb062447640 1 -- 172.21.15.19:0/1042992775 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb054050400 con 0x7fb0540e7030 2024-08-31T12:15:32.852 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.849+0000 7fb061c46640 1 --2- 172.21.15.19:0/1042992775 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb0540e7030 0x7fb0540e3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:32.853 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.847+0000 7febaf096640 1 -- start start 2024-08-31T12:15:32.853 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.849+0000 7febaf096640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feba8073510 0x7feba81422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.853 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.849+0000 7fb062447640 1 -- 172.21.15.19:0/1042992775 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fb054039720 con 0x7fb05404dd30 2024-08-31T12:15:32.854 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.849+0000 7febaf096640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feba80c4520 0x7feba8140f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.854 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.849+0000 7febaf096640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7feba8157ab0 0x7feba80ca260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.854 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.849+0000 7febaf096640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7feba80c8430 con 0x7feba8157ab0 2024-08-31T12:15:32.854 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.849+0000 7febace32640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feba8073510 0x7feba81422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:32.854 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.849+0000 7febace32640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feba8073510 0x7feba81422a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:41602/0 (socket says 172.21.15.19:41602) 2024-08-31T12:15:32.854 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.849+0000 7febace32640 1 -- 172.21.15.19:0/3036730349 learned_addr learned my addr 172.21.15.19:0/3036730349 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:15:32.855 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.850+0000 7faff8162640 1 Processor -- start 2024-08-31T12:15:32.855 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.850+0000 7f7456825640 1 -- start start 2024-08-31T12:15:32.855 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.850+0000 7fd85e34e640 1 -- 172.21.15.19:0/1229158451 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd858150e30 msgr2=0x7fd8580c53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.855 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.850+0000 7fd85e34e640 1 --2- 172.21.15.19:0/1229158451 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd858150e30 0x7fd8580c53e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd85400e600 tx=0x7fd85400eac0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.856 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.850+0000 7fd841d2b640 1 -- 172.21.15.19:0/422799422 shutdown_connections 2024-08-31T12:15:32.856 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.850+0000 7fb060c44640 1 -- 172.21.15.19:0/1042992775 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb05404dd30 msgr2=0x7fb05400d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.850+0000 7fd85e34e640 1 -- 172.21.15.19:0/1229158451 shutdown_connections 2024-08-31T12:15:32.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.850+0000 7fd85e34e640 1 --2- 172.21.15.19:0/1229158451 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd8581fe180 0x7fd8581fe590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.850+0000 7fd85e34e640 1 --2- 172.21.15.19:0/1229158451 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd858151240 0x7fd8580c5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.850+0000 7fd85e34e640 1 --2- 172.21.15.19:0/1229158451 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd858150e30 0x7fd8580c53e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.850+0000 7fd85e34e640 1 -- 172.21.15.19:0/1229158451 >> 172.21.15.19:0/1229158451 conn(0x7fd8580c9020 msgr2=0x7fd858142a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:32.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.850+0000 7fd85e34e640 1 -- 172.21.15.19:0/1229158451 shutdown_connections 2024-08-31T12:15:32.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.850+0000 7fd85e34e640 1 -- 172.21.15.19:0/1229158451 wait complete. 2024-08-31T12:15:32.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.852+0000 7f7456825640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7438004800 0x7f74502061d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.858 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.850+0000 7faff8162640 1 -- start start 2024-08-31T12:15:32.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.852+0000 7f7456825640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74501f33c0 0x7f745015db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.852+0000 7f7456825640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f74501f37d0 0x7f745015e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.852+0000 7f744ffff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7438004800 0x7f74502061d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:32.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.852+0000 7f744ffff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7438004800 0x7f74502061d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.19:43282/0 (socket says 172.21.15.19:43282) 2024-08-31T12:15:32.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.852+0000 7f744f7fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74501f33c0 0x7f745015db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:32.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.852+0000 7f744ffff640 1 -- 172.21.15.19:0/2368691699 learned_addr learned my addr 172.21.15.19:0/2368691699 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:15:32.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.852+0000 7f7456825640 1 -- 172.21.15.19:0/2368691699 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f745008c6b0 con 0x7f74501f37d0 2024-08-31T12:15:32.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.852+0000 7f7456825640 1 -- 172.21.15.19:0/2368691699 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f745006eeb0 con 0x7f74501f33c0 2024-08-31T12:15:32.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.852+0000 7f7456825640 1 -- 172.21.15.19:0/2368691699 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f745008c830 con 0x7f7438004800 2024-08-31T12:15:32.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.852+0000 7f7454dc2640 1 --2- 172.21.15.19:0/2368691699 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f74501f37d0 0x7f745015e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:32.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.855+0000 7faff8162640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faff0074590 0x7faff00cff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.855+0000 7f744ffff640 1 -- 172.21.15.19:0/2368691699 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74501f33c0 msgr2=0x7f745015db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.855+0000 7f744ffff640 1 --2- 172.21.15.19:0/2368691699 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74501f33c0 0x7f745015db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.855+0000 7f744ffff640 1 -- 172.21.15.19:0/2368691699 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f74501f37d0 msgr2=0x7f745015e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.855+0000 7f744ffff640 1 --2- 172.21.15.19:0/2368691699 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f74501f37d0 0x7f745015e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.855+0000 7f744ffff640 1 -- 172.21.15.19:0/2368691699 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7450206710 con 0x7f7438004800 2024-08-31T12:15:32.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.855+0000 7f744ffff640 1 --2- 172.21.15.19:0/2368691699 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7438004800 0x7f74502061d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7440009780 tx=0x7f7440009c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:32.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.856+0000 7febad633640 1 --2- 172.21.15.19:0/3036730349 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7feba8157ab0 0x7feba80ca260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:32.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.856+0000 7febaf096640 1 -- 172.21.15.19:0/3036730349 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7feba80c70a0 con 0x7feba8073510 2024-08-31T12:15:32.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.856+0000 7febaf096640 1 -- 172.21.15.19:0/3036730349 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7feba806cae0 con 0x7feba80c4520 2024-08-31T12:15:32.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.859+0000 7feba7fff640 1 --2- 172.21.15.19:0/3036730349 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feba80c4520 0x7feba8140f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:32.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.850+0000 7fd841d2b640 1 -- 172.21.15.19:0/422799422 wait complete. 2024-08-31T12:15:32.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.857+0000 7fd841d2b640 1 Processor -- start 2024-08-31T12:15:32.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.856+0000 7f7454dc2640 1 --2- 172.21.15.19:0/2368691699 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f74501f37d0 0x7f745015e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:15:32.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.857+0000 7f744cff9640 1 -- 172.21.15.19:0/2368691699 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7440018030 con 0x7f7438004800 2024-08-31T12:15:32.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.857+0000 7f7456825640 1 -- 172.21.15.19:0/2368691699 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f745015e580 con 0x7f7438004800 2024-08-31T12:15:32.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.857+0000 7f7456825640 1 -- 172.21.15.19:0/2368691699 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f74501f7220 con 0x7f7438004800 2024-08-31T12:15:32.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.857+0000 7f744cff9640 1 -- 172.21.15.19:0/2368691699 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7440005020 con 0x7f7438004800 2024-08-31T12:15:32.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.857+0000 7f744cff9640 1 -- 172.21.15.19:0/2368691699 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7440005450 con 0x7f7438004800 2024-08-31T12:15:32.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.859+0000 7f744cff9640 1 -- 172.21.15.19:0/2368691699 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f744001b070 con 0x7f7438004800 2024-08-31T12:15:32.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.859+0000 7f744cff9640 1 -- 172.21.15.19:0/2368691699 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f744001d410 con 0x7f7438004800 2024-08-31T12:15:32.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.859+0000 7febace32640 1 -- 172.21.15.19:0/3036730349 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feba80c4520 msgr2=0x7feba8140f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.859+0000 7febace32640 1 --2- 172.21.15.19:0/3036730349 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feba80c4520 0x7feba8140f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.859+0000 7febace32640 1 -- 172.21.15.19:0/3036730349 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7feba8157ab0 msgr2=0x7feba80ca260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.859+0000 7febace32640 1 --2- 172.21.15.19:0/3036730349 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7feba8157ab0 0x7feba80ca260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.859+0000 7febace32640 1 -- 172.21.15.19:0/3036730349 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feba814e740 con 0x7feba8073510 2024-08-31T12:15:32.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.859+0000 7faff8162640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faff00b63a0 0x7faff015b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.860+0000 7febace32640 1 --2- 172.21.15.19:0/3036730349 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feba8073510 0x7feba81422a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7feb9c00c7a0 tx=0x7feb9c00cc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:32.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.860+0000 7f715e7fc640 1 -- 172.21.15.19:0/809483613 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f71580074a0 con 0x7f71600d0480 2024-08-31T12:15:32.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.860+0000 7feba57fa640 1 -- 172.21.15.19:0/3036730349 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb9c017070 con 0x7feba8073510 2024-08-31T12:15:32.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.860+0000 7feba57fa640 1 -- 172.21.15.19:0/3036730349 <== mon.1 v2:172.21.15.43:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feb9c0035f0 con 0x7feba8073510 2024-08-31T12:15:32.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.860+0000 7feba57fa640 1 -- 172.21.15.19:0/3036730349 <== mon.1 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb9c003a00 con 0x7feba8073510 2024-08-31T12:15:32.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.860+0000 7f7456825640 1 -- 172.21.15.19:0/2368691699 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7430005290 con 0x7f7438004800 2024-08-31T12:15:32.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.860+0000 7faff5efe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faff0074590 0x7faff00cff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:32.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.862+0000 7faff5efe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faff0074590 0x7faff00cff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.19:43292/0 (socket says 172.21.15.19:43292) 2024-08-31T12:15:32.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.862+0000 7faff5efe640 1 -- 172.21.15.19:0/2708612165 learned_addr learned my addr 172.21.15.19:0/2708612165 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:15:32.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.862+0000 7faff8162640 1 --2- 172.21.15.19:0/2708612165 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faff00c7130 0x7faff00d0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.862+0000 7faff8162640 1 -- 172.21.15.19:0/2708612165 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7faff00c61a0 con 0x7faff00c7130 2024-08-31T12:15:32.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.862+0000 7faff8162640 1 -- 172.21.15.19:0/2708612165 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7faff00c5ec0 con 0x7faff00b63a0 2024-08-31T12:15:32.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.862+0000 7faff8162640 1 -- 172.21.15.19:0/2708612165 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7faff00749a0 con 0x7faff0074590 2024-08-31T12:15:32.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.857+0000 7fd841d2b640 1 -- start start 2024-08-31T12:15:32.871 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.863+0000 7f744cff9640 1 -- 172.21.15.19:0/2368691699 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f7440011070 con 0x7f7438004800 2024-08-31T12:15:32.871 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.864+0000 7febaf096640 1 -- 172.21.15.19:0/3036730349 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feba8156380 con 0x7feba8073510 2024-08-31T12:15:32.871 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.864+0000 7febaf096640 1 -- 172.21.15.19:0/3036730349 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feba813f2f0 con 0x7feba8073510 2024-08-31T12:15:32.871 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.864+0000 7feba57fa640 1 -- 172.21.15.19:0/3036730349 <== mon.1 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7feb9c01a070 con 0x7feba8073510 2024-08-31T12:15:32.871 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.865+0000 7feba57fa640 1 -- 172.21.15.19:0/3036730349 <== mon.1 v2:172.21.15.43:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7feb9c01c780 con 0x7feba8073510 2024-08-31T12:15:32.871 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.865+0000 7febaf096640 1 -- 172.21.15.19:0/3036730349 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feba813f4f0 con 0x7feba8073510 2024-08-31T12:15:32.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.864+0000 7fd841d2b640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd83c08c840 0x7fd83c20f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.865+0000 7fd841d2b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd83c168750 0x7fd83c160c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.865+0000 7fd841d2b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd83c169140 0x7fd83c0c57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.865+0000 7fd841d2b640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fd83c0c41b0 con 0x7fd83c168750 2024-08-31T12:15:32.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.865+0000 7fd841d2b640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd83c0c7340 con 0x7fd83c08c840 2024-08-31T12:15:32.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.865+0000 7fd841d2b640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fd83c168b60 con 0x7fd83c169140 2024-08-31T12:15:32.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.864+0000 7faff66ff640 1 --2- 172.21.15.19:0/2708612165 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faff00c7130 0x7faff00d0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:32.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.864+0000 7faff5efe640 1 -- 172.21.15.19:0/2708612165 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faff00b63a0 msgr2=0x7faff015b150 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-08-31T12:15:32.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.864+0000 7faff5efe640 1 --2- 172.21.15.19:0/2708612165 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faff00b63a0 0x7faff015b150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.864+0000 7faff5efe640 1 -- 172.21.15.19:0/2708612165 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faff00c7130 msgr2=0x7faff00d0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.864+0000 7faff5efe640 1 --2- 172.21.15.19:0/2708612165 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faff00c7130 0x7faff00d0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.864+0000 7faff5efe640 1 -- 172.21.15.19:0/2708612165 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faff015b690 con 0x7faff0074590 2024-08-31T12:15:32.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.865+0000 7faff5efe640 1 --2- 172.21.15.19:0/2708612165 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faff0074590 0x7faff00cff10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fafe8009dc0 tx=0x7fafe8004530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:32.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.865+0000 7fafe67fc640 1 -- 172.21.15.19:0/2708612165 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafe8045070 con 0x7faff0074590 2024-08-31T12:15:32.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.865+0000 7faff8162640 1 -- 172.21.15.19:0/2708612165 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faff015a3d0 con 0x7faff0074590 2024-08-31T12:15:32.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.865+0000 7faff8162640 1 -- 172.21.15.19:0/2708612165 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7faff015a950 con 0x7faff0074590 2024-08-31T12:15:32.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.866+0000 7fafe67fc640 1 -- 172.21.15.19:0/2708612165 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fafe8038740 con 0x7faff0074590 2024-08-31T12:15:32.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.866+0000 7fafe67fc640 1 -- 172.21.15.19:0/2708612165 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafe8033050 con 0x7faff0074590 2024-08-31T12:15:32.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.866+0000 7fd83affd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd83c168750 0x7fd83c160c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:32.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.866+0000 7fd83affd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd83c168750 0x7fd83c160c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:55340/0 (socket says 172.21.15.19:55340) 2024-08-31T12:15:32.875 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.866+0000 7fd83affd640 1 -- 172.21.15.19:0/1420162358 learned_addr learned my addr 172.21.15.19:0/1420162358 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:15:32.875 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.866+0000 7fd83affd640 1 -- 172.21.15.19:0/1420162358 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd83c169140 msgr2=0x7fd83c0c57b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:15:32.875 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.866+0000 7fd83affd640 1 --2- 172.21.15.19:0/1420162358 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd83c169140 0x7fd83c0c57b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.875 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.866+0000 7fd83affd640 1 -- 172.21.15.19:0/1420162358 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd83c08c840 msgr2=0x7fd83c20f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.875 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.866+0000 7fd83affd640 1 --2- 172.21.15.19:0/1420162358 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd83c08c840 0x7fd83c20f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.875 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.867+0000 7fafe67fc640 1 -- 172.21.15.19:0/2708612165 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fafe8048070 con 0x7faff0074590 2024-08-31T12:15:32.875 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.867+0000 7fafe67fc640 1 -- 172.21.15.19:0/2708612165 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fafe804a780 con 0x7faff0074590 2024-08-31T12:15:32.876 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.866+0000 7fd83affd640 1 -- 172.21.15.19:0/1420162358 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd83c0cbf90 con 0x7fd83c168750 2024-08-31T12:15:32.876 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.868+0000 7feba57fa640 1 -- 172.21.15.19:0/3036730349 <== mon.1 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7feba813f4f0 con 0x7feba8073510 2024-08-31T12:15:32.876 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.867+0000 7fd83affd640 1 --2- 172.21.15.19:0/1420162358 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd83c168750 0x7fd83c160c00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd82c00e970 tx=0x7fd82c00ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:32.876 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.868+0000 7fb060c44640 1 --2- 172.21.15.19:0/1042992775 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb05404dd30 0x7fb05400d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.877 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.868+0000 7fb060c44640 1 -- 172.21.15.19:0/1042992775 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb0540e7030 msgr2=0x7fb0540e3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.877 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.868+0000 7fd823fff640 1 -- 172.21.15.19:0/1420162358 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd82c00cb70 con 0x7fd83c168750 2024-08-31T12:15:32.877 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.868+0000 7fb060c44640 1 --2- 172.21.15.19:0/1042992775 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb0540e7030 0x7fb0540e3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.877 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.868+0000 7fb060c44640 1 -- 172.21.15.19:0/1042992775 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb054038600 con 0x7fb05404fb90 2024-08-31T12:15:32.878 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.868+0000 7fd841d2b640 1 -- 172.21.15.19:0/1420162358 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd83c0cbcf0 con 0x7fd83c168750 2024-08-31T12:15:32.878 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.868+0000 7fd841d2b640 1 -- 172.21.15.19:0/1420162358 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd83c165270 con 0x7fd83c168750 2024-08-31T12:15:32.878 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.868+0000 7fb060c44640 1 --2- 172.21.15.19:0/1042992775 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb05404fb90 0x7fb05400db00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb05c06b990 tx=0x7fb05c0ae5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:32.878 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.868+0000 7fb061c46640 1 --2- 172.21.15.19:0/1042992775 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb0540e7030 0x7fb0540e3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-31T12:15:32.879 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.869+0000 7fd823fff640 1 -- 172.21.15.19:0/1420162358 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd82c00cd10 con 0x7fd83c168750 2024-08-31T12:15:32.879 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.871+0000 7fd823fff640 1 -- 172.21.15.19:0/1420162358 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd82c005280 con 0x7fd83c168750 2024-08-31T12:15:32.879 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.868+0000 7faff8162640 1 -- 172.21.15.19:0/2708612165 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafd8005290 con 0x7faff0074590 2024-08-31T12:15:32.879 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.871+0000 7fd841d2b640 1 -- 172.21.15.19:0/1420162358 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd81c005290 con 0x7fd83c168750 2024-08-31T12:15:32.880 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.875+0000 7fb051ffb640 1 -- 172.21.15.19:0/1042992775 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb05c0b4050 con 0x7fb05404fb90 2024-08-31T12:15:32.880 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.875+0000 7fb062447640 1 -- 172.21.15.19:0/1042992775 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb054009bb0 con 0x7fb05404fb90 2024-08-31T12:15:32.880 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.875+0000 7fb062447640 1 -- 172.21.15.19:0/1042992775 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb05400af60 con 0x7fb05404fb90 2024-08-31T12:15:32.880 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.877+0000 7fb051ffb640 1 -- 172.21.15.19:0/1042992775 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb05c0b5040 con 0x7fb05404fb90 2024-08-31T12:15:32.880 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.875+0000 7fd823fff640 1 -- 172.21.15.19:0/1420162358 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd82c0047a0 con 0x7fd83c168750 2024-08-31T12:15:32.880 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.875+0000 7fd823fff640 1 -- 172.21.15.19:0/1420162358 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd82c018430 con 0x7fd83c168750 2024-08-31T12:15:32.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.874+0000 7fafe67fc640 1 -- 172.21.15.19:0/2708612165 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fafe804f080 con 0x7faff0074590 2024-08-31T12:15:32.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.878+0000 7fb051ffb640 1 -- 172.21.15.19:0/1042992775 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb05c0aebd0 con 0x7fb05404fb90 2024-08-31T12:15:32.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.878+0000 7fb0477fe640 1 -- 172.21.15.19:0/1042992775 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb03c005290 con 0x7fb05404fb90 2024-08-31T12:15:32.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.879+0000 7fd823fff640 1 -- 172.21.15.19:0/1420162358 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fd82c038860 con 0x7fd83c168750 2024-08-31T12:15:32.882 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.879+0000 7fb051ffb640 1 -- 172.21.15.19:0/1042992775 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb05c0c2020 con 0x7fb05404fb90 2024-08-31T12:15:32.882 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.879+0000 7fb051ffb640 1 -- 172.21.15.19:0/1042992775 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb05c0bb610 con 0x7fb05404fb90 2024-08-31T12:15:32.882 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.880+0000 7f3669b32640 1 -- 172.21.15.19:0/1470407277 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3664072930 msgr2=0x7f36640c5eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.882 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.880+0000 7f3669b32640 1 --2- 172.21.15.19:0/1470407277 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3664072930 0x7f36640c5eb0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3654005fd0 tx=0x7f365402a560 comp rx=0 tx=0).stop 2024-08-31T12:15:32.882 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.880+0000 7f3669b32640 1 -- 172.21.15.19:0/1470407277 shutdown_connections 2024-08-31T12:15:32.883 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.880+0000 7f3669b32640 1 --2- 172.21.15.19:0/1470407277 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3664151860 0x7f366415bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.883 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.880+0000 7f3669b32640 1 --2- 172.21.15.19:0/1470407277 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3664072930 0x7f36640c5eb0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.883 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.880+0000 7f3669b32640 1 -- 172.21.15.19:0/1470407277 >> 172.21.15.19:0/1470407277 conn(0x7f36640cc9e0 msgr2=0x7f36640c8d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:32.883 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_bal_fragment_size_max', '10000'], check_status=True, hostname='smithi019') 2024-08-31T12:15:32.884 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_bal_merge_size', '5'], check_status=True, hostname='smithi019') 2024-08-31T12:15:32.884 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.883+0000 7fb051ffb640 1 -- 172.21.15.19:0/1042992775 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fb05c0af440 con 0x7fb05404fb90 2024-08-31T12:15:32.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.888+0000 7f3669b32640 1 -- 172.21.15.19:0/1470407277 shutdown_connections 2024-08-31T12:15:32.892 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.892+0000 7f3669b32640 1 -- 172.21.15.19:0/1470407277 wait complete. 2024-08-31T12:15:32.895 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.894+0000 7f3669b32640 1 Processor -- start 2024-08-31T12:15:32.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.899+0000 7f3669b32640 1 -- start start 2024-08-31T12:15:32.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.901+0000 7f3669b32640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3664072930 0x7f366413f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.901+0000 7f3669b32640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3664151450 0x7f3664144f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.901+0000 7f3669b32640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3664151860 0x7f36641454d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:32.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.901+0000 7f3669b32640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f36640742a0 con 0x7f3664072930 2024-08-31T12:15:32.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.901+0000 7f3669b32640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3664074b90 con 0x7f3664151860 2024-08-31T12:15:32.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.901+0000 7f3669b32640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f36640c5990 con 0x7f3664151450 2024-08-31T12:15:32.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.901+0000 7f36637fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3664072930 0x7f366413f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:32.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.901+0000 7f36637fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3664072930 0x7f366413f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:55342/0 (socket says 172.21.15.19:55342) 2024-08-31T12:15:32.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.901+0000 7f36637fe640 1 -- 172.21.15.19:0/4260280992 learned_addr learned my addr 172.21.15.19:0/4260280992 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:15:32.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.902+0000 7f36637fe640 1 -- 172.21.15.19:0/4260280992 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3664151450 msgr2=0x7f3664144f90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-08-31T12:15:32.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.902+0000 7f36637fe640 1 --2- 172.21.15.19:0/4260280992 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3664151450 0x7f3664144f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.902+0000 7f36637fe640 1 -- 172.21.15.19:0/4260280992 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3664151860 msgr2=0x7f36641454d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-08-31T12:15:32.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.902+0000 7f36637fe640 1 --2- 172.21.15.19:0/4260280992 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3664151860 0x7f36641454d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.902+0000 7f36637fe640 1 -- 172.21.15.19:0/4260280992 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36641fe2a0 con 0x7f3664072930 2024-08-31T12:15:32.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.903+0000 7f36637fe640 1 --2- 172.21.15.19:0/4260280992 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3664072930 0x7f366413f880 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3654009850 tx=0x7f3654004260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:32.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.907+0000 7f364bfff640 1 -- 172.21.15.19:0/4260280992 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3654041070 con 0x7f3664072930 2024-08-31T12:15:32.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.907+0000 7f364bfff640 1 -- 172.21.15.19:0/4260280992 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3654031940 con 0x7f3664072930 2024-08-31T12:15:32.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.907+0000 7f3669b32640 1 -- 172.21.15.19:0/4260280992 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36640ca080 con 0x7f3664072930 2024-08-31T12:15:32.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.907+0000 7f3669b32640 1 -- 172.21.15.19:0/4260280992 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36641fe520 con 0x7f3664072930 2024-08-31T12:15:32.910 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.910+0000 7f364bfff640 1 -- 172.21.15.19:0/4260280992 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3654031d50 con 0x7f3664072930 2024-08-31T12:15:32.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.912+0000 7f364bfff640 1 -- 172.21.15.19:0/4260280992 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3654044070 con 0x7f3664072930 2024-08-31T12:15:32.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.912+0000 7f364bfff640 1 -- 172.21.15.19:0/4260280992 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f365403b430 con 0x7f3664072930 2024-08-31T12:15:32.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.916+0000 7f3649ffb640 1 -- 172.21.15.19:0/4260280992 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3644005290 con 0x7f3664072930 2024-08-31T12:15:32.927 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.923+0000 7f364bfff640 1 -- 172.21.15.19:0/4260280992 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f36641fe2a0 con 0x7f3664072930 2024-08-31T12:15:32.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.933+0000 7fc25db6e640 1 -- 172.21.15.19:0/1631849490 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command([{prefix=config set, name=mds_bal_merge_size}] v 0) -- 0x7fc25814b420 con 0x7fc25815a500 2024-08-31T12:15:32.939 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.938+0000 7fc24ffff640 1 -- 172.21.15.19:0/1631849490 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_merge_size}]=0 v11)=0 v11) ==== 116+0+0 (secure 0 0 0) 0x7fc2400108c0 con 0x7fc25815a500 2024-08-31T12:15:32.942 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.942+0000 7ff78568a640 1 -- 172.21.15.19:0/2515398365 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command([{prefix=config set, name=mds_bal_split_bits}] v 0) -- 0x7ff7780dc5b0 con 0x7ff7780df610 2024-08-31T12:15:32.948 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.947+0000 7fc25db6e640 1 -- 172.21.15.19:0/1631849490 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc25815a500 msgr2=0x7fc258143660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.948 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.947+0000 7fc25db6e640 1 --2- 172.21.15.19:0/1631849490 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc25815a500 0x7fc258143660 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc24000d770 tx=0x7fc24000dc30 comp rx=0 tx=0).stop 2024-08-31T12:15:32.948 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.947+0000 7fc25db6e640 1 -- 172.21.15.19:0/1631849490 shutdown_connections 2024-08-31T12:15:32.948 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.947+0000 7fc25db6e640 1 --2- 172.21.15.19:0/1631849490 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc258143ba0 0x7fc2581f4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.947+0000 7fc25db6e640 1 --2- 172.21.15.19:0/1631849490 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc25815a500 0x7fc258143660 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.947+0000 7fc25db6e640 1 --2- 172.21.15.19:0/1631849490 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc244003a50 0x7fc25814caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.947+0000 7fc25db6e640 1 -- 172.21.15.19:0/1631849490 >> 172.21.15.19:0/1631849490 conn(0x7fc258070d90 msgr2=0x7fc2580c8ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:32.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.947+0000 7fc25db6e640 1 -- 172.21.15.19:0/1631849490 shutdown_connections 2024-08-31T12:15:32.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.947+0000 7fc25db6e640 1 -- 172.21.15.19:0/1631849490 wait complete. 2024-08-31T12:15:32.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.948+0000 7ff77cff9640 1 -- 172.21.15.19:0/2515398365 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_split_bits}]=0 v12)=0 v12) ==== 116+0+0 (secure 0 0 0) 0x7ff780094cd0 con 0x7ff7780df610 2024-08-31T12:15:32.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.949+0000 7ff78568a640 1 -- 172.21.15.19:0/2515398365 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff7780df610 msgr2=0x7ff7780cd620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:32.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.949+0000 7ff78568a640 1 --2- 172.21.15.19:0/2515398365 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff7780df610 0x7ff7780cd620 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff78006c3d0 tx=0x7ff7800513f0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.949+0000 7ff78568a640 1 -- 172.21.15.19:0/2515398365 shutdown_connections 2024-08-31T12:15:32.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.949+0000 7ff78568a640 1 --2- 172.21.15.19:0/2515398365 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff7780df610 0x7ff7780cd620 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.949+0000 7ff78568a640 1 --2- 172.21.15.19:0/2515398365 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff778044d10 0x7ff7780d2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.949+0000 7ff78568a640 1 --2- 172.21.15.19:0/2515398365 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff778005c40 0x7ff778181fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:32.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.949+0000 7ff78568a640 1 -- 172.21.15.19:0/2515398365 >> 172.21.15.19:0/2515398365 conn(0x7ff77804fe20 msgr2=0x7ff7780d7390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:32.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.950+0000 7ff78568a640 1 -- 172.21.15.19:0/2515398365 shutdown_connections 2024-08-31T12:15:32.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.950+0000 7ff78568a640 1 -- 172.21.15.19:0/2515398365 wait complete. 2024-08-31T12:15:32.974 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_bal_split_bits', '3'], check_status=True, hostname='smithi019') 2024-08-31T12:15:32.974 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_bal_split_size', '100'], check_status=True, hostname='smithi019') 2024-08-31T12:15:32.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.983+0000 7fd4fd7fa640 1 -- 172.21.15.19:0/4196592595 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command([{prefix=config set, name=mds_debug_scatterstat}] v 0) -- 0x7fd51806eca0 con 0x7fd51820f190 2024-08-31T12:15:32.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.984+0000 7ffb5dffb640 1 -- 172.21.15.19:0/3691795524 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=mds_bal_split_size}] v 0) -- 0x7ffb58005850 con 0x7ffb7815e240 2024-08-31T12:15:32.991 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.990+0000 7fd4ff7fe640 1 -- 172.21.15.19:0/4196592595 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_scatterstat}]=0 v13)=0 v13) ==== 122+0+0 (secure 0 0 0) 0x7fd50c044060 con 0x7fd51820f190 2024-08-31T12:15:32.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:32.996+0000 7ffb5ffff640 1 -- 172.21.15.19:0/3691795524 <== mon.1 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_split_size}]=0 v14)=0 v14) ==== 118+0+0 (secure 0 0 0) 0x7ffb6c012db0 con 0x7ffb7815e240 2024-08-31T12:15:33.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.000+0000 7fd4fd7fa640 1 -- 172.21.15.19:0/4196592595 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd51820f190 msgr2=0x7fd51820c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:33.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.000+0000 7fd4fd7fa640 1 --2- 172.21.15.19:0/4196592595 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd51820f190 0x7fd51820c440 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd50c007440 tx=0x7fd50c00a300 comp rx=0 tx=0).stop 2024-08-31T12:15:33.001 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.000+0000 7fd4fd7fa640 1 -- 172.21.15.19:0/4196592595 shutdown_connections 2024-08-31T12:15:33.001 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.000+0000 7fd4fd7fa640 1 --2- 172.21.15.19:0/4196592595 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd51820f190 0x7fd51820c440 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.001 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.000+0000 7fd4fd7fa640 1 --2- 172.21.15.19:0/4196592595 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd5180c8950 0x7fd5181689c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.001 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.000+0000 7fd4fd7fa640 1 --2- 172.21.15.19:0/4196592595 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd51808c6b0 0x7fd5181f2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.001 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.000+0000 7fd4fd7fa640 1 -- 172.21.15.19:0/4196592595 >> 172.21.15.19:0/4196592595 conn(0x7fd518070ba0 msgr2=0x7fd518072a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:33.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.007+0000 7ffb7d98b640 1 -- 172.21.15.19:0/3691795524 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb7815e240 msgr2=0x7ffb7820a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:33.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.007+0000 7ffb7d98b640 1 --2- 172.21.15.19:0/3691795524 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb7815e240 0x7ffb7820a010 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c00c390 tx=0x7ffb6c00c850 comp rx=0 tx=0).stop 2024-08-31T12:15:33.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.007+0000 7ffb7d98b640 1 -- 172.21.15.19:0/3691795524 shutdown_connections 2024-08-31T12:15:33.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.007+0000 7ffb7d98b640 1 --2- 172.21.15.19:0/3691795524 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb7815e240 0x7ffb7820a010 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.007+0000 7ffb7d98b640 1 --2- 172.21.15.19:0/3691795524 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffb78167c80 0x7ffb781630e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.007+0000 7ffb7d98b640 1 --2- 172.21.15.19:0/3691795524 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffb7808c6b0 0x7ffb78164f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.007+0000 7ffb7d98b640 1 -- 172.21.15.19:0/3691795524 >> 172.21.15.19:0/3691795524 conn(0x7ffb78070ba0 msgr2=0x7ffb78166a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:33.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.007+0000 7fd4fd7fa640 1 -- 172.21.15.19:0/4196592595 shutdown_connections 2024-08-31T12:15:33.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.007+0000 7ffb7d98b640 1 -- 172.21.15.19:0/3691795524 shutdown_connections 2024-08-31T12:15:33.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.007+0000 7ffb7d98b640 1 -- 172.21.15.19:0/3691795524 wait complete. 2024-08-31T12:15:33.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.007+0000 7fd4fd7fa640 1 -- 172.21.15.19:0/4196592595 wait complete. 2024-08-31T12:15:33.021 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.021+0000 7fd5adffb640 1 -- 172.21.15.19:0/4035385145 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command([{prefix=config set, name=mds_debug_frag}] v 0) -- 0x7fd5a8005580 con 0x7fd5c00cbc50 2024-08-31T12:15:33.024 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_debug_frag', 'True'], check_status=True, hostname='smithi019') 2024-08-31T12:15:33.033 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.033+0000 7fd5affff640 1 -- 172.21.15.19:0/4035385145 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_frag}]=0 v15)=0 v15) ==== 115+0+0 (secure 0 0 0) 0x7fd5b8004ef0 con 0x7fd5c00cbc50 2024-08-31T12:15:33.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.054+0000 7f2aee7fc640 1 -- 172.21.15.19:0/1731406799 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=mds_verify_scatter}] v 0) -- 0x7f2adc005580 con 0x7f2af8146d80 2024-08-31T12:15:33.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.054+0000 7fd5adffb640 1 -- 172.21.15.19:0/4035385145 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd5c00cbc50 msgr2=0x7fd5c00cc060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:33.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.054+0000 7fd5adffb640 1 --2- 172.21.15.19:0/4035385145 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd5c00cbc50 0x7fd5c00cc060 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd5c0007740 tx=0x7fd5b800ac40 comp rx=0 tx=0).stop 2024-08-31T12:15:33.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.054+0000 7fd5adffb640 1 -- 172.21.15.19:0/4035385145 shutdown_connections 2024-08-31T12:15:33.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.054+0000 7fd5adffb640 1 --2- 172.21.15.19:0/4035385145 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd5c00cbc50 0x7fd5c00cc060 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.054+0000 7fd5adffb640 1 --2- 172.21.15.19:0/4035385145 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd5c00d2d70 0x7fd5c00d3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.054+0000 7fd5adffb640 1 --2- 172.21.15.19:0/4035385145 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd5c00da1a0 0x7fd5c00c7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.054+0000 7fd5adffb640 1 -- 172.21.15.19:0/4035385145 >> 172.21.15.19:0/4035385145 conn(0x7fd5c004fe20 msgr2=0x7fd5c004b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:33.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.056+0000 7fd5adffb640 1 -- 172.21.15.19:0/4035385145 shutdown_connections 2024-08-31T12:15:33.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.056+0000 7fd5adffb640 1 -- 172.21.15.19:0/4035385145 wait complete. 2024-08-31T12:15:33.067 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.066+0000 7faff8162640 1 -- 172.21.15.19:0/2708612165 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command([{prefix=config set, name=mon_warn_on_pool_no_app}] v 0) -- 0x7fafd8005580 con 0x7faff0074590 2024-08-31T12:15:33.071 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.071+0000 7fc6217fa640 1 -- 172.21.15.19:0/3011667190 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) -- 0x7fc61c005580 con 0x7fc63c08c6b0 2024-08-31T12:15:33.073 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_debug_scatterstat', 'True'], check_status=True, hostname='smithi019') 2024-08-31T12:15:33.074 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_op_complaint_time', '180'], check_status=True, hostname='smithi019') 2024-08-31T12:15:33.077 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.077+0000 7f2af57fa640 1 -- 172.21.15.19:0/1731406799 <== mon.1 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_verify_scatter}]=0 v16)=0 v16) ==== 119+0+0 (secure 0 0 0) 0x7f2af002f910 con 0x7f2af8146d80 2024-08-31T12:15:33.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.078+0000 7f2aee7fc640 1 -- 172.21.15.19:0/1731406799 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2af8146d80 msgr2=0x7f2af8147190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:33.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.078+0000 7f2aee7fc640 1 --2- 172.21.15.19:0/1731406799 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2af8146d80 0x7f2af8147190 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2af0009d60 tx=0x7f2af0039c10 comp rx=0 tx=0).stop 2024-08-31T12:15:33.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.078+0000 7f2aee7fc640 1 -- 172.21.15.19:0/1731406799 shutdown_connections 2024-08-31T12:15:33.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.078+0000 7f2aee7fc640 1 --2- 172.21.15.19:0/1731406799 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2af8146d80 0x7f2af8147190 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.078+0000 7f2aee7fc640 1 --2- 172.21.15.19:0/1731406799 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2af815b9d0 0x7f2af8074240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.078+0000 7f2aee7fc640 1 --2- 172.21.15.19:0/1731406799 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2af80c7a80 0x7f2af814f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.078+0000 7f2aee7fc640 1 -- 172.21.15.19:0/1731406799 >> 172.21.15.19:0/1731406799 conn(0x7f2af80cc9e0 msgr2=0x7f2af80c5ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:33.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.078+0000 7f2aee7fc640 1 -- 172.21.15.19:0/1731406799 shutdown_connections 2024-08-31T12:15:33.080 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.078+0000 7f2aee7fc640 1 -- 172.21.15.19:0/1731406799 wait complete. 2024-08-31T12:15:33.081 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.080+0000 7fc6237fe640 1 -- 172.21.15.19:0/3011667190 <== mon.1 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v17)=0 v17) ==== 120+0+0 (secure 0 0 0) 0x7fc62c005610 con 0x7fc63c08c6b0 2024-08-31T12:15:33.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.083+0000 7fc6217fa640 1 -- 172.21.15.19:0/3011667190 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc63c08c6b0 msgr2=0x7fc63c169550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:33.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.083+0000 7fc6217fa640 1 --2- 172.21.15.19:0/3011667190 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc63c08c6b0 0x7fc63c169550 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc62c00ea60 tx=0x7fc62c00ee60 comp rx=0 tx=0).stop 2024-08-31T12:15:33.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.083+0000 7fc6217fa640 1 -- 172.21.15.19:0/3011667190 shutdown_connections 2024-08-31T12:15:33.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.083+0000 7fc6217fa640 1 --2- 172.21.15.19:0/3011667190 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc63c0c0790 0x7fc63c15ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.083+0000 7fc6217fa640 1 --2- 172.21.15.19:0/3011667190 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc63c167c80 0x7fc63c169d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.083+0000 7fc6217fa640 1 --2- 172.21.15.19:0/3011667190 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc63c08c6b0 0x7fc63c169550 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.083+0000 7fc6217fa640 1 -- 172.21.15.19:0/3011667190 >> 172.21.15.19:0/3011667190 conn(0x7fc63c070ba0 msgr2=0x7fc63c164e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:33.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.084+0000 7fc6217fa640 1 -- 172.21.15.19:0/3011667190 shutdown_connections 2024-08-31T12:15:33.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.084+0000 7fc6217fa640 1 -- 172.21.15.19:0/3011667190 wait complete. 2024-08-31T12:15:33.096 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.095+0000 7fafe67fc640 1 -- 172.21.15.19:0/2708612165 <== mon.2 v2:172.21.15.43:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_pool_no_app}]=0 v18)=0 v18) ==== 125+0+0 (secure 0 0 0) 0x7fafe8038b10 con 0x7faff0074590 2024-08-31T12:15:33.098 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.097+0000 7faff8162640 1 -- 172.21.15.19:0/2708612165 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faff0074590 msgr2=0x7faff00cff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:33.098 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.097+0000 7faff8162640 1 --2- 172.21.15.19:0/2708612165 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faff0074590 0x7faff00cff10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fafe8009dc0 tx=0x7fafe8004530 comp rx=0 tx=0).stop 2024-08-31T12:15:33.099 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.097+0000 7faff8162640 1 -- 172.21.15.19:0/2708612165 shutdown_connections 2024-08-31T12:15:33.099 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.097+0000 7faff8162640 1 --2- 172.21.15.19:0/2708612165 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faff00c7130 0x7faff00d0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.099 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.098+0000 7faff8162640 1 --2- 172.21.15.19:0/2708612165 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faff00b63a0 0x7faff015b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.099 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.098+0000 7faff8162640 1 --2- 172.21.15.19:0/2708612165 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faff0074590 0x7faff00cff10 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.099 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.098+0000 7faff8162640 1 -- 172.21.15.19:0/2708612165 >> 172.21.15.19:0/2708612165 conn(0x7faff0069260 msgr2=0x7faff00cae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:33.099 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.098+0000 7faff8162640 1 -- 172.21.15.19:0/2708612165 shutdown_connections 2024-08-31T12:15:33.099 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.098+0000 7faff8162640 1 -- 172.21.15.19:0/2708612165 wait complete. 2024-08-31T12:15:33.106 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.106+0000 7fd841d2b640 1 -- 172.21.15.19:0/1420162358 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command([{prefix=config set, name=mon_op_complaint_time}] v 0) -- 0x7fd81c005580 con 0x7fd83c168750 2024-08-31T12:15:33.116 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.116+0000 7fd823fff640 1 -- 172.21.15.19:0/1420162358 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_op_complaint_time}]=0 v19)=0 v19) ==== 121+0+0 (secure 0 0 0) 0x7fd82c026020 con 0x7fd83c168750 2024-08-31T12:15:33.117 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.116+0000 7f716faa9640 1 -- 172.21.15.19:0/809483613 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command([{prefix=config set, name=mds_op_complaint_time}] v 0) -- 0x7f7150005580 con 0x7f71600d0480 2024-08-31T12:15:33.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.117+0000 7fd841d2b640 1 -- 172.21.15.19:0/1420162358 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd83c168750 msgr2=0x7fd83c160c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:33.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.117+0000 7fd841d2b640 1 --2- 172.21.15.19:0/1420162358 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd83c168750 0x7fd83c160c00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd82c00e970 tx=0x7fd82c00ee30 comp rx=0 tx=0).stop 2024-08-31T12:15:33.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.117+0000 7fd841d2b640 1 -- 172.21.15.19:0/1420162358 shutdown_connections 2024-08-31T12:15:33.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.117+0000 7fd841d2b640 1 --2- 172.21.15.19:0/1420162358 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd83c169140 0x7fd83c0c57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.118+0000 7fd841d2b640 1 --2- 172.21.15.19:0/1420162358 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd83c168750 0x7fd83c160c00 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.118+0000 7fd841d2b640 1 --2- 172.21.15.19:0/1420162358 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd83c08c840 0x7fd83c20f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.118+0000 7fd841d2b640 1 -- 172.21.15.19:0/1420162358 >> 172.21.15.19:0/1420162358 conn(0x7fd83c0cc9e0 msgr2=0x7fd83c15ef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:33.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.118+0000 7fd841d2b640 1 -- 172.21.15.19:0/1420162358 shutdown_connections 2024-08-31T12:15:33.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.118+0000 7fd841d2b640 1 -- 172.21.15.19:0/1420162358 wait complete. 2024-08-31T12:15:33.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.120+0000 7f715e7fc640 1 -- 172.21.15.19:0/809483613 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_op_complaint_time}]=0 v20)=0 v20) ==== 121+0+0 (secure 0 0 0) 0x7f7158003640 con 0x7f71600d0480 2024-08-31T12:15:33.123 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.122+0000 7f716faa9640 1 -- 172.21.15.19:0/809483613 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f71600d0480 msgr2=0x7f716004a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:33.123 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.122+0000 7f716faa9640 1 --2- 172.21.15.19:0/809483613 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f71600d0480 0x7f716004a4a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f715800c3f0 tx=0x7f715800c8b0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.123 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.122+0000 7f716faa9640 1 -- 172.21.15.19:0/809483613 shutdown_connections 2024-08-31T12:15:33.123 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.122+0000 7f716faa9640 1 --2- 172.21.15.19:0/809483613 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f71600d0480 0x7f716004a4a0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.123 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.122+0000 7f716faa9640 1 --2- 172.21.15.19:0/809483613 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7160044d40 0x7f716017c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.123 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.122+0000 7f716faa9640 1 --2- 172.21.15.19:0/809483613 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7160005c70 0x7f71600dbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.122+0000 7f716faa9640 1 -- 172.21.15.19:0/809483613 >> 172.21.15.19:0/809483613 conn(0x7f716004fe50 msgr2=0x7f716004ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:33.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.122+0000 7f716faa9640 1 -- 172.21.15.19:0/809483613 shutdown_connections 2024-08-31T12:15:33.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.122+0000 7f716faa9640 1 -- 172.21.15.19:0/809483613 wait complete. 2024-08-31T12:15:33.135 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.135+0000 7fb0477fe640 1 -- 172.21.15.19:0/1042992775 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command([{prefix=config set, name=mds_dir_prefetch}] v 0) -- 0x7fb03c005580 con 0x7fb05404fb90 2024-08-31T12:15:33.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.135+0000 7f7456825640 1 -- 172.21.15.19:0/2368691699 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7f7430005580 con 0x7f7438004800 2024-08-31T12:15:33.138 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_verify_scatter', 'True'], check_status=True, hostname='smithi019') 2024-08-31T12:15:33.139 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_dir_prefetch', 'True'], check_status=True, hostname='smithi019') 2024-08-31T12:15:33.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.138+0000 7fb051ffb640 1 -- 172.21.15.19:0/1042992775 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_dir_prefetch}]=0 v21)=0 v21) ==== 117+0+0 (secure 0 0 0) 0x7fb05c0aed70 con 0x7fb05404fb90 2024-08-31T12:15:33.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.139+0000 7fb0477fe640 1 -- 172.21.15.19:0/1042992775 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb05404fb90 msgr2=0x7fb05400db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:33.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.139+0000 7fb0477fe640 1 --2- 172.21.15.19:0/1042992775 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb05404fb90 0x7fb05400db00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb05c06b990 tx=0x7fb05c0ae5d0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.139+0000 7fb0477fe640 1 -- 172.21.15.19:0/1042992775 shutdown_connections 2024-08-31T12:15:33.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.139+0000 7fb0477fe640 1 --2- 172.21.15.19:0/1042992775 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb0540e7030 0x7fb0540e3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.139+0000 7fb0477fe640 1 --2- 172.21.15.19:0/1042992775 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb05404fb90 0x7fb05400db00 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.139+0000 7fb0477fe640 1 --2- 172.21.15.19:0/1042992775 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb05404dd30 0x7fb05400d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.139+0000 7fb0477fe640 1 -- 172.21.15.19:0/1042992775 >> 172.21.15.19:0/1042992775 conn(0x7fb0540543a0 msgr2=0x7fb05404ca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:33.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.140+0000 7fb0477fe640 1 -- 172.21.15.19:0/1042992775 shutdown_connections 2024-08-31T12:15:33.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.140+0000 7fb0477fe640 1 -- 172.21.15.19:0/1042992775 wait complete. 2024-08-31T12:15:33.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.142+0000 7febaf096640 1 -- 172.21.15.19:0/3036730349 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7feba80c78a0 con 0x7feba8073510 2024-08-31T12:15:33.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.143+0000 7f744cff9640 1 -- 172.21.15.19:0/2368691699 <== mon.2 v2:172.21.15.43:3301/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v22)=0 v22) ==== 121+0+0 (secure 0 0 0) 0x7f744000e630 con 0x7f7438004800 2024-08-31T12:15:33.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.144+0000 7f7456825640 1 -- 172.21.15.19:0/2368691699 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7438004800 msgr2=0x7f74502061d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:33.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.144+0000 7f7456825640 1 --2- 172.21.15.19:0/2368691699 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7438004800 0x7f74502061d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7440009780 tx=0x7f7440009c40 comp rx=0 tx=0).stop 2024-08-31T12:15:33.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.145+0000 7f7456825640 1 -- 172.21.15.19:0/2368691699 shutdown_connections 2024-08-31T12:15:33.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.145+0000 7f7456825640 1 --2- 172.21.15.19:0/2368691699 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f74501f37d0 0x7f745015e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.145+0000 7f7456825640 1 --2- 172.21.15.19:0/2368691699 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74501f33c0 0x7f745015db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.145+0000 7f7456825640 1 --2- 172.21.15.19:0/2368691699 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7438004800 0x7f74502061d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.145+0000 7f7456825640 1 -- 172.21.15.19:0/2368691699 >> 172.21.15.19:0/2368691699 conn(0x7f7450070ba0 msgr2=0x7f74501f4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:33.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.145+0000 7f7456825640 1 -- 172.21.15.19:0/2368691699 shutdown_connections 2024-08-31T12:15:33.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.145+0000 7f7456825640 1 -- 172.21.15.19:0/2368691699 wait complete. 2024-08-31T12:15:33.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.147+0000 7feba57fa640 1 -- 172.21.15.19:0/3036730349 <== mon.1 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v23)=0 v23) ==== 121+0+0 (secure 0 0 0) 0x7feb9c021020 con 0x7feba8073510 2024-08-31T12:15:33.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.148+0000 7f3649ffb640 1 -- 172.21.15.19:0/4260280992 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command([{prefix=config set, name=rados_osd_op_timeout}] v 0) -- 0x7f3644005580 con 0x7f3664072930 2024-08-31T12:15:33.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.149+0000 7febaf096640 1 -- 172.21.15.19:0/3036730349 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feba8073510 msgr2=0x7feba81422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:33.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.149+0000 7febaf096640 1 --2- 172.21.15.19:0/3036730349 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feba8073510 0x7feba81422a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7feb9c00c7a0 tx=0x7feb9c00cc60 comp rx=0 tx=0).stop 2024-08-31T12:15:33.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.149+0000 7febaf096640 1 -- 172.21.15.19:0/3036730349 shutdown_connections 2024-08-31T12:15:33.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.149+0000 7febaf096640 1 --2- 172.21.15.19:0/3036730349 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7feba8157ab0 0x7feba80ca260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.149+0000 7febaf096640 1 --2- 172.21.15.19:0/3036730349 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feba80c4520 0x7feba8140f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.151 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.149+0000 7febaf096640 1 --2- 172.21.15.19:0/3036730349 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feba8073510 0x7feba81422a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.151 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.149+0000 7febaf096640 1 -- 172.21.15.19:0/3036730349 >> 172.21.15.19:0/3036730349 conn(0x7feba80cc9e0 msgr2=0x7feba8155010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:33.151 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.149+0000 7febaf096640 1 -- 172.21.15.19:0/3036730349 shutdown_connections 2024-08-31T12:15:33.151 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.149+0000 7febaf096640 1 -- 172.21.15.19:0/3036730349 wait complete. 2024-08-31T12:15:33.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.152+0000 7f364bfff640 1 -- 172.21.15.19:0/4260280992 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v24)=0 v24) ==== 120+0+0 (secure 0 0 0) 0x7f3654046450 con 0x7f3664072930 2024-08-31T12:15:33.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.153+0000 7f3649ffb640 1 -- 172.21.15.19:0/4260280992 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3664072930 msgr2=0x7f366413f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:33.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.153+0000 7f3649ffb640 1 --2- 172.21.15.19:0/4260280992 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3664072930 0x7f366413f880 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3654009850 tx=0x7f3654004260 comp rx=0 tx=0).stop 2024-08-31T12:15:33.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.153+0000 7f3649ffb640 1 -- 172.21.15.19:0/4260280992 shutdown_connections 2024-08-31T12:15:33.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.153+0000 7f3649ffb640 1 --2- 172.21.15.19:0/4260280992 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3664151860 0x7f36641454d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.155 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.153+0000 7f3649ffb640 1 --2- 172.21.15.19:0/4260280992 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3664151450 0x7f3664144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.155 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.153+0000 7f3649ffb640 1 --2- 172.21.15.19:0/4260280992 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3664072930 0x7f366413f880 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.155 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.153+0000 7f3649ffb640 1 -- 172.21.15.19:0/4260280992 >> 172.21.15.19:0/4260280992 conn(0x7f36640cc9e0 msgr2=0x7f3664155fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:33.155 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.153+0000 7f3649ffb640 1 -- 172.21.15.19:0/4260280992 shutdown_connections 2024-08-31T12:15:33.155 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.153+0000 7f3649ffb640 1 -- 172.21.15.19:0/4260280992 wait complete. 2024-08-31T12:15:33.156 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi019') 2024-08-31T12:15:33.164 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi019') 2024-08-31T12:15:33.164 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi019') 2024-08-31T12:15:33.166 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mon', 'mon_op_complaint_time', '120'], check_status=True, hostname='smithi019') 2024-08-31T12:15:33.167 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mon', 'mon_warn_on_pool_no_app', 'False'], check_status=True, hostname='smithi019') 2024-08-31T12:15:33.167 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'osd', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi019') 2024-08-31T12:15:33.167 INFO:tasks.ceph:Setting crush tunables to default 2024-08-31T12:15:33.167 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph osd crush tunables default 2024-08-31T12:15:33.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.319+0000 7f6950fad640 1 -- 172.21.15.19:0/1049807868 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f694c073fd0 msgr2=0x7f694c0c3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:33.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.319+0000 7f6950fad640 1 --2- 172.21.15.19:0/1049807868 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f694c073fd0 0x7f694c0c3a30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f694c0c95b0 tx=0x7f694002a5f0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.319+0000 7f6950fad640 1 -- 172.21.15.19:0/1049807868 shutdown_connections 2024-08-31T12:15:33.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.319+0000 7f6950fad640 1 --2- 172.21.15.19:0/1049807868 >> v2:172.21.15.43:3300/0 conn(0x7f694c160660 0x7f694c162a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.319+0000 7f6950fad640 1 --2- 172.21.15.19:0/1049807868 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f694c073fd0 0x7f694c0c3a30 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.319+0000 7f6950fad640 1 -- 172.21.15.19:0/1049807868 >> 172.21.15.19:0/1049807868 conn(0x7f694c070d90 msgr2=0x7f694c0711a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:33.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.319+0000 7f6950fad640 1 -- 172.21.15.19:0/1049807868 shutdown_connections 2024-08-31T12:15:33.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.319+0000 7f6950fad640 1 -- 172.21.15.19:0/1049807868 wait complete. 2024-08-31T12:15:33.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.321+0000 7f6950fad640 1 Processor -- start 2024-08-31T12:15:33.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.321+0000 7f6950fad640 1 -- start start 2024-08-31T12:15:33.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.322+0000 7f6950fad640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f694c073fd0 0x7f694c0cb360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:33.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.322+0000 7f6950fad640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f694c14e860 0x7f694c15fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:33.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.322+0000 7f6950fad640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f694c160660 0x7f694c06cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:33.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.322+0000 7f6950fad640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f694c158a50 con 0x7f694c073fd0 2024-08-31T12:15:33.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.322+0000 7f6950fad640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f694c0743e0 con 0x7f694c14e860 2024-08-31T12:15:33.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.322+0000 7f6950fad640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f694c0c3510 con 0x7f694c160660 2024-08-31T12:15:33.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.322+0000 7f694a59c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f694c073fd0 0x7f694c0cb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:33.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.322+0000 7f694a59c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f694c073fd0 0x7f694c0cb360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:55360/0 (socket says 172.21.15.19:55360) 2024-08-31T12:15:33.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.322+0000 7f6949d9b640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f694c14e860 0x7f694c15fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:33.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.322+0000 7f694a59c640 1 -- 172.21.15.19:0/3933782599 learned_addr learned my addr 172.21.15.19:0/3933782599 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:15:33.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.322+0000 7f694ad9d640 1 --2- 172.21.15.19:0/3933782599 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f694c160660 0x7f694c06cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:33.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.323+0000 7f694a59c640 1 -- 172.21.15.19:0/3933782599 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f694c160660 msgr2=0x7f694c06cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:33.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.323+0000 7f694a59c640 1 --2- 172.21.15.19:0/3933782599 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f694c160660 0x7f694c06cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.323+0000 7f694a59c640 1 -- 172.21.15.19:0/3933782599 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f694c14e860 msgr2=0x7f694c15fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:33.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.323+0000 7f694a59c640 1 --2- 172.21.15.19:0/3933782599 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f694c14e860 0x7f694c15fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.323+0000 7f694a59c640 1 -- 172.21.15.19:0/3933782599 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f694c0c5d70 con 0x7f694c073fd0 2024-08-31T12:15:33.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.323+0000 7f694a59c640 1 --2- 172.21.15.19:0/3933782599 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f694c073fd0 0x7f694c0cb360 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f693c00c3a0 tx=0x7f693c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:33.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.323+0000 7f693affd640 1 -- 172.21.15.19:0/3933782599 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f693c017070 con 0x7f694c073fd0 2024-08-31T12:15:33.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.323+0000 7f693affd640 1 -- 172.21.15.19:0/3933782599 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f693c0053c0 con 0x7f694c073fd0 2024-08-31T12:15:33.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.323+0000 7f6950fad640 1 -- 172.21.15.19:0/3933782599 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f694c0cb8a0 con 0x7f694c073fd0 2024-08-31T12:15:33.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.323+0000 7f693affd640 1 -- 172.21.15.19:0/3933782599 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f693c003850 con 0x7f694c073fd0 2024-08-31T12:15:33.327 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.323+0000 7f6950fad640 1 -- 172.21.15.19:0/3933782599 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f694c0c4b40 con 0x7f694c073fd0 2024-08-31T12:15:33.327 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.323+0000 7f693affd640 1 -- 172.21.15.19:0/3933782599 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f693c01a070 con 0x7f694c073fd0 2024-08-31T12:15:33.327 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.323+0000 7f693affd640 1 -- 172.21.15.19:0/3933782599 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f693c01c780 con 0x7f694c073fd0 2024-08-31T12:15:33.327 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.324+0000 7f6950fad640 1 -- 172.21.15.19:0/3933782599 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f694c157a60 con 0x7f694c073fd0 2024-08-31T12:15:33.327 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.326+0000 7f693affd640 1 -- 172.21.15.19:0/3933782599 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f694c157a60 con 0x7f694c073fd0 2024-08-31T12:15:33.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.392+0000 7f6950fad640 1 -- 172.21.15.19:0/3933782599 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f694c0c8470 con 0x7f694c073fd0 2024-08-31T12:15:33.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.394+0000 7f693affd640 1 -- 172.21.15.19:0/3933782599 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f693c021020 con 0x7f694c073fd0 2024-08-31T12:15:33.395 INFO:teuthology.orchestra.run.smithi019.stderr:adjusted tunables profile to default 2024-08-31T12:15:33.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.395+0000 7f6950fad640 1 -- 172.21.15.19:0/3933782599 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f694c073fd0 msgr2=0x7f694c0cb360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:33.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.395+0000 7f6950fad640 1 --2- 172.21.15.19:0/3933782599 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f694c073fd0 0x7f694c0cb360 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f693c00c3a0 tx=0x7f693c00c860 comp rx=0 tx=0).stop 2024-08-31T12:15:33.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.395+0000 7f6950fad640 1 -- 172.21.15.19:0/3933782599 shutdown_connections 2024-08-31T12:15:33.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.395+0000 7f6950fad640 1 --2- 172.21.15.19:0/3933782599 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f694c160660 0x7f694c06cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.395+0000 7f6950fad640 1 --2- 172.21.15.19:0/3933782599 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f694c14e860 0x7f694c15fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.395+0000 7f6950fad640 1 --2- 172.21.15.19:0/3933782599 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f694c073fd0 0x7f694c0cb360 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:33.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.395+0000 7f6950fad640 1 -- 172.21.15.19:0/3933782599 >> 172.21.15.19:0/3933782599 conn(0x7f694c070d90 msgr2=0x7f694c14bba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:33.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.395+0000 7f6950fad640 1 -- 172.21.15.19:0/3933782599 shutdown_connections 2024-08-31T12:15:33.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:33.395+0000 7f6950fad640 1 -- 172.21.15.19:0/3933782599 wait complete. 2024-08-31T12:15:33.407 INFO:tasks.ceph:check_enable_crimson: False 2024-08-31T12:15:33.407 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-08-31T12:15:33.407 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-08-31T12:15:33.407 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-08-31T12:15:33.431 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-08-31T12:15:33.431 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-08-31T12:15:33.496 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-08-31T12:15:33.496 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-08-31T12:15:33.560 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-08-31T12:15:33.561 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-08-31T12:15:33.625 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-08-31T12:15:33.625 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-08-31T12:15:33.650 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-08-31T12:15:33.650 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-08-31T12:15:33.714 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-08-31T12:15:33.714 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-08-31T12:15:33.781 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-08-31T12:15:33.781 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-08-31T12:15:33.846 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --cluster ceph osd new b80d6a3f-7e4e-4d4d-b1b4-0df71c9176ae 0 2024-08-31T12:15:34.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:33.998+0000 7f727ac27640 1 -- 172.21.15.43:0/2065283052 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f72741520c0 msgr2=0x7f727415b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:34.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:33.998+0000 7f727ac27640 1 --2- 172.21.15.43:0/2065283052 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f72741520c0 0x7f727415b270 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7268009860 tx=0x7f726802a620 comp rx=0 tx=0).stop 2024-08-31T12:15:34.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:33.999+0000 7f727ac27640 1 -- 172.21.15.43:0/2065283052 shutdown_connections 2024-08-31T12:15:34.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:33.999+0000 7f727ac27640 1 --2- 172.21.15.43:0/2065283052 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f72741520c0 0x7f727415b270 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:34.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:33.999+0000 7f727ac27640 1 -- 172.21.15.43:0/2065283052 >> 172.21.15.43:0/2065283052 conn(0x7f7274071a00 msgr2=0x7f7274071e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:34.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:33.999+0000 7f727ac27640 1 -- 172.21.15.43:0/2065283052 shutdown_connections 2024-08-31T12:15:34.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:33.999+0000 7f727ac27640 1 -- 172.21.15.43:0/2065283052 wait complete. 2024-08-31T12:15:34.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.000+0000 7f727ac27640 1 Processor -- start 2024-08-31T12:15:34.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.000+0000 7f727ac27640 1 -- start start 2024-08-31T12:15:34.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.002+0000 7f727ac27640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f72740c7520 0x7f72741e45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:34.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.002+0000 7f727ac27640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f72741410f0 0x7f7274154930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:34.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.002+0000 7f727ac27640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f72741520c0 0x7f72740635a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:34.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.002+0000 7f727ac27640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f72740c7000 con 0x7f72740c7520 2024-08-31T12:15:34.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.002+0000 7f727ac27640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f72740c6d20 con 0x7f72741520c0 2024-08-31T12:15:34.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.002+0000 7f727ac27640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f72740c6a20 con 0x7f72741410f0 2024-08-31T12:15:34.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.002+0000 7f7273fff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f72741410f0 0x7f7274154930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:34.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.002+0000 7f72791c4640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f72741520c0 0x7f72740635a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:34.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.002+0000 7f7273fff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f72741410f0 0x7f7274154930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:45812/0 (socket says 172.21.15.43:45812) 2024-08-31T12:15:34.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.002+0000 7f72791c4640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f72741520c0 0x7f72740635a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48024/0 (socket says 172.21.15.43:48024) 2024-08-31T12:15:34.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.002+0000 7f72789c3640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f72740c7520 0x7f72741e45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:34.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.002+0000 7f7273fff640 1 -- 172.21.15.43:0/485997638 learned_addr learned my addr 172.21.15.43:0/485997638 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-08-31T12:15:34.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.002+0000 7f7273fff640 1 -- 172.21.15.43:0/485997638 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f72741520c0 msgr2=0x7f72740635a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:34.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.002+0000 7f7273fff640 1 --2- 172.21.15.43:0/485997638 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f72741520c0 0x7f72740635a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:34.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.002+0000 7f7273fff640 1 -- 172.21.15.43:0/485997638 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f72740c7520 msgr2=0x7f72741e45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:34.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.002+0000 7f7273fff640 1 --2- 172.21.15.43:0/485997638 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f72740c7520 0x7f72741e45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:34.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.002+0000 7f7273fff640 1 -- 172.21.15.43:0/485997638 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72741e3e20 con 0x7f72741410f0 2024-08-31T12:15:34.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.002+0000 7f72791c4640 1 --2- 172.21.15.43:0/485997638 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f72741520c0 0x7f72740635a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:15:34.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.002+0000 7f7273fff640 1 --2- 172.21.15.43:0/485997638 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f72741410f0 0x7f7274154930 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f726000c460 tx=0x7f726000c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:34.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.003+0000 7f72717fa640 1 -- 172.21.15.43:0/485997638 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72600052d0 con 0x7f72741410f0 2024-08-31T12:15:34.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.003+0000 7f72717fa640 1 -- 172.21.15.43:0/485997638 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7260005470 con 0x7f72741410f0 2024-08-31T12:15:34.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.003+0000 7f72717fa640 1 -- 172.21.15.43:0/485997638 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7260012890 con 0x7f72741410f0 2024-08-31T12:15:34.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.003+0000 7f727ac27640 1 -- 172.21.15.43:0/485997638 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72741e4af0 con 0x7f72741410f0 2024-08-31T12:15:34.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.003+0000 7f727ac27640 1 -- 172.21.15.43:0/485997638 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f727415ac80 con 0x7f72741410f0 2024-08-31T12:15:34.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.003+0000 7f72717fa640 1 -- 172.21.15.43:0/485997638 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7260012a30 con 0x7f72741410f0 2024-08-31T12:15:34.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.003+0000 7f72717fa640 1 -- 172.21.15.43:0/485997638 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f726001b8f0 con 0x7f72741410f0 2024-08-31T12:15:34.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.004+0000 7f727ac27640 1 -- 172.21.15.43:0/485997638 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72740c5330 con 0x7f72741410f0 2024-08-31T12:15:34.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.006+0000 7f72717fa640 1 -- 172.21.15.43:0/485997638 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f72740c7930 con 0x7f72741410f0 2024-08-31T12:15:34.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.078+0000 7f727ac27640 1 -- 172.21.15.43:0/485997638 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "osd new", "uuid": "b80d6a3f-7e4e-4d4d-b1b4-0df71c9176ae", "id": 0} v 0) -- 0x7f72740c8870 con 0x7f72741410f0 2024-08-31T12:15:34.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.082+0000 7f72717fa640 1 -- 172.21.15.43:0/485997638 <== mon.2 v2:172.21.15.43:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b80d6a3f-7e4e-4d4d-b1b4-0df71c9176ae", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f726001bb40 con 0x7f72741410f0 2024-08-31T12:15:34.084 INFO:teuthology.orchestra.run.smithi043.stdout:0 2024-08-31T12:15:34.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.083+0000 7f727ac27640 1 -- 172.21.15.43:0/485997638 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f72741410f0 msgr2=0x7f7274154930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:34.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.083+0000 7f727ac27640 1 --2- 172.21.15.43:0/485997638 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f72741410f0 0x7f7274154930 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f726000c460 tx=0x7f726000c920 comp rx=0 tx=0).stop 2024-08-31T12:15:34.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.083+0000 7f727ac27640 1 -- 172.21.15.43:0/485997638 shutdown_connections 2024-08-31T12:15:34.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.084+0000 7f727ac27640 1 --2- 172.21.15.43:0/485997638 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f72741520c0 0x7f72740635a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:34.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.084+0000 7f727ac27640 1 --2- 172.21.15.43:0/485997638 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f72741410f0 0x7f7274154930 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:34.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.084+0000 7f727ac27640 1 --2- 172.21.15.43:0/485997638 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f72740c7520 0x7f72741e45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:34.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.084+0000 7f727ac27640 1 -- 172.21.15.43:0/485997638 >> 172.21.15.43:0/485997638 conn(0x7f7274071a00 msgr2=0x7f727413fff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:34.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.084+0000 7f727ac27640 1 -- 172.21.15.43:0/485997638 shutdown_connections 2024-08-31T12:15:34.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.084+0000 7f727ac27640 1 -- 172.21.15.43:0/485997638 wait complete. 2024-08-31T12:15:34.096 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --cluster ceph osd new 6f986366-82a6-4133-9f2e-43f84c7b03bc 1 2024-08-31T12:15:34.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.208+0000 7f0273ea9640 1 -- 172.21.15.43:0/729806991 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f026c1520c0 msgr2=0x7f026c15b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:34.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.208+0000 7f0273ea9640 1 --2- 172.21.15.43:0/729806991 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f026c1520c0 0x7f026c15b270 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f026c072610 tx=0x7f026802a5f0 comp rx=0 tx=0).stop 2024-08-31T12:15:34.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.208+0000 7f0273ea9640 1 -- 172.21.15.43:0/729806991 shutdown_connections 2024-08-31T12:15:34.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.208+0000 7f0273ea9640 1 --2- 172.21.15.43:0/729806991 >> v2:172.21.15.19:3300/0 conn(0x7f026c0cd3b0 0x7f026c0cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:34.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.208+0000 7f0273ea9640 1 --2- 172.21.15.43:0/729806991 >> v2:172.21.15.43:3300/0 conn(0x7f026c1410f0 0x7f026c0c7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:34.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.208+0000 7f0273ea9640 1 --2- 172.21.15.43:0/729806991 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f026c1520c0 0x7f026c15b270 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:34.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.208+0000 7f0273ea9640 1 -- 172.21.15.43:0/729806991 >> 172.21.15.43:0/729806991 conn(0x7f026c071a00 msgr2=0x7f026c071e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:34.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.208+0000 7f0273ea9640 1 -- 172.21.15.43:0/729806991 shutdown_connections 2024-08-31T12:15:34.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.208+0000 7f0273ea9640 1 -- 172.21.15.43:0/729806991 wait complete. 2024-08-31T12:15:34.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.210+0000 7f0273ea9640 1 Processor -- start 2024-08-31T12:15:34.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.210+0000 7f0273ea9640 1 -- start start 2024-08-31T12:15:34.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.211+0000 7f0273ea9640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f026c0cd3b0 0x7f026c1e1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:34.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.211+0000 7f0273ea9640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f026c1410f0 0x7f026c13d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:34.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.211+0000 7f0273ea9640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f026c1520c0 0x7f026c154e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:34.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.211+0000 7f0273ea9640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f026c0c6a20 con 0x7f026c1520c0 2024-08-31T12:15:34.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.211+0000 7f0273ea9640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f026c0c7000 con 0x7f026c1410f0 2024-08-31T12:15:34.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.211+0000 7f0273ea9640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f026c0c5330 con 0x7f026c0cd3b0 2024-08-31T12:15:34.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.211+0000 7f0271c45640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f026c0cd3b0 0x7f026c1e1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:34.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.211+0000 7f0271444640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f026c1410f0 0x7f026c13d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:34.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.211+0000 7f0271c45640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f026c0cd3b0 0x7f026c1e1060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:45834/0 (socket says 172.21.15.43:45834) 2024-08-31T12:15:34.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.211+0000 7f0271c45640 1 -- 172.21.15.43:0/3858418458 learned_addr learned my addr 172.21.15.43:0/3858418458 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-08-31T12:15:34.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.211+0000 7f0272446640 1 --2- 172.21.15.43:0/3858418458 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f026c1520c0 0x7f026c154e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:34.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.211+0000 7f0271c45640 1 -- 172.21.15.43:0/3858418458 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f026c1410f0 msgr2=0x7f026c13d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:34.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.211+0000 7f0271c45640 1 --2- 172.21.15.43:0/3858418458 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f026c1410f0 0x7f026c13d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:34.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.211+0000 7f0271c45640 1 -- 172.21.15.43:0/3858418458 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f026c1520c0 msgr2=0x7f026c154e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:34.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.211+0000 7f0271c45640 1 --2- 172.21.15.43:0/3858418458 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f026c1520c0 0x7f026c154e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:34.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.211+0000 7f0271c45640 1 -- 172.21.15.43:0/3858418458 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f026c134d60 con 0x7f026c0cd3b0 2024-08-31T12:15:34.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.212+0000 7f0271c45640 1 --2- 172.21.15.43:0/3858418458 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f026c0cd3b0 0x7f026c1e1060 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0268034040 tx=0x7f02680038a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:34.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.212+0000 7f02627fc640 1 -- 172.21.15.43:0/3858418458 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f026803e030 con 0x7f026c0cd3b0 2024-08-31T12:15:34.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.212+0000 7f02627fc640 1 -- 172.21.15.43:0/3858418458 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0268003ca0 con 0x7f026c0cd3b0 2024-08-31T12:15:34.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.212+0000 7f0273ea9640 1 -- 172.21.15.43:0/3858418458 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f026c1e19d0 con 0x7f026c0cd3b0 2024-08-31T12:15:34.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.212+0000 7f02627fc640 1 -- 172.21.15.43:0/3858418458 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0268035040 con 0x7f026c0cd3b0 2024-08-31T12:15:34.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.212+0000 7f0273ea9640 1 -- 172.21.15.43:0/3858418458 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f026c1eeca0 con 0x7f026c0cd3b0 2024-08-31T12:15:34.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.212+0000 7f02627fc640 1 -- 172.21.15.43:0/3858418458 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0268041030 con 0x7f026c0cd3b0 2024-08-31T12:15:34.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.212+0000 7f02627fc640 1 -- 172.21.15.43:0/3858418458 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f0268043b60 con 0x7f026c0cd3b0 2024-08-31T12:15:34.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.213+0000 7f0273ea9640 1 -- 172.21.15.43:0/3858418458 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f026c0c8870 con 0x7f026c0cd3b0 2024-08-31T12:15:34.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.215+0000 7f02627fc640 1 -- 172.21.15.43:0/3858418458 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f0268043db0 con 0x7f026c0cd3b0 2024-08-31T12:15:34.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.287+0000 7f0273ea9640 1 -- 172.21.15.43:0/3858418458 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "osd new", "uuid": "6f986366-82a6-4133-9f2e-43f84c7b03bc", "id": 1} v 0) -- 0x7f026c0c90f0 con 0x7f026c0cd3b0 2024-08-31T12:15:34.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.291+0000 7f02627fc640 1 -- 172.21.15.43:0/3858418458 <== mon.2 v2:172.21.15.43:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6f986366-82a6-4133-9f2e-43f84c7b03bc", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f0268004ee0 con 0x7f026c0cd3b0 2024-08-31T12:15:34.293 INFO:teuthology.orchestra.run.smithi043.stdout:1 2024-08-31T12:15:34.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.292+0000 7f0273ea9640 1 -- 172.21.15.43:0/3858418458 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f026c0cd3b0 msgr2=0x7f026c1e1060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:34.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.292+0000 7f0273ea9640 1 --2- 172.21.15.43:0/3858418458 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f026c0cd3b0 0x7f026c1e1060 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0268034040 tx=0x7f02680038a0 comp rx=0 tx=0).stop 2024-08-31T12:15:34.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.292+0000 7f0273ea9640 1 -- 172.21.15.43:0/3858418458 shutdown_connections 2024-08-31T12:15:34.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.292+0000 7f0273ea9640 1 --2- 172.21.15.43:0/3858418458 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f026c1520c0 0x7f026c154e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:34.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.292+0000 7f0273ea9640 1 --2- 172.21.15.43:0/3858418458 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f026c1410f0 0x7f026c13d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:34.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.292+0000 7f0273ea9640 1 --2- 172.21.15.43:0/3858418458 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f026c0cd3b0 0x7f026c1e1060 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:34.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.292+0000 7f0273ea9640 1 -- 172.21.15.43:0/3858418458 >> 172.21.15.43:0/3858418458 conn(0x7f026c071a00 msgr2=0x7f026c139560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:34.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.293+0000 7f0273ea9640 1 -- 172.21.15.43:0/3858418458 shutdown_connections 2024-08-31T12:15:34.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.293+0000 7f0273ea9640 1 -- 172.21.15.43:0/3858418458 wait complete. 2024-08-31T12:15:34.305 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --cluster ceph osd new ce8c2068-845c-4cbb-a4d1-077375d605ab 2 2024-08-31T12:15:34.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.417+0000 7efecd157640 1 -- 172.21.15.43:0/2043572091 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efec80c7520 msgr2=0x7efec8162600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:34.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.417+0000 7efecd157640 1 --2- 172.21.15.43:0/2043572091 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efec80c7520 0x7efec8162600 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7efec0005fd0 tx=0x7efec002ab20 comp rx=0 tx=0).stop 2024-08-31T12:15:34.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.417+0000 7efecd157640 1 -- 172.21.15.43:0/2043572091 shutdown_connections 2024-08-31T12:15:34.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.417+0000 7efecd157640 1 --2- 172.21.15.43:0/2043572091 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efec80c7520 0x7efec8162600 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:34.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.417+0000 7efecd157640 1 --2- 172.21.15.43:0/2043572091 >> v2:172.21.15.43:3300/0 conn(0x7efec81520c0 0x7efec815b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:34.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.417+0000 7efecd157640 1 -- 172.21.15.43:0/2043572091 >> 172.21.15.43:0/2043572091 conn(0x7efec8071a00 msgr2=0x7efec8071e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:34.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.417+0000 7efecd157640 1 -- 172.21.15.43:0/2043572091 shutdown_connections 2024-08-31T12:15:34.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.417+0000 7efecd157640 1 -- 172.21.15.43:0/2043572091 wait complete. 2024-08-31T12:15:34.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.419+0000 7efecd157640 1 Processor -- start 2024-08-31T12:15:34.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.419+0000 7efecd157640 1 -- start start 2024-08-31T12:15:34.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.421+0000 7efecd157640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efec80c7520 0x7efec81f0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:34.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.421+0000 7efecd157640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efec81410f0 0x7efec81f3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:34.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.421+0000 7efecd157640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efec81520c0 0x7efec80cd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:34.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.421+0000 7efecd157640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7efec80c7000 con 0x7efec81520c0 2024-08-31T12:15:34.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.421+0000 7efecd157640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7efec80c6d20 con 0x7efec81410f0 2024-08-31T12:15:34.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.421+0000 7efecd157640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7efec80c6a20 con 0x7efec80c7520 2024-08-31T12:15:34.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.421+0000 7efec6d9d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efec80c7520 0x7efec81f0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:34.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.421+0000 7efec659c640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efec81410f0 0x7efec81f3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:34.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.421+0000 7efec6d9d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efec80c7520 0x7efec81f0060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:45844/0 (socket says 172.21.15.43:45844) 2024-08-31T12:15:34.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.421+0000 7efec6d9d640 1 -- 172.21.15.43:0/4111900607 learned_addr learned my addr 172.21.15.43:0/4111900607 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-08-31T12:15:34.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.421+0000 7efec759e640 1 --2- 172.21.15.43:0/4111900607 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efec81520c0 0x7efec80cd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:34.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.421+0000 7efec6d9d640 1 -- 172.21.15.43:0/4111900607 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efec81410f0 msgr2=0x7efec81f3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:34.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.421+0000 7efec6d9d640 1 --2- 172.21.15.43:0/4111900607 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efec81410f0 0x7efec81f3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:34.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.421+0000 7efec6d9d640 1 -- 172.21.15.43:0/4111900607 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efec81520c0 msgr2=0x7efec80cd6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:34.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.421+0000 7efec6d9d640 1 --2- 172.21.15.43:0/4111900607 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efec81520c0 0x7efec80cd6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:34.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.421+0000 7efec6d9d640 1 -- 172.21.15.43:0/4111900607 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efec81f36c0 con 0x7efec80c7520 2024-08-31T12:15:34.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.421+0000 7efec6d9d640 1 --2- 172.21.15.43:0/4111900607 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efec80c7520 0x7efec81f0060 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efeb800e7f0 tx=0x7efeb800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:34.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.421+0000 7efeaf7fe640 1 -- 172.21.15.43:0/4111900607 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb800ccc0 con 0x7efec80c7520 2024-08-31T12:15:34.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.421+0000 7efeaf7fe640 1 -- 172.21.15.43:0/4111900607 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efeb8010430 con 0x7efec80c7520 2024-08-31T12:15:34.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.421+0000 7efecd157640 1 -- 172.21.15.43:0/4111900607 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efec81f2ca0 con 0x7efec80c7520 2024-08-31T12:15:34.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.421+0000 7efeaf7fe640 1 -- 172.21.15.43:0/4111900607 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb8010860 con 0x7efec80c7520 2024-08-31T12:15:34.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.422+0000 7efecd157640 1 -- 172.21.15.43:0/4111900607 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efec815b150 con 0x7efec80c7520 2024-08-31T12:15:34.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.423+0000 7efeaf7fe640 1 -- 172.21.15.43:0/4111900607 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7efeb8010a00 con 0x7efec80c7520 2024-08-31T12:15:34.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.423+0000 7efeaf7fe640 1 -- 172.21.15.43:0/4111900607 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7efeb8010cf0 con 0x7efec80c7520 2024-08-31T12:15:34.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.423+0000 7efecd157640 1 -- 172.21.15.43:0/4111900607 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efec80c8870 con 0x7efec80c7520 2024-08-31T12:15:34.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.426+0000 7efeaf7fe640 1 -- 172.21.15.43:0/4111900607 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7efeb8018bf0 con 0x7efec80c7520 2024-08-31T12:15:34.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.499+0000 7efecd157640 1 -- 172.21.15.43:0/4111900607 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "osd new", "uuid": "ce8c2068-845c-4cbb-a4d1-077375d605ab", "id": 2} v 0) -- 0x7efec813d8f0 con 0x7efec80c7520 2024-08-31T12:15:34.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.503+0000 7efeaf7fe640 1 -- 172.21.15.43:0/4111900607 <== mon.2 v2:172.21.15.43:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ce8c2068-845c-4cbb-a4d1-077375d605ab", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7efeb8018420 con 0x7efec80c7520 2024-08-31T12:15:34.505 INFO:teuthology.orchestra.run.smithi043.stdout:2 2024-08-31T12:15:34.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.504+0000 7efecd157640 1 -- 172.21.15.43:0/4111900607 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efec80c7520 msgr2=0x7efec81f0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:34.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.504+0000 7efecd157640 1 --2- 172.21.15.43:0/4111900607 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efec80c7520 0x7efec81f0060 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efeb800e7f0 tx=0x7efeb800ecb0 comp rx=0 tx=0).stop 2024-08-31T12:15:34.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.504+0000 7efecd157640 1 -- 172.21.15.43:0/4111900607 shutdown_connections 2024-08-31T12:15:34.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.504+0000 7efecd157640 1 --2- 172.21.15.43:0/4111900607 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efec81520c0 0x7efec80cd6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:34.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.504+0000 7efecd157640 1 --2- 172.21.15.43:0/4111900607 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efec81410f0 0x7efec81f3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:34.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.504+0000 7efecd157640 1 --2- 172.21.15.43:0/4111900607 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efec80c7520 0x7efec81f0060 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:34.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.504+0000 7efecd157640 1 -- 172.21.15.43:0/4111900607 >> 172.21.15.43:0/4111900607 conn(0x7efec8071a00 msgr2=0x7efec80c8a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:34.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.504+0000 7efecd157640 1 -- 172.21.15.43:0/4111900607 shutdown_connections 2024-08-31T12:15:34.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.504+0000 7efecd157640 1 -- 172.21.15.43:0/4111900607 wait complete. 2024-08-31T12:15:34.516 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --cluster ceph osd new b1cb0ef4-aa9b-421b-a94e-82de0c46e5cc 3 2024-08-31T12:15:34.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.628+0000 7f529a8bd640 1 -- 172.21.15.43:0/1539805626 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5294159410 msgr2=0x7f5294159820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:34.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.628+0000 7f529a8bd640 1 --2- 172.21.15.43:0/1539805626 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5294159410 0x7f5294159820 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f52940c9e40 tx=0x7f528802a5d0 comp rx=0 tx=0).stop 2024-08-31T12:15:34.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.628+0000 7f529a8bd640 1 -- 172.21.15.43:0/1539805626 shutdown_connections 2024-08-31T12:15:34.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.628+0000 7f529a8bd640 1 --2- 172.21.15.43:0/1539805626 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5294159410 0x7f5294159820 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:34.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.628+0000 7f529a8bd640 1 --2- 172.21.15.43:0/1539805626 >> v2:172.21.15.19:3300/0 conn(0x7f52940cdfb0 0x7f52940c3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:34.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.628+0000 7f529a8bd640 1 -- 172.21.15.43:0/1539805626 >> 172.21.15.43:0/1539805626 conn(0x7f5294071a00 msgr2=0x7f5294071e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:34.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.628+0000 7f529a8bd640 1 -- 172.21.15.43:0/1539805626 shutdown_connections 2024-08-31T12:15:34.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.628+0000 7f529a8bd640 1 -- 172.21.15.43:0/1539805626 wait complete. 2024-08-31T12:15:34.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.630+0000 7f529a8bd640 1 Processor -- start 2024-08-31T12:15:34.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.630+0000 7f529a8bd640 1 -- start start 2024-08-31T12:15:34.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.631+0000 7f529a8bd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f52940ca460 0x7f52941f0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:34.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.631+0000 7f529a8bd640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f52940cdfb0 0x7f52941e2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:34.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.631+0000 7f529a8bd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5294159410 0x7f529415b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:34.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.631+0000 7f529a8bd640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f52940c9a90 con 0x7f5294159410 2024-08-31T12:15:34.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.631+0000 7f529a8bd640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f52940c86c0 con 0x7f52940cdfb0 2024-08-31T12:15:34.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.631+0000 7f529a8bd640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f5294153330 con 0x7f52940ca460 2024-08-31T12:15:34.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.632+0000 7f52937fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f52940cdfb0 0x7f52941e2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:34.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.631+0000 7f5293fff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f52940ca460 0x7f52941f0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:34.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.632+0000 7f52937fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f52940cdfb0 0x7f52941e2ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48086/0 (socket says 172.21.15.43:48086) 2024-08-31T12:15:34.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.632+0000 7f5293fff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f52940ca460 0x7f52941f0170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:45860/0 (socket says 172.21.15.43:45860) 2024-08-31T12:15:34.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.632+0000 7f52937fe640 1 -- 172.21.15.43:0/4047231148 learned_addr learned my addr 172.21.15.43:0/4047231148 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-08-31T12:15:34.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.632+0000 7f5298e5a640 1 --2- 172.21.15.43:0/4047231148 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5294159410 0x7f529415b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:34.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.632+0000 7f5293fff640 1 -- 172.21.15.43:0/4047231148 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f52940cdfb0 msgr2=0x7f52941e2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:34.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.632+0000 7f5293fff640 1 --2- 172.21.15.43:0/4047231148 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f52940cdfb0 0x7f52941e2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:34.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.632+0000 7f5293fff640 1 -- 172.21.15.43:0/4047231148 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5294159410 msgr2=0x7f529415b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:34.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.632+0000 7f5293fff640 1 --2- 172.21.15.43:0/4047231148 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5294159410 0x7f529415b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:34.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.632+0000 7f5293fff640 1 -- 172.21.15.43:0/4047231148 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52941e33e0 con 0x7f52940ca460 2024-08-31T12:15:34.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.632+0000 7f52937fe640 1 --2- 172.21.15.43:0/4047231148 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f52940cdfb0 0x7f52941e2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:15:34.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.632+0000 7f5293fff640 1 --2- 172.21.15.43:0/4047231148 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f52940ca460 0x7f52941f0170 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f528400e6d0 tx=0x7f528400eb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:34.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.632+0000 7f5290ff9640 1 -- 172.21.15.43:0/4047231148 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f528400ca50 con 0x7f52940ca460 2024-08-31T12:15:34.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.632+0000 7f5290ff9640 1 -- 172.21.15.43:0/4047231148 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f528400cce0 con 0x7f52940ca460 2024-08-31T12:15:34.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.632+0000 7f529a8bd640 1 -- 172.21.15.43:0/4047231148 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5294143e50 con 0x7f52940ca460 2024-08-31T12:15:34.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.632+0000 7f5290ff9640 1 -- 172.21.15.43:0/4047231148 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5284010690 con 0x7f52940ca460 2024-08-31T12:15:34.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.632+0000 7f529a8bd640 1 -- 172.21.15.43:0/4047231148 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f52941e26a0 con 0x7f52940ca460 2024-08-31T12:15:34.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.633+0000 7f5290ff9640 1 -- 172.21.15.43:0/4047231148 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5284010830 con 0x7f52940ca460 2024-08-31T12:15:34.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.633+0000 7f5290ff9640 1 -- 172.21.15.43:0/4047231148 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f5284018bd0 con 0x7f52940ca460 2024-08-31T12:15:34.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.634+0000 7f529a8bd640 1 -- 172.21.15.43:0/4047231148 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5294144370 con 0x7f52940ca460 2024-08-31T12:15:34.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.636+0000 7f5290ff9640 1 -- 172.21.15.43:0/4047231148 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f5294144370 con 0x7f52940ca460 2024-08-31T12:15:34.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.708+0000 7f529a8bd640 1 -- 172.21.15.43:0/4047231148 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "osd new", "uuid": "b1cb0ef4-aa9b-421b-a94e-82de0c46e5cc", "id": 3} v 0) -- 0x7f52940c4500 con 0x7f52940ca460 2024-08-31T12:15:34.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.712+0000 7f5290ff9640 1 -- 172.21.15.43:0/4047231148 <== mon.2 v2:172.21.15.43:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b1cb0ef4-aa9b-421b-a94e-82de0c46e5cc", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f5284010c10 con 0x7f52940ca460 2024-08-31T12:15:34.713 INFO:teuthology.orchestra.run.smithi043.stdout:3 2024-08-31T12:15:34.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.713+0000 7f529a8bd640 1 -- 172.21.15.43:0/4047231148 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f52940ca460 msgr2=0x7f52941f0170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:34.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.713+0000 7f529a8bd640 1 --2- 172.21.15.43:0/4047231148 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f52940ca460 0x7f52941f0170 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f528400e6d0 tx=0x7f528400eb90 comp rx=0 tx=0).stop 2024-08-31T12:15:34.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.713+0000 7f529a8bd640 1 -- 172.21.15.43:0/4047231148 shutdown_connections 2024-08-31T12:15:34.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.713+0000 7f529a8bd640 1 --2- 172.21.15.43:0/4047231148 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5294159410 0x7f529415b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:34.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.713+0000 7f529a8bd640 1 --2- 172.21.15.43:0/4047231148 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f52940cdfb0 0x7f52941e2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:34.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.713+0000 7f529a8bd640 1 --2- 172.21.15.43:0/4047231148 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f52940ca460 0x7f52941f0170 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:34.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.713+0000 7f529a8bd640 1 -- 172.21.15.43:0/4047231148 >> 172.21.15.43:0/4047231148 conn(0x7f5294071a00 msgr2=0x7f52941485c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:34.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.713+0000 7f529a8bd640 1 -- 172.21.15.43:0/4047231148 shutdown_connections 2024-08-31T12:15:34.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.713+0000 7f529a8bd640 1 -- 172.21.15.43:0/4047231148 wait complete. 2024-08-31T12:15:34.725 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --cluster ceph osd new 825b58ec-a6da-4515-b6a5-81debe327011 4 2024-08-31T12:15:34.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.836+0000 7f6dd8eec640 1 -- 172.21.15.43:0/1745255371 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6dd40c9c70 msgr2=0x7f6dd41530e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:34.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.836+0000 7f6dd8eec640 1 --2- 172.21.15.43:0/1745255371 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6dd40c9c70 0x7f6dd41530e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6dc4009830 tx=0x7f6dc402a510 comp rx=0 tx=0).stop 2024-08-31T12:15:34.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.837+0000 7f6dd8eec640 1 -- 172.21.15.43:0/1745255371 shutdown_connections 2024-08-31T12:15:34.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.837+0000 7f6dd8eec640 1 --2- 172.21.15.43:0/1745255371 >> v2:172.21.15.19:3300/0 conn(0x7f6dd40ce580 0x7f6dd40ce990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:34.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.837+0000 7f6dd8eec640 1 --2- 172.21.15.43:0/1745255371 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6dd40c9c70 0x7f6dd41530e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:34.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.837+0000 7f6dd8eec640 1 -- 172.21.15.43:0/1745255371 >> 172.21.15.43:0/1745255371 conn(0x7f6dd4069490 msgr2=0x7f6dd4072810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:34.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.837+0000 7f6dd8eec640 1 -- 172.21.15.43:0/1745255371 shutdown_connections 2024-08-31T12:15:34.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.837+0000 7f6dd8eec640 1 -- 172.21.15.43:0/1745255371 wait complete. 2024-08-31T12:15:34.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.839+0000 7f6dd8eec640 1 Processor -- start 2024-08-31T12:15:34.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.839+0000 7f6dd8eec640 1 -- start start 2024-08-31T12:15:34.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.840+0000 7f6dd8eec640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6dd40c94a0 0x7f6dd41ef450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:34.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.840+0000 7f6dd8eec640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6dd40c9c70 0x7f6dd41f1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:34.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.840+0000 7f6dd8eec640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6dd40ce580 0x7f6dd4149290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:34.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.840+0000 7f6dd8eec640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd40c98b0 con 0x7f6dd40c9c70 2024-08-31T12:15:34.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.840+0000 7f6dd8eec640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd40b1bf0 con 0x7f6dd40c94a0 2024-08-31T12:15:34.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.840+0000 7f6dd8eec640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f6dd40c3650 con 0x7f6dd40ce580 2024-08-31T12:15:34.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.840+0000 7f6dd259c640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6dd40c94a0 0x7f6dd41ef450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:34.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.840+0000 7f6dd2d9d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6dd40ce580 0x7f6dd4149290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:34.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.840+0000 7f6dd259c640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6dd40c94a0 0x7f6dd41ef450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48098/0 (socket says 172.21.15.43:48098) 2024-08-31T12:15:34.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.840+0000 7f6dd259c640 1 -- 172.21.15.43:0/4128718329 learned_addr learned my addr 172.21.15.43:0/4128718329 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-08-31T12:15:34.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.840+0000 7f6dd2d9d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6dd40ce580 0x7f6dd4149290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:45868/0 (socket says 172.21.15.43:45868) 2024-08-31T12:15:34.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.840+0000 7f6dd1d9b640 1 --2- 172.21.15.43:0/4128718329 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6dd40c9c70 0x7f6dd41f1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:34.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.840+0000 7f6dd2d9d640 1 -- 172.21.15.43:0/4128718329 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6dd40c94a0 msgr2=0x7f6dd41ef450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:34.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.840+0000 7f6dd2d9d640 1 --2- 172.21.15.43:0/4128718329 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6dd40c94a0 0x7f6dd41ef450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:34.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.840+0000 7f6dd2d9d640 1 -- 172.21.15.43:0/4128718329 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6dd40c9c70 msgr2=0x7f6dd41f1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:34.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.840+0000 7f6dd2d9d640 1 --2- 172.21.15.43:0/4128718329 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6dd40c9c70 0x7f6dd41f1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:34.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.840+0000 7f6dd2d9d640 1 -- 172.21.15.43:0/4128718329 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dd40af2f0 con 0x7f6dd40ce580 2024-08-31T12:15:34.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.840+0000 7f6dd259c640 1 --2- 172.21.15.43:0/4128718329 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6dd40c94a0 0x7f6dd41ef450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:15:34.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.841+0000 7f6dd2d9d640 1 --2- 172.21.15.43:0/4128718329 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6dd40ce580 0x7f6dd4149290 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc005fd0 tx=0x7f6dcc00ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:34.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.841+0000 7f6dc2ffd640 1 -- 172.21.15.43:0/4128718329 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dcc00cd80 con 0x7f6dd40ce580 2024-08-31T12:15:34.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.841+0000 7f6dc2ffd640 1 -- 172.21.15.43:0/4128718329 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6dcc004b20 con 0x7f6dd40ce580 2024-08-31T12:15:34.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.841+0000 7f6dd8eec640 1 -- 172.21.15.43:0/4128718329 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dd41f4b80 con 0x7f6dd40ce580 2024-08-31T12:15:34.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.841+0000 7f6dc2ffd640 1 -- 172.21.15.43:0/4128718329 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dcc005020 con 0x7f6dd40ce580 2024-08-31T12:15:34.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.841+0000 7f6dd8eec640 1 -- 172.21.15.43:0/4128718329 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dd41f5140 con 0x7f6dd40ce580 2024-08-31T12:15:34.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.841+0000 7f6dc2ffd640 1 -- 172.21.15.43:0/4128718329 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6dcc005550 con 0x7f6dd40ce580 2024-08-31T12:15:34.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.841+0000 7f6dc2ffd640 1 -- 172.21.15.43:0/4128718329 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f6dcc018dd0 con 0x7f6dd40ce580 2024-08-31T12:15:34.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.842+0000 7f6dd8eec640 1 -- 172.21.15.43:0/4128718329 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6dd40af5d0 con 0x7f6dd40ce580 2024-08-31T12:15:34.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.844+0000 7f6dc2ffd640 1 -- 172.21.15.43:0/4128718329 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f6dcc026020 con 0x7f6dd40ce580 2024-08-31T12:15:34.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.915+0000 7f6dd8eec640 1 -- 172.21.15.43:0/4128718329 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "osd new", "uuid": "825b58ec-a6da-4515-b6a5-81debe327011", "id": 4} v 0) -- 0x7f6dd41f4990 con 0x7f6dd40ce580 2024-08-31T12:15:34.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.919+0000 7f6dc2ffd640 1 -- 172.21.15.43:0/4128718329 <== mon.2 v2:172.21.15.43:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "825b58ec-a6da-4515-b6a5-81debe327011", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f6dcc010430 con 0x7f6dd40ce580 2024-08-31T12:15:34.920 INFO:teuthology.orchestra.run.smithi043.stdout:4 2024-08-31T12:15:34.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.920+0000 7f6dd8eec640 1 -- 172.21.15.43:0/4128718329 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6dd40ce580 msgr2=0x7f6dd4149290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:34.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.920+0000 7f6dd8eec640 1 --2- 172.21.15.43:0/4128718329 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6dd40ce580 0x7f6dd4149290 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc005fd0 tx=0x7f6dcc00ee40 comp rx=0 tx=0).stop 2024-08-31T12:15:34.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.920+0000 7f6dd8eec640 1 -- 172.21.15.43:0/4128718329 shutdown_connections 2024-08-31T12:15:34.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.920+0000 7f6dd8eec640 1 --2- 172.21.15.43:0/4128718329 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6dd40ce580 0x7f6dd4149290 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:34.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.920+0000 7f6dd8eec640 1 --2- 172.21.15.43:0/4128718329 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6dd40c9c70 0x7f6dd41f1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:34.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.920+0000 7f6dd8eec640 1 --2- 172.21.15.43:0/4128718329 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6dd40c94a0 0x7f6dd41ef450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:34.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.920+0000 7f6dd8eec640 1 -- 172.21.15.43:0/4128718329 >> 172.21.15.43:0/4128718329 conn(0x7f6dd4069490 msgr2=0x7f6dd40c3910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:34.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.920+0000 7f6dd8eec640 1 -- 172.21.15.43:0/4128718329 shutdown_connections 2024-08-31T12:15:34.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:34.920+0000 7f6dd8eec640 1 -- 172.21.15.43:0/4128718329 wait complete. 2024-08-31T12:15:34.931 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --cluster ceph osd new 7f34e537-5997-40e4-92e7-2570507160c7 5 2024-08-31T12:15:35.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.043+0000 7fb5c3e15640 1 -- 172.21.15.43:0/3572354589 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5bc15c7e0 msgr2=0x7fb5bc15cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:35.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.043+0000 7fb5c3e15640 1 --2- 172.21.15.43:0/3572354589 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5bc15c7e0 0x7fb5bc15cbf0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc0cb2b0 tx=0x7fb5b802a680 comp rx=0 tx=0).stop 2024-08-31T12:15:35.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.043+0000 7fb5c3e15640 1 -- 172.21.15.43:0/3572354589 shutdown_connections 2024-08-31T12:15:35.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.043+0000 7fb5c3e15640 1 --2- 172.21.15.43:0/3572354589 >> v2:172.21.15.19:3300/0 conn(0x7fb5bc161710 0x7fb5bc163b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:35.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.043+0000 7fb5c3e15640 1 --2- 172.21.15.43:0/3572354589 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5bc15c7e0 0x7fb5bc15cbf0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:35.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.043+0000 7fb5c3e15640 1 -- 172.21.15.43:0/3572354589 >> 172.21.15.43:0/3572354589 conn(0x7fb5bc071a00 msgr2=0x7fb5bc071e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:35.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.043+0000 7fb5c3e15640 1 -- 172.21.15.43:0/3572354589 shutdown_connections 2024-08-31T12:15:35.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.043+0000 7fb5c3e15640 1 -- 172.21.15.43:0/3572354589 wait complete. 2024-08-31T12:15:35.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.046+0000 7fb5c3e15640 1 Processor -- start 2024-08-31T12:15:35.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.046+0000 7fb5c3e15640 1 -- start start 2024-08-31T12:15:35.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.048+0000 7fb5c3e15640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb5bc142120 0x7fb5bc13c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:35.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.048+0000 7fb5c3e15640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb5bc15c7e0 0x7fb5bc15b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:35.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.048+0000 7fb5c3e15640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5bc161710 0x7fb5bc133c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:35.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.048+0000 7fb5c3e15640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fb5bc0c8850 con 0x7fb5bc15c7e0 2024-08-31T12:15:35.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.048+0000 7fb5c3e15640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb5bc15d130 con 0x7fb5bc161710 2024-08-31T12:15:35.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.048+0000 7fb5c3e15640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fb5bc073ad0 con 0x7fb5bc142120 2024-08-31T12:15:35.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.048+0000 7fb5c23b2640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5bc161710 0x7fb5bc133c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:35.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.048+0000 7fb5c1bb1640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb5bc142120 0x7fb5bc13c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:35.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.048+0000 7fb5c23b2640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5bc161710 0x7fb5bc133c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48112/0 (socket says 172.21.15.43:48112) 2024-08-31T12:15:35.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.048+0000 7fb5c23b2640 1 -- 172.21.15.43:0/2467430026 learned_addr learned my addr 172.21.15.43:0/2467430026 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-08-31T12:15:35.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.048+0000 7fb5c13b0640 1 --2- 172.21.15.43:0/2467430026 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb5bc15c7e0 0x7fb5bc15b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:35.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.049+0000 7fb5c23b2640 1 -- 172.21.15.43:0/2467430026 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb5bc142120 msgr2=0x7fb5bc13c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:35.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.049+0000 7fb5c23b2640 1 --2- 172.21.15.43:0/2467430026 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb5bc142120 0x7fb5bc13c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:35.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.049+0000 7fb5c23b2640 1 -- 172.21.15.43:0/2467430026 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb5bc15c7e0 msgr2=0x7fb5bc15b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:35.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.049+0000 7fb5c23b2640 1 --2- 172.21.15.43:0/2467430026 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb5bc15c7e0 0x7fb5bc15b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:35.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.049+0000 7fb5c23b2640 1 -- 172.21.15.43:0/2467430026 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5bc13de70 con 0x7fb5bc161710 2024-08-31T12:15:35.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.049+0000 7fb5c23b2640 1 --2- 172.21.15.43:0/2467430026 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5bc161710 0x7fb5bc133c00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb5b400ea70 tx=0x7fb5b400ef30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:35.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.049+0000 7fb5ae7fc640 1 -- 172.21.15.43:0/2467430026 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5b400ce40 con 0x7fb5bc161710 2024-08-31T12:15:35.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.049+0000 7fb5ae7fc640 1 -- 172.21.15.43:0/2467430026 <== mon.1 v2:172.21.15.43:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb5b4004c10 con 0x7fb5bc161710 2024-08-31T12:15:35.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.049+0000 7fb5c3e15640 1 -- 172.21.15.43:0/2467430026 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5bc153420 con 0x7fb5bc161710 2024-08-31T12:15:35.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.049+0000 7fb5ae7fc640 1 -- 172.21.15.43:0/2467430026 <== mon.1 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5b4005020 con 0x7fb5bc161710 2024-08-31T12:15:35.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.049+0000 7fb5c3e15640 1 -- 172.21.15.43:0/2467430026 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5bc155fe0 con 0x7fb5bc161710 2024-08-31T12:15:35.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.049+0000 7fb5ae7fc640 1 -- 172.21.15.43:0/2467430026 <== mon.1 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb5b4005550 con 0x7fb5bc161710 2024-08-31T12:15:35.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.049+0000 7fb5ae7fc640 1 -- 172.21.15.43:0/2467430026 <== mon.1 v2:172.21.15.43:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7fb5b40109b0 con 0x7fb5bc161710 2024-08-31T12:15:35.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.050+0000 7fb5c3e15640 1 -- 172.21.15.43:0/2467430026 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5bc141ba0 con 0x7fb5bc161710 2024-08-31T12:15:35.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.054+0000 7fb5ae7fc640 1 -- 172.21.15.43:0/2467430026 <== mon.1 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fb5b40059d0 con 0x7fb5bc161710 2024-08-31T12:15:35.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.128+0000 7fb5c3e15640 1 -- 172.21.15.43:0/2467430026 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd new", "uuid": "7f34e537-5997-40e4-92e7-2570507160c7", "id": 5} v 0) -- 0x7fb5bc1567c0 con 0x7fb5bc161710 2024-08-31T12:15:35.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.132+0000 7fb5ae7fc640 1 -- 172.21.15.43:0/2467430026 <== mon.1 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7f34e537-5997-40e4-92e7-2570507160c7", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7fb5b4010d80 con 0x7fb5bc161710 2024-08-31T12:15:35.133 INFO:teuthology.orchestra.run.smithi043.stdout:5 2024-08-31T12:15:35.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.133+0000 7fb5c3e15640 1 -- 172.21.15.43:0/2467430026 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5bc161710 msgr2=0x7fb5bc133c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:35.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.133+0000 7fb5c3e15640 1 --2- 172.21.15.43:0/2467430026 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5bc161710 0x7fb5bc133c00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb5b400ea70 tx=0x7fb5b400ef30 comp rx=0 tx=0).stop 2024-08-31T12:15:35.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.133+0000 7fb5c3e15640 1 -- 172.21.15.43:0/2467430026 shutdown_connections 2024-08-31T12:15:35.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.133+0000 7fb5c3e15640 1 --2- 172.21.15.43:0/2467430026 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5bc161710 0x7fb5bc133c00 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:35.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.133+0000 7fb5c3e15640 1 --2- 172.21.15.43:0/2467430026 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb5bc15c7e0 0x7fb5bc15b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:35.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.133+0000 7fb5c3e15640 1 --2- 172.21.15.43:0/2467430026 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb5bc142120 0x7fb5bc13c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:35.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.133+0000 7fb5c3e15640 1 -- 172.21.15.43:0/2467430026 >> 172.21.15.43:0/2467430026 conn(0x7fb5bc071a00 msgr2=0x7fb5bc0cb300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:35.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.133+0000 7fb5c3e15640 1 -- 172.21.15.43:0/2467430026 shutdown_connections 2024-08-31T12:15:35.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.133+0000 7fb5c3e15640 1 -- 172.21.15.43:0/2467430026 wait complete. 2024-08-31T12:15:35.145 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --cluster ceph osd new 195dd8ad-640a-4fa3-9ee4-06e4172e0464 6 2024-08-31T12:15:35.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.256+0000 7f02b4453640 1 -- 172.21.15.43:0/694393438 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f02ac0c9a70 msgr2=0x7f02ac157d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:35.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.256+0000 7f02b4453640 1 --2- 172.21.15.43:0/694393438 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f02ac0c9a70 0x7f02ac157d30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f029c005fd0 tx=0x7f029c02a5d0 comp rx=0 tx=0).stop 2024-08-31T12:15:35.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.256+0000 7f02b4453640 1 -- 172.21.15.43:0/694393438 shutdown_connections 2024-08-31T12:15:35.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.256+0000 7f02b4453640 1 --2- 172.21.15.43:0/694393438 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f02ac0c9a70 0x7f02ac157d30 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:35.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.256+0000 7f02b4453640 1 --2- 172.21.15.43:0/694393438 >> v2:172.21.15.19:3300/0 conn(0x7f02ac152170 0x7f02ac0c9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:35.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.256+0000 7f02b4453640 1 -- 172.21.15.43:0/694393438 >> 172.21.15.43:0/694393438 conn(0x7f02ac070d90 msgr2=0x7f02ac0711a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:35.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.256+0000 7f02b4453640 1 -- 172.21.15.43:0/694393438 shutdown_connections 2024-08-31T12:15:35.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.256+0000 7f02b4453640 1 -- 172.21.15.43:0/694393438 wait complete. 2024-08-31T12:15:35.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.258+0000 7f02b4453640 1 Processor -- start 2024-08-31T12:15:35.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.258+0000 7f02b4453640 1 -- start start 2024-08-31T12:15:35.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.259+0000 7f02b4453640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f02ac0c9a70 0x7f02ac13fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:35.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.259+0000 7f02b4453640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f02ac152170 0x7f02ac1f2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:35.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.259+0000 7f02b4453640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f02ac158270 0x7f02ac1470b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:35.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.259+0000 7f02b4453640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f02ac0c9e80 con 0x7f02ac158270 2024-08-31T12:15:35.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.259+0000 7f02b4453640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f02ac074de0 con 0x7f02ac152170 2024-08-31T12:15:35.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.259+0000 7f02b4453640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f02ac074a70 con 0x7f02ac0c9a70 2024-08-31T12:15:35.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.259+0000 7f02b21ef640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f02ac0c9a70 0x7f02ac13fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:35.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.259+0000 7f02b19ee640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f02ac152170 0x7f02ac1f2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:35.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.259+0000 7f02b21ef640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f02ac0c9a70 0x7f02ac13fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:45888/0 (socket says 172.21.15.43:45888) 2024-08-31T12:15:35.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.259+0000 7f02b21ef640 1 -- 172.21.15.43:0/3748482002 learned_addr learned my addr 172.21.15.43:0/3748482002 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-08-31T12:15:35.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.260+0000 7f02b29f0640 1 --2- 172.21.15.43:0/3748482002 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f02ac158270 0x7f02ac1470b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:35.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.260+0000 7f02b21ef640 1 -- 172.21.15.43:0/3748482002 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f02ac152170 msgr2=0x7f02ac1f2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:35.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.260+0000 7f02b21ef640 1 --2- 172.21.15.43:0/3748482002 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f02ac152170 0x7f02ac1f2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:35.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.260+0000 7f02b21ef640 1 -- 172.21.15.43:0/3748482002 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f02ac158270 msgr2=0x7f02ac1470b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:35.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.260+0000 7f02b21ef640 1 --2- 172.21.15.43:0/3748482002 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f02ac158270 0x7f02ac1470b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:35.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.260+0000 7f02b21ef640 1 -- 172.21.15.43:0/3748482002 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02ac14be60 con 0x7f02ac0c9a70 2024-08-31T12:15:35.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.260+0000 7f02b19ee640 1 --2- 172.21.15.43:0/3748482002 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f02ac152170 0x7f02ac1f2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:15:35.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.260+0000 7f02b29f0640 1 --2- 172.21.15.43:0/3748482002 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f02ac158270 0x7f02ac1470b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:15:35.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.260+0000 7f02b21ef640 1 --2- 172.21.15.43:0/3748482002 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f02ac0c9a70 0x7f02ac13fce0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f02a800e790 tx=0x7f02a800ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:35.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.260+0000 7f02a2ffd640 1 -- 172.21.15.43:0/3748482002 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a800eea0 con 0x7f02ac0c9a70 2024-08-31T12:15:35.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.260+0000 7f02a2ffd640 1 -- 172.21.15.43:0/3748482002 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f02a800cdb0 con 0x7f02ac0c9a70 2024-08-31T12:15:35.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.260+0000 7f02a2ffd640 1 -- 172.21.15.43:0/3748482002 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a8010690 con 0x7f02ac0c9a70 2024-08-31T12:15:35.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.260+0000 7f02b4453640 1 -- 172.21.15.43:0/3748482002 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02ac1f3350 con 0x7f02ac0c9a70 2024-08-31T12:15:35.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.260+0000 7f02b4453640 1 -- 172.21.15.43:0/3748482002 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f02ac1f04e0 con 0x7f02ac0c9a70 2024-08-31T12:15:35.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.260+0000 7f02a2ffd640 1 -- 172.21.15.43:0/3748482002 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f02a8010830 con 0x7f02ac0c9a70 2024-08-31T12:15:35.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.260+0000 7f02a2ffd640 1 -- 172.21.15.43:0/3748482002 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f02a8018c40 con 0x7f02ac0c9a70 2024-08-31T12:15:35.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.262+0000 7f02b4453640 1 -- 172.21.15.43:0/3748482002 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02ac0c1b60 con 0x7f02ac0c9a70 2024-08-31T12:15:35.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.264+0000 7f02a2ffd640 1 -- 172.21.15.43:0/3748482002 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f02a8010d00 con 0x7f02ac0c9a70 2024-08-31T12:15:35.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.335+0000 7f02b4453640 1 -- 172.21.15.43:0/3748482002 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "osd new", "uuid": "195dd8ad-640a-4fa3-9ee4-06e4172e0464", "id": 6} v 0) -- 0x7f02ac149140 con 0x7f02ac0c9a70 2024-08-31T12:15:35.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.339+0000 7f02a2ffd640 1 -- 172.21.15.43:0/3748482002 <== mon.2 v2:172.21.15.43:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "195dd8ad-640a-4fa3-9ee4-06e4172e0464", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f02a8018420 con 0x7f02ac0c9a70 2024-08-31T12:15:35.340 INFO:teuthology.orchestra.run.smithi043.stdout:6 2024-08-31T12:15:35.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.340+0000 7f02b4453640 1 -- 172.21.15.43:0/3748482002 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f02ac0c9a70 msgr2=0x7f02ac13fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:35.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.340+0000 7f02b4453640 1 --2- 172.21.15.43:0/3748482002 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f02ac0c9a70 0x7f02ac13fce0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f02a800e790 tx=0x7f02a800ec50 comp rx=0 tx=0).stop 2024-08-31T12:15:35.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.340+0000 7f02b4453640 1 -- 172.21.15.43:0/3748482002 shutdown_connections 2024-08-31T12:15:35.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.340+0000 7f02b4453640 1 --2- 172.21.15.43:0/3748482002 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f02ac158270 0x7f02ac1470b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:35.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.340+0000 7f02b4453640 1 --2- 172.21.15.43:0/3748482002 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f02ac152170 0x7f02ac1f2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:35.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.340+0000 7f02b4453640 1 --2- 172.21.15.43:0/3748482002 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f02ac0c9a70 0x7f02ac13fce0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:35.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.340+0000 7f02b4453640 1 -- 172.21.15.43:0/3748482002 >> 172.21.15.43:0/3748482002 conn(0x7f02ac070d90 msgr2=0x7f02ac14c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:35.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.340+0000 7f02b4453640 1 -- 172.21.15.43:0/3748482002 shutdown_connections 2024-08-31T12:15:35.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.340+0000 7f02b4453640 1 -- 172.21.15.43:0/3748482002 wait complete. 2024-08-31T12:15:35.352 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --cluster ceph osd new 4645661f-f430-4f05-bdd1-69138dcd1bbc 7 2024-08-31T12:15:35.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.463+0000 7f4a07bad640 1 -- 172.21.15.43:0/1643791502 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4a001410f0 msgr2=0x7f4a000c7520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:35.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.463+0000 7f4a07bad640 1 --2- 172.21.15.43:0/1643791502 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4a001410f0 0x7f4a000c7520 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f49f0009830 tx=0x7f49f002a5a0 comp rx=0 tx=0).stop 2024-08-31T12:15:35.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.463+0000 7f4a07bad640 1 -- 172.21.15.43:0/1643791502 shutdown_connections 2024-08-31T12:15:35.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.463+0000 7f4a07bad640 1 --2- 172.21.15.43:0/1643791502 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4a001410f0 0x7f4a000c7520 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:35.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.463+0000 7f4a07bad640 1 --2- 172.21.15.43:0/1643791502 >> v2:172.21.15.43:3300/0 conn(0x7f4a001520c0 0x7f4a0015b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:35.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.463+0000 7f4a07bad640 1 -- 172.21.15.43:0/1643791502 >> 172.21.15.43:0/1643791502 conn(0x7f4a00071a00 msgr2=0x7f4a00071e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:35.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.463+0000 7f4a07bad640 1 -- 172.21.15.43:0/1643791502 shutdown_connections 2024-08-31T12:15:35.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.464+0000 7f4a07bad640 1 -- 172.21.15.43:0/1643791502 wait complete. 2024-08-31T12:15:35.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.465+0000 7f4a07bad640 1 Processor -- start 2024-08-31T12:15:35.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.465+0000 7f4a07bad640 1 -- start start 2024-08-31T12:15:35.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.466+0000 7f4a07bad640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4a000cd3b0 0x7f4a000c4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:35.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.466+0000 7f4a07bad640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4a001410f0 0x7f4a001e44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:35.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.466+0000 7f4a07bad640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4a001520c0 0x7f4a001e49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:35.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.466+0000 7f4a07bad640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f4a000c6a20 con 0x7f4a001520c0 2024-08-31T12:15:35.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.466+0000 7f4a07bad640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4a000c7000 con 0x7f4a000cd3b0 2024-08-31T12:15:35.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.466+0000 7f4a07bad640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f4a000c5330 con 0x7f4a001410f0 2024-08-31T12:15:35.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.467+0000 7f4a05949640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4a000cd3b0 0x7f4a000c4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:35.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.467+0000 7f4a05148640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4a001410f0 0x7f4a001e44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:35.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.467+0000 7f4a05949640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4a000cd3b0 0x7f4a000c4880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48148/0 (socket says 172.21.15.43:48148) 2024-08-31T12:15:35.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.467+0000 7f4a05949640 1 -- 172.21.15.43:0/2005995377 learned_addr learned my addr 172.21.15.43:0/2005995377 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-08-31T12:15:35.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.467+0000 7f4a05148640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4a001410f0 0x7f4a001e44a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:45904/0 (socket says 172.21.15.43:45904) 2024-08-31T12:15:35.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.467+0000 7f4a0614a640 1 --2- 172.21.15.43:0/2005995377 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4a001520c0 0x7f4a001e49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:35.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.467+0000 7f4a05949640 1 -- 172.21.15.43:0/2005995377 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4a001410f0 msgr2=0x7f4a001e44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:35.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.467+0000 7f4a05949640 1 --2- 172.21.15.43:0/2005995377 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4a001410f0 0x7f4a001e44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:35.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.467+0000 7f4a05949640 1 -- 172.21.15.43:0/2005995377 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4a001520c0 msgr2=0x7f4a001e49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:35.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.467+0000 7f4a05949640 1 --2- 172.21.15.43:0/2005995377 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4a001520c0 0x7f4a001e49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:35.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.467+0000 7f4a05949640 1 -- 172.21.15.43:0/2005995377 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a001e53a0 con 0x7f4a000cd3b0 2024-08-31T12:15:35.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.467+0000 7f4a05148640 1 --2- 172.21.15.43:0/2005995377 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4a001410f0 0x7f4a001e44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:15:35.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.467+0000 7f4a05949640 1 --2- 172.21.15.43:0/2005995377 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4a000cd3b0 0x7f4a000c4880 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f49fc00e9c0 tx=0x7f49fc00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:35.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.467+0000 7f49f67fc640 1 -- 172.21.15.43:0/2005995377 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49fc00cd80 con 0x7f4a000cd3b0 2024-08-31T12:15:35.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.467+0000 7f49f67fc640 1 -- 172.21.15.43:0/2005995377 <== mon.1 v2:172.21.15.43:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f49fc004020 con 0x7f4a000cd3b0 2024-08-31T12:15:35.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.467+0000 7f4a07bad640 1 -- 172.21.15.43:0/2005995377 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a001f0310 con 0x7f4a000cd3b0 2024-08-31T12:15:35.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.467+0000 7f49f67fc640 1 -- 172.21.15.43:0/2005995377 <== mon.1 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49fc004450 con 0x7f4a000cd3b0 2024-08-31T12:15:35.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.467+0000 7f4a07bad640 1 -- 172.21.15.43:0/2005995377 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a001ef830 con 0x7f4a000cd3b0 2024-08-31T12:15:35.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.468+0000 7f49f67fc640 1 -- 172.21.15.43:0/2005995377 <== mon.1 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f49fc0053b0 con 0x7f4a000cd3b0 2024-08-31T12:15:35.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.468+0000 7f49f67fc640 1 -- 172.21.15.43:0/2005995377 <== mon.1 v2:172.21.15.43:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f49fc003de0 con 0x7f4a000cd3b0 2024-08-31T12:15:35.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.468+0000 7f4a07bad640 1 -- 172.21.15.43:0/2005995377 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a000c8870 con 0x7f4a000cd3b0 2024-08-31T12:15:35.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.470+0000 7f49f67fc640 1 -- 172.21.15.43:0/2005995377 <== mon.1 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f49fc026080 con 0x7f4a000cd3b0 2024-08-31T12:15:35.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.542+0000 7f4a07bad640 1 -- 172.21.15.43:0/2005995377 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4645661f-f430-4f05-bdd1-69138dcd1bbc", "id": 7} v 0) -- 0x7f4a001e1c10 con 0x7f4a000cd3b0 2024-08-31T12:15:35.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.546+0000 7f49f67fc640 1 -- 172.21.15.43:0/2005995377 <== mon.1 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4645661f-f430-4f05-bdd1-69138dcd1bbc", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f49fc003780 con 0x7f4a000cd3b0 2024-08-31T12:15:35.548 INFO:teuthology.orchestra.run.smithi043.stdout:7 2024-08-31T12:15:35.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.547+0000 7f4a07bad640 1 -- 172.21.15.43:0/2005995377 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4a000cd3b0 msgr2=0x7f4a000c4880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:35.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.547+0000 7f4a07bad640 1 --2- 172.21.15.43:0/2005995377 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4a000cd3b0 0x7f4a000c4880 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f49fc00e9c0 tx=0x7f49fc00ee80 comp rx=0 tx=0).stop 2024-08-31T12:15:35.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.547+0000 7f4a07bad640 1 -- 172.21.15.43:0/2005995377 shutdown_connections 2024-08-31T12:15:35.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.547+0000 7f4a07bad640 1 --2- 172.21.15.43:0/2005995377 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4a001520c0 0x7f4a001e49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:35.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.547+0000 7f4a07bad640 1 --2- 172.21.15.43:0/2005995377 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4a001410f0 0x7f4a001e44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:35.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.547+0000 7f4a07bad640 1 --2- 172.21.15.43:0/2005995377 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4a000cd3b0 0x7f4a000c4880 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:35.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.547+0000 7f4a07bad640 1 -- 172.21.15.43:0/2005995377 >> 172.21.15.43:0/2005995377 conn(0x7f4a00071a00 msgr2=0x7f4a001e1060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:35.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.547+0000 7f4a07bad640 1 -- 172.21.15.43:0/2005995377 shutdown_connections 2024-08-31T12:15:35.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-08-31T12:15:35.547+0000 7f4a07bad640 1 -- 172.21.15.43:0/2005995377 wait complete. 2024-08-31T12:15:35.559 INFO:tasks.ceph.osd.0:Restarting daemon 2024-08-31T12:15:35.559 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-08-31T12:15:35.562 INFO:tasks.ceph.osd.0:Started 2024-08-31T12:15:35.562 INFO:tasks.ceph.osd.1:Restarting daemon 2024-08-31T12:15:35.562 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-08-31T12:15:35.564 INFO:tasks.ceph.osd.1:Started 2024-08-31T12:15:35.564 INFO:tasks.ceph.osd.2:Restarting daemon 2024-08-31T12:15:35.565 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-08-31T12:15:35.567 INFO:tasks.ceph.osd.2:Started 2024-08-31T12:15:35.567 INFO:tasks.ceph.osd.3:Restarting daemon 2024-08-31T12:15:35.567 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-08-31T12:15:35.569 INFO:tasks.ceph.osd.3:Started 2024-08-31T12:15:35.569 INFO:tasks.ceph.osd.4:Restarting daemon 2024-08-31T12:15:35.570 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-08-31T12:15:35.572 INFO:tasks.ceph.osd.4:Started 2024-08-31T12:15:35.572 INFO:tasks.ceph.osd.5:Restarting daemon 2024-08-31T12:15:35.572 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-08-31T12:15:35.574 INFO:tasks.ceph.osd.5:Started 2024-08-31T12:15:35.574 INFO:tasks.ceph.osd.6:Restarting daemon 2024-08-31T12:15:35.574 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-08-31T12:15:35.577 INFO:tasks.ceph.osd.6:Started 2024-08-31T12:15:35.577 INFO:tasks.ceph.osd.7:Restarting daemon 2024-08-31T12:15:35.577 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-08-31T12:15:35.579 INFO:tasks.ceph.osd.7:Started 2024-08-31T12:15:35.579 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-31T12:15:35.737 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.736+0000 7fb36b580640 1 -- 172.21.15.19:0/995500016 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb350003a50 msgr2=0x7fb350003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:35.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.736+0000 7fb36b580640 1 --2- 172.21.15.19:0/995500016 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb350003a50 0x7fb350003ec0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb360009b50 tx=0x7fb350005830 comp rx=0 tx=0).stop 2024-08-31T12:15:35.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.736+0000 7fb36b580640 1 -- 172.21.15.19:0/995500016 shutdown_connections 2024-08-31T12:15:35.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.736+0000 7fb36b580640 1 --2- 172.21.15.19:0/995500016 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb350003a50 0x7fb350003ec0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:35.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.736+0000 7fb36b580640 1 -- 172.21.15.19:0/995500016 >> 172.21.15.19:0/995500016 conn(0x7fb3640690d0 msgr2=0x7fb3640d0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:35.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.736+0000 7fb36b580640 1 -- 172.21.15.19:0/995500016 shutdown_connections 2024-08-31T12:15:35.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.736+0000 7fb36b580640 1 -- 172.21.15.19:0/995500016 wait complete. 2024-08-31T12:15:35.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.738+0000 7fb36b580640 1 Processor -- start 2024-08-31T12:15:35.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.738+0000 7fb36b580640 1 -- start start 2024-08-31T12:15:35.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.739+0000 7fb36b580640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb350003a50 0x7fb3641e52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:35.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.739+0000 7fb36b580640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb3641e43a0 0x7fb36415d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:35.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.739+0000 7fb36b580640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb3640cd320 0x7fb3640c24d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:35.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.739+0000 7fb36b580640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fb36415b250 con 0x7fb3640cd320 2024-08-31T12:15:35.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.739+0000 7fb36b580640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb36415af70 con 0x7fb350003a50 2024-08-31T12:15:35.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.739+0000 7fb36b580640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fb3640c8610 con 0x7fb3641e43a0 2024-08-31T12:15:35.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.739+0000 7fb368b1b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb3641e43a0 0x7fb36415d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:35.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.739+0000 7fb369b1d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb3640cd320 0x7fb3640c24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:35.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.739+0000 7fb368b1b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb3641e43a0 0x7fb36415d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.19:43368/0 (socket says 172.21.15.19:43368) 2024-08-31T12:15:35.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.739+0000 7fb368b1b640 1 -- 172.21.15.19:0/3941326189 learned_addr learned my addr 172.21.15.19:0/3941326189 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:15:35.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.739+0000 7fb36931c640 1 --2- 172.21.15.19:0/3941326189 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb350003a50 0x7fb3641e52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:35.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.740+0000 7fb368b1b640 1 -- 172.21.15.19:0/3941326189 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb350003a50 msgr2=0x7fb3641e52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:35.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.740+0000 7fb368b1b640 1 --2- 172.21.15.19:0/3941326189 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb350003a50 0x7fb3641e52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:35.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.740+0000 7fb368b1b640 1 -- 172.21.15.19:0/3941326189 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb3640cd320 msgr2=0x7fb3640c24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:35.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.740+0000 7fb368b1b640 1 --2- 172.21.15.19:0/3941326189 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb3640cd320 0x7fb3640c24d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:35.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.740+0000 7fb368b1b640 1 -- 172.21.15.19:0/3941326189 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3641e5830 con 0x7fb3641e43a0 2024-08-31T12:15:35.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.740+0000 7fb369b1d640 1 --2- 172.21.15.19:0/3941326189 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb3640cd320 0x7fb3640c24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:15:35.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.740+0000 7fb36931c640 1 --2- 172.21.15.19:0/3941326189 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb350003a50 0x7fb3641e52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:15:35.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.740+0000 7fb368b1b640 1 --2- 172.21.15.19:0/3941326189 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb3641e43a0 0x7fb36415d9b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb358003e40 tx=0x7fb358001850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:35.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.740+0000 7fb355ffb640 1 -- 172.21.15.19:0/3941326189 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb358015070 con 0x7fb3641e43a0 2024-08-31T12:15:35.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.740+0000 7fb355ffb640 1 -- 172.21.15.19:0/3941326189 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb358011900 con 0x7fb3641e43a0 2024-08-31T12:15:35.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.740+0000 7fb36b580640 1 -- 172.21.15.19:0/3941326189 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb36415def0 con 0x7fb3641e43a0 2024-08-31T12:15:35.744 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.740+0000 7fb355ffb640 1 -- 172.21.15.19:0/3941326189 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3580012e0 con 0x7fb3641e43a0 2024-08-31T12:15:35.744 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.740+0000 7fb355ffb640 1 -- 172.21.15.19:0/3941326189 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb358018070 con 0x7fb3641e43a0 2024-08-31T12:15:35.744 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.740+0000 7fb36b580640 1 -- 172.21.15.19:0/3941326189 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3641f42b0 con 0x7fb3641e43a0 2024-08-31T12:15:35.744 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.741+0000 7fb355ffb640 1 -- 172.21.15.19:0/3941326189 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fb35801ab20 con 0x7fb3641e43a0 2024-08-31T12:15:35.744 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.741+0000 7fb36b580640 1 -- 172.21.15.19:0/3941326189 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3640c9a10 con 0x7fb3641e43a0 2024-08-31T12:15:35.744 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.743+0000 7fb355ffb640 1 -- 172.21.15.19:0/3941326189 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fb35801a450 con 0x7fb3641e43a0 2024-08-31T12:15:35.807 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.806+0000 7fb36b580640 1 -- 172.21.15.19:0/3941326189 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb3641f4830 con 0x7fb3641e43a0 2024-08-31T12:15:35.807 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.806+0000 7fb355ffb640 1 -- 172.21.15.19:0/3941326189 <== mon.2 v2:172.21.15.43:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fb35801a630 con 0x7fb3641e43a0 2024-08-31T12:15:35.807 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:15:35.807 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":10,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","created":"2024-08-31T12:15:32.084309+0000","modified":"2024-08-31T12:15:35.545202+0000","last_up_change":"0.000000","last_in_change":"2024-08-31T12:15:35.545202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b80d6a3f-7e4e-4d4d-b1b4-0df71c9176ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6f986366-82a6-4133-9f2e-43f84c7b03bc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ce8c2068-845c-4cbb-a4d1-077375d605ab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1cb0ef4-aa9b-421b-a94e-82de0c46e5cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"825b58ec-a6da-4515-b6a5-81debe327011","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7f34e537-5997-40e4-92e7-2570507160c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"195dd8ad-640a-4fa3-9ee4-06e4172e0464","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4645661f-f430-4f05-bdd1-69138dcd1bbc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-31T12:15:35.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.807+0000 7fb36b580640 1 -- 172.21.15.19:0/3941326189 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb3641e43a0 msgr2=0x7fb36415d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:35.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.807+0000 7fb36b580640 1 --2- 172.21.15.19:0/3941326189 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb3641e43a0 0x7fb36415d9b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb358003e40 tx=0x7fb358001850 comp rx=0 tx=0).stop 2024-08-31T12:15:35.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.807+0000 7fb36b580640 1 -- 172.21.15.19:0/3941326189 shutdown_connections 2024-08-31T12:15:35.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.807+0000 7fb36b580640 1 --2- 172.21.15.19:0/3941326189 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb3640cd320 0x7fb3640c24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:35.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.807+0000 7fb36b580640 1 --2- 172.21.15.19:0/3941326189 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb3641e43a0 0x7fb36415d9b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:35.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.807+0000 7fb36b580640 1 --2- 172.21.15.19:0/3941326189 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb350003a50 0x7fb3641e52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:35.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.807+0000 7fb36b580640 1 -- 172.21.15.19:0/3941326189 >> 172.21.15.19:0/3941326189 conn(0x7fb3640690d0 msgr2=0x7fb3640cc690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:35.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.807+0000 7fb36b580640 1 -- 172.21.15.19:0/3941326189 shutdown_connections 2024-08-31T12:15:35.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:35.808+0000 7fb36b580640 1 -- 172.21.15.19:0/3941326189 wait complete. 2024-08-31T12:15:35.819 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-08-31T12:15:35.819 INFO:tasks.ceph:Waiting for OSDs to come up 2024-08-31T12:15:36.625 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-08-31T12:15:36.731 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.730+0000 7f05a6f9d640 1 -- 172.21.15.19:0/1877845041 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05a00ca3f0 msgr2=0x7f05a00c3c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:36.731 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.730+0000 7f05a6f9d640 1 --2- 172.21.15.19:0/1877845041 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05a00ca3f0 0x7f05a00c3c90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f05a0159320 tx=0x7f059c02a640 comp rx=0 tx=0).stop 2024-08-31T12:15:36.731 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.730+0000 7f05a6f9d640 1 -- 172.21.15.19:0/1877845041 shutdown_connections 2024-08-31T12:15:36.731 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.730+0000 7f05a6f9d640 1 --2- 172.21.15.19:0/1877845041 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f05a0152e70 0x7f05a00ce870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:36.731 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.730+0000 7f05a6f9d640 1 --2- 172.21.15.19:0/1877845041 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05a00ca3f0 0x7f05a00c3c90 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:36.731 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.730+0000 7f05a6f9d640 1 -- 172.21.15.19:0/1877845041 >> 172.21.15.19:0/1877845041 conn(0x7f05a0068ec0 msgr2=0x7f05a00692d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:36.731 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.730+0000 7f05a6f9d640 1 -- 172.21.15.19:0/1877845041 shutdown_connections 2024-08-31T12:15:36.732 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.730+0000 7f05a6f9d640 1 -- 172.21.15.19:0/1877845041 wait complete. 2024-08-31T12:15:36.732 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.732+0000 7f05a6f9d640 1 Processor -- start 2024-08-31T12:15:36.732 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.732+0000 7f05a6f9d640 1 -- start start 2024-08-31T12:15:36.735 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.733+0000 7f05a6f9d640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05a00710d0 0x7f05a01555b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:36.735 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.733+0000 7f05a6f9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f05a00ca3f0 0x7f05a0155af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:36.735 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.733+0000 7f05a6f9d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f05a0152e70 0x7f05a0156c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:36.735 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.733+0000 7f05a6f9d640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f05a0153280 con 0x7f05a00ca3f0 2024-08-31T12:15:36.735 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.733+0000 7f05a6f9d640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f05a0070e60 con 0x7f05a00710d0 2024-08-31T12:15:36.735 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.733+0000 7f05a6f9d640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f05a00c4c70 con 0x7f05a0152e70 2024-08-31T12:15:36.735 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.733+0000 7f05a579a640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f05a00ca3f0 0x7f05a0155af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:36.735 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.733+0000 7f05a579a640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f05a00ca3f0 0x7f05a0155af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:55398/0 (socket says 172.21.15.19:55398) 2024-08-31T12:15:36.736 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.733+0000 7f05a579a640 1 -- 172.21.15.19:0/1558943718 learned_addr learned my addr 172.21.15.19:0/1558943718 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:15:36.736 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.733+0000 7f05a5f9b640 1 --2- 172.21.15.19:0/1558943718 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05a00710d0 0x7f05a01555b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:36.736 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.733+0000 7f05a679c640 1 --2- 172.21.15.19:0/1558943718 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f05a0152e70 0x7f05a0156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:36.736 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.733+0000 7f05a579a640 1 -- 172.21.15.19:0/1558943718 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f05a0152e70 msgr2=0x7f05a0156c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:36.736 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.733+0000 7f05a579a640 1 --2- 172.21.15.19:0/1558943718 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f05a0152e70 0x7f05a0156c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:36.736 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.733+0000 7f05a579a640 1 -- 172.21.15.19:0/1558943718 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05a00710d0 msgr2=0x7f05a01555b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:36.736 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.733+0000 7f05a579a640 1 --2- 172.21.15.19:0/1558943718 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05a00710d0 0x7f05a01555b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:36.736 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.733+0000 7f05a579a640 1 -- 172.21.15.19:0/1558943718 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05a01562e0 con 0x7f05a00ca3f0 2024-08-31T12:15:36.736 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.733+0000 7f05a5f9b640 1 --2- 172.21.15.19:0/1558943718 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05a00710d0 0x7f05a01555b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:15:36.736 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.733+0000 7f05a679c640 1 --2- 172.21.15.19:0/1558943718 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f05a0152e70 0x7f05a0156c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:15:36.736 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.734+0000 7f05a579a640 1 --2- 172.21.15.19:0/1558943718 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f05a00ca3f0 0x7f05a0155af0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f059400c3c0 tx=0x7f059400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:36.736 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.734+0000 7f05927fc640 1 -- 172.21.15.19:0/1558943718 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0594017070 con 0x7f05a00ca3f0 2024-08-31T12:15:36.737 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.734+0000 7f05927fc640 1 -- 172.21.15.19:0/1558943718 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0594003640 con 0x7f05a00ca3f0 2024-08-31T12:15:36.737 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.734+0000 7f05a6f9d640 1 -- 172.21.15.19:0/1558943718 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05a01f0380 con 0x7f05a00ca3f0 2024-08-31T12:15:36.737 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.734+0000 7f05927fc640 1 -- 172.21.15.19:0/1558943718 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0594003a90 con 0x7f05a00ca3f0 2024-08-31T12:15:36.737 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.734+0000 7f05a6f9d640 1 -- 172.21.15.19:0/1558943718 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05a01e39d0 con 0x7f05a00ca3f0 2024-08-31T12:15:36.737 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.735+0000 7f05927fc640 1 -- 172.21.15.19:0/1558943718 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 2) ==== 50097+0+0 (secure 0 0 0) 0x7f059401a070 con 0x7f05a00ca3f0 2024-08-31T12:15:36.737 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.735+0000 7f05a6f9d640 1 -- 172.21.15.19:0/1558943718 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05a00c3c90 con 0x7f05a00ca3f0 2024-08-31T12:15:36.738 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.735+0000 7f05927fc640 1 -- 172.21.15.19:0/1558943718 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f05940509c0 con 0x7f05a00ca3f0 2024-08-31T12:15:36.739 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.739+0000 7f05927fc640 1 -- 172.21.15.19:0/1558943718 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f0594033860 con 0x7f05a00ca3f0 2024-08-31T12:15:36.801 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.801+0000 7f05a6f9d640 1 -- 172.21.15.19:0/1558943718 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f05a0156640 con 0x7f05a00ca3f0 2024-08-31T12:15:36.802 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.801+0000 7f05927fc640 1 -- 172.21.15.19:0/1558943718 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f059401ad80 con 0x7f05a00ca3f0 2024-08-31T12:15:36.802 INFO:teuthology.misc.health.smithi019.stdout: 2024-08-31T12:15:36.802 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","created":"2024-08-31T12:15:32.084309+0000","modified":"2024-08-31T12:15:35.545202+0000","last_up_change":"0.000000","last_in_change":"2024-08-31T12:15:35.545202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b80d6a3f-7e4e-4d4d-b1b4-0df71c9176ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6f986366-82a6-4133-9f2e-43f84c7b03bc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ce8c2068-845c-4cbb-a4d1-077375d605ab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1cb0ef4-aa9b-421b-a94e-82de0c46e5cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"825b58ec-a6da-4515-b6a5-81debe327011","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7f34e537-5997-40e4-92e7-2570507160c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"195dd8ad-640a-4fa3-9ee4-06e4172e0464","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4645661f-f430-4f05-bdd1-69138dcd1bbc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-31T12:15:36.803 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.802+0000 7f05a6f9d640 1 -- 172.21.15.19:0/1558943718 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f05a00ca3f0 msgr2=0x7f05a0155af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:36.803 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.802+0000 7f05a6f9d640 1 --2- 172.21.15.19:0/1558943718 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f05a00ca3f0 0x7f05a0155af0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f059400c3c0 tx=0x7f059400c880 comp rx=0 tx=0).stop 2024-08-31T12:15:36.803 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.802+0000 7f05a6f9d640 1 -- 172.21.15.19:0/1558943718 shutdown_connections 2024-08-31T12:15:36.804 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.802+0000 7f05a6f9d640 1 --2- 172.21.15.19:0/1558943718 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f05a0152e70 0x7f05a0156c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:36.804 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.802+0000 7f05a6f9d640 1 --2- 172.21.15.19:0/1558943718 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f05a00ca3f0 0x7f05a0155af0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:36.804 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.802+0000 7f05a6f9d640 1 --2- 172.21.15.19:0/1558943718 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05a00710d0 0x7f05a01555b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:36.804 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.802+0000 7f05a6f9d640 1 -- 172.21.15.19:0/1558943718 >> 172.21.15.19:0/1558943718 conn(0x7f05a0068ec0 msgr2=0x7f05a0151240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:36.804 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.802+0000 7f05a6f9d640 1 -- 172.21.15.19:0/1558943718 shutdown_connections 2024-08-31T12:15:36.804 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:36.802+0000 7f05a6f9d640 1 -- 172.21.15.19:0/1558943718 wait complete. 2024-08-31T12:15:36.813 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-08-31T12:15:37.054 INFO:tasks.ceph.osd.5.smithi043.stderr:2024-08-31T12:15:37.053+0000 7fe919ab4640 -1 Falling back to public interface 2024-08-31T12:15:37.069 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-08-31T12:15:37.069+0000 7fe393345640 -1 Falling back to public interface 2024-08-31T12:15:37.270 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-08-31T12:15:37.270+0000 7fc3261a4640 -1 Falling back to public interface 2024-08-31T12:15:37.275 INFO:tasks.ceph.osd.7.smithi043.stderr:2024-08-31T12:15:37.274+0000 7faae5a14640 -1 Falling back to public interface 2024-08-31T12:15:37.287 INFO:tasks.ceph.osd.4.smithi043.stderr:2024-08-31T12:15:37.286+0000 7f6423af1640 -1 Falling back to public interface 2024-08-31T12:15:37.287 INFO:tasks.ceph.osd.1.smithi019.stderr:2024-08-31T12:15:37.287+0000 7fb2e6a40640 -1 Falling back to public interface 2024-08-31T12:15:37.497 INFO:tasks.ceph.mgr.x.smithi043.stderr:2024-08-31T12:15:37.496+0000 7ff716040640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-08-31T12:15:37.497 INFO:tasks.ceph.mgr.x.smithi043.stderr:2024-08-31T12:15:37.496+0000 7ff716040640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-08-31T12:15:37.561 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-08-31T12:15:37.561+0000 7f717a80b640 -1 Falling back to public interface 2024-08-31T12:15:37.563 INFO:tasks.ceph.osd.6.smithi043.stderr:2024-08-31T12:15:37.562+0000 7f95f9ba8640 -1 Falling back to public interface 2024-08-31T12:15:40.801 INFO:tasks.ceph.osd.7.smithi043.stderr:2024-08-31T12:15:40.799+0000 7faae5a14640 -1 osd.7 0 log_to_monitors true 2024-08-31T12:15:40.882 INFO:tasks.ceph.osd.5.smithi043.stderr:2024-08-31T12:15:40.881+0000 7fe919ab4640 -1 osd.5 0 log_to_monitors true 2024-08-31T12:15:41.030 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-08-31T12:15:41.029+0000 7fc3261a4640 -1 osd.2 0 log_to_monitors true 2024-08-31T12:15:41.111 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-08-31T12:15:41.110+0000 7fe393345640 -1 osd.0 0 log_to_monitors true 2024-08-31T12:15:41.580 INFO:tasks.ceph.osd.4.smithi043.stderr:2024-08-31T12:15:41.579+0000 7f6423af1640 -1 osd.4 0 log_to_monitors true 2024-08-31T12:15:41.616 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-08-31T12:15:41.615+0000 7f717a80b640 -1 osd.3 0 log_to_monitors true 2024-08-31T12:15:41.616 INFO:tasks.ceph.osd.1.smithi019.stderr:2024-08-31T12:15:41.616+0000 7fb2e6a40640 -1 osd.1 0 log_to_monitors true 2024-08-31T12:15:41.629 INFO:tasks.ceph.osd.6.smithi043.stderr:2024-08-31T12:15:41.628+0000 7f95f9ba8640 -1 osd.6 0 log_to_monitors true 2024-08-31T12:15:43.010 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-08-31T12:15:43.009+0000 7fc322132640 -1 osd.2 0 waiting for initial osdmap 2024-08-31T12:15:43.028 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-08-31T12:15:43.028+0000 7fc319f28640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-31T12:15:43.180 INFO:tasks.ceph.osd.5.smithi043.stderr:2024-08-31T12:15:43.179+0000 7fe915a42640 -1 osd.5 0 waiting for initial osdmap 2024-08-31T12:15:43.190 INFO:tasks.ceph.osd.7.smithi043.stderr:2024-08-31T12:15:43.189+0000 7faae1379640 -1 osd.7 0 waiting for initial osdmap 2024-08-31T12:15:43.200 INFO:tasks.ceph.osd.5.smithi043.stderr:2024-08-31T12:15:43.199+0000 7fe90d838640 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-31T12:15:43.208 INFO:tasks.ceph.osd.7.smithi043.stderr:2024-08-31T12:15:43.207+0000 7faad916f640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-31T12:15:43.618 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-08-31T12:15:43.733 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.732+0000 7f761a4f2640 1 -- 172.21.15.19:0/1406001925 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f76140c8f20 msgr2=0x7f76140c9330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:43.733 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.732+0000 7f761a4f2640 1 --2- 172.21.15.19:0/1406001925 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f76140c8f20 0x7f76140c9330 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f760c005fd0 tx=0x7f760c02ab20 comp rx=0 tx=0).stop 2024-08-31T12:15:43.733 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.732+0000 7f761a4f2640 1 -- 172.21.15.19:0/1406001925 shutdown_connections 2024-08-31T12:15:43.733 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.732+0000 7f761a4f2640 1 --2- 172.21.15.19:0/1406001925 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f76140c8f20 0x7f76140c9330 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:43.733 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.732+0000 7f761a4f2640 1 --2- 172.21.15.19:0/1406001925 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f76140ca6d0 0x7f76140c7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:43.733 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.732+0000 7f761a4f2640 1 -- 172.21.15.19:0/1406001925 >> 172.21.15.19:0/1406001925 conn(0x7f76140cda50 msgr2=0x7f76140c9ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:43.733 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.732+0000 7f761a4f2640 1 -- 172.21.15.19:0/1406001925 shutdown_connections 2024-08-31T12:15:43.733 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.732+0000 7f761a4f2640 1 -- 172.21.15.19:0/1406001925 wait complete. 2024-08-31T12:15:43.734 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.733+0000 7f761a4f2640 1 Processor -- start 2024-08-31T12:15:43.734 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.734+0000 7f761a4f2640 1 -- start start 2024-08-31T12:15:43.737 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.735+0000 7f761a4f2640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f76140c8f20 0x7f7614073cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:43.737 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.735+0000 7f761a4f2640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76140ca6d0 0x7f761414d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:43.737 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.735+0000 7f761a4f2640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7614151d30 0x7f761414dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:43.737 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.735+0000 7f761a4f2640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f76140c6590 con 0x7f7614151d30 2024-08-31T12:15:43.737 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.735+0000 7f761a4f2640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f76140c9870 con 0x7f76140ca6d0 2024-08-31T12:15:43.738 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.735+0000 7f761a4f2640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f7614152140 con 0x7f76140c8f20 2024-08-31T12:15:43.738 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.735+0000 7f7618a8f640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7614151d30 0x7f761414dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:43.738 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.735+0000 7f7618a8f640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7614151d30 0x7f761414dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:43732/0 (socket says 172.21.15.19:43732) 2024-08-31T12:15:43.738 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.735+0000 7f7618a8f640 1 -- 172.21.15.19:0/3749536620 learned_addr learned my addr 172.21.15.19:0/3749536620 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:15:43.738 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.735+0000 7f7613fff640 1 --2- 172.21.15.19:0/3749536620 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f76140c8f20 0x7f7614073cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:43.738 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.735+0000 7f76137fe640 1 --2- 172.21.15.19:0/3749536620 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76140ca6d0 0x7f761414d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:43.738 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.735+0000 7f7618a8f640 1 -- 172.21.15.19:0/3749536620 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f76140c8f20 msgr2=0x7f7614073cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:43.738 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.735+0000 7f7618a8f640 1 --2- 172.21.15.19:0/3749536620 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f76140c8f20 0x7f7614073cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:43.739 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.735+0000 7f7618a8f640 1 -- 172.21.15.19:0/3749536620 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76140ca6d0 msgr2=0x7f761414d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:43.739 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.735+0000 7f7618a8f640 1 --2- 172.21.15.19:0/3749536620 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76140ca6d0 0x7f761414d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:43.739 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.735+0000 7f7618a8f640 1 -- 172.21.15.19:0/3749536620 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f761414fd50 con 0x7f7614151d30 2024-08-31T12:15:43.739 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.736+0000 7f7618a8f640 1 --2- 172.21.15.19:0/3749536620 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7614151d30 0x7f761414dc70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f760c009d90 tx=0x7f760c003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:43.739 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.736+0000 7f7610ff9640 1 -- 172.21.15.19:0/3749536620 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f760c044070 con 0x7f7614151d30 2024-08-31T12:15:43.739 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.736+0000 7f7610ff9640 1 -- 172.21.15.19:0/3749536620 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f760c003e00 con 0x7f7614151d30 2024-08-31T12:15:43.739 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.736+0000 7f761a4f2640 1 -- 172.21.15.19:0/3749536620 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7614146060 con 0x7f7614151d30 2024-08-31T12:15:43.739 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.736+0000 7f7610ff9640 1 -- 172.21.15.19:0/3749536620 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f760c0044b0 con 0x7f7614151d30 2024-08-31T12:15:43.739 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.736+0000 7f761a4f2640 1 -- 172.21.15.19:0/3749536620 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76141eee00 con 0x7f7614151d30 2024-08-31T12:15:43.740 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.737+0000 7f761a4f2640 1 -- 172.21.15.19:0/3749536620 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76140c5230 con 0x7f7614151d30 2024-08-31T12:15:43.740 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.738+0000 7f7610ff9640 1 -- 172.21.15.19:0/3749536620 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 4) ==== 100095+0+0 (secure 0 0 0) 0x7f760c004650 con 0x7f7614151d30 2024-08-31T12:15:43.740 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.738+0000 7f7610ff9640 1 --2- 172.21.15.19:0/3749536620 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f76000779c0 0x7f7600079ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:43.740 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.738+0000 7f7610ff9640 1 -- 172.21.15.19:0/3749536620 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 2511+0+0 (secure 0 0 0) 0x7f760c0b90c0 con 0x7f7614151d30 2024-08-31T12:15:43.740 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.739+0000 7f7613fff640 1 --2- 172.21.15.19:0/3749536620 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f76000779c0 0x7f7600079ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:43.740 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.739+0000 7f7613fff640 1 --2- 172.21.15.19:0/3749536620 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f76000779c0 0x7f7600079ed0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7604009b30 tx=0x7f7604005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:43.744 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.743+0000 7f7610ff9640 1 -- 172.21.15.19:0/3749536620 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f760c032030 con 0x7f7614151d30 2024-08-31T12:15:43.950 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.949+0000 7f761a4f2640 1 -- 172.21.15.19:0/3749536620 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f76140c88b0 con 0x7f7614151d30 2024-08-31T12:15:43.951 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.950+0000 7f7610ff9640 1 -- 172.21.15.19:0/3749536620 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) ==== 74+0+7555 (secure 0 0 0) 0x7f760c0098b0 con 0x7f7614151d30 2024-08-31T12:15:43.951 INFO:teuthology.misc.health.smithi019.stdout: 2024-08-31T12:15:43.951 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":13,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","created":"2024-08-31T12:15:32.084309+0000","modified":"2024-08-31T12:15:43.115521+0000","last_up_change":"2024-08-31T12:15:43.115521+0000","last_in_change":"2024-08-31T12:15:35.545202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b80d6a3f-7e4e-4d4d-b1b4-0df71c9176ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6f986366-82a6-4133-9f2e-43f84c7b03bc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ce8c2068-845c-4cbb-a4d1-077375d605ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6809","nonce":2332368824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6811","nonce":2332368824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6815","nonce":2332368824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6813","nonce":2332368824}]},"public_addr":"172.21.15.19:6809/2332368824","cluster_addr":"172.21.15.19:6811/2332368824","heartbeat_back_addr":"172.21.15.19:6815/2332368824","heartbeat_front_addr":"172.21.15.19:6813/2332368824","state":["exists","up"]},{"osd":3,"uuid":"b1cb0ef4-aa9b-421b-a94e-82de0c46e5cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"825b58ec-a6da-4515-b6a5-81debe327011","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7f34e537-5997-40e4-92e7-2570507160c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"195dd8ad-640a-4fa3-9ee4-06e4172e0464","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4645661f-f430-4f05-bdd1-69138dcd1bbc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-31T12:15:43.954 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.952+0000 7f761a4f2640 1 -- 172.21.15.19:0/3749536620 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f76000779c0 msgr2=0x7f7600079ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:43.954 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.952+0000 7f761a4f2640 1 --2- 172.21.15.19:0/3749536620 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f76000779c0 0x7f7600079ed0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7604009b30 tx=0x7f7604005f90 comp rx=0 tx=0).stop 2024-08-31T12:15:43.954 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.952+0000 7f761a4f2640 1 -- 172.21.15.19:0/3749536620 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7614151d30 msgr2=0x7f761414dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:43.954 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.952+0000 7f761a4f2640 1 --2- 172.21.15.19:0/3749536620 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7614151d30 0x7f761414dc70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f760c009d90 tx=0x7f760c003c10 comp rx=0 tx=0).stop 2024-08-31T12:15:43.954 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.952+0000 7f761a4f2640 1 -- 172.21.15.19:0/3749536620 shutdown_connections 2024-08-31T12:15:43.954 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.952+0000 7f761a4f2640 1 --2- 172.21.15.19:0/3749536620 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f76000779c0 0x7f7600079ed0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:43.954 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.952+0000 7f761a4f2640 1 --2- 172.21.15.19:0/3749536620 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7614151d30 0x7f761414dc70 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:43.954 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.952+0000 7f761a4f2640 1 --2- 172.21.15.19:0/3749536620 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76140ca6d0 0x7f761414d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:43.954 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.952+0000 7f761a4f2640 1 --2- 172.21.15.19:0/3749536620 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f76140c8f20 0x7f7614073cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:43.954 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.952+0000 7f761a4f2640 1 -- 172.21.15.19:0/3749536620 >> 172.21.15.19:0/3749536620 conn(0x7f76140cda50 msgr2=0x7f7614148100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:43.954 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.953+0000 7f761a4f2640 1 -- 172.21.15.19:0/3749536620 shutdown_connections 2024-08-31T12:15:43.955 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:43.953+0000 7f761a4f2640 1 -- 172.21.15.19:0/3749536620 wait complete. 2024-08-31T12:15:43.964 DEBUG:teuthology.misc:1 of 8 OSDs are up 2024-08-31T12:15:44.142 INFO:tasks.ceph.osd.4.smithi043.stderr:2024-08-31T12:15:44.141+0000 7f641fa7f640 -1 osd.4 0 waiting for initial osdmap 2024-08-31T12:15:44.154 INFO:tasks.ceph.osd.4.smithi043.stderr:2024-08-31T12:15:44.152+0000 7f6417875640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-31T12:15:44.154 INFO:tasks.ceph.osd.6.smithi043.stderr:2024-08-31T12:15:44.152+0000 7f95f5b36640 -1 osd.6 0 waiting for initial osdmap 2024-08-31T12:15:44.167 INFO:tasks.ceph.osd.6.smithi043.stderr:2024-08-31T12:15:44.166+0000 7f95ed92c640 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-31T12:15:44.891 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-08-31T12:15:44.890+0000 7fe38f2d3640 -1 osd.0 0 waiting for initial osdmap 2024-08-31T12:15:44.907 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-08-31T12:15:44.906+0000 7fe3870c9640 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-31T12:15:44.913 INFO:tasks.ceph.osd.1.smithi019.stderr:2024-08-31T12:15:44.912+0000 7fb2e29ce640 -1 osd.1 0 waiting for initial osdmap 2024-08-31T12:15:44.924 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-08-31T12:15:44.923+0000 7f717621b640 -1 osd.3 0 waiting for initial osdmap 2024-08-31T12:15:44.927 INFO:tasks.ceph.osd.1.smithi019.stderr:2024-08-31T12:15:44.926+0000 7fb2da7c4640 -1 osd.1 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-31T12:15:44.940 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-08-31T12:15:44.940+0000 7f716e011640 -1 osd.3 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-31T12:15:50.771 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-08-31T12:15:50.891 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:50.889+0000 7ff09f512640 1 -- 172.21.15.19:0/3815605322 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff0980c2220 msgr2=0x7ff098074a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:50.891 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:50.889+0000 7ff09f512640 1 --2- 172.21.15.19:0/3815605322 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff0980c2220 0x7ff098074a40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff08c009860 tx=0x7ff08c02a5f0 comp rx=0 tx=0).stop 2024-08-31T12:15:50.891 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:50.889+0000 7ff09f512640 1 -- 172.21.15.19:0/3815605322 shutdown_connections 2024-08-31T12:15:50.892 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:50.889+0000 7ff09f512640 1 --2- 172.21.15.19:0/3815605322 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff0980c2220 0x7ff098074a40 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:50.892 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:50.889+0000 7ff09f512640 1 -- 172.21.15.19:0/3815605322 >> 172.21.15.19:0/3815605322 conn(0x7ff098071450 msgr2=0x7ff0980ccc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:50.892 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:50.889+0000 7ff09f512640 1 -- 172.21.15.19:0/3815605322 shutdown_connections 2024-08-31T12:15:50.892 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:50.889+0000 7ff09f512640 1 -- 172.21.15.19:0/3815605322 wait complete. 2024-08-31T12:15:50.892 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:50.891+0000 7ff09f512640 1 Processor -- start 2024-08-31T12:15:50.893 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:50.891+0000 7ff09f512640 1 -- start start 2024-08-31T12:15:50.895 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:50.892+0000 7ff09f512640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff0980c2220 0x7ff0980cc620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:50.895 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:50.892+0000 7ff09f512640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff0980c2d10 0x7ff0981f2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:50.895 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:50.892+0000 7ff09f512640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff098158860 0x7ff0981e3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:50.895 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:50.892+0000 7ff09f512640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7ff0980c3120 con 0x7ff0980c2220 2024-08-31T12:15:50.895 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:50.892+0000 7ff09f512640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff0980b2cb0 con 0x7ff0980c2d10 2024-08-31T12:15:50.895 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:50.892+0000 7ff09f512640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7ff0980c32a0 con 0x7ff098158860 2024-08-31T12:15:50.896 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:50.892+0000 7ff09d2ae640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff0980c2220 0x7ff0980cc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:50.896 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:50.892+0000 7ff09d2ae640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff0980c2220 0x7ff0980cc620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:34926/0 (socket says 172.21.15.19:34926) 2024-08-31T12:15:50.896 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:50.892+0000 7ff09caad640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff0980c2d10 0x7ff0981f2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:50.896 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:50.892+0000 7ff09d2ae640 1 -- 172.21.15.19:0/3027213752 learned_addr learned my addr 172.21.15.19:0/3027213752 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:15:50.896 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:50.893+0000 7ff09daaf640 1 --2- 172.21.15.19:0/3027213752 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff098158860 0x7ff0981e3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:50.896 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:50.893+0000 7ff09d2ae640 1 -- 172.21.15.19:0/3027213752 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff098158860 msgr2=0x7ff0981e3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:50.896 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:50.893+0000 7ff09d2ae640 1 --2- 172.21.15.19:0/3027213752 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff098158860 0x7ff0981e3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:50.896 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:50.893+0000 7ff09d2ae640 1 -- 172.21.15.19:0/3027213752 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff0980c2d10 msgr2=0x7ff0981f2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:50.897 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:50.893+0000 7ff09d2ae640 1 --2- 172.21.15.19:0/3027213752 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff0980c2d10 0x7ff0981f2060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:50.897 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:50.893+0000 7ff09d2ae640 1 -- 172.21.15.19:0/3027213752 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0981e1200 con 0x7ff0980c2220 2024-08-31T12:15:50.897 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:50.893+0000 7ff09d2ae640 1 --2- 172.21.15.19:0/3027213752 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff0980c2220 0x7ff0980cc620 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff08800cbc0 tx=0x7ff08800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:50.897 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:50.893+0000 7ff095ffb640 1 -- 172.21.15.19:0/3027213752 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff088016020 con 0x7ff0980c2220 2024-08-31T12:15:50.897 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:50.893+0000 7ff095ffb640 1 -- 172.21.15.19:0/3027213752 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff08800add0 con 0x7ff0980c2220 2024-08-31T12:15:50.897 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:50.893+0000 7ff09f512640 1 -- 172.21.15.19:0/3027213752 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0981f25a0 con 0x7ff0980c2220 2024-08-31T12:15:50.897 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:50.893+0000 7ff095ffb640 1 -- 172.21.15.19:0/3027213752 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff088012680 con 0x7ff0980c2220 2024-08-31T12:15:50.897 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:50.894+0000 7ff09f512640 1 -- 172.21.15.19:0/3027213752 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0980cbe20 con 0x7ff0980c2220 2024-08-31T12:15:50.898 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:50.895+0000 7ff09f512640 1 -- 172.21.15.19:0/3027213752 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0980c2000 con 0x7ff0980c2220 2024-08-31T12:15:50.899 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:50.895+0000 7ff095ffb640 1 -- 172.21.15.19:0/3027213752 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7ff088019020 con 0x7ff0980c2220 2024-08-31T12:15:50.899 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:50.896+0000 7ff095ffb640 1 --2- 172.21.15.19:0/3027213752 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff084077a50 0x7ff084079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:50.899 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:50.896+0000 7ff095ffb640 1 -- 172.21.15.19:0/3027213752 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7ff088099520 con 0x7ff0980c2220 2024-08-31T12:15:50.899 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:50.896+0000 7ff09caad640 1 --2- 172.21.15.19:0/3027213752 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff084077a50 0x7ff084079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:50.899 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:50.897+0000 7ff09caad640 1 --2- 172.21.15.19:0/3027213752 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff084077a50 0x7ff084079f60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff08c034000 tx=0x7ff08c004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:50.901 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:50.901+0000 7ff095ffb640 1 -- 172.21.15.19:0/3027213752 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7ff088020030 con 0x7ff0980c2220 2024-08-31T12:15:51.086 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:51.085+0000 7ff09f512640 1 -- 172.21.15.19:0/3027213752 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff09814dd10 con 0x7ff0980c2220 2024-08-31T12:15:51.087 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:51.086+0000 7ff095ffb640 1 -- 172.21.15.19:0/3027213752 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12848 (secure 0 0 0) 0x7ff088061f90 con 0x7ff0980c2220 2024-08-31T12:15:51.087 INFO:teuthology.misc.health.smithi019.stdout: 2024-08-31T12:15:51.088 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":17,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","created":"2024-08-31T12:15:32.084309+0000","modified":"2024-08-31T12:15:47.228281+0000","last_up_change":"2024-08-31T12:15:45.213688+0000","last_in_change":"2024-08-31T12:15:35.545202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-31T12:15:44.508197+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b80d6a3f-7e4e-4d4d-b1b4-0df71c9176ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6800","nonce":680447146},{"type":"v1","addr":"172.21.15.19:6801","nonce":680447146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":680447146},{"type":"v1","addr":"172.21.15.19:6803","nonce":680447146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":680447146},{"type":"v1","addr":"172.21.15.19:6807","nonce":680447146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":680447146},{"type":"v1","addr":"172.21.15.19:6805","nonce":680447146}]},"public_addr":"172.21.15.19:6801/680447146","cluster_addr":"172.21.15.19:6803/680447146","heartbeat_back_addr":"172.21.15.19:6807/680447146","heartbeat_front_addr":"172.21.15.19:6805/680447146","state":["exists","up"]},{"osd":1,"uuid":"6f986366-82a6-4133-9f2e-43f84c7b03bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":1522610034},{"type":"v1","addr":"172.21.15.19:6817","nonce":1522610034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":1522610034},{"type":"v1","addr":"172.21.15.19:6819","nonce":1522610034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":1522610034},{"type":"v1","addr":"172.21.15.19:6823","nonce":1522610034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":1522610034},{"type":"v1","addr":"172.21.15.19:6821","nonce":1522610034}]},"public_addr":"172.21.15.19:6817/1522610034","cluster_addr":"172.21.15.19:6819/1522610034","heartbeat_back_addr":"172.21.15.19:6823/1522610034","heartbeat_front_addr":"172.21.15.19:6821/1522610034","state":["exists","up"]},{"osd":2,"uuid":"ce8c2068-845c-4cbb-a4d1-077375d605ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6809","nonce":2332368824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6811","nonce":2332368824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6815","nonce":2332368824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6813","nonce":2332368824}]},"public_addr":"172.21.15.19:6809/2332368824","cluster_addr":"172.21.15.19:6811/2332368824","heartbeat_back_addr":"172.21.15.19:6815/2332368824","heartbeat_front_addr":"172.21.15.19:6813/2332368824","state":["exists","up"]},{"osd":3,"uuid":"b1cb0ef4-aa9b-421b-a94e-82de0c46e5cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":3785550598},{"type":"v1","addr":"172.21.15.19:6825","nonce":3785550598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":3785550598},{"type":"v1","addr":"172.21.15.19:6827","nonce":3785550598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":3785550598},{"type":"v1","addr":"172.21.15.19:6831","nonce":3785550598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":3785550598},{"type":"v1","addr":"172.21.15.19:6829","nonce":3785550598}]},"public_addr":"172.21.15.19:6825/3785550598","cluster_addr":"172.21.15.19:6827/3785550598","heartbeat_back_addr":"172.21.15.19:6831/3785550598","heartbeat_front_addr":"172.21.15.19:6829/3785550598","state":["exists","up"]},{"osd":4,"uuid":"825b58ec-a6da-4515-b6a5-81debe327011","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":1985519349},{"type":"v1","addr":"172.21.15.43:6819","nonce":1985519349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":1985519349},{"type":"v1","addr":"172.21.15.43:6821","nonce":1985519349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":1985519349},{"type":"v1","addr":"172.21.15.43:6825","nonce":1985519349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":1985519349},{"type":"v1","addr":"172.21.15.43:6823","nonce":1985519349}]},"public_addr":"172.21.15.43:6819/1985519349","cluster_addr":"172.21.15.43:6821/1985519349","heartbeat_back_addr":"172.21.15.43:6825/1985519349","heartbeat_front_addr":"172.21.15.43:6823/1985519349","state":["exists","up"]},{"osd":5,"uuid":"7f34e537-5997-40e4-92e7-2570507160c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":4271971521},{"type":"v1","addr":"172.21.15.43:6803","nonce":4271971521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":4271971521},{"type":"v1","addr":"172.21.15.43:6805","nonce":4271971521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":4271971521},{"type":"v1","addr":"172.21.15.43:6809","nonce":4271971521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":4271971521},{"type":"v1","addr":"172.21.15.43:6807","nonce":4271971521}]},"public_addr":"172.21.15.43:6803/4271971521","cluster_addr":"172.21.15.43:6805/4271971521","heartbeat_back_addr":"172.21.15.43:6809/4271971521","heartbeat_front_addr":"172.21.15.43:6807/4271971521","state":["exists","up"]},{"osd":6,"uuid":"195dd8ad-640a-4fa3-9ee4-06e4172e0464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2863384694},{"type":"v1","addr":"172.21.15.43:6827","nonce":2863384694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2863384694},{"type":"v1","addr":"172.21.15.43:6829","nonce":2863384694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":2863384694},{"type":"v1","addr":"172.21.15.43:6833","nonce":2863384694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2863384694},{"type":"v1","addr":"172.21.15.43:6831","nonce":2863384694}]},"public_addr":"172.21.15.43:6827/2863384694","cluster_addr":"172.21.15.43:6829/2863384694","heartbeat_back_addr":"172.21.15.43:6833/2863384694","heartbeat_front_addr":"172.21.15.43:6831/2863384694","state":["exists","up"]},{"osd":7,"uuid":"4645661f-f430-4f05-bdd1-69138dcd1bbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":792572234},{"type":"v1","addr":"172.21.15.43:6811","nonce":792572234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":792572234},{"type":"v1","addr":"172.21.15.43:6813","nonce":792572234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":792572234},{"type":"v1","addr":"172.21.15.43:6817","nonce":792572234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":792572234},{"type":"v1","addr":"172.21.15.43:6815","nonce":792572234}]},"public_addr":"172.21.15.43:6811/792572234","cluster_addr":"172.21.15.43:6813/792572234","heartbeat_back_addr":"172.21.15.43:6817/792572234","heartbeat_front_addr":"172.21.15.43:6815/792572234","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.156485+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.578126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.075440+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.639867+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.543602+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:41.919592+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.615557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:41.813337+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-31T12:15:51.090 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:51.088+0000 7ff09f512640 1 -- 172.21.15.19:0/3027213752 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff084077a50 msgr2=0x7ff084079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:51.090 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:51.088+0000 7ff09f512640 1 --2- 172.21.15.19:0/3027213752 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff084077a50 0x7ff084079f60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff08c034000 tx=0x7ff08c004730 comp rx=0 tx=0).stop 2024-08-31T12:15:51.091 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:51.089+0000 7ff09f512640 1 -- 172.21.15.19:0/3027213752 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff0980c2220 msgr2=0x7ff0980cc620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:51.091 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:51.089+0000 7ff09f512640 1 --2- 172.21.15.19:0/3027213752 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff0980c2220 0x7ff0980cc620 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff08800cbc0 tx=0x7ff08800a430 comp rx=0 tx=0).stop 2024-08-31T12:15:51.091 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:51.089+0000 7ff09f512640 1 -- 172.21.15.19:0/3027213752 shutdown_connections 2024-08-31T12:15:51.091 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:51.089+0000 7ff09f512640 1 --2- 172.21.15.19:0/3027213752 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff084077a50 0x7ff084079f60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:51.091 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:51.089+0000 7ff09f512640 1 --2- 172.21.15.19:0/3027213752 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff098158860 0x7ff0981e3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:51.091 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:51.089+0000 7ff09f512640 1 --2- 172.21.15.19:0/3027213752 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff0980c2d10 0x7ff0981f2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:51.092 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:51.089+0000 7ff09f512640 1 --2- 172.21.15.19:0/3027213752 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff0980c2220 0x7ff0980cc620 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:51.092 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:51.089+0000 7ff09f512640 1 -- 172.21.15.19:0/3027213752 >> 172.21.15.19:0/3027213752 conn(0x7ff098071450 msgr2=0x7ff098073150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:51.092 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:51.089+0000 7ff09f512640 1 -- 172.21.15.19:0/3027213752 shutdown_connections 2024-08-31T12:15:51.092 INFO:teuthology.misc.health.smithi019.stderr:2024-08-31T12:15:51.089+0000 7ff09f512640 1 -- 172.21.15.19:0/3027213752 wait complete. 2024-08-31T12:15:51.100 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-08-31T12:15:51.100 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-08-31T12:15:51.100 INFO:tasks.ceph.mds.a:Restarting daemon 2024-08-31T12:15:51.101 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-08-31T12:15:51.104 INFO:tasks.ceph.mds.a:Started 2024-08-31T12:15:51.104 INFO:tasks.ceph.mds.c:Restarting daemon 2024-08-31T12:15:51.105 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-08-31T12:15:51.107 INFO:tasks.ceph.mds.c:Started 2024-08-31T12:15:51.107 INFO:tasks.ceph.mds.e:Restarting daemon 2024-08-31T12:15:51.108 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2024-08-31T12:15:51.111 INFO:tasks.ceph.mds.e:Started 2024-08-31T12:15:51.111 INFO:tasks.ceph.mds.b:Restarting daemon 2024-08-31T12:15:51.111 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-08-31T12:15:51.114 INFO:tasks.ceph.mds.b:Started 2024-08-31T12:15:51.114 INFO:tasks.ceph.mds.d:Restarting daemon 2024-08-31T12:15:51.114 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-08-31T12:15:51.116 INFO:tasks.ceph.mds.d:Started 2024-08-31T12:15:51.117 INFO:tasks.ceph.mds.f:Restarting daemon 2024-08-31T12:15:51.117 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2024-08-31T12:15:51.119 INFO:tasks.ceph.mds.f:Started 2024-08-31T12:15:51.119 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-08-31T12:15:51.119 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-31T12:15:51.194 INFO:tasks.ceph.mds.c.smithi019.stdout:starting mds.c at 2024-08-31T12:15:51.195 INFO:tasks.ceph.mds.a.smithi019.stdout:starting mds.a at 2024-08-31T12:15:51.202 INFO:tasks.ceph.mds.e.smithi019.stdout:starting mds.e at 2024-08-31T12:15:51.219 INFO:tasks.ceph.mds.b.smithi043.stdout:starting mds.b at 2024-08-31T12:15:51.219 INFO:tasks.ceph.mds.d.smithi043.stdout:starting mds.d at 2024-08-31T12:15:51.219 INFO:tasks.ceph.mds.f.smithi043.stdout:starting mds.f at 2024-08-31T12:15:51.220 INFO:tasks.ceph.mds.b.smithi043.stderr:2024-08-31T12:15:51.218+0000 7f3b29c02640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-08-31T12:15:51.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.245+0000 7f59b34a9640 1 -- 172.21.15.19:0/1519204883 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f59ac0cb580 msgr2=0x7f59ac070bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:51.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.245+0000 7f59b34a9640 1 --2- 172.21.15.19:0/1519204883 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f59ac0cb580 0x7f59ac070bf0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f59a4005fd0 tx=0x7f59a402a940 comp rx=0 tx=0).stop 2024-08-31T12:15:51.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.245+0000 7f59b34a9640 1 -- 172.21.15.19:0/1519204883 shutdown_connections 2024-08-31T12:15:51.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.245+0000 7f59b34a9640 1 --2- 172.21.15.19:0/1519204883 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f59ac0cb580 0x7f59ac070bf0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:51.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.245+0000 7f59b34a9640 1 --2- 172.21.15.19:0/1519204883 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f59ac159820 0x7f59ac0cb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:51.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.245+0000 7f59b34a9640 1 -- 172.21.15.19:0/1519204883 >> 172.21.15.19:0/1519204883 conn(0x7f59ac0728b0 msgr2=0x7f59ac072cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:51.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.245+0000 7f59b34a9640 1 -- 172.21.15.19:0/1519204883 shutdown_connections 2024-08-31T12:15:51.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.245+0000 7f59b34a9640 1 -- 172.21.15.19:0/1519204883 wait complete. 2024-08-31T12:15:51.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.247+0000 7f59b34a9640 1 Processor -- start 2024-08-31T12:15:51.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.247+0000 7f59b34a9640 1 -- start start 2024-08-31T12:15:51.251 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.249+0000 7f59b34a9640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f59ac0cb580 0x7f59ac148510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:51.251 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.249+0000 7f59b34a9640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f59ac153190 0x7f59ac0cd860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:51.251 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.249+0000 7f59b34a9640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f59ac159820 0x7f59ac070b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:51.251 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.249+0000 7f59b34a9640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f59ac15cdc0 con 0x7f59ac159820 2024-08-31T12:15:51.251 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.249+0000 7f59b34a9640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f59ac1535a0 con 0x7f59ac153190 2024-08-31T12:15:51.251 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.249+0000 7f59b34a9640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f59ac159c30 con 0x7f59ac0cb580 2024-08-31T12:15:51.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.249+0000 7f59b1245640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f59ac0cb580 0x7f59ac148510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:51.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.249+0000 7f59b0a44640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f59ac153190 0x7f59ac0cd860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:51.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.249+0000 7f59b1245640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f59ac0cb580 0x7f59ac148510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.19:43504/0 (socket says 172.21.15.19:43504) 2024-08-31T12:15:51.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.249+0000 7f59b1245640 1 -- 172.21.15.19:0/3314489699 learned_addr learned my addr 172.21.15.19:0/3314489699 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:15:51.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.249+0000 7f59b1a46640 1 --2- 172.21.15.19:0/3314489699 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f59ac159820 0x7f59ac070b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:51.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.249+0000 7f59b1245640 1 -- 172.21.15.19:0/3314489699 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f59ac153190 msgr2=0x7f59ac0cd860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:51.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.249+0000 7f59b1245640 1 --2- 172.21.15.19:0/3314489699 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f59ac153190 0x7f59ac0cd860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:51.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.249+0000 7f59b1245640 1 -- 172.21.15.19:0/3314489699 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f59ac159820 msgr2=0x7f59ac070b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:51.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.249+0000 7f59b1245640 1 --2- 172.21.15.19:0/3314489699 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f59ac159820 0x7f59ac070b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:51.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.249+0000 7f59b1245640 1 -- 172.21.15.19:0/3314489699 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59ac153f40 con 0x7f59ac0cb580 2024-08-31T12:15:51.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.249+0000 7f59b1a46640 1 --2- 172.21.15.19:0/3314489699 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f59ac159820 0x7f59ac070b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:15:51.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.249+0000 7f59b0a44640 1 --2- 172.21.15.19:0/3314489699 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f59ac153190 0x7f59ac0cd860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:15:51.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.249+0000 7f59b1245640 1 --2- 172.21.15.19:0/3314489699 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f59ac0cb580 0x7f59ac148510 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f59a000c3c0 tx=0x7f59a000c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:51.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.250+0000 7f599dffb640 1 -- 172.21.15.19:0/3314489699 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a0017070 con 0x7f59ac0cb580 2024-08-31T12:15:51.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.250+0000 7f599dffb640 1 -- 172.21.15.19:0/3314489699 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f59a00035f0 con 0x7f59ac0cb580 2024-08-31T12:15:51.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.250+0000 7f599dffb640 1 -- 172.21.15.19:0/3314489699 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a0003a80 con 0x7f59ac0cb580 2024-08-31T12:15:51.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.250+0000 7f59b34a9640 1 -- 172.21.15.19:0/3314489699 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59ac148cf0 con 0x7f59ac0cb580 2024-08-31T12:15:51.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.250+0000 7f59b34a9640 1 -- 172.21.15.19:0/3314489699 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f59ac153ca0 con 0x7f59ac0cb580 2024-08-31T12:15:51.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.251+0000 7f59b34a9640 1 -- 172.21.15.19:0/3314489699 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59ac0cb040 con 0x7f59ac0cb580 2024-08-31T12:15:51.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.254+0000 7f599dffb640 1 -- 172.21.15.19:0/3314489699 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f59a001a070 con 0x7f59ac0cb580 2024-08-31T12:15:51.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.254+0000 7f599dffb640 1 --2- 172.21.15.19:0/3314489699 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5990077a50 0x7f5990079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:51.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.254+0000 7f599dffb640 1 -- 172.21.15.19:0/3314489699 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f59a0098a90 con 0x7f59ac0cb580 2024-08-31T12:15:51.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.254+0000 7f59b0a44640 1 --2- 172.21.15.19:0/3314489699 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5990077a50 0x7f5990079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:51.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.255+0000 7f59b0a44640 1 --2- 172.21.15.19:0/3314489699 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5990077a50 0x7f5990079f60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f59ac0b35c0 tx=0x7f59a8005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:51.259 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.258+0000 7f599dffb640 1 -- 172.21.15.19:0/3314489699 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f59a0021020 con 0x7f59ac0cb580 2024-08-31T12:15:51.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.480+0000 7f59b34a9640 1 -- 172.21.15.19:0/3314489699 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f59ac0cf3b0 con 0x7f59ac0cb580 2024-08-31T12:15:51.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.480+0000 7f599dffb640 1 -- 172.21.15.19:0/3314489699 <== mon.2 v2:172.21.15.43:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1434 (secure 0 0 0) 0x7f59a0061450 con 0x7f59ac0cb580 2024-08-31T12:15:51.481 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:15:51.482 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":2,"btime":"2024-08-31T12:15:51:199654+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4371,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6833/4215834946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":4215834946},{"type":"v1","addr":"172.21.15.19:6833","nonce":4215834946}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-08-31T12:15:51.482 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 2 2024-08-31T12:15:51.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.483+0000 7f59b34a9640 1 -- 172.21.15.19:0/3314489699 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5990077a50 msgr2=0x7f5990079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:51.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.483+0000 7f59b34a9640 1 --2- 172.21.15.19:0/3314489699 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5990077a50 0x7f5990079f60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f59ac0b35c0 tx=0x7f59a8005f90 comp rx=0 tx=0).stop 2024-08-31T12:15:51.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.483+0000 7f59b34a9640 1 -- 172.21.15.19:0/3314489699 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f59ac0cb580 msgr2=0x7f59ac148510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:51.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.483+0000 7f59b34a9640 1 --2- 172.21.15.19:0/3314489699 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f59ac0cb580 0x7f59ac148510 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f59a000c3c0 tx=0x7f59a000c880 comp rx=0 tx=0).stop 2024-08-31T12:15:51.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.483+0000 7f59b34a9640 1 -- 172.21.15.19:0/3314489699 shutdown_connections 2024-08-31T12:15:51.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.483+0000 7f59b34a9640 1 --2- 172.21.15.19:0/3314489699 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5990077a50 0x7f5990079f60 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:51.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.483+0000 7f59b34a9640 1 --2- 172.21.15.19:0/3314489699 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f59ac159820 0x7f59ac070b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:51.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.483+0000 7f59b34a9640 1 --2- 172.21.15.19:0/3314489699 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f59ac153190 0x7f59ac0cd860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:51.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.483+0000 7f59b34a9640 1 --2- 172.21.15.19:0/3314489699 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f59ac0cb580 0x7f59ac148510 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:51.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.483+0000 7f59b34a9640 1 -- 172.21.15.19:0/3314489699 >> 172.21.15.19:0/3314489699 conn(0x7f59ac0728b0 msgr2=0x7f59ac1513b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:51.485 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.483+0000 7f59b34a9640 1 -- 172.21.15.19:0/3314489699 shutdown_connections 2024-08-31T12:15:51.485 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:51.483+0000 7f59b34a9640 1 -- 172.21.15.19:0/3314489699 wait complete. 2024-08-31T12:15:53.498 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-31T12:15:53.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.626+0000 7f330e04d640 1 -- 172.21.15.19:0/53915089 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f32f0003a00 msgr2=0x7f32f0003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:53.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.626+0000 7f330e04d640 1 --2- 172.21.15.19:0/53915089 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f32f0003a00 0x7f32f0003e70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f33080cb2b0 tx=0x7f32f0005830 comp rx=0 tx=0).stop 2024-08-31T12:15:53.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.626+0000 7f3307fff640 1 -- 172.21.15.19:0/53915089 reap_dead start 2024-08-31T12:15:53.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.626+0000 7f330e04d640 1 -- 172.21.15.19:0/53915089 shutdown_connections 2024-08-31T12:15:53.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.626+0000 7f330e04d640 1 -- 172.21.15.19:0/53915089 >> 172.21.15.19:0/53915089 conn(0x7f33080690d0 msgr2=0x7f33080d0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:53.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.626+0000 7f330e04d640 1 -- 172.21.15.19:0/53915089 shutdown_connections 2024-08-31T12:15:53.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.626+0000 7f330e04d640 1 -- 172.21.15.19:0/53915089 wait complete. 2024-08-31T12:15:53.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.628+0000 7f330e04d640 1 Processor -- start 2024-08-31T12:15:53.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.628+0000 7f330e04d640 1 -- start start 2024-08-31T12:15:53.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.629+0000 7f330e04d640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32f0003a00 0x7f33080c5d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:53.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.629+0000 7f330e04d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f33080c2f60 0x7f330806d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:53.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.629+0000 7f330e04d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f33080c3370 0x7f330806d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:53.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.629+0000 7f330e04d640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f33080c8330 con 0x7f33080c2f60 2024-08-31T12:15:53.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.629+0000 7f330e04d640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f330815b250 con 0x7f32f0003a00 2024-08-31T12:15:53.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.629+0000 7f330e04d640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f33080c8610 con 0x7f33080c3370 2024-08-31T12:15:53.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.629+0000 7f3306ffd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f33080c2f60 0x7f330806d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:53.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.629+0000 7f3306ffd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f33080c2f60 0x7f330806d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:35002/0 (socket says 172.21.15.19:35002) 2024-08-31T12:15:53.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.629+0000 7f3306ffd640 1 -- 172.21.15.19:0/895484115 learned_addr learned my addr 172.21.15.19:0/895484115 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:15:53.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.629+0000 7f3307fff640 1 --2- 172.21.15.19:0/895484115 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f33080c3370 0x7f330806d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:53.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.629+0000 7f33077fe640 1 --2- 172.21.15.19:0/895484115 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32f0003a00 0x7f33080c5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:53.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.630+0000 7f3306ffd640 1 -- 172.21.15.19:0/895484115 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f33080c3370 msgr2=0x7f330806d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:53.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.630+0000 7f3306ffd640 1 --2- 172.21.15.19:0/895484115 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f33080c3370 0x7f330806d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:53.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.630+0000 7f3306ffd640 1 -- 172.21.15.19:0/895484115 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32f0003a00 msgr2=0x7f33080c5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:53.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.630+0000 7f3306ffd640 1 --2- 172.21.15.19:0/895484115 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32f0003a00 0x7f33080c5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:53.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.630+0000 7f3306ffd640 1 -- 172.21.15.19:0/895484115 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33080c6290 con 0x7f33080c2f60 2024-08-31T12:15:53.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.630+0000 7f3306ffd640 1 --2- 172.21.15.19:0/895484115 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f33080c2f60 0x7f330806d000 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f32fc011ae0 tx=0x7f32fc009eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:53.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.630+0000 7f32effff640 1 -- 172.21.15.19:0/895484115 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32fc01e070 con 0x7f33080c2f60 2024-08-31T12:15:53.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.630+0000 7f32effff640 1 -- 172.21.15.19:0/895484115 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f32fc00fc70 con 0x7f33080c2f60 2024-08-31T12:15:53.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.630+0000 7f32effff640 1 -- 172.21.15.19:0/895484115 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32fc019410 con 0x7f33080c2f60 2024-08-31T12:15:53.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.630+0000 7f330e04d640 1 -- 172.21.15.19:0/895484115 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33080c6f80 con 0x7f33080c2f60 2024-08-31T12:15:53.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.630+0000 7f330e04d640 1 -- 172.21.15.19:0/895484115 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33081f5220 con 0x7f33080c2f60 2024-08-31T12:15:53.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.631+0000 7f330e04d640 1 -- 172.21.15.19:0/895484115 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33081f4d60 con 0x7f33080c2f60 2024-08-31T12:15:53.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.632+0000 7f32effff640 1 -- 172.21.15.19:0/895484115 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f32fc021070 con 0x7f33080c2f60 2024-08-31T12:15:53.635 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.632+0000 7f32effff640 1 --2- 172.21.15.19:0/895484115 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f32f4077a80 0x7f32f4079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:53.635 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.632+0000 7f32effff640 1 -- 172.21.15.19:0/895484115 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f32fc0999e0 con 0x7f33080c2f60 2024-08-31T12:15:53.635 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.633+0000 7f33077fe640 1 --2- 172.21.15.19:0/895484115 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f32f4077a80 0x7f32f4079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:53.635 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.633+0000 7f33077fe640 1 --2- 172.21.15.19:0/895484115 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f32f4077a80 0x7f32f4079f90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f32f8033000 tx=0x7f32f802d000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:53.637 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.637+0000 7f32effff640 1 -- 172.21.15.19:0/895484115 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f33081f4d60 con 0x7f33080c2f60 2024-08-31T12:15:53.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.860+0000 7f330e04d640 1 -- 172.21.15.19:0/895484115 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f33080c9850 con 0x7f33080c2f60 2024-08-31T12:15:53.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.860+0000 7f32effff640 1 -- 172.21.15.19:0/895484115 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+5548 (secure 0 0 0) 0x7f32fc0624e0 con 0x7f33080c2f60 2024-08-31T12:15:53.861 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:15:53.861 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":3,"btime":"2024-08-31T12:15:52:201056+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4371,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6833/4215834946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":4215834946},{"type":"v1","addr":"172.21.15.19:6833","nonce":4215834946}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/543344732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":543344732},{"type":"v1","addr":"172.21.15.19:6835","nonce":543344732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4380,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6837/699077015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6836","nonce":699077015},{"type":"v1","addr":"172.21.15.19:6837","nonce":699077015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4398,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/2103490885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6835","nonce":2103490885},{"type":"v1","addr":"172.21.15.43:6837","nonce":2103490885}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4426,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6839/1196858061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1196858061},{"type":"v1","addr":"172.21.15.43:6839","nonce":1196858061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4429,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6838/3981040655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":3981040655},{"type":"v1","addr":"172.21.15.43:6838","nonce":3981040655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-08-31T12:15:53.862 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 3 2024-08-31T12:15:53.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.863+0000 7f330e04d640 1 -- 172.21.15.19:0/895484115 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f32f4077a80 msgr2=0x7f32f4079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:53.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.863+0000 7f330e04d640 1 --2- 172.21.15.19:0/895484115 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f32f4077a80 0x7f32f4079f90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f32f8033000 tx=0x7f32f802d000 comp rx=0 tx=0).stop 2024-08-31T12:15:53.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.863+0000 7f330e04d640 1 -- 172.21.15.19:0/895484115 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f33080c2f60 msgr2=0x7f330806d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:53.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.863+0000 7f330e04d640 1 --2- 172.21.15.19:0/895484115 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f33080c2f60 0x7f330806d000 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f32fc011ae0 tx=0x7f32fc009eb0 comp rx=0 tx=0).stop 2024-08-31T12:15:53.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.863+0000 7f330e04d640 1 -- 172.21.15.19:0/895484115 shutdown_connections 2024-08-31T12:15:53.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.863+0000 7f330e04d640 1 --2- 172.21.15.19:0/895484115 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f32f4077a80 0x7f32f4079f90 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:53.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.863+0000 7f330e04d640 1 --2- 172.21.15.19:0/895484115 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f33080c3370 0x7f330806d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:53.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.863+0000 7f330e04d640 1 --2- 172.21.15.19:0/895484115 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f33080c2f60 0x7f330806d000 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:53.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.863+0000 7f330e04d640 1 --2- 172.21.15.19:0/895484115 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32f0003a00 0x7f33080c5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:53.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.863+0000 7f330e04d640 1 -- 172.21.15.19:0/895484115 >> 172.21.15.19:0/895484115 conn(0x7f33080690d0 msgr2=0x7f33080d0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:53.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.863+0000 7f330e04d640 1 -- 172.21.15.19:0/895484115 shutdown_connections 2024-08-31T12:15:53.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.863+0000 7f330e04d640 1 -- 172.21.15.19:0/895484115 wait complete. 2024-08-31T12:15:53.874 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-08-31T12:15:53.992 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.987+0000 7f95acf0b640 1 -- 172.21.15.19:0/1408718483 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9594004800 msgr2=0x7f9594004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:53.992 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.987+0000 7f95acf0b640 1 --2- 172.21.15.19:0/1408718483 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9594004800 0x7f9594004c10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9590006fc0 tx=0x7f9594005830 comp rx=0 tx=0).stop 2024-08-31T12:15:53.992 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.987+0000 7f95acf0b640 1 -- 172.21.15.19:0/1408718483 shutdown_connections 2024-08-31T12:15:53.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.987+0000 7f95acf0b640 1 --2- 172.21.15.19:0/1408718483 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9594004800 0x7f9594004c10 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:53.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.987+0000 7f95acf0b640 1 -- 172.21.15.19:0/1408718483 >> 172.21.15.19:0/1408718483 conn(0x7f95a80690d0 msgr2=0x7f95a80d0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:53.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.987+0000 7f95acf0b640 1 -- 172.21.15.19:0/1408718483 shutdown_connections 2024-08-31T12:15:53.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.987+0000 7f95acf0b640 1 -- 172.21.15.19:0/1408718483 wait complete. 2024-08-31T12:15:53.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.989+0000 7f95acf0b640 1 Processor -- start 2024-08-31T12:15:53.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.989+0000 7f95acf0b640 1 -- start start 2024-08-31T12:15:53.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.990+0000 7f95acf0b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9594004800 0x7f95a80c5d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:53.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.990+0000 7f95acf0b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f95a80cfbc0 0x7f95a815d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:53.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.990+0000 7f95acf0b640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f95a8157db0 0x7f95a81581c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:53.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.990+0000 7f95acf0b640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f95a815b250 con 0x7f9594004800 2024-08-31T12:15:53.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.990+0000 7f95acf0b640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f95a815af70 con 0x7f95a8157db0 2024-08-31T12:15:53.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.990+0000 7f95acf0b640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f95a80c8610 con 0x7f95a80cfbc0 2024-08-31T12:15:53.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.990+0000 7f95a659c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9594004800 0x7f95a80c5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:53.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.990+0000 7f95a659c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9594004800 0x7f95a80c5d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:35020/0 (socket says 172.21.15.19:35020) 2024-08-31T12:15:53.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.990+0000 7f95a659c640 1 -- 172.21.15.19:0/2066947591 learned_addr learned my addr 172.21.15.19:0/2066947591 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:15:53.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.990+0000 7f95a6d9d640 1 --2- 172.21.15.19:0/2066947591 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f95a8157db0 0x7f95a81581c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:53.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.990+0000 7f959dd9b640 1 --2- 172.21.15.19:0/2066947591 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f95a80cfbc0 0x7f95a815d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:53.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.991+0000 7f95a659c640 1 -- 172.21.15.19:0/2066947591 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f95a80cfbc0 msgr2=0x7f95a815d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:53.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.991+0000 7f95a659c640 1 --2- 172.21.15.19:0/2066947591 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f95a80cfbc0 0x7f95a815d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:53.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.991+0000 7f95a659c640 1 -- 172.21.15.19:0/2066947591 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f95a8157db0 msgr2=0x7f95a81581c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:53.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.991+0000 7f95a659c640 1 --2- 172.21.15.19:0/2066947591 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f95a8157db0 0x7f95a81581c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:53.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.991+0000 7f95a659c640 1 -- 172.21.15.19:0/2066947591 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95a80c6280 con 0x7f9594004800 2024-08-31T12:15:53.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.991+0000 7f95a659c640 1 --2- 172.21.15.19:0/2066947591 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9594004800 0x7f95a80c5d40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9598011b10 tx=0x7f9598009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:53.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.991+0000 7f959f7fe640 1 -- 172.21.15.19:0/2066947591 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f959800fce0 con 0x7f9594004800 2024-08-31T12:15:53.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.991+0000 7f959f7fe640 1 -- 172.21.15.19:0/2066947591 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f959800fe80 con 0x7f9594004800 2024-08-31T12:15:53.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.991+0000 7f95acf0b640 1 -- 172.21.15.19:0/2066947591 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95a80c5840 con 0x7f9594004800 2024-08-31T12:15:53.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.991+0000 7f959f7fe640 1 -- 172.21.15.19:0/2066947591 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9598004280 con 0x7f9594004800 2024-08-31T12:15:53.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.991+0000 7f95acf0b640 1 -- 172.21.15.19:0/2066947591 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95a81f21c0 con 0x7f9594004800 2024-08-31T12:15:53.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.992+0000 7f95acf0b640 1 -- 172.21.15.19:0/2066947591 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95a806a440 con 0x7f9594004800 2024-08-31T12:15:53.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.993+0000 7f959f7fe640 1 -- 172.21.15.19:0/2066947591 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f95980035f0 con 0x7f9594004800 2024-08-31T12:15:53.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.994+0000 7f959f7fe640 1 --2- 172.21.15.19:0/2066947591 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9588077a80 0x7f9588079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:53.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.994+0000 7f959f7fe640 1 -- 172.21.15.19:0/2066947591 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f9598099eb0 con 0x7f9594004800 2024-08-31T12:15:53.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.994+0000 7f959dd9b640 1 --2- 172.21.15.19:0/2066947591 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9588077a80 0x7f9588079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:53.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.994+0000 7f959dd9b640 1 --2- 172.21.15.19:0/2066947591 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9588077a80 0x7f9588079f90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9590033000 tx=0x7f9590038040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:53.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:53.998+0000 7f959f7fe640 1 -- 172.21.15.19:0/2066947591 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f95a806a440 con 0x7f9594004800 2024-08-31T12:15:54.185 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.184+0000 7f95acf0b640 1 -- 172.21.15.19:0/2066947591 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f95a80c9a10 con 0x7f9594004800 2024-08-31T12:15:54.185 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.184+0000 7f959f7fe640 1 -- 172.21.15.19:0/2066947591 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) ==== 84+0+65 (secure 0 0 0) 0x7f9598062880 con 0x7f9594004800 2024-08-31T12:15:54.185 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:15:54.185 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-08-31T12:15:54.185 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:15:54.185 INFO:teuthology.orchestra.run.smithi019.stdout: "poolnum": 1, 2024-08-31T12:15:54.185 INFO:teuthology.orchestra.run.smithi019.stdout: "poolname": ".mgr" 2024-08-31T12:15:54.186 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:15:54.186 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-08-31T12:15:54.187 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.186+0000 7f95acf0b640 1 -- 172.21.15.19:0/2066947591 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9588077a80 msgr2=0x7f9588079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:54.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.186+0000 7f95acf0b640 1 --2- 172.21.15.19:0/2066947591 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9588077a80 0x7f9588079f90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9590033000 tx=0x7f9590038040 comp rx=0 tx=0).stop 2024-08-31T12:15:54.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.186+0000 7f95acf0b640 1 -- 172.21.15.19:0/2066947591 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9594004800 msgr2=0x7f95a80c5d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:54.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.187+0000 7f95acf0b640 1 --2- 172.21.15.19:0/2066947591 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9594004800 0x7f95a80c5d40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9598011b10 tx=0x7f9598009f10 comp rx=0 tx=0).stop 2024-08-31T12:15:54.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.187+0000 7f95acf0b640 1 -- 172.21.15.19:0/2066947591 shutdown_connections 2024-08-31T12:15:54.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.187+0000 7f95acf0b640 1 --2- 172.21.15.19:0/2066947591 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9588077a80 0x7f9588079f90 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:54.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.187+0000 7f95acf0b640 1 --2- 172.21.15.19:0/2066947591 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f95a8157db0 0x7f95a81581c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:54.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.187+0000 7f95acf0b640 1 --2- 172.21.15.19:0/2066947591 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f95a80cfbc0 0x7f95a815d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:54.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.187+0000 7f95acf0b640 1 --2- 172.21.15.19:0/2066947591 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9594004800 0x7f95a80c5d40 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:54.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.187+0000 7f95acf0b640 1 -- 172.21.15.19:0/2066947591 >> 172.21.15.19:0/2066947591 conn(0x7f95a80690d0 msgr2=0x7f95a815ec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:54.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.187+0000 7f95acf0b640 1 -- 172.21.15.19:0/2066947591 shutdown_connections 2024-08-31T12:15:54.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.187+0000 7f95acf0b640 1 -- 172.21.15.19:0/2066947591 wait complete. 2024-08-31T12:15:54.198 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-08-31T12:15:54.199 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-08-31T12:15:54.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.315+0000 7fa68cd39640 1 -- 172.21.15.19:0/1338254917 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa6880ca6c0 msgr2=0x7fa6880d0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:54.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.315+0000 7fa68cd39640 1 --2- 172.21.15.19:0/1338254917 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa6880ca6c0 0x7fa6880d0bf0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa68806d010 tx=0x7fa67002a5f0 comp rx=0 tx=0).stop 2024-08-31T12:15:54.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.316+0000 7fa68cd39640 1 -- 172.21.15.19:0/1338254917 shutdown_connections 2024-08-31T12:15:54.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.316+0000 7fa68cd39640 1 --2- 172.21.15.19:0/1338254917 >> v2:172.21.15.43:3300/0 conn(0x7fa6880d1130 0x7fa68815e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:54.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.316+0000 7fa68cd39640 1 --2- 172.21.15.19:0/1338254917 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa6880ca6c0 0x7fa6880d0bf0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:54.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.316+0000 7fa68cd39640 1 --2- 172.21.15.19:0/1338254917 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa6880cbcf0 0x7fa6880cc150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:54.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.316+0000 7fa68cd39640 1 -- 172.21.15.19:0/1338254917 >> 172.21.15.19:0/1338254917 conn(0x7fa688074210 msgr2=0x7fa688074620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:54.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.316+0000 7fa68cd39640 1 -- 172.21.15.19:0/1338254917 shutdown_connections 2024-08-31T12:15:54.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.316+0000 7fa68cd39640 1 -- 172.21.15.19:0/1338254917 wait complete. 2024-08-31T12:15:54.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.317+0000 7fa68cd39640 1 Processor -- start 2024-08-31T12:15:54.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.318+0000 7fa68cd39640 1 -- start start 2024-08-31T12:15:54.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.319+0000 7fa68cd39640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa6880ca6c0 0x7fa6881f1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:54.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.319+0000 7fa68cd39640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6880cbcf0 0x7fa688152630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:54.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.319+0000 7fa68cd39640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa6880d1130 0x7fa68814cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:54.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.319+0000 7fa68cd39640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fa6880b3880 con 0x7fa6880ca6c0 2024-08-31T12:15:54.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.319+0000 7fa68cd39640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa6880b4680 con 0x7fa6880cbcf0 2024-08-31T12:15:54.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.319+0000 7fa68cd39640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fa6880b3a00 con 0x7fa6880d1130 2024-08-31T12:15:54.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.319+0000 7fa68659c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa6880ca6c0 0x7fa6881f1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:54.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.319+0000 7fa68659c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa6880ca6c0 0x7fa6881f1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:35026/0 (socket says 172.21.15.19:35026) 2024-08-31T12:15:54.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.319+0000 7fa68659c640 1 -- 172.21.15.19:0/488900005 learned_addr learned my addr 172.21.15.19:0/488900005 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:15:54.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.319+0000 7fa685d9b640 1 --2- 172.21.15.19:0/488900005 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6880cbcf0 0x7fa688152630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:54.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.319+0000 7fa686d9d640 1 --2- 172.21.15.19:0/488900005 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa6880d1130 0x7fa68814cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:54.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.319+0000 7fa68659c640 1 -- 172.21.15.19:0/488900005 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa6880d1130 msgr2=0x7fa68814cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:54.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.319+0000 7fa68659c640 1 --2- 172.21.15.19:0/488900005 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa6880d1130 0x7fa68814cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:54.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.319+0000 7fa68659c640 1 -- 172.21.15.19:0/488900005 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6880cbcf0 msgr2=0x7fa688152630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:54.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.319+0000 7fa68659c640 1 --2- 172.21.15.19:0/488900005 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6880cbcf0 0x7fa688152630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:54.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.319+0000 7fa68659c640 1 -- 172.21.15.19:0/488900005 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6881f1e90 con 0x7fa6880ca6c0 2024-08-31T12:15:54.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.319+0000 7fa68659c640 1 --2- 172.21.15.19:0/488900005 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa6880ca6c0 0x7fa6881f1950 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa68000ccb0 tx=0x7fa680007590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:54.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.320+0000 7fa67effd640 1 -- 172.21.15.19:0/488900005 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa680004b20 con 0x7fa6880ca6c0 2024-08-31T12:15:54.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.320+0000 7fa67effd640 1 -- 172.21.15.19:0/488900005 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa680004db0 con 0x7fa6880ca6c0 2024-08-31T12:15:54.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.320+0000 7fa67effd640 1 -- 172.21.15.19:0/488900005 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa680003bd0 con 0x7fa6880ca6c0 2024-08-31T12:15:54.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.320+0000 7fa68cd39640 1 -- 172.21.15.19:0/488900005 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa688158390 con 0x7fa6880ca6c0 2024-08-31T12:15:54.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.320+0000 7fa68cd39640 1 -- 172.21.15.19:0/488900005 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6881e6bb0 con 0x7fa6880ca6c0 2024-08-31T12:15:54.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.321+0000 7fa68cd39640 1 -- 172.21.15.19:0/488900005 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa688050e20 con 0x7fa6880ca6c0 2024-08-31T12:15:54.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.321+0000 7fa67effd640 1 -- 172.21.15.19:0/488900005 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fa68000f450 con 0x7fa6880ca6c0 2024-08-31T12:15:54.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.322+0000 7fa67effd640 1 --2- 172.21.15.19:0/488900005 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa668077a50 0x7fa668079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:54.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.322+0000 7fa67effd640 1 -- 172.21.15.19:0/488900005 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fa680099900 con 0x7fa6880ca6c0 2024-08-31T12:15:54.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.322+0000 7fa685d9b640 1 --2- 172.21.15.19:0/488900005 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa668077a50 0x7fa668079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:54.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.323+0000 7fa685d9b640 1 --2- 172.21.15.19:0/488900005 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa668077a50 0x7fa668079f60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa670033000 tx=0x7fa670004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:54.327 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.326+0000 7fa67effd640 1 -- 172.21.15.19:0/488900005 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fa688050e20 con 0x7fa6880ca6c0 2024-08-31T12:15:54.509 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:54.508+0000 7fa68cd39640 1 -- 172.21.15.19:0/488900005 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fa6880caad0 con 0x7fa6880ca6c0 2024-08-31T12:15:55.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.264+0000 7fa67effd640 1 -- 172.21.15.19:0/488900005 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) ==== 138+0+0 (secure 0 0 0) 0x7fa680069e20 con 0x7fa6880ca6c0 2024-08-31T12:15:55.265 INFO:teuthology.orchestra.run.smithi019.stderr:pool 'cephfs_metadata' created 2024-08-31T12:15:55.269 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.268+0000 7fa68cd39640 1 -- 172.21.15.19:0/488900005 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa668077a50 msgr2=0x7fa668079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:55.269 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.268+0000 7fa68cd39640 1 --2- 172.21.15.19:0/488900005 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa668077a50 0x7fa668079f60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa670033000 tx=0x7fa670004730 comp rx=0 tx=0).stop 2024-08-31T12:15:55.269 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.268+0000 7fa68cd39640 1 -- 172.21.15.19:0/488900005 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa6880ca6c0 msgr2=0x7fa6881f1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:55.269 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.268+0000 7fa68cd39640 1 --2- 172.21.15.19:0/488900005 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa6880ca6c0 0x7fa6881f1950 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa68000ccb0 tx=0x7fa680007590 comp rx=0 tx=0).stop 2024-08-31T12:15:55.269 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.268+0000 7fa68cd39640 1 -- 172.21.15.19:0/488900005 shutdown_connections 2024-08-31T12:15:55.269 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.268+0000 7fa68cd39640 1 --2- 172.21.15.19:0/488900005 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa668077a50 0x7fa668079f60 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:55.269 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.268+0000 7fa68cd39640 1 --2- 172.21.15.19:0/488900005 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa6880d1130 0x7fa68814cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:55.269 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.268+0000 7fa68cd39640 1 --2- 172.21.15.19:0/488900005 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6880cbcf0 0x7fa688152630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:55.269 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.268+0000 7fa68cd39640 1 --2- 172.21.15.19:0/488900005 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa6880ca6c0 0x7fa6881f1950 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:55.269 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.268+0000 7fa68cd39640 1 -- 172.21.15.19:0/488900005 >> 172.21.15.19:0/488900005 conn(0x7fa688074210 msgr2=0x7fa688063680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:55.269 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.268+0000 7fa68cd39640 1 -- 172.21.15.19:0/488900005 shutdown_connections 2024-08-31T12:15:55.269 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.268+0000 7fa68cd39640 1 -- 172.21.15.19:0/488900005 wait complete. 2024-08-31T12:15:55.287 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-08-31T12:15:55.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.413+0000 7fd6509e4640 1 -- 172.21.15.19:0/719916759 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd64c0d0b50 msgr2=0x7fd64c0d0f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:55.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.413+0000 7fd6509e4640 1 --2- 172.21.15.19:0/719916759 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd64c0d0b50 0x7fd64c0d0f60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd64c0b4b80 tx=0x7fd63c02a680 comp rx=0 tx=0).stop 2024-08-31T12:15:55.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.413+0000 7fd6509e4640 1 -- 172.21.15.19:0/719916759 shutdown_connections 2024-08-31T12:15:55.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.413+0000 7fd6509e4640 1 --2- 172.21.15.19:0/719916759 >> v2:172.21.15.43:3300/0 conn(0x7fd64c15f080 0x7fd64c0cd8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:55.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.413+0000 7fd6509e4640 1 --2- 172.21.15.19:0/719916759 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd64c0d0b50 0x7fd64c0d0f60 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:55.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.413+0000 7fd6509e4640 1 -- 172.21.15.19:0/719916759 >> 172.21.15.19:0/719916759 conn(0x7fd64c074210 msgr2=0x7fd64c074620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:55.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.413+0000 7fd6509e4640 1 -- 172.21.15.19:0/719916759 shutdown_connections 2024-08-31T12:15:55.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.413+0000 7fd6509e4640 1 -- 172.21.15.19:0/719916759 wait complete. 2024-08-31T12:15:55.416 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.415+0000 7fd6509e4640 1 Processor -- start 2024-08-31T12:15:55.416 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.415+0000 7fd6509e4640 1 -- start start 2024-08-31T12:15:55.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.416+0000 7fd6509e4640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd64c0d0b50 0x7fd64c0c7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:55.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.416+0000 7fd6509e4640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd64c15ec70 0x7fd64c15d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:55.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.416+0000 7fd6509e4640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd64c15f080 0x7fd64c14f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:55.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.416+0000 7fd6509e4640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fd64c15f490 con 0x7fd64c15ec70 2024-08-31T12:15:55.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.416+0000 7fd6509e4640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd64c0cc130 con 0x7fd64c15f080 2024-08-31T12:15:55.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.416+0000 7fd6509e4640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fd64c152170 con 0x7fd64c0d0b50 2024-08-31T12:15:55.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.417+0000 7fd649d9b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd64c15ec70 0x7fd64c15d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:55.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.417+0000 7fd649d9b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd64c15ec70 0x7fd64c15d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:35044/0 (socket says 172.21.15.19:35044) 2024-08-31T12:15:55.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.417+0000 7fd649d9b640 1 -- 172.21.15.19:0/271846615 learned_addr learned my addr 172.21.15.19:0/271846615 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:15:55.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.417+0000 7fd64ad9d640 1 --2- 172.21.15.19:0/271846615 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd64c15f080 0x7fd64c14f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:55.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.417+0000 7fd64a59c640 1 --2- 172.21.15.19:0/271846615 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd64c0d0b50 0x7fd64c0c7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:55.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.417+0000 7fd649d9b640 1 -- 172.21.15.19:0/271846615 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd64c0d0b50 msgr2=0x7fd64c0c7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:55.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.417+0000 7fd649d9b640 1 --2- 172.21.15.19:0/271846615 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd64c0d0b50 0x7fd64c0c7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:55.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.417+0000 7fd649d9b640 1 -- 172.21.15.19:0/271846615 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd64c15f080 msgr2=0x7fd64c14f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:55.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.417+0000 7fd649d9b640 1 --2- 172.21.15.19:0/271846615 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd64c15f080 0x7fd64c14f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:55.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.417+0000 7fd649d9b640 1 -- 172.21.15.19:0/271846615 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd64c0c80c0 con 0x7fd64c15ec70 2024-08-31T12:15:55.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.417+0000 7fd64ad9d640 1 --2- 172.21.15.19:0/271846615 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd64c15f080 0x7fd64c14f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:15:55.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.417+0000 7fd649d9b640 1 --2- 172.21.15.19:0/271846615 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd64c15ec70 0x7fd64c15d5e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd64000c370 tx=0x7fd64000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:55.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.417+0000 7fd63affd640 1 -- 172.21.15.19:0/271846615 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd640017070 con 0x7fd64c15ec70 2024-08-31T12:15:55.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.417+0000 7fd63affd640 1 -- 172.21.15.19:0/271846615 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd6400035f0 con 0x7fd64c15ec70 2024-08-31T12:15:55.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.417+0000 7fd6509e4640 1 -- 172.21.15.19:0/271846615 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd64c0c95e0 con 0x7fd64c15ec70 2024-08-31T12:15:55.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.417+0000 7fd63affd640 1 -- 172.21.15.19:0/271846615 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd640003a40 con 0x7fd64c15ec70 2024-08-31T12:15:55.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.417+0000 7fd6509e4640 1 -- 172.21.15.19:0/271846615 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd64c1e4600 con 0x7fd64c15ec70 2024-08-31T12:15:55.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.418+0000 7fd6509e4640 1 -- 172.21.15.19:0/271846615 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd64c0b33a0 con 0x7fd64c15ec70 2024-08-31T12:15:55.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.419+0000 7fd63affd640 1 -- 172.21.15.19:0/271846615 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fd64001a070 con 0x7fd64c15ec70 2024-08-31T12:15:55.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.420+0000 7fd63affd640 1 --2- 172.21.15.19:0/271846615 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd62c077a80 0x7fd62c079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:55.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.420+0000 7fd63affd640 1 -- 172.21.15.19:0/271846615 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5227+0+0 (secure 0 0 0) 0x7fd640098640 con 0x7fd64c15ec70 2024-08-31T12:15:55.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.420+0000 7fd64a59c640 1 --2- 172.21.15.19:0/271846615 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd62c077a80 0x7fd62c079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:55.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.421+0000 7fd64a59c640 1 --2- 172.21.15.19:0/271846615 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd62c077a80 0x7fd62c079f90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd63c033000 tx=0x7fd63c004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:55.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.425+0000 7fd63affd640 1 -- 172.21.15.19:0/271846615 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fd640061620 con 0x7fd64c15ec70 2024-08-31T12:15:55.581 DEBUG:teuthology.orchestra.run.smithi019:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:15:55.583 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:15:55.615 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:55.614+0000 7fd6509e4640 1 -- 172.21.15.19:0/271846615 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fd64c155660 con 0x7fd64c15ec70 2024-08-31T12:15:56.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.267+0000 7fd63affd640 1 -- 172.21.15.19:0/271846615 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v19) ==== 170+0+0 (secure 0 0 0) 0x7fd640060f20 con 0x7fd64c15ec70 2024-08-31T12:15:56.268 INFO:teuthology.orchestra.run.smithi019.stderr:pool 'cephfs_data' created 2024-08-31T12:15:56.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.270+0000 7fd6509e4640 1 -- 172.21.15.19:0/271846615 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd62c077a80 msgr2=0x7fd62c079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:56.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.270+0000 7fd6509e4640 1 --2- 172.21.15.19:0/271846615 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd62c077a80 0x7fd62c079f90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd63c033000 tx=0x7fd63c004730 comp rx=0 tx=0).stop 2024-08-31T12:15:56.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.270+0000 7fd6509e4640 1 -- 172.21.15.19:0/271846615 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd64c15ec70 msgr2=0x7fd64c15d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:56.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.270+0000 7fd6509e4640 1 --2- 172.21.15.19:0/271846615 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd64c15ec70 0x7fd64c15d5e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd64000c370 tx=0x7fd64000c830 comp rx=0 tx=0).stop 2024-08-31T12:15:56.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.270+0000 7fd6509e4640 1 -- 172.21.15.19:0/271846615 shutdown_connections 2024-08-31T12:15:56.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.270+0000 7fd6509e4640 1 --2- 172.21.15.19:0/271846615 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd62c077a80 0x7fd62c079f90 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:56.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.270+0000 7fd6509e4640 1 --2- 172.21.15.19:0/271846615 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd64c15f080 0x7fd64c14f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:56.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.270+0000 7fd6509e4640 1 --2- 172.21.15.19:0/271846615 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd64c15ec70 0x7fd64c15d5e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:56.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.270+0000 7fd6509e4640 1 --2- 172.21.15.19:0/271846615 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd64c0d0b50 0x7fd64c0c7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:56.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.270+0000 7fd6509e4640 1 -- 172.21.15.19:0/271846615 >> 172.21.15.19:0/271846615 conn(0x7fd64c074210 msgr2=0x7fd64c0c6470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:56.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.270+0000 7fd6509e4640 1 -- 172.21.15.19:0/271846615 shutdown_connections 2024-08-31T12:15:56.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.270+0000 7fd6509e4640 1 -- 172.21.15.19:0/271846615 wait complete. 2024-08-31T12:15:56.301 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-08-31T12:15:56.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.421+0000 7f7b45b20640 1 -- 172.21.15.19:0/3472412158 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7b40163f70 msgr2=0x7f7b40166390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:56.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.421+0000 7f7b45b20640 1 --2- 172.21.15.19:0/3472412158 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7b40163f70 0x7f7b40166390 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7b400b4050 tx=0x7f7b3802aac0 comp rx=0 tx=0).stop 2024-08-31T12:15:56.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.421+0000 7f7b45b20640 1 -- 172.21.15.19:0/3472412158 shutdown_connections 2024-08-31T12:15:56.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.421+0000 7f7b45b20640 1 --2- 172.21.15.19:0/3472412158 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7b40163f70 0x7f7b40166390 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:56.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.421+0000 7f7b45b20640 1 --2- 172.21.15.19:0/3472412158 >> v2:172.21.15.43:3300/0 conn(0x7f7b400d0f60 0x7f7b40163a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:56.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.421+0000 7f7b45b20640 1 -- 172.21.15.19:0/3472412158 >> 172.21.15.19:0/3472412158 conn(0x7f7b40074210 msgr2=0x7f7b40074620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:56.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.422+0000 7f7b45b20640 1 -- 172.21.15.19:0/3472412158 shutdown_connections 2024-08-31T12:15:56.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.422+0000 7f7b45b20640 1 -- 172.21.15.19:0/3472412158 wait complete. 2024-08-31T12:15:56.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.423+0000 7f7b45b20640 1 Processor -- start 2024-08-31T12:15:56.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.423+0000 7f7b45b20640 1 -- start start 2024-08-31T12:15:56.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.425+0000 7f7b45b20640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7b400d0b50 0x7f7b4014d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:56.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.425+0000 7f7b45b20640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7b400d0f60 0x7f7b4014b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:56.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.425+0000 7f7b45b20640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b40163f70 0x7f7b40149030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:56.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.425+0000 7f7b45b20640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f7b400cc130 con 0x7f7b400d0f60 2024-08-31T12:15:56.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.425+0000 7f7b45b20640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7b400d1370 con 0x7f7b40163f70 2024-08-31T12:15:56.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.425+0000 7f7b45b20640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f7b40152170 con 0x7f7b400d0b50 2024-08-31T12:15:56.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.425+0000 7f7b3effd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7b400d0f60 0x7f7b4014b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:56.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.425+0000 7f7b3effd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7b400d0f60 0x7f7b4014b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:35058/0 (socket says 172.21.15.19:35058) 2024-08-31T12:15:56.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.425+0000 7f7b3effd640 1 -- 172.21.15.19:0/2676659024 learned_addr learned my addr 172.21.15.19:0/2676659024 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:15:56.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.425+0000 7f7b3ffff640 1 --2- 172.21.15.19:0/2676659024 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b40163f70 0x7f7b40149030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:56.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.425+0000 7f7b3f7fe640 1 --2- 172.21.15.19:0/2676659024 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7b400d0b50 0x7f7b4014d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:56.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.425+0000 7f7b3effd640 1 -- 172.21.15.19:0/2676659024 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7b400d0b50 msgr2=0x7f7b4014d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:56.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.425+0000 7f7b3effd640 1 --2- 172.21.15.19:0/2676659024 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7b400d0b50 0x7f7b4014d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:56.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.425+0000 7f7b3effd640 1 -- 172.21.15.19:0/2676659024 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b40163f70 msgr2=0x7f7b40149030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:56.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.425+0000 7f7b3effd640 1 --2- 172.21.15.19:0/2676659024 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b40163f70 0x7f7b40149030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:56.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.425+0000 7f7b3effd640 1 -- 172.21.15.19:0/2676659024 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b40156d70 con 0x7f7b400d0f60 2024-08-31T12:15:56.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.425+0000 7f7b3ffff640 1 --2- 172.21.15.19:0/2676659024 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b40163f70 0x7f7b40149030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-31T12:15:56.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.425+0000 7f7b3effd640 1 --2- 172.21.15.19:0/2676659024 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7b400d0f60 0x7f7b4014b170 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7b3400d780 tx=0x7f7b3400dc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:56.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.425+0000 7f7b27fff640 1 -- 172.21.15.19:0/2676659024 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b34017070 con 0x7f7b400d0f60 2024-08-31T12:15:56.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.425+0000 7f7b27fff640 1 -- 172.21.15.19:0/2676659024 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7b34009b10 con 0x7f7b400d0f60 2024-08-31T12:15:56.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.425+0000 7f7b45b20640 1 -- 172.21.15.19:0/2676659024 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b401f2440 con 0x7f7b400d0f60 2024-08-31T12:15:56.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.425+0000 7f7b27fff640 1 -- 172.21.15.19:0/2676659024 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b34010460 con 0x7f7b400d0f60 2024-08-31T12:15:56.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.425+0000 7f7b45b20640 1 -- 172.21.15.19:0/2676659024 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b401569b0 con 0x7f7b400d0f60 2024-08-31T12:15:56.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.426+0000 7f7b45b20640 1 -- 172.21.15.19:0/2676659024 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b4015de70 con 0x7f7b400d0f60 2024-08-31T12:15:56.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.427+0000 7f7b27fff640 1 -- 172.21.15.19:0/2676659024 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f7b3401a070 con 0x7f7b400d0f60 2024-08-31T12:15:56.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.428+0000 7f7b27fff640 1 --2- 172.21.15.19:0/2676659024 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7b2c077a50 0x7f7b2c079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:56.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.428+0000 7f7b27fff640 1 -- 172.21.15.19:0/2676659024 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5609+0+0 (secure 0 0 0) 0x7f7b34099780 con 0x7f7b400d0f60 2024-08-31T12:15:56.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.428+0000 7f7b3f7fe640 1 --2- 172.21.15.19:0/2676659024 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7b2c077a50 0x7f7b2c079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:56.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.428+0000 7f7b3f7fe640 1 --2- 172.21.15.19:0/2676659024 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7b2c077a50 0x7f7b2c079f60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7b3000a750 tx=0x7f7b30008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:56.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.432+0000 7f7b27fff640 1 -- 172.21.15.19:0/2676659024 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f7b34062540 con 0x7f7b400d0f60 2024-08-31T12:15:56.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:56.652+0000 7f7b45b20640 1 -- 172.21.15.19:0/2676659024 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f7b40155680 con 0x7f7b400d0f60 2024-08-31T12:15:57.293 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.291+0000 7f7b27fff640 1 -- 172.21.15.19:0/2676659024 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7f7b34061e40 con 0x7f7b400d0f60 2024-08-31T12:15:57.293 INFO:teuthology.orchestra.run.smithi019.stderr:new fs with metadata pool 2 and data pool 3 2024-08-31T12:15:57.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.301+0000 7f7b45b20640 1 -- 172.21.15.19:0/2676659024 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7b2c077a50 msgr2=0x7f7b2c079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:57.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.301+0000 7f7b45b20640 1 --2- 172.21.15.19:0/2676659024 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7b2c077a50 0x7f7b2c079f60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7b3000a750 tx=0x7f7b30008040 comp rx=0 tx=0).stop 2024-08-31T12:15:57.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.301+0000 7f7b45b20640 1 -- 172.21.15.19:0/2676659024 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7b400d0f60 msgr2=0x7f7b4014b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:57.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.301+0000 7f7b45b20640 1 --2- 172.21.15.19:0/2676659024 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7b400d0f60 0x7f7b4014b170 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7b3400d780 tx=0x7f7b3400dc40 comp rx=0 tx=0).stop 2024-08-31T12:15:57.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.302+0000 7f7b45b20640 1 -- 172.21.15.19:0/2676659024 shutdown_connections 2024-08-31T12:15:57.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.302+0000 7f7b45b20640 1 --2- 172.21.15.19:0/2676659024 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7b2c077a50 0x7f7b2c079f60 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:57.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.302+0000 7f7b45b20640 1 --2- 172.21.15.19:0/2676659024 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b40163f70 0x7f7b40149030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:57.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.302+0000 7f7b45b20640 1 --2- 172.21.15.19:0/2676659024 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7b400d0f60 0x7f7b4014b170 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:57.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.302+0000 7f7b45b20640 1 --2- 172.21.15.19:0/2676659024 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7b400d0b50 0x7f7b4014d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:57.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.302+0000 7f7b45b20640 1 -- 172.21.15.19:0/2676659024 >> 172.21.15.19:0/2676659024 conn(0x7f7b40074210 msgr2=0x7f7b4014c970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:57.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.303+0000 7f7b45b20640 1 -- 172.21.15.19:0/2676659024 shutdown_connections 2024-08-31T12:15:57.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.303+0000 7f7b45b20640 1 -- 172.21.15.19:0/2676659024 wait complete. 2024-08-31T12:15:57.326 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-31T12:15:57.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.445+0000 7f29bb007640 1 -- 172.21.15.19:0/3736746262 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f299c005250 msgr2=0x7f299c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:57.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.445+0000 7f29bb007640 1 --2- 172.21.15.19:0/3736746262 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f299c005250 0x7f299c00b9c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f29a40016c0 tx=0x7f299c005830 comp rx=0 tx=0).stop 2024-08-31T12:15:57.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.445+0000 7f29bb007640 1 -- 172.21.15.19:0/3736746262 shutdown_connections 2024-08-31T12:15:57.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.445+0000 7f29bb007640 1 --2- 172.21.15.19:0/3736746262 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f299c005250 0x7f299c00b9c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:57.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.445+0000 7f29bb007640 1 -- 172.21.15.19:0/3736746262 >> 172.21.15.19:0/3736746262 conn(0x7f29b40690d0 msgr2=0x7f29b40d0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:57.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.445+0000 7f29bb007640 1 -- 172.21.15.19:0/3736746262 shutdown_connections 2024-08-31T12:15:57.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.445+0000 7f29bb007640 1 -- 172.21.15.19:0/3736746262 wait complete. 2024-08-31T12:15:57.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.446+0000 7f29bb007640 1 Processor -- start 2024-08-31T12:15:57.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.446+0000 7f29bb007640 1 -- start start 2024-08-31T12:15:57.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.448+0000 7f29bb007640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f299c005250 0x7f29b40d0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:57.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.448+0000 7f29bb007640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29b41f44a0 0x7f29b4149e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:57.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.448+0000 7f29bb007640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f29b4153cd0 0x7f29b41540e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:57.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.448+0000 7f29bb007640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f29b415b250 con 0x7f299c005250 2024-08-31T12:15:57.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.448+0000 7f29bb007640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f29b415af70 con 0x7f29b41f44a0 2024-08-31T12:15:57.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.448+0000 7f29bb007640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f29b40c8610 con 0x7f29b4153cd0 2024-08-31T12:15:57.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.448+0000 7f29b8da3640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f299c005250 0x7f29b40d0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:57.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.448+0000 7f29b8da3640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f299c005250 0x7f29b40d0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:35076/0 (socket says 172.21.15.19:35076) 2024-08-31T12:15:57.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.448+0000 7f29b8da3640 1 -- 172.21.15.19:0/2826645668 learned_addr learned my addr 172.21.15.19:0/2826645668 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:15:57.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.448+0000 7f29b95a4640 1 --2- 172.21.15.19:0/2826645668 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f29b4153cd0 0x7f29b41540e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:57.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.448+0000 7f29b3fff640 1 --2- 172.21.15.19:0/2826645668 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29b41f44a0 0x7f29b4149e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:57.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.448+0000 7f29b8da3640 1 -- 172.21.15.19:0/2826645668 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f29b4153cd0 msgr2=0x7f29b41540e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:57.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.448+0000 7f29b8da3640 1 --2- 172.21.15.19:0/2826645668 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f29b4153cd0 0x7f29b41540e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:57.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.448+0000 7f29b8da3640 1 -- 172.21.15.19:0/2826645668 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29b41f44a0 msgr2=0x7f29b4149e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:57.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.448+0000 7f29b8da3640 1 --2- 172.21.15.19:0/2826645668 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29b41f44a0 0x7f29b4149e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:57.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.448+0000 7f29b8da3640 1 -- 172.21.15.19:0/2826645668 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29b414de00 con 0x7f299c005250 2024-08-31T12:15:57.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.448+0000 7f29b8da3640 1 --2- 172.21.15.19:0/2826645668 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f299c005250 0x7f29b40d0890 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f29ac011c70 tx=0x7f29ac00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:57.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.449+0000 7f29b17fa640 1 -- 172.21.15.19:0/2826645668 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29ac003e40 con 0x7f299c005250 2024-08-31T12:15:57.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.449+0000 7f29b17fa640 1 -- 172.21.15.19:0/2826645668 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f29ac004b20 con 0x7f299c005250 2024-08-31T12:15:57.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.449+0000 7f29bb007640 1 -- 172.21.15.19:0/2826645668 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29b414ee80 con 0x7f299c005250 2024-08-31T12:15:57.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.449+0000 7f29b17fa640 1 -- 172.21.15.19:0/2826645668 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29ac004050 con 0x7f299c005250 2024-08-31T12:15:57.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.449+0000 7f29bb007640 1 -- 172.21.15.19:0/2826645668 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29b414e5e0 con 0x7f299c005250 2024-08-31T12:15:57.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.450+0000 7f29bb007640 1 -- 172.21.15.19:0/2826645668 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29b40c9a30 con 0x7f299c005250 2024-08-31T12:15:57.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.450+0000 7f29b17fa640 1 -- 172.21.15.19:0/2826645668 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f29ac0041f0 con 0x7f299c005250 2024-08-31T12:15:57.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.451+0000 7f29b17fa640 1 --2- 172.21.15.19:0/2826645668 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f29a0077a80 0x7f29a0079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:57.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.451+0000 7f29b17fa640 1 -- 172.21.15.19:0/2826645668 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f29ac09a880 con 0x7f299c005250 2024-08-31T12:15:57.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.451+0000 7f29b3fff640 1 --2- 172.21.15.19:0/2826645668 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f29a0077a80 0x7f29a0079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:57.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.452+0000 7f29b3fff640 1 --2- 172.21.15.19:0/2826645668 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f29a0077a80 0x7f29a0079f90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f29b40c76e0 tx=0x7f29a80067a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:57.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.455+0000 7f29b17fa640 1 -- 172.21.15.19:0/2826645668 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f29ac0635f0 con 0x7f299c005250 2024-08-31T12:15:57.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.642+0000 7f29bb007640 1 -- 172.21.15.19:0/2826645668 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f29b40ccf20 con 0x7f299c005250 2024-08-31T12:15:57.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.643+0000 7f29b17fa640 1 -- 172.21.15.19:0/2826645668 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16630 (secure 0 0 0) 0x7f29ac062ef0 con 0x7f299c005250 2024-08-31T12:15:57.644 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:15:57.644 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":20,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","created":"2024-08-31T12:15:32.084309+0000","modified":"2024-08-31T12:15:56.652979+0000","last_up_change":"2024-08-31T12:15:45.213688+0000","last_in_change":"2024-08-31T12:15:35.545202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-31T12:15:44.508197+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-31T12:15:54.512584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-31T12:15:55.618251+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b80d6a3f-7e4e-4d4d-b1b4-0df71c9176ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6800","nonce":680447146},{"type":"v1","addr":"172.21.15.19:6801","nonce":680447146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":680447146},{"type":"v1","addr":"172.21.15.19:6803","nonce":680447146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":680447146},{"type":"v1","addr":"172.21.15.19:6807","nonce":680447146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":680447146},{"type":"v1","addr":"172.21.15.19:6805","nonce":680447146}]},"public_addr":"172.21.15.19:6801/680447146","cluster_addr":"172.21.15.19:6803/680447146","heartbeat_back_addr":"172.21.15.19:6807/680447146","heartbeat_front_addr":"172.21.15.19:6805/680447146","state":["exists","up"]},{"osd":1,"uuid":"6f986366-82a6-4133-9f2e-43f84c7b03bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":1522610034},{"type":"v1","addr":"172.21.15.19:6817","nonce":1522610034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":1522610034},{"type":"v1","addr":"172.21.15.19:6819","nonce":1522610034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":1522610034},{"type":"v1","addr":"172.21.15.19:6823","nonce":1522610034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":1522610034},{"type":"v1","addr":"172.21.15.19:6821","nonce":1522610034}]},"public_addr":"172.21.15.19:6817/1522610034","cluster_addr":"172.21.15.19:6819/1522610034","heartbeat_back_addr":"172.21.15.19:6823/1522610034","heartbeat_front_addr":"172.21.15.19:6821/1522610034","state":["exists","up"]},{"osd":2,"uuid":"ce8c2068-845c-4cbb-a4d1-077375d605ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6809","nonce":2332368824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6811","nonce":2332368824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6815","nonce":2332368824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6813","nonce":2332368824}]},"public_addr":"172.21.15.19:6809/2332368824","cluster_addr":"172.21.15.19:6811/2332368824","heartbeat_back_addr":"172.21.15.19:6815/2332368824","heartbeat_front_addr":"172.21.15.19:6813/2332368824","state":["exists","up"]},{"osd":3,"uuid":"b1cb0ef4-aa9b-421b-a94e-82de0c46e5cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":3785550598},{"type":"v1","addr":"172.21.15.19:6825","nonce":3785550598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":3785550598},{"type":"v1","addr":"172.21.15.19:6827","nonce":3785550598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":3785550598},{"type":"v1","addr":"172.21.15.19:6831","nonce":3785550598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":3785550598},{"type":"v1","addr":"172.21.15.19:6829","nonce":3785550598}]},"public_addr":"172.21.15.19:6825/3785550598","cluster_addr":"172.21.15.19:6827/3785550598","heartbeat_back_addr":"172.21.15.19:6831/3785550598","heartbeat_front_addr":"172.21.15.19:6829/3785550598","state":["exists","up"]},{"osd":4,"uuid":"825b58ec-a6da-4515-b6a5-81debe327011","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":1985519349},{"type":"v1","addr":"172.21.15.43:6819","nonce":1985519349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":1985519349},{"type":"v1","addr":"172.21.15.43:6821","nonce":1985519349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":1985519349},{"type":"v1","addr":"172.21.15.43:6825","nonce":1985519349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":1985519349},{"type":"v1","addr":"172.21.15.43:6823","nonce":1985519349}]},"public_addr":"172.21.15.43:6819/1985519349","cluster_addr":"172.21.15.43:6821/1985519349","heartbeat_back_addr":"172.21.15.43:6825/1985519349","heartbeat_front_addr":"172.21.15.43:6823/1985519349","state":["exists","up"]},{"osd":5,"uuid":"7f34e537-5997-40e4-92e7-2570507160c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":4271971521},{"type":"v1","addr":"172.21.15.43:6803","nonce":4271971521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":4271971521},{"type":"v1","addr":"172.21.15.43:6805","nonce":4271971521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":4271971521},{"type":"v1","addr":"172.21.15.43:6809","nonce":4271971521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":4271971521},{"type":"v1","addr":"172.21.15.43:6807","nonce":4271971521}]},"public_addr":"172.21.15.43:6803/4271971521","cluster_addr":"172.21.15.43:6805/4271971521","heartbeat_back_addr":"172.21.15.43:6809/4271971521","heartbeat_front_addr":"172.21.15.43:6807/4271971521","state":["exists","up"]},{"osd":6,"uuid":"195dd8ad-640a-4fa3-9ee4-06e4172e0464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2863384694},{"type":"v1","addr":"172.21.15.43:6827","nonce":2863384694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2863384694},{"type":"v1","addr":"172.21.15.43:6829","nonce":2863384694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":2863384694},{"type":"v1","addr":"172.21.15.43:6833","nonce":2863384694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2863384694},{"type":"v1","addr":"172.21.15.43:6831","nonce":2863384694}]},"public_addr":"172.21.15.43:6827/2863384694","cluster_addr":"172.21.15.43:6829/2863384694","heartbeat_back_addr":"172.21.15.43:6833/2863384694","heartbeat_front_addr":"172.21.15.43:6831/2863384694","state":["exists","up"]},{"osd":7,"uuid":"4645661f-f430-4f05-bdd1-69138dcd1bbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":792572234},{"type":"v1","addr":"172.21.15.43:6811","nonce":792572234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":792572234},{"type":"v1","addr":"172.21.15.43:6813","nonce":792572234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":792572234},{"type":"v1","addr":"172.21.15.43:6817","nonce":792572234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":792572234},{"type":"v1","addr":"172.21.15.43:6815","nonce":792572234}]},"public_addr":"172.21.15.43:6811/792572234","cluster_addr":"172.21.15.43:6813/792572234","heartbeat_back_addr":"172.21.15.43:6817/792572234","heartbeat_front_addr":"172.21.15.43:6815/792572234","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.156485+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.578126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.075440+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.639867+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.543602+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:41.919592+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.615557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:41.813337+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-31T12:15:57.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.645+0000 7f29bb007640 1 -- 172.21.15.19:0/2826645668 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f29a0077a80 msgr2=0x7f29a0079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:57.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.645+0000 7f29bb007640 1 --2- 172.21.15.19:0/2826645668 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f29a0077a80 0x7f29a0079f90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f29b40c76e0 tx=0x7f29a80067a0 comp rx=0 tx=0).stop 2024-08-31T12:15:57.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.645+0000 7f29bb007640 1 -- 172.21.15.19:0/2826645668 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f299c005250 msgr2=0x7f29b40d0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:57.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.645+0000 7f29bb007640 1 --2- 172.21.15.19:0/2826645668 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f299c005250 0x7f29b40d0890 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f29ac011c70 tx=0x7f29ac00f790 comp rx=0 tx=0).stop 2024-08-31T12:15:57.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.645+0000 7f29bb007640 1 -- 172.21.15.19:0/2826645668 shutdown_connections 2024-08-31T12:15:57.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.645+0000 7f29bb007640 1 --2- 172.21.15.19:0/2826645668 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f29a0077a80 0x7f29a0079f90 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:57.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.645+0000 7f29bb007640 1 --2- 172.21.15.19:0/2826645668 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f29b4153cd0 0x7f29b41540e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:57.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.645+0000 7f29bb007640 1 --2- 172.21.15.19:0/2826645668 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29b41f44a0 0x7f29b4149e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:57.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.645+0000 7f29bb007640 1 --2- 172.21.15.19:0/2826645668 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f299c005250 0x7f29b40d0890 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:57.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.645+0000 7f29bb007640 1 -- 172.21.15.19:0/2826645668 >> 172.21.15.19:0/2826645668 conn(0x7f29b40690d0 msgr2=0x7f29b40cc7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:57.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.645+0000 7f29bb007640 1 -- 172.21.15.19:0/2826645668 shutdown_connections 2024-08-31T12:15:57.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.645+0000 7f29bb007640 1 -- 172.21.15.19:0/2826645668 wait complete. 2024-08-31T12:15:57.657 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-31T12:15:57.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.770+0000 7f62de48d640 1 -- 172.21.15.19:0/1376639784 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62c0003c10 msgr2=0x7f62c0004060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:57.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.770+0000 7f62de48d640 1 --2- 172.21.15.19:0/1376639784 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62c0003c10 0x7f62c0004060 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f62c8009b50 tx=0x7f62c0005830 comp rx=0 tx=0).stop 2024-08-31T12:15:57.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.770+0000 7f62dca2a640 1 -- 172.21.15.19:0/1376639784 reap_dead start 2024-08-31T12:15:57.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.770+0000 7f62de48d640 1 -- 172.21.15.19:0/1376639784 shutdown_connections 2024-08-31T12:15:57.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.770+0000 7f62de48d640 1 -- 172.21.15.19:0/1376639784 >> 172.21.15.19:0/1376639784 conn(0x7f62d806e6e0 msgr2=0x7f62d806eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:57.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.770+0000 7f62de48d640 1 -- 172.21.15.19:0/1376639784 shutdown_connections 2024-08-31T12:15:57.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.770+0000 7f62de48d640 1 -- 172.21.15.19:0/1376639784 wait complete. 2024-08-31T12:15:57.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.772+0000 7f62de48d640 1 Processor -- start 2024-08-31T12:15:57.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.772+0000 7f62de48d640 1 -- start start 2024-08-31T12:15:57.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.773+0000 7f62de48d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f62c0003c10 0x7f62d814f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:57.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.773+0000 7f62de48d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62d814c0a0 0x7f62d81f16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:57.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.773+0000 7f62de48d640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62d8151170 0x7f62d8151580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:57.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.773+0000 7f62de48d640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f62d815d090 con 0x7f62d814c0a0 2024-08-31T12:15:57.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.773+0000 7f62de48d640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f62d815cdb0 con 0x7f62d8151170 2024-08-31T12:15:57.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.773+0000 7f62de48d640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f62d80cc320 con 0x7f62c0003c10 2024-08-31T12:15:57.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.773+0000 7f62d77fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62d814c0a0 0x7f62d81f16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:57.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.773+0000 7f62d77fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62d814c0a0 0x7f62d81f16c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:35096/0 (socket says 172.21.15.19:35096) 2024-08-31T12:15:57.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.774+0000 7f62d77fe640 1 -- 172.21.15.19:0/433235752 learned_addr learned my addr 172.21.15.19:0/433235752 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:15:57.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.774+0000 7f62dca2a640 1 --2- 172.21.15.19:0/433235752 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62d8151170 0x7f62d8151580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:57.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.774+0000 7f62d7fff640 1 --2- 172.21.15.19:0/433235752 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f62c0003c10 0x7f62d814f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:57.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.774+0000 7f62d77fe640 1 -- 172.21.15.19:0/433235752 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f62c0003c10 msgr2=0x7f62d814f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:57.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.774+0000 7f62d77fe640 1 --2- 172.21.15.19:0/433235752 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f62c0003c10 0x7f62d814f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:57.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.774+0000 7f62d77fe640 1 -- 172.21.15.19:0/433235752 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62d8151170 msgr2=0x7f62d8151580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:57.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.774+0000 7f62d77fe640 1 --2- 172.21.15.19:0/433235752 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62d8151170 0x7f62d8151580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:57.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.774+0000 7f62d77fe640 1 -- 172.21.15.19:0/433235752 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62d81f1c00 con 0x7f62d814c0a0 2024-08-31T12:15:57.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.774+0000 7f62d7fff640 1 --2- 172.21.15.19:0/433235752 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f62c0003c10 0x7f62d814f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-31T12:15:57.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.774+0000 7f62d77fe640 1 --2- 172.21.15.19:0/433235752 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62d814c0a0 0x7f62d81f16c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f62cc004e00 tx=0x7f62cc001800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:57.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.774+0000 7f62d4ff9640 1 -- 172.21.15.19:0/433235752 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62cc010890 con 0x7f62d814c0a0 2024-08-31T12:15:57.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.774+0000 7f62d4ff9640 1 -- 172.21.15.19:0/433235752 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f62cc016070 con 0x7f62d814c0a0 2024-08-31T12:15:57.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.774+0000 7f62de48d640 1 -- 172.21.15.19:0/433235752 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62d814cc40 con 0x7f62d814c0a0 2024-08-31T12:15:57.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.774+0000 7f62d4ff9640 1 -- 172.21.15.19:0/433235752 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62cc010dc0 con 0x7f62d814c0a0 2024-08-31T12:15:57.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.774+0000 7f62de48d640 1 -- 172.21.15.19:0/433235752 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62d80cfd40 con 0x7f62d814c0a0 2024-08-31T12:15:57.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.775+0000 7f62de48d640 1 -- 172.21.15.19:0/433235752 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62d80cab80 con 0x7f62d814c0a0 2024-08-31T12:15:57.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.776+0000 7f62d4ff9640 1 -- 172.21.15.19:0/433235752 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f62cc01b460 con 0x7f62d814c0a0 2024-08-31T12:15:57.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.776+0000 7f62d4ff9640 1 --2- 172.21.15.19:0/433235752 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f62c4077a50 0x7f62c4079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:57.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.777+0000 7f62d4ff9640 1 -- 172.21.15.19:0/433235752 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f62cc09a170 con 0x7f62d814c0a0 2024-08-31T12:15:57.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.777+0000 7f62d7fff640 1 --2- 172.21.15.19:0/433235752 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f62c4077a50 0x7f62c4079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:57.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.777+0000 7f62d7fff640 1 --2- 172.21.15.19:0/433235752 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f62c4077a50 0x7f62c4079f60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f62c8035000 tx=0x7f62c802d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:57.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.781+0000 7f62d4ff9640 1 -- 172.21.15.19:0/433235752 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f62cc062db0 con 0x7f62d814c0a0 2024-08-31T12:15:57.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.967+0000 7f62de48d640 1 -- 172.21.15.19:0/433235752 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f62d8158e90 con 0x7f62d814c0a0 2024-08-31T12:15:57.969 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.968+0000 7f62d4ff9640 1 -- 172.21.15.19:0/433235752 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16630 (secure 0 0 0) 0x7f62cc06b2f0 con 0x7f62d814c0a0 2024-08-31T12:15:57.970 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:15:57.970 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":20,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","created":"2024-08-31T12:15:32.084309+0000","modified":"2024-08-31T12:15:56.652979+0000","last_up_change":"2024-08-31T12:15:45.213688+0000","last_in_change":"2024-08-31T12:15:35.545202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-31T12:15:44.508197+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-31T12:15:54.512584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-31T12:15:55.618251+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b80d6a3f-7e4e-4d4d-b1b4-0df71c9176ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6800","nonce":680447146},{"type":"v1","addr":"172.21.15.19:6801","nonce":680447146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":680447146},{"type":"v1","addr":"172.21.15.19:6803","nonce":680447146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":680447146},{"type":"v1","addr":"172.21.15.19:6807","nonce":680447146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":680447146},{"type":"v1","addr":"172.21.15.19:6805","nonce":680447146}]},"public_addr":"172.21.15.19:6801/680447146","cluster_addr":"172.21.15.19:6803/680447146","heartbeat_back_addr":"172.21.15.19:6807/680447146","heartbeat_front_addr":"172.21.15.19:6805/680447146","state":["exists","up"]},{"osd":1,"uuid":"6f986366-82a6-4133-9f2e-43f84c7b03bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":1522610034},{"type":"v1","addr":"172.21.15.19:6817","nonce":1522610034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":1522610034},{"type":"v1","addr":"172.21.15.19:6819","nonce":1522610034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":1522610034},{"type":"v1","addr":"172.21.15.19:6823","nonce":1522610034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":1522610034},{"type":"v1","addr":"172.21.15.19:6821","nonce":1522610034}]},"public_addr":"172.21.15.19:6817/1522610034","cluster_addr":"172.21.15.19:6819/1522610034","heartbeat_back_addr":"172.21.15.19:6823/1522610034","heartbeat_front_addr":"172.21.15.19:6821/1522610034","state":["exists","up"]},{"osd":2,"uuid":"ce8c2068-845c-4cbb-a4d1-077375d605ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6809","nonce":2332368824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6811","nonce":2332368824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6815","nonce":2332368824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6813","nonce":2332368824}]},"public_addr":"172.21.15.19:6809/2332368824","cluster_addr":"172.21.15.19:6811/2332368824","heartbeat_back_addr":"172.21.15.19:6815/2332368824","heartbeat_front_addr":"172.21.15.19:6813/2332368824","state":["exists","up"]},{"osd":3,"uuid":"b1cb0ef4-aa9b-421b-a94e-82de0c46e5cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":3785550598},{"type":"v1","addr":"172.21.15.19:6825","nonce":3785550598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":3785550598},{"type":"v1","addr":"172.21.15.19:6827","nonce":3785550598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":3785550598},{"type":"v1","addr":"172.21.15.19:6831","nonce":3785550598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":3785550598},{"type":"v1","addr":"172.21.15.19:6829","nonce":3785550598}]},"public_addr":"172.21.15.19:6825/3785550598","cluster_addr":"172.21.15.19:6827/3785550598","heartbeat_back_addr":"172.21.15.19:6831/3785550598","heartbeat_front_addr":"172.21.15.19:6829/3785550598","state":["exists","up"]},{"osd":4,"uuid":"825b58ec-a6da-4515-b6a5-81debe327011","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":1985519349},{"type":"v1","addr":"172.21.15.43:6819","nonce":1985519349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":1985519349},{"type":"v1","addr":"172.21.15.43:6821","nonce":1985519349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":1985519349},{"type":"v1","addr":"172.21.15.43:6825","nonce":1985519349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":1985519349},{"type":"v1","addr":"172.21.15.43:6823","nonce":1985519349}]},"public_addr":"172.21.15.43:6819/1985519349","cluster_addr":"172.21.15.43:6821/1985519349","heartbeat_back_addr":"172.21.15.43:6825/1985519349","heartbeat_front_addr":"172.21.15.43:6823/1985519349","state":["exists","up"]},{"osd":5,"uuid":"7f34e537-5997-40e4-92e7-2570507160c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":4271971521},{"type":"v1","addr":"172.21.15.43:6803","nonce":4271971521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":4271971521},{"type":"v1","addr":"172.21.15.43:6805","nonce":4271971521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":4271971521},{"type":"v1","addr":"172.21.15.43:6809","nonce":4271971521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":4271971521},{"type":"v1","addr":"172.21.15.43:6807","nonce":4271971521}]},"public_addr":"172.21.15.43:6803/4271971521","cluster_addr":"172.21.15.43:6805/4271971521","heartbeat_back_addr":"172.21.15.43:6809/4271971521","heartbeat_front_addr":"172.21.15.43:6807/4271971521","state":["exists","up"]},{"osd":6,"uuid":"195dd8ad-640a-4fa3-9ee4-06e4172e0464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2863384694},{"type":"v1","addr":"172.21.15.43:6827","nonce":2863384694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2863384694},{"type":"v1","addr":"172.21.15.43:6829","nonce":2863384694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":2863384694},{"type":"v1","addr":"172.21.15.43:6833","nonce":2863384694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2863384694},{"type":"v1","addr":"172.21.15.43:6831","nonce":2863384694}]},"public_addr":"172.21.15.43:6827/2863384694","cluster_addr":"172.21.15.43:6829/2863384694","heartbeat_back_addr":"172.21.15.43:6833/2863384694","heartbeat_front_addr":"172.21.15.43:6831/2863384694","state":["exists","up"]},{"osd":7,"uuid":"4645661f-f430-4f05-bdd1-69138dcd1bbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":792572234},{"type":"v1","addr":"172.21.15.43:6811","nonce":792572234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":792572234},{"type":"v1","addr":"172.21.15.43:6813","nonce":792572234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":792572234},{"type":"v1","addr":"172.21.15.43:6817","nonce":792572234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":792572234},{"type":"v1","addr":"172.21.15.43:6815","nonce":792572234}]},"public_addr":"172.21.15.43:6811/792572234","cluster_addr":"172.21.15.43:6813/792572234","heartbeat_back_addr":"172.21.15.43:6817/792572234","heartbeat_front_addr":"172.21.15.43:6815/792572234","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.156485+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.578126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.075440+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.639867+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.543602+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:41.919592+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.615557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:41.813337+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-31T12:15:57.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.971+0000 7f62de48d640 1 -- 172.21.15.19:0/433235752 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f62c4077a50 msgr2=0x7f62c4079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:57.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.971+0000 7f62de48d640 1 --2- 172.21.15.19:0/433235752 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f62c4077a50 0x7f62c4079f60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f62c8035000 tx=0x7f62c802d040 comp rx=0 tx=0).stop 2024-08-31T12:15:57.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.971+0000 7f62de48d640 1 -- 172.21.15.19:0/433235752 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62d814c0a0 msgr2=0x7f62d81f16c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:57.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.971+0000 7f62de48d640 1 --2- 172.21.15.19:0/433235752 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62d814c0a0 0x7f62d81f16c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f62cc004e00 tx=0x7f62cc001800 comp rx=0 tx=0).stop 2024-08-31T12:15:57.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.971+0000 7f62de48d640 1 -- 172.21.15.19:0/433235752 shutdown_connections 2024-08-31T12:15:57.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.971+0000 7f62de48d640 1 --2- 172.21.15.19:0/433235752 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f62c4077a50 0x7f62c4079f60 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:57.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.971+0000 7f62de48d640 1 --2- 172.21.15.19:0/433235752 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62d8151170 0x7f62d8151580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:57.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.971+0000 7f62de48d640 1 --2- 172.21.15.19:0/433235752 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62d814c0a0 0x7f62d81f16c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:57.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.971+0000 7f62de48d640 1 --2- 172.21.15.19:0/433235752 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f62c0003c10 0x7f62d814f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:57.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.971+0000 7f62de48d640 1 -- 172.21.15.19:0/433235752 >> 172.21.15.19:0/433235752 conn(0x7f62d806e6e0 msgr2=0x7f62d80c94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:57.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.971+0000 7f62de48d640 1 -- 172.21.15.19:0/433235752 shutdown_connections 2024-08-31T12:15:57.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:57.971+0000 7f62de48d640 1 -- 172.21.15.19:0/433235752 wait complete. 2024-08-31T12:15:57.983 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-08-31T12:15:58.099 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:58.098+0000 7f81dcde6640 1 -- 172.21.15.19:0/2368542446 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81d80cab00 msgr2=0x7f81d8072650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:58.099 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:58.098+0000 7f81dcde6640 1 --2- 172.21.15.19:0/2368542446 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81d80cab00 0x7f81d8072650 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f81c0009830 tx=0x7f81c002a5a0 comp rx=0 tx=0).stop 2024-08-31T12:15:58.099 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:58.098+0000 7f81dcde6640 1 -- 172.21.15.19:0/2368542446 shutdown_connections 2024-08-31T12:15:58.100 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:58.098+0000 7f81dcde6640 1 --2- 172.21.15.19:0/2368542446 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81d80cab00 0x7f81d8072650 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:58.100 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:58.098+0000 7f81dcde6640 1 --2- 172.21.15.19:0/2368542446 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f81d80c8fc0 0x7f81d8153140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:58.100 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:58.098+0000 7f81dcde6640 1 -- 172.21.15.19:0/2368542446 >> 172.21.15.19:0/2368542446 conn(0x7f81d8071150 msgr2=0x7f81d806cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:58.100 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:58.098+0000 7f81dcde6640 1 -- 172.21.15.19:0/2368542446 shutdown_connections 2024-08-31T12:15:58.100 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:58.098+0000 7f81dcde6640 1 -- 172.21.15.19:0/2368542446 wait complete. 2024-08-31T12:15:58.101 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:58.100+0000 7f81dcde6640 1 Processor -- start 2024-08-31T12:15:58.101 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:58.100+0000 7f81dcde6640 1 -- start start 2024-08-31T12:15:58.104 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:58.102+0000 7f81dcde6640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f81d80c51b0 0x7f81d814d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:58.104 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:58.102+0000 7f81dcde6640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f81d80c8fc0 0x7f81d8143540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:58.104 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:58.102+0000 7f81dcde6640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81d80cab00 0x7f81d8143a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:58.104 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:58.102+0000 7f81dcde6640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f81d80cef10 con 0x7f81d80c8fc0 2024-08-31T12:15:58.104 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:58.102+0000 7f81dcde6640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f81d8159650 con 0x7f81d80cab00 2024-08-31T12:15:58.104 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:58.102+0000 7f81dcde6640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f81d81e28b0 con 0x7f81d80c51b0 2024-08-31T12:15:58.104 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:58.102+0000 7f81cdd9b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f81d80c8fc0 0x7f81d8143540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:58.104 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:58.102+0000 7f81cdd9b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f81d80c8fc0 0x7f81d8143540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:35106/0 (socket says 172.21.15.19:35106) 2024-08-31T12:15:58.104 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:58.102+0000 7f81cdd9b640 1 -- 172.21.15.19:0/3191995792 learned_addr learned my addr 172.21.15.19:0/3191995792 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:15:58.104 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:58.102+0000 7f81d659c640 1 --2- 172.21.15.19:0/3191995792 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f81d80c51b0 0x7f81d814d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:58.104 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:58.102+0000 7f81d6d9d640 1 --2- 172.21.15.19:0/3191995792 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81d80cab00 0x7f81d8143a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:58.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:58.102+0000 7f81cdd9b640 1 -- 172.21.15.19:0/3191995792 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f81d80c51b0 msgr2=0x7f81d814d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:58.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:58.102+0000 7f81cdd9b640 1 --2- 172.21.15.19:0/3191995792 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f81d80c51b0 0x7f81d814d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:58.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:58.102+0000 7f81cdd9b640 1 -- 172.21.15.19:0/3191995792 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81d80cab00 msgr2=0x7f81d8143a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:58.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:58.102+0000 7f81cdd9b640 1 --2- 172.21.15.19:0/3191995792 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81d80cab00 0x7f81d8143a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:58.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:58.102+0000 7f81cdd9b640 1 -- 172.21.15.19:0/3191995792 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81d814de90 con 0x7f81d80c8fc0 2024-08-31T12:15:58.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:58.102+0000 7f81cdd9b640 1 --2- 172.21.15.19:0/3191995792 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f81d80c8fc0 0x7f81d8143540 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f81c0000c00 tx=0x7f81c0033ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:58.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:58.102+0000 7f81cf7fe640 1 -- 172.21.15.19:0/3191995792 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c0031b50 con 0x7f81d80c8fc0 2024-08-31T12:15:58.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:58.102+0000 7f81cf7fe640 1 -- 172.21.15.19:0/3191995792 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f81c0031de0 con 0x7f81d80c8fc0 2024-08-31T12:15:58.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:58.102+0000 7f81dcde6640 1 -- 172.21.15.19:0/3191995792 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81d81521d0 con 0x7f81d80c8fc0 2024-08-31T12:15:58.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:58.102+0000 7f81cf7fe640 1 -- 172.21.15.19:0/3191995792 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c0033330 con 0x7f81d80c8fc0 2024-08-31T12:15:58.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:58.103+0000 7f81dcde6640 1 -- 172.21.15.19:0/3191995792 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81d8152470 con 0x7f81d80c8fc0 2024-08-31T12:15:58.106 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:58.103+0000 7f81dcde6640 1 -- 172.21.15.19:0/3191995792 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81d8153140 con 0x7f81d80c8fc0 2024-08-31T12:15:58.106 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:58.104+0000 7f81cf7fe640 1 -- 172.21.15.19:0/3191995792 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f81c00334d0 con 0x7f81d80c8fc0 2024-08-31T12:15:58.106 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:58.105+0000 7f81cf7fe640 1 --2- 172.21.15.19:0/3191995792 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f81b8077a50 0x7f81b8079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:58.106 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:58.105+0000 7f81cf7fe640 1 -- 172.21.15.19:0/3191995792 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f81c00ba860 con 0x7f81d80c8fc0 2024-08-31T12:15:58.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:58.105+0000 7f81d659c640 1 --2- 172.21.15.19:0/3191995792 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f81b8077a50 0x7f81b8079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:58.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:58.105+0000 7f81d659c640 1 --2- 172.21.15.19:0/3191995792 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f81b8077a50 0x7f81b8079f60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f81c8005fd0 tx=0x7f81c8009260 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:58.110 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:58.109+0000 7f81cf7fe640 1 -- 172.21.15.19:0/3191995792 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f81c003f030 con 0x7f81d80c8fc0 2024-08-31T12:15:58.336 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:58.335+0000 7f81dcde6640 1 -- 172.21.15.19:0/3191995792 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f81d80c9b20 con 0x7f81d80c8fc0 2024-08-31T12:15:59.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.303+0000 7f81cf7fe640 1 -- 172.21.15.19:0/3191995792 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v5) ==== 118+0+0 (secure 0 0 0) 0x7f81c00ba070 con 0x7f81d80c8fc0 2024-08-31T12:15:59.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.307+0000 7f81dcde6640 1 -- 172.21.15.19:0/3191995792 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f81b8077a50 msgr2=0x7f81b8079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:59.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.307+0000 7f81dcde6640 1 --2- 172.21.15.19:0/3191995792 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f81b8077a50 0x7f81b8079f60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f81c8005fd0 tx=0x7f81c8009260 comp rx=0 tx=0).stop 2024-08-31T12:15:59.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.307+0000 7f81dcde6640 1 -- 172.21.15.19:0/3191995792 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f81d80c8fc0 msgr2=0x7f81d8143540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:59.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.307+0000 7f81dcde6640 1 --2- 172.21.15.19:0/3191995792 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f81d80c8fc0 0x7f81d8143540 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f81c0000c00 tx=0x7f81c0033ba0 comp rx=0 tx=0).stop 2024-08-31T12:15:59.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.308+0000 7f81dcde6640 1 -- 172.21.15.19:0/3191995792 shutdown_connections 2024-08-31T12:15:59.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.308+0000 7f81dcde6640 1 --2- 172.21.15.19:0/3191995792 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f81b8077a50 0x7f81b8079f60 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:59.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.308+0000 7f81dcde6640 1 --2- 172.21.15.19:0/3191995792 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81d80cab00 0x7f81d8143a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:59.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.308+0000 7f81dcde6640 1 --2- 172.21.15.19:0/3191995792 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f81d80c8fc0 0x7f81d8143540 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:59.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.308+0000 7f81dcde6640 1 --2- 172.21.15.19:0/3191995792 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f81d80c51b0 0x7f81d814d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:59.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.308+0000 7f81dcde6640 1 -- 172.21.15.19:0/3191995792 >> 172.21.15.19:0/3191995792 conn(0x7f81d8071150 msgr2=0x7f81d8149340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:59.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.308+0000 7f81dcde6640 1 -- 172.21.15.19:0/3191995792 shutdown_connections 2024-08-31T12:15:59.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.308+0000 7f81dcde6640 1 -- 172.21.15.19:0/3191995792 wait complete. 2024-08-31T12:15:59.323 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300} 2024-08-31T12:15:59.323 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-08-31T12:15:59.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.449+0000 7feaf06a4640 1 -- 172.21.15.19:0/2069012586 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7feae80d0b50 msgr2=0x7feae80d0f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:59.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.449+0000 7feaf06a4640 1 --2- 172.21.15.19:0/2069012586 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7feae80d0b50 0x7feae80d0f60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7feae808e5b0 tx=0x7feae002a5f0 comp rx=0 tx=0).stop 2024-08-31T12:15:59.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.449+0000 7feaf06a4640 1 -- 172.21.15.19:0/2069012586 shutdown_connections 2024-08-31T12:15:59.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.449+0000 7feaf06a4640 1 --2- 172.21.15.19:0/2069012586 >> v2:172.21.15.43:3300/0 conn(0x7feae80cd1b0 0x7feae80cd5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:59.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.449+0000 7feaf06a4640 1 --2- 172.21.15.19:0/2069012586 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feae815ec70 0x7feae815f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:59.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.449+0000 7feaf06a4640 1 --2- 172.21.15.19:0/2069012586 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7feae80d0b50 0x7feae80d0f60 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:59.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.449+0000 7feaf06a4640 1 -- 172.21.15.19:0/2069012586 >> 172.21.15.19:0/2069012586 conn(0x7feae8074210 msgr2=0x7feae8074620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:15:59.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.449+0000 7feaf06a4640 1 -- 172.21.15.19:0/2069012586 shutdown_connections 2024-08-31T12:15:59.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.449+0000 7feaf06a4640 1 -- 172.21.15.19:0/2069012586 wait complete. 2024-08-31T12:15:59.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.451+0000 7feaf06a4640 1 Processor -- start 2024-08-31T12:15:59.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.451+0000 7feaf06a4640 1 -- start start 2024-08-31T12:15:59.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.452+0000 7feaf06a4640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feae80cd1b0 0x7feae815c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:59.459 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.452+0000 7feaf06a4640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feae80d0b50 0x7feae81e71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:59.459 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.452+0000 7feaf06a4640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7feae815ec70 0x7feae814b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:59.459 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.452+0000 7feaf06a4640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7feae8152170 con 0x7feae815ec70 2024-08-31T12:15:59.459 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.452+0000 7feaf06a4640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7feae80cc130 con 0x7feae80cd1b0 2024-08-31T12:15:59.459 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.452+0000 7feaf06a4640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7feae8155680 con 0x7feae80d0b50 2024-08-31T12:15:59.459 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.452+0000 7feaeec41640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7feae815ec70 0x7feae814b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:59.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.452+0000 7feaeec41640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7feae815ec70 0x7feae814b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:56420/0 (socket says 172.21.15.19:56420) 2024-08-31T12:15:59.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.452+0000 7feaeec41640 1 -- 172.21.15.19:0/2440565703 learned_addr learned my addr 172.21.15.19:0/2440565703 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:15:59.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.452+0000 7feaee440640 1 --2- 172.21.15.19:0/2440565703 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feae80cd1b0 0x7feae815c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:59.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.452+0000 7feaedc3f640 1 --2- 172.21.15.19:0/2440565703 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feae80d0b50 0x7feae81e71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:59.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.453+0000 7feaeec41640 1 -- 172.21.15.19:0/2440565703 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feae80d0b50 msgr2=0x7feae81e71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:59.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.453+0000 7feaeec41640 1 --2- 172.21.15.19:0/2440565703 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feae80d0b50 0x7feae81e71f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:59.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.453+0000 7feaeec41640 1 -- 172.21.15.19:0/2440565703 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feae80cd1b0 msgr2=0x7feae815c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:15:59.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.453+0000 7feaeec41640 1 --2- 172.21.15.19:0/2440565703 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feae80cd1b0 0x7feae815c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:15:59.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.453+0000 7feaeec41640 1 -- 172.21.15.19:0/2440565703 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feae815c850 con 0x7feae815ec70 2024-08-31T12:15:59.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.453+0000 7feaeec41640 1 --2- 172.21.15.19:0/2440565703 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7feae815ec70 0x7feae814b330 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fead800e9c0 tx=0x7fead800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:59.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.453+0000 7feae6ffd640 1 -- 172.21.15.19:0/2440565703 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead8004d20 con 0x7feae815ec70 2024-08-31T12:15:59.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.453+0000 7feae6ffd640 1 -- 172.21.15.19:0/2440565703 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fead800cac0 con 0x7feae815ec70 2024-08-31T12:15:59.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.453+0000 7feae6ffd640 1 -- 172.21.15.19:0/2440565703 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead8005020 con 0x7feae815ec70 2024-08-31T12:15:59.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.453+0000 7feaf06a4640 1 -- 172.21.15.19:0/2440565703 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feae80cf7c0 con 0x7feae815ec70 2024-08-31T12:15:59.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.453+0000 7feaf06a4640 1 -- 172.21.15.19:0/2440565703 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feae815e320 con 0x7feae815ec70 2024-08-31T12:15:59.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.454+0000 7feaf06a4640 1 -- 172.21.15.19:0/2440565703 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feae80b33a0 con 0x7feae815ec70 2024-08-31T12:15:59.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.455+0000 7feae6ffd640 1 -- 172.21.15.19:0/2440565703 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fead80051c0 con 0x7feae815ec70 2024-08-31T12:15:59.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.455+0000 7feae6ffd640 1 --2- 172.21.15.19:0/2440565703 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fead4077a50 0x7fead4079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:15:59.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.455+0000 7feae6ffd640 1 -- 172.21.15.19:0/2440565703 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fead8099d30 con 0x7feae815ec70 2024-08-31T12:15:59.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.455+0000 7feaee440640 1 --2- 172.21.15.19:0/2440565703 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fead4077a50 0x7fead4079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:15:59.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.456+0000 7feaee440640 1 --2- 172.21.15.19:0/2440565703 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fead4077a50 0x7fead4079f60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7feae0033000 tx=0x7feae0004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:15:59.464 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.460+0000 7feae6ffd640 1 -- 172.21.15.19:0/2440565703 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fead8016030 con 0x7feae815ec70 2024-08-31T12:15:59.682 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:15:59.681+0000 7feaf06a4640 1 -- 172.21.15.19:0/2440565703 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7feae80b5730 con 0x7feae815ec70 2024-08-31T12:16:00.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.305+0000 7feae6ffd640 1 -- 172.21.15.19:0/2440565703 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v6) ==== 105+0+0 (secure 0 0 0) 0x7fead80624e0 con 0x7feae815ec70 2024-08-31T12:16:00.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.308+0000 7feaf06a4640 1 -- 172.21.15.19:0/2440565703 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fead4077a50 msgr2=0x7fead4079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:00.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.308+0000 7feaf06a4640 1 --2- 172.21.15.19:0/2440565703 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fead4077a50 0x7fead4079f60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7feae0033000 tx=0x7feae0004730 comp rx=0 tx=0).stop 2024-08-31T12:16:00.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.308+0000 7feaf06a4640 1 -- 172.21.15.19:0/2440565703 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7feae815ec70 msgr2=0x7feae814b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:00.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.308+0000 7feaf06a4640 1 --2- 172.21.15.19:0/2440565703 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7feae815ec70 0x7feae814b330 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fead800e9c0 tx=0x7fead800ee80 comp rx=0 tx=0).stop 2024-08-31T12:16:00.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.308+0000 7feaf06a4640 1 -- 172.21.15.19:0/2440565703 shutdown_connections 2024-08-31T12:16:00.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.308+0000 7feaf06a4640 1 --2- 172.21.15.19:0/2440565703 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fead4077a50 0x7fead4079f60 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:00.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.308+0000 7feaf06a4640 1 --2- 172.21.15.19:0/2440565703 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7feae815ec70 0x7feae814b330 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:00.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.308+0000 7feaf06a4640 1 --2- 172.21.15.19:0/2440565703 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feae80d0b50 0x7feae81e71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:00.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.308+0000 7feaf06a4640 1 --2- 172.21.15.19:0/2440565703 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feae80cd1b0 0x7feae815c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:00.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.308+0000 7feaf06a4640 1 -- 172.21.15.19:0/2440565703 >> 172.21.15.19:0/2440565703 conn(0x7feae8074210 msgr2=0x7feae815a200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:00.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.308+0000 7feaf06a4640 1 -- 172.21.15.19:0/2440565703 shutdown_connections 2024-08-31T12:16:00.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.308+0000 7feaf06a4640 1 -- 172.21.15.19:0/2440565703 wait complete. 2024-08-31T12:16:00.323 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-08-31T12:16:00.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.446+0000 7faecfa7f640 1 -- 172.21.15.19:0/769538297 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faec80d0b50 msgr2=0x7faec80d0f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:00.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.446+0000 7faecfa7f640 1 --2- 172.21.15.19:0/769538297 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faec80d0b50 0x7faec80d0f60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7faec0005fd0 tx=0x7faec002a670 comp rx=0 tx=0).stop 2024-08-31T12:16:00.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.446+0000 7faecfa7f640 1 -- 172.21.15.19:0/769538297 shutdown_connections 2024-08-31T12:16:00.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.446+0000 7faecfa7f640 1 --2- 172.21.15.19:0/769538297 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faec815f080 0x7faec80cd8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:00.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.446+0000 7faecfa7f640 1 --2- 172.21.15.19:0/769538297 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faec80d0b50 0x7faec80d0f60 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:00.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.446+0000 7faecfa7f640 1 -- 172.21.15.19:0/769538297 >> 172.21.15.19:0/769538297 conn(0x7faec8074210 msgr2=0x7faec8074620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:00.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.446+0000 7faecfa7f640 1 -- 172.21.15.19:0/769538297 shutdown_connections 2024-08-31T12:16:00.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.446+0000 7faecfa7f640 1 -- 172.21.15.19:0/769538297 wait complete. 2024-08-31T12:16:00.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.448+0000 7faecfa7f640 1 Processor -- start 2024-08-31T12:16:00.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.448+0000 7faecfa7f640 1 -- start start 2024-08-31T12:16:00.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.449+0000 7faecfa7f640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faec80d0b50 0x7faec814e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:00.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.449+0000 7faecfa7f640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faec815ec70 0x7faec8153e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:00.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.449+0000 7faecfa7f640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faec815f080 0x7faec815d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:00.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.449+0000 7faecfa7f640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7faec815f490 con 0x7faec815ec70 2024-08-31T12:16:00.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.449+0000 7faecfa7f640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7faec80cc130 con 0x7faec815f080 2024-08-31T12:16:00.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.449+0000 7faecfa7f640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7faec8152170 con 0x7faec80d0b50 2024-08-31T12:16:00.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.449+0000 7faecd01a640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faec815ec70 0x7faec8153e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:00.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.449+0000 7faecd01a640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faec815ec70 0x7faec8153e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:56442/0 (socket says 172.21.15.19:56442) 2024-08-31T12:16:00.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.449+0000 7faecd01a640 1 -- 172.21.15.19:0/3247043249 learned_addr learned my addr 172.21.15.19:0/3247043249 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:00.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.449+0000 7faecd81b640 1 --2- 172.21.15.19:0/3247043249 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faec80d0b50 0x7faec814e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:00.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.449+0000 7faece01c640 1 --2- 172.21.15.19:0/3247043249 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faec815f080 0x7faec815d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:00.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.450+0000 7faecd01a640 1 -- 172.21.15.19:0/3247043249 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faec80d0b50 msgr2=0x7faec814e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:00.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.450+0000 7faecd01a640 1 --2- 172.21.15.19:0/3247043249 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faec80d0b50 0x7faec814e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:00.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.450+0000 7faecd01a640 1 -- 172.21.15.19:0/3247043249 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faec815f080 msgr2=0x7faec815d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:00.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.450+0000 7faecd01a640 1 --2- 172.21.15.19:0/3247043249 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faec815f080 0x7faec815d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:00.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.450+0000 7faecd01a640 1 -- 172.21.15.19:0/3247043249 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faec80c9f20 con 0x7faec815ec70 2024-08-31T12:16:00.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.450+0000 7faecd81b640 1 --2- 172.21.15.19:0/3247043249 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faec80d0b50 0x7faec814e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:16:00.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.450+0000 7faece01c640 1 --2- 172.21.15.19:0/3247043249 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faec815f080 0x7faec815d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:16:00.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.450+0000 7faecd01a640 1 --2- 172.21.15.19:0/3247043249 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faec815ec70 0x7faec8153e30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7faebc00c370 tx=0x7faebc00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:00.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.450+0000 7faec67fc640 1 -- 172.21.15.19:0/3247043249 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faebc017070 con 0x7faec815ec70 2024-08-31T12:16:00.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.450+0000 7faec67fc640 1 -- 172.21.15.19:0/3247043249 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faebc0053c0 con 0x7faec815ec70 2024-08-31T12:16:00.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.450+0000 7faecfa7f640 1 -- 172.21.15.19:0/3247043249 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faec8154370 con 0x7faec815ec70 2024-08-31T12:16:00.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.450+0000 7faec67fc640 1 -- 172.21.15.19:0/3247043249 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faebc003850 con 0x7faec815ec70 2024-08-31T12:16:00.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.450+0000 7faecfa7f640 1 -- 172.21.15.19:0/3247043249 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faec8153630 con 0x7faec815ec70 2024-08-31T12:16:00.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.451+0000 7faecfa7f640 1 -- 172.21.15.19:0/3247043249 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faec8155680 con 0x7faec815ec70 2024-08-31T12:16:00.454 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.452+0000 7faec67fc640 1 -- 172.21.15.19:0/3247043249 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7faebc01a070 con 0x7faec815ec70 2024-08-31T12:16:00.454 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.452+0000 7faec67fc640 1 --2- 172.21.15.19:0/3247043249 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7faeb4077a50 0x7faeb4079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:00.454 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.452+0000 7faec67fc640 1 -- 172.21.15.19:0/3247043249 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7faebc098b30 con 0x7faec815ec70 2024-08-31T12:16:00.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.452+0000 7faecd81b640 1 --2- 172.21.15.19:0/3247043249 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7faeb4077a50 0x7faeb4079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:00.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.453+0000 7faecd81b640 1 --2- 172.21.15.19:0/3247043249 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7faeb4077a50 0x7faeb4079f60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7faec0033000 tx=0x7faec0004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:00.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.457+0000 7faec67fc640 1 -- 172.21.15.19:0/3247043249 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7faebc061790 con 0x7faec815ec70 2024-08-31T12:16:00.680 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:00.680+0000 7faecfa7f640 1 -- 172.21.15.19:0/3247043249 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7faec80b33a0 con 0x7faec815ec70 2024-08-31T12:16:01.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.314+0000 7faec67fc640 1 -- 172.21.15.19:0/3247043249 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v7) ==== 122+0+0 (secure 0 0 0) 0x7faebc061090 con 0x7faec815ec70 2024-08-31T12:16:01.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.316+0000 7faecfa7f640 1 -- 172.21.15.19:0/3247043249 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7faeb4077a50 msgr2=0x7faeb4079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:01.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.316+0000 7faecfa7f640 1 --2- 172.21.15.19:0/3247043249 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7faeb4077a50 0x7faeb4079f60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7faec0033000 tx=0x7faec0004730 comp rx=0 tx=0).stop 2024-08-31T12:16:01.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.316+0000 7faecfa7f640 1 -- 172.21.15.19:0/3247043249 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faec815ec70 msgr2=0x7faec8153e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:01.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.316+0000 7faecfa7f640 1 --2- 172.21.15.19:0/3247043249 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faec815ec70 0x7faec8153e30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7faebc00c370 tx=0x7faebc00c830 comp rx=0 tx=0).stop 2024-08-31T12:16:01.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.316+0000 7faecfa7f640 1 -- 172.21.15.19:0/3247043249 shutdown_connections 2024-08-31T12:16:01.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.316+0000 7faecfa7f640 1 --2- 172.21.15.19:0/3247043249 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7faeb4077a50 0x7faeb4079f60 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:01.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.316+0000 7faecfa7f640 1 --2- 172.21.15.19:0/3247043249 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faec815f080 0x7faec815d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:01.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.316+0000 7faecfa7f640 1 --2- 172.21.15.19:0/3247043249 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faec815ec70 0x7faec8153e30 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:01.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.316+0000 7faecfa7f640 1 --2- 172.21.15.19:0/3247043249 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faec80d0b50 0x7faec814e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:01.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.316+0000 7faecfa7f640 1 -- 172.21.15.19:0/3247043249 >> 172.21.15.19:0/3247043249 conn(0x7faec8074210 msgr2=0x7faec8159c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:01.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.316+0000 7faecfa7f640 1 -- 172.21.15.19:0/3247043249 shutdown_connections 2024-08-31T12:16:01.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.316+0000 7faecfa7f640 1 -- 172.21.15.19:0/3247043249 wait complete. 2024-08-31T12:16:01.328 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-08-31T12:16:01.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.445+0000 7f2ef03cc640 1 -- 172.21.15.19:0/1558195862 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2ee8151dc0 msgr2=0x7f2ee8074560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:01.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.445+0000 7f2ef03cc640 1 --2- 172.21.15.19:0/1558195862 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2ee8151dc0 0x7f2ee8074560 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2ee4009830 tx=0x7f2ee402a5a0 comp rx=0 tx=0).stop 2024-08-31T12:16:01.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.446+0000 7f2ef03cc640 1 -- 172.21.15.19:0/1558195862 shutdown_connections 2024-08-31T12:16:01.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.446+0000 7f2ef03cc640 1 --2- 172.21.15.19:0/1558195862 >> v2:172.21.15.43:3300/0 conn(0x7f2ee8157e80 0x7f2ee8158290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:01.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.446+0000 7f2ef03cc640 1 --2- 172.21.15.19:0/1558195862 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2ee8151dc0 0x7f2ee8074560 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:01.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.446+0000 7f2ef03cc640 1 -- 172.21.15.19:0/1558195862 >> 172.21.15.19:0/1558195862 conn(0x7f2ee80cd840 msgr2=0x7f2ee80cdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:01.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.446+0000 7f2ef03cc640 1 -- 172.21.15.19:0/1558195862 shutdown_connections 2024-08-31T12:16:01.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.446+0000 7f2ef03cc640 1 -- 172.21.15.19:0/1558195862 wait complete. 2024-08-31T12:16:01.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.447+0000 7f2ef03cc640 1 Processor -- start 2024-08-31T12:16:01.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.447+0000 7f2ef03cc640 1 -- start start 2024-08-31T12:16:01.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.449+0000 7f2ef03cc640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2ee8151dc0 0x7f2ee80c8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:01.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.449+0000 7f2ef03cc640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ee81521d0 0x7f2ee80c35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:01.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.449+0000 7f2ef03cc640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2ee8157e80 0x7f2ee80c3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:01.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.449+0000 7f2ef03cc640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f2ee8072280 con 0x7f2ee8157e80 2024-08-31T12:16:01.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.449+0000 7f2ef03cc640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2ee81587d0 con 0x7f2ee81521d0 2024-08-31T12:16:01.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.449+0000 7f2ef03cc640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f2ee8072400 con 0x7f2ee8151dc0 2024-08-31T12:16:01.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.449+0000 7f2eee969640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2ee8157e80 0x7f2ee80c3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:01.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.449+0000 7f2eee969640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2ee8157e80 0x7f2ee80c3b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:56454/0 (socket says 172.21.15.19:56454) 2024-08-31T12:16:01.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.449+0000 7f2eee969640 1 -- 172.21.15.19:0/177514369 learned_addr learned my addr 172.21.15.19:0/177514369 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:01.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.449+0000 7f2eed967640 1 --2- 172.21.15.19:0/177514369 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ee81521d0 0x7f2ee80c35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:01.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.449+0000 7f2eee168640 1 --2- 172.21.15.19:0/177514369 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2ee8151dc0 0x7f2ee80c8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:01.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.449+0000 7f2eee969640 1 -- 172.21.15.19:0/177514369 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2ee8151dc0 msgr2=0x7f2ee80c8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:01.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.449+0000 7f2eee969640 1 --2- 172.21.15.19:0/177514369 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2ee8151dc0 0x7f2ee80c8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:01.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.449+0000 7f2eee969640 1 -- 172.21.15.19:0/177514369 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ee81521d0 msgr2=0x7f2ee80c35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:01.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.449+0000 7f2eee969640 1 --2- 172.21.15.19:0/177514369 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ee81521d0 0x7f2ee80c35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:01.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.449+0000 7f2eee969640 1 -- 172.21.15.19:0/177514369 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ee81f2380 con 0x7f2ee8157e80 2024-08-31T12:16:01.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.449+0000 7f2eee969640 1 --2- 172.21.15.19:0/177514369 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2ee8157e80 0x7f2ee80c3b00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2ee000ec20 tx=0x7f2ee000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:01.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.449+0000 7f2edeffd640 1 -- 172.21.15.19:0/177514369 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ee0010630 con 0x7f2ee8157e80 2024-08-31T12:16:01.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.449+0000 7f2edeffd640 1 -- 172.21.15.19:0/177514369 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2ee00107d0 con 0x7f2ee8157e80 2024-08-31T12:16:01.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.449+0000 7f2edeffd640 1 -- 172.21.15.19:0/177514369 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ee0010be0 con 0x7f2ee8157e80 2024-08-31T12:16:01.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.449+0000 7f2ef03cc640 1 -- 172.21.15.19:0/177514369 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ee81f15c0 con 0x7f2ee8157e80 2024-08-31T12:16:01.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.450+0000 7f2ef03cc640 1 -- 172.21.15.19:0/177514369 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ee81f1b80 con 0x7f2ee8157e80 2024-08-31T12:16:01.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.450+0000 7f2ef03cc640 1 -- 172.21.15.19:0/177514369 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ee80cd000 con 0x7f2ee8157e80 2024-08-31T12:16:01.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.451+0000 7f2edeffd640 1 -- 172.21.15.19:0/177514369 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f2ee0021430 con 0x7f2ee8157e80 2024-08-31T12:16:01.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.452+0000 7f2edeffd640 1 --2- 172.21.15.19:0/177514369 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2ecc077a50 0x7f2ecc079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:01.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.452+0000 7f2edeffd640 1 -- 172.21.15.19:0/177514369 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2ee009a3b0 con 0x7f2ee8157e80 2024-08-31T12:16:01.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.452+0000 7f2eee168640 1 --2- 172.21.15.19:0/177514369 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2ecc077a50 0x7f2ecc079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:01.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.453+0000 7f2eee168640 1 --2- 172.21.15.19:0/177514369 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2ecc077a50 0x7f2ecc079f60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2ee4034000 tx=0x7f2ee4004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:01.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.457+0000 7f2edeffd640 1 -- 172.21.15.19:0/177514369 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f2ee0062b60 con 0x7f2ee8157e80 2024-08-31T12:16:01.678 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:01.677+0000 7f2ef03cc640 1 -- 172.21.15.19:0/177514369 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f2ee80ccd90 con 0x7f2ee8157e80 2024-08-31T12:16:02.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.315+0000 7f2edeffd640 1 -- 172.21.15.19:0/177514369 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v8) ==== 115+0+0 (secure 0 0 0) 0x7f2ee0067320 con 0x7f2ee8157e80 2024-08-31T12:16:02.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.318+0000 7f2ef03cc640 1 -- 172.21.15.19:0/177514369 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2ecc077a50 msgr2=0x7f2ecc079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:02.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.318+0000 7f2ef03cc640 1 --2- 172.21.15.19:0/177514369 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2ecc077a50 0x7f2ecc079f60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2ee4034000 tx=0x7f2ee4004730 comp rx=0 tx=0).stop 2024-08-31T12:16:02.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.318+0000 7f2ef03cc640 1 -- 172.21.15.19:0/177514369 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2ee8157e80 msgr2=0x7f2ee80c3b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:02.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.318+0000 7f2ef03cc640 1 --2- 172.21.15.19:0/177514369 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2ee8157e80 0x7f2ee80c3b00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2ee000ec20 tx=0x7f2ee000c6a0 comp rx=0 tx=0).stop 2024-08-31T12:16:02.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.318+0000 7f2ef03cc640 1 -- 172.21.15.19:0/177514369 shutdown_connections 2024-08-31T12:16:02.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.318+0000 7f2ef03cc640 1 --2- 172.21.15.19:0/177514369 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2ecc077a50 0x7f2ecc079f60 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:02.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.318+0000 7f2ef03cc640 1 --2- 172.21.15.19:0/177514369 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2ee8157e80 0x7f2ee80c3b00 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:02.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.318+0000 7f2ef03cc640 1 --2- 172.21.15.19:0/177514369 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ee81521d0 0x7f2ee80c35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:02.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.318+0000 7f2ef03cc640 1 --2- 172.21.15.19:0/177514369 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2ee8151dc0 0x7f2ee80c8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:02.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.318+0000 7f2ef03cc640 1 -- 172.21.15.19:0/177514369 >> 172.21.15.19:0/177514369 conn(0x7f2ee80cd840 msgr2=0x7f2ee80635a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:02.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.318+0000 7f2ef03cc640 1 -- 172.21.15.19:0/177514369 shutdown_connections 2024-08-31T12:16:02.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.318+0000 7f2ef03cc640 1 -- 172.21.15.19:0/177514369 wait complete. 2024-08-31T12:16:02.332 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-08-31T12:16:02.332 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-31T12:16:02.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.451+0000 7fc11f870640 1 -- 172.21.15.19:0/2080702606 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc104004d40 msgr2=0x7fc104005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:02.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.451+0000 7fc11f870640 1 --2- 172.21.15.19:0/2080702606 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc104004d40 0x7fc104005150 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc110006fd0 tx=0x7fc104005830 comp rx=0 tx=0).stop 2024-08-31T12:16:02.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.451+0000 7fc11de0d640 1 -- 172.21.15.19:0/2080702606 reap_dead start 2024-08-31T12:16:02.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.451+0000 7fc11f870640 1 -- 172.21.15.19:0/2080702606 shutdown_connections 2024-08-31T12:16:02.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.451+0000 7fc11f870640 1 -- 172.21.15.19:0/2080702606 >> 172.21.15.19:0/2080702606 conn(0x7fc1180690d0 msgr2=0x7fc1180d0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:02.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.451+0000 7fc11f870640 1 -- 172.21.15.19:0/2080702606 shutdown_connections 2024-08-31T12:16:02.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.451+0000 7fc11f870640 1 -- 172.21.15.19:0/2080702606 wait complete. 2024-08-31T12:16:02.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.453+0000 7fc11f870640 1 Processor -- start 2024-08-31T12:16:02.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.453+0000 7fc11f870640 1 -- start start 2024-08-31T12:16:02.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.454+0000 7fc11f870640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc104004d40 0x7fc11815bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:02.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.454+0000 7fc11f870640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc118156ca0 0x7fc1181540a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:02.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.454+0000 7fc11f870640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc1181f0e50 0x7fc1180c6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:02.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.454+0000 7fc11f870640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fc1180c8330 con 0x7fc104004d40 2024-08-31T12:16:02.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.454+0000 7fc11f870640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc11815b250 con 0x7fc118156ca0 2024-08-31T12:16:02.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.454+0000 7fc11f870640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fc1180c8610 con 0x7fc1181f0e50 2024-08-31T12:16:02.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.454+0000 7fc11d60c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc104004d40 0x7fc11815bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:02.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.454+0000 7fc11d60c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc104004d40 0x7fc11815bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:56464/0 (socket says 172.21.15.19:56464) 2024-08-31T12:16:02.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.454+0000 7fc11d60c640 1 -- 172.21.15.19:0/3049857933 learned_addr learned my addr 172.21.15.19:0/3049857933 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:02.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.454+0000 7fc11ce0b640 1 --2- 172.21.15.19:0/3049857933 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc118156ca0 0x7fc1181540a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:02.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.454+0000 7fc11de0d640 1 --2- 172.21.15.19:0/3049857933 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc1181f0e50 0x7fc1180c6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:02.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.454+0000 7fc11d60c640 1 -- 172.21.15.19:0/3049857933 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc1181f0e50 msgr2=0x7fc1180c6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:02.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.454+0000 7fc11d60c640 1 --2- 172.21.15.19:0/3049857933 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc1181f0e50 0x7fc1180c6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:02.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.454+0000 7fc11d60c640 1 -- 172.21.15.19:0/3049857933 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc118156ca0 msgr2=0x7fc1181540a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:02.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.454+0000 7fc11d60c640 1 --2- 172.21.15.19:0/3049857933 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc118156ca0 0x7fc1181540a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:02.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.454+0000 7fc11d60c640 1 -- 172.21.15.19:0/3049857933 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc11815eb60 con 0x7fc104004d40 2024-08-31T12:16:02.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.454+0000 7fc11de0d640 1 --2- 172.21.15.19:0/3049857933 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc1181f0e50 0x7fc1180c6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:16:02.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.455+0000 7fc11d60c640 1 --2- 172.21.15.19:0/3049857933 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc104004d40 0x7fc11815bd40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc114011b10 tx=0x7fc114011f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:02.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.455+0000 7fc10dffb640 1 -- 172.21.15.19:0/3049857933 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc11401e070 con 0x7fc104004d40 2024-08-31T12:16:02.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.455+0000 7fc10dffb640 1 -- 172.21.15.19:0/3049857933 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc114004460 con 0x7fc104004d40 2024-08-31T12:16:02.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.455+0000 7fc11f870640 1 -- 172.21.15.19:0/3049857933 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc11815e5c0 con 0x7fc104004d40 2024-08-31T12:16:02.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.455+0000 7fc10dffb640 1 -- 172.21.15.19:0/3049857933 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc11400fc40 con 0x7fc104004d40 2024-08-31T12:16:02.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.455+0000 7fc11f870640 1 -- 172.21.15.19:0/3049857933 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc118157200 con 0x7fc104004d40 2024-08-31T12:16:02.459 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.456+0000 7fc11f870640 1 -- 172.21.15.19:0/3049857933 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1180c4210 con 0x7fc104004d40 2024-08-31T12:16:02.459 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.457+0000 7fc10dffb640 1 -- 172.21.15.19:0/3049857933 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fc114021070 con 0x7fc104004d40 2024-08-31T12:16:02.459 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.457+0000 7fc10dffb640 1 --2- 172.21.15.19:0/3049857933 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc0fc077a50 0x7fc0fc079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:02.459 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.457+0000 7fc10dffb640 1 -- 172.21.15.19:0/3049857933 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc11409a060 con 0x7fc104004d40 2024-08-31T12:16:02.459 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.457+0000 7fc11ce0b640 1 --2- 172.21.15.19:0/3049857933 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc0fc077a50 0x7fc0fc079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:02.459 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.458+0000 7fc11ce0b640 1 --2- 172.21.15.19:0/3049857933 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc0fc077a50 0x7fc0fc079f60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc1180c3f70 tx=0x7fc108005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:02.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.462+0000 7fc10dffb640 1 -- 172.21.15.19:0/3049857933 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fc1180c4210 con 0x7fc104004d40 2024-08-31T12:16:02.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.683+0000 7fc11f870640 1 -- 172.21.15.19:0/3049857933 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc118155300 con 0x7fc104004d40 2024-08-31T12:16:02.685 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.684+0000 7fc10dffb640 1 -- 172.21.15.19:0/3049857933 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+6876 (secure 0 0 0) 0x7fc114062840 con 0x7fc104004d40 2024-08-31T12:16:02.685 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:16:02.685 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":9,"btime":"2024-08-31T12:16:02:319023+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4371,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6833/4215834946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":4215834946},{"type":"v1","addr":"172.21.15.19:6833","nonce":4215834946}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/543344732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":543344732},{"type":"v1","addr":"172.21.15.19:6835","nonce":543344732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4380,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6837/699077015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6836","nonce":699077015},{"type":"v1","addr":"172.21.15.19:6837","nonce":699077015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4398,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/2103490885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6835","nonce":2103490885},{"type":"v1","addr":"172.21.15.43:6837","nonce":2103490885}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-31T12:15:56.652936+0000","modified":"2024-08-31T12:16:02.318998+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1],"up":{"mds_0":4429,"mds_1":4426},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4426":{"gid":4426,"name":"b","rank":1,"incarnation":9,"state":"up:creating","state_seq":1,"addr":"172.21.15.43:6839/1196858061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1196858061},{"type":"v1","addr":"172.21.15.43:6839","nonce":1196858061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4429":{"gid":4429,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.43:6838/3981040655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":3981040655},{"type":"v1","addr":"172.21.15.43:6838","nonce":3981040655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4429,"qdb_cluster":[4429]},"id":1}]} 2024-08-31T12:16:02.685 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 9 2024-08-31T12:16:02.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.686+0000 7fc11f870640 1 -- 172.21.15.19:0/3049857933 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc0fc077a50 msgr2=0x7fc0fc079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:02.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.686+0000 7fc11f870640 1 --2- 172.21.15.19:0/3049857933 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc0fc077a50 0x7fc0fc079f60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc1180c3f70 tx=0x7fc108005f90 comp rx=0 tx=0).stop 2024-08-31T12:16:02.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.686+0000 7fc11f870640 1 -- 172.21.15.19:0/3049857933 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc104004d40 msgr2=0x7fc11815bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:02.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.686+0000 7fc11f870640 1 --2- 172.21.15.19:0/3049857933 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc104004d40 0x7fc11815bd40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc114011b10 tx=0x7fc114011f10 comp rx=0 tx=0).stop 2024-08-31T12:16:02.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.686+0000 7fc11f870640 1 -- 172.21.15.19:0/3049857933 shutdown_connections 2024-08-31T12:16:02.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.686+0000 7fc11f870640 1 --2- 172.21.15.19:0/3049857933 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc0fc077a50 0x7fc0fc079f60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:02.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.686+0000 7fc11f870640 1 --2- 172.21.15.19:0/3049857933 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc1181f0e50 0x7fc1180c6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:02.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.686+0000 7fc11f870640 1 --2- 172.21.15.19:0/3049857933 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc118156ca0 0x7fc1181540a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:02.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.686+0000 7fc11f870640 1 --2- 172.21.15.19:0/3049857933 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc104004d40 0x7fc11815bd40 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:02.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.686+0000 7fc11f870640 1 -- 172.21.15.19:0/3049857933 >> 172.21.15.19:0/3049857933 conn(0x7fc1180690d0 msgr2=0x7fc118159a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:02.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.686+0000 7fc11f870640 1 -- 172.21.15.19:0/3049857933 shutdown_connections 2024-08-31T12:16:02.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.686+0000 7fc11f870640 1 -- 172.21.15.19:0/3049857933 wait complete. 2024-08-31T12:16:02.698 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-31T12:16:02.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.811+0000 7fbb128d5640 1 -- 172.21.15.19:0/2946283157 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbb0c155300 msgr2=0x7fbb0c06d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:02.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.811+0000 7fbb128d5640 1 --2- 172.21.15.19:0/2946283157 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbb0c155300 0x7fbb0c06d000 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c15b0f0 tx=0x7fbb0402ab40 comp rx=0 tx=0).stop 2024-08-31T12:16:02.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.811+0000 7fbb128d5640 1 -- 172.21.15.19:0/2946283157 shutdown_connections 2024-08-31T12:16:02.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.811+0000 7fbb128d5640 1 --2- 172.21.15.19:0/2946283157 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbb0c155300 0x7fbb0c06d000 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:02.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.811+0000 7fbb128d5640 1 --2- 172.21.15.19:0/2946283157 >> v2:172.21.15.43:3300/0 conn(0x7fbb0c06e0e0 0x7fbb0c0c9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:02.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.811+0000 7fbb128d5640 1 -- 172.21.15.19:0/2946283157 >> 172.21.15.19:0/2946283157 conn(0x7fbb0c0690d0 msgr2=0x7fbb0c0d0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:02.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.811+0000 7fbb128d5640 1 -- 172.21.15.19:0/2946283157 shutdown_connections 2024-08-31T12:16:02.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.811+0000 7fbb128d5640 1 -- 172.21.15.19:0/2946283157 wait complete. 2024-08-31T12:16:02.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.812+0000 7fbb128d5640 1 Processor -- start 2024-08-31T12:16:02.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.813+0000 7fbb128d5640 1 -- start start 2024-08-31T12:16:02.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.814+0000 7fbb128d5640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbb0c06e0e0 0x7fbb0c1e4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:02.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.814+0000 7fbb128d5640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbb0c0c8a40 0x7fbb0c1592e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:02.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.814+0000 7fbb128d5640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbb0c155300 0x7fbb0c1e1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:02.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.814+0000 7fbb128d5640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fbb0c0c8330 con 0x7fbb0c06e0e0 2024-08-31T12:16:02.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.814+0000 7fbb128d5640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fbb0c15b250 con 0x7fbb0c0c8a40 2024-08-31T12:16:02.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.814+0000 7fbb128d5640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fbb0c0c8610 con 0x7fbb0c155300 2024-08-31T12:16:02.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.814+0000 7fbb0bfff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbb0c06e0e0 0x7fbb0c1e4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:02.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.814+0000 7fbb0bfff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbb0c06e0e0 0x7fbb0c1e4d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:56476/0 (socket says 172.21.15.19:56476) 2024-08-31T12:16:02.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.814+0000 7fbb0bfff640 1 -- 172.21.15.19:0/2314863011 learned_addr learned my addr 172.21.15.19:0/2314863011 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:02.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.814+0000 7fbb0b7fe640 1 --2- 172.21.15.19:0/2314863011 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbb0c0c8a40 0x7fbb0c1592e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:02.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.814+0000 7fbb10e72640 1 --2- 172.21.15.19:0/2314863011 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbb0c155300 0x7fbb0c1e1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:02.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.814+0000 7fbb0bfff640 1 -- 172.21.15.19:0/2314863011 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbb0c155300 msgr2=0x7fbb0c1e1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:02.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.814+0000 7fbb0bfff640 1 --2- 172.21.15.19:0/2314863011 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbb0c155300 0x7fbb0c1e1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:02.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.814+0000 7fbb0bfff640 1 -- 172.21.15.19:0/2314863011 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbb0c0c8a40 msgr2=0x7fbb0c1592e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:02.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.814+0000 7fbb0bfff640 1 --2- 172.21.15.19:0/2314863011 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbb0c0c8a40 0x7fbb0c1592e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:02.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.814+0000 7fbb0bfff640 1 -- 172.21.15.19:0/2314863011 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb0c1e52b0 con 0x7fbb0c06e0e0 2024-08-31T12:16:02.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.814+0000 7fbb0bfff640 1 --2- 172.21.15.19:0/2314863011 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbb0c06e0e0 0x7fbb0c1e4d70 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbafc00c3d0 tx=0x7fbafc00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:02.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.814+0000 7fbb08ff9640 1 -- 172.21.15.19:0/2314863011 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbafc017070 con 0x7fbb0c06e0e0 2024-08-31T12:16:02.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.815+0000 7fbb08ff9640 1 -- 172.21.15.19:0/2314863011 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbafc0053c0 con 0x7fbb0c06e0e0 2024-08-31T12:16:02.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.815+0000 7fbb128d5640 1 -- 172.21.15.19:0/2314863011 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb0c1f5820 con 0x7fbb0c06e0e0 2024-08-31T12:16:02.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.815+0000 7fbb08ff9640 1 -- 172.21.15.19:0/2314863011 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbafc003850 con 0x7fbb0c06e0e0 2024-08-31T12:16:02.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.815+0000 7fbb128d5640 1 -- 172.21.15.19:0/2314863011 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb0c0cf870 con 0x7fbb0c06e0e0 2024-08-31T12:16:02.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.816+0000 7fbb128d5640 1 -- 172.21.15.19:0/2314863011 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb0c0c9850 con 0x7fbb0c06e0e0 2024-08-31T12:16:02.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.816+0000 7fbb08ff9640 1 -- 172.21.15.19:0/2314863011 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fbafc01a070 con 0x7fbb0c06e0e0 2024-08-31T12:16:02.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.817+0000 7fbb08ff9640 1 --2- 172.21.15.19:0/2314863011 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fbaf8077a50 0x7fbaf8079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:02.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.817+0000 7fbb08ff9640 1 -- 172.21.15.19:0/2314863011 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fbafc0988c0 con 0x7fbb0c06e0e0 2024-08-31T12:16:02.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.817+0000 7fbb0b7fe640 1 --2- 172.21.15.19:0/2314863011 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fbaf8077a50 0x7fbaf8079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:02.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.818+0000 7fbb0b7fe640 1 --2- 172.21.15.19:0/2314863011 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fbaf8077a50 0x7fbaf8079f60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c0c97b0 tx=0x7fbb0000b2e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:02.822 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:02.822+0000 7fbb08ff9640 1 -- 172.21.15.19:0/2314863011 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fbafc061070 con 0x7fbb0c06e0e0 2024-08-31T12:16:03.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.008+0000 7fbb128d5640 1 -- 172.21.15.19:0/2314863011 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbb0c0ccf20 con 0x7fbb0c06e0e0 2024-08-31T12:16:03.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.009+0000 7fbb08ff9640 1 -- 172.21.15.19:0/2314863011 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16617 (secure 0 0 0) 0x7fbafc065830 con 0x7fbb0c06e0e0 2024-08-31T12:16:03.010 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:16:03.010 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":21,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","created":"2024-08-31T12:15:32.084309+0000","modified":"2024-08-31T12:15:58.292413+0000","last_up_change":"2024-08-31T12:15:45.213688+0000","last_in_change":"2024-08-31T12:15:35.545202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-31T12:15:44.508197+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-31T12:15:54.512584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-31T12:15:55.618251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b80d6a3f-7e4e-4d4d-b1b4-0df71c9176ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6800","nonce":680447146},{"type":"v1","addr":"172.21.15.19:6801","nonce":680447146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":680447146},{"type":"v1","addr":"172.21.15.19:6803","nonce":680447146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":680447146},{"type":"v1","addr":"172.21.15.19:6807","nonce":680447146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":680447146},{"type":"v1","addr":"172.21.15.19:6805","nonce":680447146}]},"public_addr":"172.21.15.19:6801/680447146","cluster_addr":"172.21.15.19:6803/680447146","heartbeat_back_addr":"172.21.15.19:6807/680447146","heartbeat_front_addr":"172.21.15.19:6805/680447146","state":["exists","up"]},{"osd":1,"uuid":"6f986366-82a6-4133-9f2e-43f84c7b03bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":1522610034},{"type":"v1","addr":"172.21.15.19:6817","nonce":1522610034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":1522610034},{"type":"v1","addr":"172.21.15.19:6819","nonce":1522610034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":1522610034},{"type":"v1","addr":"172.21.15.19:6823","nonce":1522610034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":1522610034},{"type":"v1","addr":"172.21.15.19:6821","nonce":1522610034}]},"public_addr":"172.21.15.19:6817/1522610034","cluster_addr":"172.21.15.19:6819/1522610034","heartbeat_back_addr":"172.21.15.19:6823/1522610034","heartbeat_front_addr":"172.21.15.19:6821/1522610034","state":["exists","up"]},{"osd":2,"uuid":"ce8c2068-845c-4cbb-a4d1-077375d605ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6809","nonce":2332368824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6811","nonce":2332368824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6815","nonce":2332368824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6813","nonce":2332368824}]},"public_addr":"172.21.15.19:6809/2332368824","cluster_addr":"172.21.15.19:6811/2332368824","heartbeat_back_addr":"172.21.15.19:6815/2332368824","heartbeat_front_addr":"172.21.15.19:6813/2332368824","state":["exists","up"]},{"osd":3,"uuid":"b1cb0ef4-aa9b-421b-a94e-82de0c46e5cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":3785550598},{"type":"v1","addr":"172.21.15.19:6825","nonce":3785550598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":3785550598},{"type":"v1","addr":"172.21.15.19:6827","nonce":3785550598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":3785550598},{"type":"v1","addr":"172.21.15.19:6831","nonce":3785550598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":3785550598},{"type":"v1","addr":"172.21.15.19:6829","nonce":3785550598}]},"public_addr":"172.21.15.19:6825/3785550598","cluster_addr":"172.21.15.19:6827/3785550598","heartbeat_back_addr":"172.21.15.19:6831/3785550598","heartbeat_front_addr":"172.21.15.19:6829/3785550598","state":["exists","up"]},{"osd":4,"uuid":"825b58ec-a6da-4515-b6a5-81debe327011","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":1985519349},{"type":"v1","addr":"172.21.15.43:6819","nonce":1985519349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":1985519349},{"type":"v1","addr":"172.21.15.43:6821","nonce":1985519349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":1985519349},{"type":"v1","addr":"172.21.15.43:6825","nonce":1985519349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":1985519349},{"type":"v1","addr":"172.21.15.43:6823","nonce":1985519349}]},"public_addr":"172.21.15.43:6819/1985519349","cluster_addr":"172.21.15.43:6821/1985519349","heartbeat_back_addr":"172.21.15.43:6825/1985519349","heartbeat_front_addr":"172.21.15.43:6823/1985519349","state":["exists","up"]},{"osd":5,"uuid":"7f34e537-5997-40e4-92e7-2570507160c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":4271971521},{"type":"v1","addr":"172.21.15.43:6803","nonce":4271971521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":4271971521},{"type":"v1","addr":"172.21.15.43:6805","nonce":4271971521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":4271971521},{"type":"v1","addr":"172.21.15.43:6809","nonce":4271971521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":4271971521},{"type":"v1","addr":"172.21.15.43:6807","nonce":4271971521}]},"public_addr":"172.21.15.43:6803/4271971521","cluster_addr":"172.21.15.43:6805/4271971521","heartbeat_back_addr":"172.21.15.43:6809/4271971521","heartbeat_front_addr":"172.21.15.43:6807/4271971521","state":["exists","up"]},{"osd":6,"uuid":"195dd8ad-640a-4fa3-9ee4-06e4172e0464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2863384694},{"type":"v1","addr":"172.21.15.43:6827","nonce":2863384694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2863384694},{"type":"v1","addr":"172.21.15.43:6829","nonce":2863384694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":2863384694},{"type":"v1","addr":"172.21.15.43:6833","nonce":2863384694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2863384694},{"type":"v1","addr":"172.21.15.43:6831","nonce":2863384694}]},"public_addr":"172.21.15.43:6827/2863384694","cluster_addr":"172.21.15.43:6829/2863384694","heartbeat_back_addr":"172.21.15.43:6833/2863384694","heartbeat_front_addr":"172.21.15.43:6831/2863384694","state":["exists","up"]},{"osd":7,"uuid":"4645661f-f430-4f05-bdd1-69138dcd1bbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":792572234},{"type":"v1","addr":"172.21.15.43:6811","nonce":792572234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":792572234},{"type":"v1","addr":"172.21.15.43:6813","nonce":792572234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":792572234},{"type":"v1","addr":"172.21.15.43:6817","nonce":792572234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":792572234},{"type":"v1","addr":"172.21.15.43:6815","nonce":792572234}]},"public_addr":"172.21.15.43:6811/792572234","cluster_addr":"172.21.15.43:6813/792572234","heartbeat_back_addr":"172.21.15.43:6817/792572234","heartbeat_front_addr":"172.21.15.43:6815/792572234","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.156485+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.578126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.075440+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.639867+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.543602+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:41.919592+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.615557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:41.813337+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-31T12:16:03.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.011+0000 7fbb128d5640 1 -- 172.21.15.19:0/2314863011 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fbaf8077a50 msgr2=0x7fbaf8079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:03.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.011+0000 7fbb128d5640 1 --2- 172.21.15.19:0/2314863011 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fbaf8077a50 0x7fbaf8079f60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c0c97b0 tx=0x7fbb0000b2e0 comp rx=0 tx=0).stop 2024-08-31T12:16:03.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.011+0000 7fbb128d5640 1 -- 172.21.15.19:0/2314863011 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbb0c06e0e0 msgr2=0x7fbb0c1e4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:03.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.011+0000 7fbb128d5640 1 --2- 172.21.15.19:0/2314863011 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbb0c06e0e0 0x7fbb0c1e4d70 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbafc00c3d0 tx=0x7fbafc00c890 comp rx=0 tx=0).stop 2024-08-31T12:16:03.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.011+0000 7fbb128d5640 1 -- 172.21.15.19:0/2314863011 shutdown_connections 2024-08-31T12:16:03.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.011+0000 7fbb128d5640 1 --2- 172.21.15.19:0/2314863011 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fbaf8077a50 0x7fbaf8079f60 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:03.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.011+0000 7fbb128d5640 1 --2- 172.21.15.19:0/2314863011 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbb0c155300 0x7fbb0c1e1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:03.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.011+0000 7fbb128d5640 1 --2- 172.21.15.19:0/2314863011 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbb0c0c8a40 0x7fbb0c1592e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:03.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.011+0000 7fbb128d5640 1 --2- 172.21.15.19:0/2314863011 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbb0c06e0e0 0x7fbb0c1e4d70 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:03.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.011+0000 7fbb128d5640 1 -- 172.21.15.19:0/2314863011 >> 172.21.15.19:0/2314863011 conn(0x7fbb0c0690d0 msgr2=0x7fbb0c15a3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:03.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.012+0000 7fbb128d5640 1 -- 172.21.15.19:0/2314863011 shutdown_connections 2024-08-31T12:16:03.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.012+0000 7fbb128d5640 1 -- 172.21.15.19:0/2314863011 wait complete. 2024-08-31T12:16:03.024 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-08-31T12:16:03.024 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-31T12:16:03.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.137+0000 7f6809e9b640 1 -- 172.21.15.19:0/248750862 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f68040c8a40 msgr2=0x7f68040c9850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:03.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.137+0000 7f6809e9b640 1 --2- 172.21.15.19:0/248750862 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f68040c8a40 0x7f68040c9850 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f67f4009830 tx=0x7f67f402a530 comp rx=0 tx=0).stop 2024-08-31T12:16:03.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.137+0000 7f6809e9b640 1 -- 172.21.15.19:0/248750862 shutdown_connections 2024-08-31T12:16:03.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.137+0000 7f6809e9b640 1 --2- 172.21.15.19:0/248750862 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6804155300 0x7f680406d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:03.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.137+0000 7f6809e9b640 1 --2- 172.21.15.19:0/248750862 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f68040c8a40 0x7f68040c9850 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:03.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.137+0000 7f6809e9b640 1 -- 172.21.15.19:0/248750862 >> 172.21.15.19:0/248750862 conn(0x7f68040690d0 msgr2=0x7f68040d0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:03.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.137+0000 7f6809e9b640 1 -- 172.21.15.19:0/248750862 shutdown_connections 2024-08-31T12:16:03.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.137+0000 7f6809e9b640 1 -- 172.21.15.19:0/248750862 wait complete. 2024-08-31T12:16:03.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.139+0000 7f6809e9b640 1 Processor -- start 2024-08-31T12:16:03.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.139+0000 7f6809e9b640 1 -- start start 2024-08-31T12:16:03.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.140+0000 7f6809e9b640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f680406e0e0 0x7f680414b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:03.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.140+0000 7f6809e9b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f68040c8a40 0x7f68041e6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:03.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.140+0000 7f6809e9b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6804155300 0x7f6804158fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:03.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.140+0000 7f6809e9b640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f68040c8330 con 0x7f6804155300 2024-08-31T12:16:03.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.140+0000 7f6809e9b640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f680415b250 con 0x7f680406e0e0 2024-08-31T12:16:03.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.140+0000 7f6809e9b640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f68040c8610 con 0x7f68040c8a40 2024-08-31T12:16:03.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.140+0000 7f6803fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6804155300 0x7f6804158fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:03.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.140+0000 7f6803fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6804155300 0x7f6804158fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:56486/0 (socket says 172.21.15.19:56486) 2024-08-31T12:16:03.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.140+0000 7f6803fff640 1 -- 172.21.15.19:0/3590053708 learned_addr learned my addr 172.21.15.19:0/3590053708 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:03.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.140+0000 7f68037fe640 1 --2- 172.21.15.19:0/3590053708 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f680406e0e0 0x7f680414b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:03.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.140+0000 7f6802ffd640 1 --2- 172.21.15.19:0/3590053708 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f68040c8a40 0x7f68041e6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:03.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.141+0000 7f6803fff640 1 -- 172.21.15.19:0/3590053708 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f68040c8a40 msgr2=0x7f68041e6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:03.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.141+0000 7f6803fff640 1 --2- 172.21.15.19:0/3590053708 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f68040c8a40 0x7f68041e6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:03.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.141+0000 7f6803fff640 1 -- 172.21.15.19:0/3590053708 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f680406e0e0 msgr2=0x7f680414b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:03.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.141+0000 7f6803fff640 1 --2- 172.21.15.19:0/3590053708 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f680406e0e0 0x7f680414b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:03.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.141+0000 7f6803fff640 1 -- 172.21.15.19:0/3590053708 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f680414b6e0 con 0x7f6804155300 2024-08-31T12:16:03.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.141+0000 7f6803fff640 1 --2- 172.21.15.19:0/3590053708 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6804155300 0x7f6804158fc0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f67fc00ec20 tx=0x7f67fc00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:03.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.141+0000 7f67ebfff640 1 -- 172.21.15.19:0/3590053708 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67fc003b70 con 0x7f6804155300 2024-08-31T12:16:03.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.141+0000 7f67ebfff640 1 -- 172.21.15.19:0/3590053708 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f67fc003d10 con 0x7f6804155300 2024-08-31T12:16:03.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.141+0000 7f67ebfff640 1 -- 172.21.15.19:0/3590053708 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67fc004280 con 0x7f6804155300 2024-08-31T12:16:03.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.141+0000 7f6809e9b640 1 -- 172.21.15.19:0/3590053708 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f680414d380 con 0x7f6804155300 2024-08-31T12:16:03.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.141+0000 7f6809e9b640 1 -- 172.21.15.19:0/3590053708 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68041e6ac0 con 0x7f6804155300 2024-08-31T12:16:03.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.142+0000 7f6809e9b640 1 -- 172.21.15.19:0/3590053708 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68040c9850 con 0x7f6804155300 2024-08-31T12:16:03.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.143+0000 7f67ebfff640 1 -- 172.21.15.19:0/3590053708 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f67fc004420 con 0x7f6804155300 2024-08-31T12:16:03.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.143+0000 7f67ebfff640 1 --2- 172.21.15.19:0/3590053708 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f67f0077a50 0x7f67f0079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:03.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.143+0000 7f67ebfff640 1 -- 172.21.15.19:0/3590053708 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f67fc09a2b0 con 0x7f6804155300 2024-08-31T12:16:03.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.143+0000 7f68037fe640 1 --2- 172.21.15.19:0/3590053708 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f67f0077a50 0x7f67f0079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:03.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.144+0000 7f68037fe640 1 --2- 172.21.15.19:0/3590053708 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f67f0077a50 0x7f67f0079f60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f67f4034000 tx=0x7f67f4004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:03.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.148+0000 7f67ebfff640 1 -- 172.21.15.19:0/3590053708 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f67fc062940 con 0x7f6804155300 2024-08-31T12:16:03.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.328+0000 7f6809e9b640 1 -- 172.21.15.19:0/3590053708 --> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f68040ccf20 con 0x7f67f0077a50 2024-08-31T12:16:03.334 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.333+0000 7f67ebfff640 1 -- 172.21.15.19:0/3590053708 <== mgr.4100 v2:172.21.15.43:6800/2824336655 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251838 (secure 0 0 0) 0x7f68040ccf20 con 0x7f67f0077a50 2024-08-31T12:16:03.337 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:16:03.339 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-08-31T12:16:03.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.336+0000 7f6809e9b640 1 -- 172.21.15.19:0/3590053708 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f67f0077a50 msgr2=0x7f67f0079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:03.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.336+0000 7f6809e9b640 1 --2- 172.21.15.19:0/3590053708 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f67f0077a50 0x7f67f0079f60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f67f4034000 tx=0x7f67f4004730 comp rx=0 tx=0).stop 2024-08-31T12:16:03.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.336+0000 7f6809e9b640 1 -- 172.21.15.19:0/3590053708 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6804155300 msgr2=0x7f6804158fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:03.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.336+0000 7f6809e9b640 1 --2- 172.21.15.19:0/3590053708 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6804155300 0x7f6804158fc0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f67fc00ec20 tx=0x7f67fc00c6a0 comp rx=0 tx=0).stop 2024-08-31T12:16:03.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.336+0000 7f6809e9b640 1 -- 172.21.15.19:0/3590053708 shutdown_connections 2024-08-31T12:16:03.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.336+0000 7f6809e9b640 1 --2- 172.21.15.19:0/3590053708 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f67f0077a50 0x7f67f0079f60 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:03.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.336+0000 7f6809e9b640 1 --2- 172.21.15.19:0/3590053708 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6804155300 0x7f6804158fc0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:03.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.336+0000 7f6809e9b640 1 --2- 172.21.15.19:0/3590053708 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f68040c8a40 0x7f68041e6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:03.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.336+0000 7f6809e9b640 1 --2- 172.21.15.19:0/3590053708 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f680406e0e0 0x7f680414b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:03.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.336+0000 7f6809e9b640 1 -- 172.21.15.19:0/3590053708 >> 172.21.15.19:0/3590053708 conn(0x7f68040690d0 msgr2=0x7f680415a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:03.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.336+0000 7f6809e9b640 1 -- 172.21.15.19:0/3590053708 shutdown_connections 2024-08-31T12:16:03.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.336+0000 7f6809e9b640 1 -- 172.21.15.19:0/3590053708 wait complete. 2024-08-31T12:16:03.348 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-08-31T12:16:02.492207+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":176,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":252316,"kb_used_data":2780,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754722404,"statfs":{"total":773094113280,"available":772835741696,"internally_reserved":0,"allocated":2846720,"data_stored":1759244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.223387"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306730+0000","last_change":"2024-08-31T12:15:57.306560+0000","last_active":"2024-08-31T12:15:58.306730+0000","last_peered":"2024-08-31T12:15:58.306730+0000","last_clean":"2024-08-31T12:15:58.306730+0000","last_became_active":"2024-08-31T12:15:57.306148+0000","last_became_peered":"2024-08-31T12:15:57.306148+0000","last_unstale":"2024-08-31T12:15:58.306730+0000","last_undegraded":"2024-08-31T12:15:58.306730+0000","last_fullsized":"2024-08-31T12:15:58.306730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:35:53.831742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300935+0000","last_change":"2024-08-31T12:15:57.307314+0000","last_active":"2024-08-31T12:15:58.300935+0000","last_peered":"2024-08-31T12:15:58.300935+0000","last_clean":"2024-08-31T12:15:58.300935+0000","last_became_active":"2024-08-31T12:15:57.306940+0000","last_became_peered":"2024-08-31T12:15:57.306940+0000","last_unstale":"2024-08-31T12:15:58.300935+0000","last_undegraded":"2024-08-31T12:15:58.300935+0000","last_fullsized":"2024-08-31T12:15:58.300935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:38:30.321214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937267+0000","last_change":"2024-08-31T12:15:57.309372+0000","last_active":"2024-08-31T12:15:58.937267+0000","last_peered":"2024-08-31T12:15:58.937267+0000","last_clean":"2024-08-31T12:15:58.937267+0000","last_became_active":"2024-08-31T12:15:57.305795+0000","last_became_peered":"2024-08-31T12:15:57.305795+0000","last_unstale":"2024-08-31T12:15:58.937267+0000","last_undegraded":"2024-08-31T12:15:58.937267+0000","last_fullsized":"2024-08-31T12:15:58.937267+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:37:45.891394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302363+0000","last_change":"2024-08-31T12:15:57.307339+0000","last_active":"2024-08-31T12:15:58.302363+0000","last_peered":"2024-08-31T12:15:58.302363+0000","last_clean":"2024-08-31T12:15:58.302363+0000","last_became_active":"2024-08-31T12:15:57.306957+0000","last_became_peered":"2024-08-31T12:15:57.306957+0000","last_unstale":"2024-08-31T12:15:58.302363+0000","last_undegraded":"2024-08-31T12:15:58.302363+0000","last_fullsized":"2024-08-31T12:15:58.302363+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:43:33.705099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306757+0000","last_change":"2024-08-31T12:15:57.317492+0000","last_active":"2024-08-31T12:15:58.306757+0000","last_peered":"2024-08-31T12:15:58.306757+0000","last_clean":"2024-08-31T12:15:58.306757+0000","last_became_active":"2024-08-31T12:15:57.317079+0000","last_became_peered":"2024-08-31T12:15:57.317079+0000","last_unstale":"2024-08-31T12:15:58.306757+0000","last_undegraded":"2024-08-31T12:15:58.306757+0000","last_fullsized":"2024-08-31T12:15:58.306757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:44:40.712699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302342+0000","last_change":"2024-08-31T12:15:57.307505+0000","last_active":"2024-08-31T12:15:58.302342+0000","last_peered":"2024-08-31T12:15:58.302342+0000","last_clean":"2024-08-31T12:15:58.302342+0000","last_became_active":"2024-08-31T12:15:57.307010+0000","last_became_peered":"2024-08-31T12:15:57.307010+0000","last_unstale":"2024-08-31T12:15:58.302342+0000","last_undegraded":"2024-08-31T12:15:58.302342+0000","last_fullsized":"2024-08-31T12:15:58.302342+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:39:31.100565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306247+0000","last_change":"2024-08-31T12:15:57.316892+0000","last_active":"2024-08-31T12:15:58.306247+0000","last_peered":"2024-08-31T12:15:58.306247+0000","last_clean":"2024-08-31T12:15:58.306247+0000","last_became_active":"2024-08-31T12:15:57.316504+0000","last_became_peered":"2024-08-31T12:15:57.316504+0000","last_unstale":"2024-08-31T12:15:58.306247+0000","last_undegraded":"2024-08-31T12:15:58.306247+0000","last_fullsized":"2024-08-31T12:15:58.306247+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:05:48.614918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937976+0000","last_change":"2024-08-31T12:15:57.309224+0000","last_active":"2024-08-31T12:15:58.937976+0000","last_peered":"2024-08-31T12:15:58.937976+0000","last_clean":"2024-08-31T12:15:58.937976+0000","last_became_active":"2024-08-31T12:15:57.308914+0000","last_became_peered":"2024-08-31T12:15:57.308914+0000","last_unstale":"2024-08-31T12:15:58.937976+0000","last_undegraded":"2024-08-31T12:15:58.937976+0000","last_fullsized":"2024-08-31T12:15:58.937976+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:37:02.435678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306762+0000","last_change":"2024-08-31T12:15:57.316723+0000","last_active":"2024-08-31T12:15:58.306762+0000","last_peered":"2024-08-31T12:15:58.306762+0000","last_clean":"2024-08-31T12:15:58.306762+0000","last_became_active":"2024-08-31T12:15:57.316354+0000","last_became_peered":"2024-08-31T12:15:57.316354+0000","last_unstale":"2024-08-31T12:15:58.306762+0000","last_undegraded":"2024-08-31T12:15:58.306762+0000","last_fullsized":"2024-08-31T12:15:58.306762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:31:19.860950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302888+0000","last_change":"2024-08-31T12:15:57.311161+0000","last_active":"2024-08-31T12:15:58.302888+0000","last_peered":"2024-08-31T12:15:58.302888+0000","last_clean":"2024-08-31T12:15:58.302888+0000","last_became_active":"2024-08-31T12:15:57.310807+0000","last_became_peered":"2024-08-31T12:15:57.310807+0000","last_unstale":"2024-08-31T12:15:58.302888+0000","last_undegraded":"2024-08-31T12:15:58.302888+0000","last_fullsized":"2024-08-31T12:15:58.302888+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:56:03.280339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302118+0000","last_change":"2024-08-31T12:15:57.300930+0000","last_active":"2024-08-31T12:15:58.302118+0000","last_peered":"2024-08-31T12:15:58.302118+0000","last_clean":"2024-08-31T12:15:58.302118+0000","last_became_active":"2024-08-31T12:15:57.300610+0000","last_became_peered":"2024-08-31T12:15:57.300610+0000","last_unstale":"2024-08-31T12:15:58.302118+0000","last_undegraded":"2024-08-31T12:15:58.302118+0000","last_fullsized":"2024-08-31T12:15:58.302118+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:42:39.731742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936435+0000","last_change":"2024-08-31T12:15:57.303151+0000","last_active":"2024-08-31T12:15:58.936435+0000","last_peered":"2024-08-31T12:15:58.936435+0000","last_clean":"2024-08-31T12:15:58.936435+0000","last_became_active":"2024-08-31T12:15:57.302622+0000","last_became_peered":"2024-08-31T12:15:57.302622+0000","last_unstale":"2024-08-31T12:15:58.936435+0000","last_undegraded":"2024-08-31T12:15:58.936435+0000","last_fullsized":"2024-08-31T12:15:58.936435+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:49:18.493629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937309+0000","last_change":"2024-08-31T12:15:57.305676+0000","last_active":"2024-08-31T12:15:58.937309+0000","last_peered":"2024-08-31T12:15:58.937309+0000","last_clean":"2024-08-31T12:15:58.937309+0000","last_became_active":"2024-08-31T12:15:57.305321+0000","last_became_peered":"2024-08-31T12:15:57.305321+0000","last_unstale":"2024-08-31T12:15:58.937309+0000","last_undegraded":"2024-08-31T12:15:58.937309+0000","last_fullsized":"2024-08-31T12:15:58.937309+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:42:39.731742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936688+0000","last_change":"2024-08-31T12:15:57.309671+0000","last_active":"2024-08-31T12:15:58.936688+0000","last_peered":"2024-08-31T12:15:58.936688+0000","last_clean":"2024-08-31T12:15:58.936688+0000","last_became_active":"2024-08-31T12:15:57.307013+0000","last_became_peered":"2024-08-31T12:15:57.307013+0000","last_unstale":"2024-08-31T12:15:58.936688+0000","last_undegraded":"2024-08-31T12:15:58.936688+0000","last_fullsized":"2024-08-31T12:15:58.936688+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:18:55.451164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300929+0000","last_change":"2024-08-31T12:15:57.298150+0000","last_active":"2024-08-31T12:15:58.300929+0000","last_peered":"2024-08-31T12:15:58.300929+0000","last_clean":"2024-08-31T12:15:58.300929+0000","last_became_active":"2024-08-31T12:15:57.297182+0000","last_became_peered":"2024-08-31T12:15:57.297182+0000","last_unstale":"2024-08-31T12:15:58.300929+0000","last_undegraded":"2024-08-31T12:15:58.300929+0000","last_fullsized":"2024-08-31T12:15:58.300929+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:16:42.625396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302762+0000","last_change":"2024-08-31T12:15:57.314224+0000","last_active":"2024-08-31T12:15:58.302762+0000","last_peered":"2024-08-31T12:15:58.302762+0000","last_clean":"2024-08-31T12:15:58.302762+0000","last_became_active":"2024-08-31T12:15:57.313887+0000","last_became_peered":"2024-08-31T12:15:57.313887+0000","last_unstale":"2024-08-31T12:15:58.302762+0000","last_undegraded":"2024-08-31T12:15:58.302762+0000","last_fullsized":"2024-08-31T12:15:58.302762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:54:45.720621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303640+0000","last_change":"2024-08-31T12:15:57.308899+0000","last_active":"2024-08-31T12:15:58.303640+0000","last_peered":"2024-08-31T12:15:58.303640+0000","last_clean":"2024-08-31T12:15:58.303640+0000","last_became_active":"2024-08-31T12:15:57.308388+0000","last_became_peered":"2024-08-31T12:15:57.308388+0000","last_unstale":"2024-08-31T12:15:58.303640+0000","last_undegraded":"2024-08-31T12:15:58.303640+0000","last_fullsized":"2024-08-31T12:15:58.303640+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:54:03.871840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-08-31T12:15:57.302751+0000","last_change":"2024-08-31T12:15:57.300201+0000","last_active":"2024-08-31T12:15:57.302751+0000","last_peered":"2024-08-31T12:15:57.302751+0000","last_clean":"2024-08-31T12:15:57.302751+0000","last_became_active":"2024-08-31T12:15:57.299826+0000","last_became_peered":"2024-08-31T12:15:57.299826+0000","last_unstale":"2024-08-31T12:15:57.302751+0000","last_undegraded":"2024-08-31T12:15:57.302751+0000","last_fullsized":"2024-08-31T12:15:57.302751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:50:24.220339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.307053+0000","last_change":"2024-08-31T12:15:57.317606+0000","last_active":"2024-08-31T12:15:58.307053+0000","last_peered":"2024-08-31T12:15:58.307053+0000","last_clean":"2024-08-31T12:15:58.307053+0000","last_became_active":"2024-08-31T12:15:57.317185+0000","last_became_peered":"2024-08-31T12:15:57.317185+0000","last_unstale":"2024-08-31T12:15:58.307053+0000","last_undegraded":"2024-08-31T12:15:58.307053+0000","last_fullsized":"2024-08-31T12:15:58.307053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:25:46.346490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302171+0000","last_change":"2024-08-31T12:15:57.298778+0000","last_active":"2024-08-31T12:15:58.302171+0000","last_peered":"2024-08-31T12:15:58.302171+0000","last_clean":"2024-08-31T12:15:58.302171+0000","last_became_active":"2024-08-31T12:15:57.298430+0000","last_became_peered":"2024-08-31T12:15:57.298430+0000","last_unstale":"2024-08-31T12:15:58.302171+0000","last_undegraded":"2024-08-31T12:15:58.302171+0000","last_fullsized":"2024-08-31T12:15:58.302171+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:22:13.435300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937190+0000","last_change":"2024-08-31T12:15:57.317176+0000","last_active":"2024-08-31T12:15:58.937190+0000","last_peered":"2024-08-31T12:15:58.937190+0000","last_clean":"2024-08-31T12:15:58.937190+0000","last_became_active":"2024-08-31T12:15:57.316870+0000","last_became_peered":"2024-08-31T12:15:57.316870+0000","last_unstale":"2024-08-31T12:15:58.937190+0000","last_undegraded":"2024-08-31T12:15:58.937190+0000","last_fullsized":"2024-08-31T12:15:58.937190+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:43:12.703971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302636+0000","last_change":"2024-08-31T12:15:57.308907+0000","last_active":"2024-08-31T12:15:58.302636+0000","last_peered":"2024-08-31T12:15:58.302636+0000","last_clean":"2024-08-31T12:15:58.302636+0000","last_became_active":"2024-08-31T12:15:57.308459+0000","last_became_peered":"2024-08-31T12:15:57.308459+0000","last_unstale":"2024-08-31T12:15:58.302636+0000","last_undegraded":"2024-08-31T12:15:58.302636+0000","last_fullsized":"2024-08-31T12:15:58.302636+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:16:46.648912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937760+0000","last_change":"2024-08-31T12:15:57.304032+0000","last_active":"2024-08-31T12:15:58.937760+0000","last_peered":"2024-08-31T12:15:58.937760+0000","last_clean":"2024-08-31T12:15:58.937760+0000","last_became_active":"2024-08-31T12:15:57.303401+0000","last_became_peered":"2024-08-31T12:15:57.303401+0000","last_unstale":"2024-08-31T12:15:58.937760+0000","last_undegraded":"2024-08-31T12:15:58.937760+0000","last_fullsized":"2024-08-31T12:15:58.937760+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:56:03.280339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303013+0000","last_change":"2024-08-31T12:15:57.305412+0000","last_active":"2024-08-31T12:15:58.303013+0000","last_peered":"2024-08-31T12:15:58.303013+0000","last_clean":"2024-08-31T12:15:58.303013+0000","last_became_active":"2024-08-31T12:15:57.305075+0000","last_became_peered":"2024-08-31T12:15:57.305075+0000","last_unstale":"2024-08-31T12:15:58.303013+0000","last_undegraded":"2024-08-31T12:15:58.303013+0000","last_fullsized":"2024-08-31T12:15:58.303013+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:30:13.715505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937828+0000","last_change":"2024-08-31T12:15:57.300179+0000","last_active":"2024-08-31T12:15:58.937828+0000","last_peered":"2024-08-31T12:15:58.937828+0000","last_clean":"2024-08-31T12:15:58.937828+0000","last_became_active":"2024-08-31T12:15:57.299861+0000","last_became_peered":"2024-08-31T12:15:57.299861+0000","last_unstale":"2024-08-31T12:15:58.937828+0000","last_undegraded":"2024-08-31T12:15:58.937828+0000","last_fullsized":"2024-08-31T12:15:58.937828+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:47:51.267409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303001+0000","last_change":"2024-08-31T12:15:57.308519+0000","last_active":"2024-08-31T12:15:58.303001+0000","last_peered":"2024-08-31T12:15:58.303001+0000","last_clean":"2024-08-31T12:15:58.303001+0000","last_became_active":"2024-08-31T12:15:57.308189+0000","last_became_peered":"2024-08-31T12:15:57.308189+0000","last_unstale":"2024-08-31T12:15:58.303001+0000","last_undegraded":"2024-08-31T12:15:58.303001+0000","last_fullsized":"2024-08-31T12:15:58.303001+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:37:02.435678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300666+0000","last_change":"2024-08-31T12:15:57.308321+0000","last_active":"2024-08-31T12:15:58.300666+0000","last_peered":"2024-08-31T12:15:58.300666+0000","last_clean":"2024-08-31T12:15:58.300666+0000","last_became_active":"2024-08-31T12:15:57.304514+0000","last_became_peered":"2024-08-31T12:15:57.304514+0000","last_unstale":"2024-08-31T12:15:58.300666+0000","last_undegraded":"2024-08-31T12:15:58.300666+0000","last_fullsized":"2024-08-31T12:15:58.300666+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:09:49.617826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936830+0000","last_change":"2024-08-31T12:15:57.299268+0000","last_active":"2024-08-31T12:15:58.936830+0000","last_peered":"2024-08-31T12:15:58.936830+0000","last_clean":"2024-08-31T12:15:58.936830+0000","last_became_active":"2024-08-31T12:15:57.298766+0000","last_became_peered":"2024-08-31T12:15:57.298766+0000","last_unstale":"2024-08-31T12:15:58.936830+0000","last_undegraded":"2024-08-31T12:15:58.936830+0000","last_fullsized":"2024-08-31T12:15:58.936830+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:37:45.891394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937525+0000","last_change":"2024-08-31T12:15:57.310121+0000","last_active":"2024-08-31T12:15:58.937525+0000","last_peered":"2024-08-31T12:15:58.937525+0000","last_clean":"2024-08-31T12:15:58.937525+0000","last_became_active":"2024-08-31T12:15:57.309440+0000","last_became_peered":"2024-08-31T12:15:57.309440+0000","last_unstale":"2024-08-31T12:15:58.937525+0000","last_undegraded":"2024-08-31T12:15:58.937525+0000","last_fullsized":"2024-08-31T12:15:58.937525+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:53:19.523811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302889+0000","last_change":"2024-08-31T12:15:57.306214+0000","last_active":"2024-08-31T12:15:58.302889+0000","last_peered":"2024-08-31T12:15:58.302889+0000","last_clean":"2024-08-31T12:15:58.302889+0000","last_became_active":"2024-08-31T12:15:57.305878+0000","last_became_peered":"2024-08-31T12:15:57.305878+0000","last_unstale":"2024-08-31T12:15:58.302889+0000","last_undegraded":"2024-08-31T12:15:58.302889+0000","last_fullsized":"2024-08-31T12:15:58.302889+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:05:48.614918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-08-31T12:15:57.312611+0000","last_change":"2024-08-31T12:15:57.307376+0000","last_active":"2024-08-31T12:15:57.312611+0000","last_peered":"2024-08-31T12:15:57.312611+0000","last_clean":"2024-08-31T12:15:57.312611+0000","last_became_active":"2024-08-31T12:15:57.307056+0000","last_became_peered":"2024-08-31T12:15:57.307056+0000","last_unstale":"2024-08-31T12:15:57.312611+0000","last_undegraded":"2024-08-31T12:15:57.312611+0000","last_fullsized":"2024-08-31T12:15:57.312611+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:39:32.951892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302319+0000","last_change":"2024-08-31T12:15:57.313798+0000","last_active":"2024-08-31T12:15:58.302319+0000","last_peered":"2024-08-31T12:15:58.302319+0000","last_clean":"2024-08-31T12:15:58.302319+0000","last_became_active":"2024-08-31T12:15:57.313463+0000","last_became_peered":"2024-08-31T12:15:57.313463+0000","last_unstale":"2024-08-31T12:15:58.302319+0000","last_undegraded":"2024-08-31T12:15:58.302319+0000","last_fullsized":"2024-08-31T12:15:58.302319+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:47:51.267409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936240+0000","last_change":"2024-08-31T12:15:57.293350+0000","last_active":"2024-08-31T12:15:58.936240+0000","last_peered":"2024-08-31T12:15:58.936240+0000","last_clean":"2024-08-31T12:15:58.936240+0000","last_became_active":"2024-08-31T12:15:57.292981+0000","last_became_peered":"2024-08-31T12:15:57.292981+0000","last_unstale":"2024-08-31T12:15:58.936240+0000","last_undegraded":"2024-08-31T12:15:58.936240+0000","last_fullsized":"2024-08-31T12:15:58.936240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:52:00.312013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937396+0000","last_change":"2024-08-31T12:15:56.293497+0000","last_active":"2024-08-31T12:15:58.937396+0000","last_peered":"2024-08-31T12:15:58.937396+0000","last_clean":"2024-08-31T12:15:58.937396+0000","last_became_active":"2024-08-31T12:15:56.293164+0000","last_became_peered":"2024-08-31T12:15:56.293164+0000","last_unstale":"2024-08-31T12:15:58.937396+0000","last_undegraded":"2024-08-31T12:15:58.937396+0000","last_fullsized":"2024-08-31T12:15:58.937396+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:43:17.443433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937814+0000","last_change":"2024-08-31T12:15:57.312738+0000","last_active":"2024-08-31T12:15:58.937814+0000","last_peered":"2024-08-31T12:15:58.937814+0000","last_clean":"2024-08-31T12:15:58.937814+0000","last_became_active":"2024-08-31T12:15:57.312360+0000","last_became_peered":"2024-08-31T12:15:57.312360+0000","last_unstale":"2024-08-31T12:15:58.937814+0000","last_undegraded":"2024-08-31T12:15:58.937814+0000","last_fullsized":"2024-08-31T12:15:58.937814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:22:13.435300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.301762+0000","last_change":"2024-08-31T12:15:56.283526+0000","last_active":"2024-08-31T12:15:58.301762+0000","last_peered":"2024-08-31T12:15:58.301762+0000","last_clean":"2024-08-31T12:15:58.301762+0000","last_became_active":"2024-08-31T12:15:56.283147+0000","last_became_peered":"2024-08-31T12:15:56.283147+0000","last_unstale":"2024-08-31T12:15:58.301762+0000","last_undegraded":"2024-08-31T12:15:58.301762+0000","last_fullsized":"2024-08-31T12:15:58.301762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:51:59.306912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937727+0000","last_change":"2024-08-31T12:15:56.291170+0000","last_active":"2024-08-31T12:15:58.937727+0000","last_peered":"2024-08-31T12:15:58.937727+0000","last_clean":"2024-08-31T12:15:58.937727+0000","last_became_active":"2024-08-31T12:15:56.290835+0000","last_became_peered":"2024-08-31T12:15:56.290835+0000","last_unstale":"2024-08-31T12:15:58.937727+0000","last_undegraded":"2024-08-31T12:15:58.937727+0000","last_fullsized":"2024-08-31T12:15:58.937727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:15:51.561831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.938053+0000","last_change":"2024-08-31T12:15:57.308183+0000","last_active":"2024-08-31T12:15:58.938053+0000","last_peered":"2024-08-31T12:15:58.938053+0000","last_clean":"2024-08-31T12:15:58.938053+0000","last_became_active":"2024-08-31T12:15:57.306962+0000","last_became_peered":"2024-08-31T12:15:57.306962+0000","last_unstale":"2024-08-31T12:15:58.938053+0000","last_undegraded":"2024-08-31T12:15:58.938053+0000","last_fullsized":"2024-08-31T12:15:58.938053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:01:18.188264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303216+0000","last_change":"2024-08-31T12:15:56.288126+0000","last_active":"2024-08-31T12:15:58.303216+0000","last_peered":"2024-08-31T12:15:58.303216+0000","last_clean":"2024-08-31T12:15:58.303216+0000","last_became_active":"2024-08-31T12:15:56.287552+0000","last_became_peered":"2024-08-31T12:15:56.287552+0000","last_unstale":"2024-08-31T12:15:58.303216+0000","last_undegraded":"2024-08-31T12:15:58.303216+0000","last_fullsized":"2024-08-31T12:15:58.303216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:54:29.988079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300810+0000","last_change":"2024-08-31T12:15:57.305029+0000","last_active":"2024-08-31T12:15:58.300810+0000","last_peered":"2024-08-31T12:15:58.300810+0000","last_clean":"2024-08-31T12:15:58.300810+0000","last_became_active":"2024-08-31T12:15:57.304700+0000","last_became_peered":"2024-08-31T12:15:57.304700+0000","last_unstale":"2024-08-31T12:15:58.300810+0000","last_undegraded":"2024-08-31T12:15:58.300810+0000","last_fullsized":"2024-08-31T12:15:58.300810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:58:17.482204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302528+0000","last_change":"2024-08-31T12:15:56.284907+0000","last_active":"2024-08-31T12:15:58.302528+0000","last_peered":"2024-08-31T12:15:58.302528+0000","last_clean":"2024-08-31T12:15:58.302528+0000","last_became_active":"2024-08-31T12:15:56.284558+0000","last_became_peered":"2024-08-31T12:15:56.284558+0000","last_unstale":"2024-08-31T12:15:58.302528+0000","last_undegraded":"2024-08-31T12:15:58.302528+0000","last_fullsized":"2024-08-31T12:15:58.302528+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:29:53.117724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303538+0000","last_change":"2024-08-31T12:15:57.308679+0000","last_active":"2024-08-31T12:15:58.303538+0000","last_peered":"2024-08-31T12:15:58.303538+0000","last_clean":"2024-08-31T12:15:58.303538+0000","last_became_active":"2024-08-31T12:15:57.308260+0000","last_became_peered":"2024-08-31T12:15:57.308260+0000","last_unstale":"2024-08-31T12:15:58.303538+0000","last_undegraded":"2024-08-31T12:15:58.303538+0000","last_fullsized":"2024-08-31T12:15:58.303538+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:31:19.860950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303564+0000","last_change":"2024-08-31T12:15:56.290763+0000","last_active":"2024-08-31T12:15:58.303564+0000","last_peered":"2024-08-31T12:15:58.303564+0000","last_clean":"2024-08-31T12:15:58.303564+0000","last_became_active":"2024-08-31T12:15:56.290400+0000","last_became_peered":"2024-08-31T12:15:56.290400+0000","last_unstale":"2024-08-31T12:15:58.303564+0000","last_undegraded":"2024-08-31T12:15:58.303564+0000","last_fullsized":"2024-08-31T12:15:58.303564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:53:08.929860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.938122+0000","last_change":"2024-08-31T12:15:57.311657+0000","last_active":"2024-08-31T12:15:58.938122+0000","last_peered":"2024-08-31T12:15:58.938122+0000","last_clean":"2024-08-31T12:15:58.938122+0000","last_became_active":"2024-08-31T12:15:57.310564+0000","last_became_peered":"2024-08-31T12:15:57.310564+0000","last_unstale":"2024-08-31T12:15:58.938122+0000","last_undegraded":"2024-08-31T12:15:58.938122+0000","last_fullsized":"2024-08-31T12:15:58.938122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:37:02.435678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300755+0000","last_change":"2024-08-31T12:15:56.286792+0000","last_active":"2024-08-31T12:15:58.300755+0000","last_peered":"2024-08-31T12:15:58.300755+0000","last_clean":"2024-08-31T12:15:58.300755+0000","last_became_active":"2024-08-31T12:15:56.286458+0000","last_became_peered":"2024-08-31T12:15:56.286458+0000","last_unstale":"2024-08-31T12:15:58.300755+0000","last_undegraded":"2024-08-31T12:15:58.300755+0000","last_fullsized":"2024-08-31T12:15:58.300755+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:49:17.488528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936603+0000","last_change":"2024-08-31T12:15:57.309485+0000","last_active":"2024-08-31T12:15:58.936603+0000","last_peered":"2024-08-31T12:15:58.936603+0000","last_clean":"2024-08-31T12:15:58.936603+0000","last_became_active":"2024-08-31T12:15:57.309102+0000","last_became_peered":"2024-08-31T12:15:57.309102+0000","last_unstale":"2024-08-31T12:15:58.936603+0000","last_undegraded":"2024-08-31T12:15:58.936603+0000","last_fullsized":"2024-08-31T12:15:58.936603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:17:34.692296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302279+0000","last_change":"2024-08-31T12:15:56.289460+0000","last_active":"2024-08-31T12:15:58.302279+0000","last_peered":"2024-08-31T12:15:58.302279+0000","last_clean":"2024-08-31T12:15:58.302279+0000","last_became_active":"2024-08-31T12:15:56.289096+0000","last_became_peered":"2024-08-31T12:15:56.289096+0000","last_unstale":"2024-08-31T12:15:58.302279+0000","last_undegraded":"2024-08-31T12:15:58.302279+0000","last_fullsized":"2024-08-31T12:15:58.302279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:06:34.313772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306865+0000","last_change":"2024-08-31T12:15:57.317968+0000","last_active":"2024-08-31T12:15:58.306865+0000","last_peered":"2024-08-31T12:15:58.306865+0000","last_clean":"2024-08-31T12:15:58.306865+0000","last_became_active":"2024-08-31T12:15:57.317632+0000","last_became_peered":"2024-08-31T12:15:57.317632+0000","last_unstale":"2024-08-31T12:15:58.306865+0000","last_undegraded":"2024-08-31T12:15:58.306865+0000","last_fullsized":"2024-08-31T12:15:58.306865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:22:02.569106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937785+0000","last_change":"2024-08-31T12:15:56.284014+0000","last_active":"2024-08-31T12:15:58.937785+0000","last_peered":"2024-08-31T12:15:58.937785+0000","last_clean":"2024-08-31T12:15:58.937785+0000","last_became_active":"2024-08-31T12:15:56.283675+0000","last_became_peered":"2024-08-31T12:15:56.283675+0000","last_unstale":"2024-08-31T12:15:58.937785+0000","last_undegraded":"2024-08-31T12:15:58.937785+0000","last_fullsized":"2024-08-31T12:15:58.937785+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:51:59.306912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937018+0000","last_change":"2024-08-31T12:15:57.312440+0000","last_active":"2024-08-31T12:15:58.937018+0000","last_peered":"2024-08-31T12:15:58.937018+0000","last_clean":"2024-08-31T12:15:58.937018+0000","last_became_active":"2024-08-31T12:15:57.312007+0000","last_became_peered":"2024-08-31T12:15:57.312007+0000","last_unstale":"2024-08-31T12:15:58.937018+0000","last_undegraded":"2024-08-31T12:15:58.937018+0000","last_fullsized":"2024-08-31T12:15:58.937018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:23:43.553625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303030+0000","last_change":"2024-08-31T12:15:56.290968+0000","last_active":"2024-08-31T12:15:58.303030+0000","last_peered":"2024-08-31T12:15:58.303030+0000","last_clean":"2024-08-31T12:15:58.303030+0000","last_became_active":"2024-08-31T12:15:56.290603+0000","last_became_peered":"2024-08-31T12:15:56.290603+0000","last_unstale":"2024-08-31T12:15:58.303030+0000","last_undegraded":"2024-08-31T12:15:58.303030+0000","last_fullsized":"2024-08-31T12:15:58.303030+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:50:23.215239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.307024+0000","last_change":"2024-08-31T12:15:57.312790+0000","last_active":"2024-08-31T12:15:58.307024+0000","last_peered":"2024-08-31T12:15:58.307024+0000","last_clean":"2024-08-31T12:15:58.307024+0000","last_became_active":"2024-08-31T12:15:57.312383+0000","last_became_peered":"2024-08-31T12:15:57.312383+0000","last_unstale":"2024-08-31T12:15:58.307024+0000","last_undegraded":"2024-08-31T12:15:58.307024+0000","last_fullsized":"2024-08-31T12:15:58.307024+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:20:41.042375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.301697+0000","last_change":"2024-08-31T12:15:56.289795+0000","last_active":"2024-08-31T12:15:58.301697+0000","last_peered":"2024-08-31T12:15:58.301697+0000","last_clean":"2024-08-31T12:15:58.301697+0000","last_became_active":"2024-08-31T12:15:56.289477+0000","last_became_peered":"2024-08-31T12:15:56.289477+0000","last_unstale":"2024-08-31T12:15:58.301697+0000","last_undegraded":"2024-08-31T12:15:58.301697+0000","last_fullsized":"2024-08-31T12:15:58.301697+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:15:51.561831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302374+0000","last_change":"2024-08-31T12:15:57.312865+0000","last_active":"2024-08-31T12:15:58.302374+0000","last_peered":"2024-08-31T12:15:58.302374+0000","last_clean":"2024-08-31T12:15:58.302374+0000","last_became_active":"2024-08-31T12:15:57.312462+0000","last_became_peered":"2024-08-31T12:15:57.312462+0000","last_unstale":"2024-08-31T12:15:58.302374+0000","last_undegraded":"2024-08-31T12:15:58.302374+0000","last_fullsized":"2024-08-31T12:15:58.302374+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:16:42.625396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-08-31T12:15:57.292892+0000","last_change":"2024-08-31T12:15:56.285412+0000","last_active":"2024-08-31T12:15:57.292892+0000","last_peered":"2024-08-31T12:15:57.292892+0000","last_clean":"2024-08-31T12:15:57.292892+0000","last_became_active":"2024-08-31T12:15:56.285108+0000","last_became_peered":"2024-08-31T12:15:56.285108+0000","last_unstale":"2024-08-31T12:15:57.292892+0000","last_undegraded":"2024-08-31T12:15:57.292892+0000","last_fullsized":"2024-08-31T12:15:57.292892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:29:45.580957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306407+0000","last_change":"2024-08-31T12:15:57.311701+0000","last_active":"2024-08-31T12:15:58.306407+0000","last_peered":"2024-08-31T12:15:58.306407+0000","last_clean":"2024-08-31T12:15:58.306407+0000","last_became_active":"2024-08-31T12:15:57.311368+0000","last_became_peered":"2024-08-31T12:15:57.311368+0000","last_unstale":"2024-08-31T12:15:58.306407+0000","last_undegraded":"2024-08-31T12:15:58.306407+0000","last_fullsized":"2024-08-31T12:15:58.306407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:30:13.715505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936411+0000","last_change":"2024-08-31T12:15:56.289979+0000","last_active":"2024-08-31T12:15:58.936411+0000","last_peered":"2024-08-31T12:15:58.936411+0000","last_clean":"2024-08-31T12:15:58.936411+0000","last_became_active":"2024-08-31T12:15:56.288289+0000","last_became_peered":"2024-08-31T12:15:56.288289+0000","last_unstale":"2024-08-31T12:15:58.936411+0000","last_undegraded":"2024-08-31T12:15:58.936411+0000","last_fullsized":"2024-08-31T12:15:58.936411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:46:55.057087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306825+0000","last_change":"2024-08-31T12:15:57.311580+0000","last_active":"2024-08-31T12:15:58.306825+0000","last_peered":"2024-08-31T12:15:58.306825+0000","last_clean":"2024-08-31T12:15:58.306825+0000","last_became_active":"2024-08-31T12:15:57.311323+0000","last_became_peered":"2024-08-31T12:15:57.311323+0000","last_unstale":"2024-08-31T12:15:58.306825+0000","last_undegraded":"2024-08-31T12:15:58.306825+0000","last_fullsized":"2024-08-31T12:15:58.306825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:54:03.871840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-08-31T12:15:57.293295+0000","last_change":"2024-08-31T12:15:56.290496+0000","last_active":"2024-08-31T12:15:57.293295+0000","last_peered":"2024-08-31T12:15:57.293295+0000","last_clean":"2024-08-31T12:15:57.293295+0000","last_became_active":"2024-08-31T12:15:56.290196+0000","last_became_peered":"2024-08-31T12:15:56.290196+0000","last_unstale":"2024-08-31T12:15:57.293295+0000","last_undegraded":"2024-08-31T12:15:57.293295+0000","last_fullsized":"2024-08-31T12:15:57.293295+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:52:20.055757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306986+0000","last_change":"2024-08-31T12:15:57.313317+0000","last_active":"2024-08-31T12:15:58.306986+0000","last_peered":"2024-08-31T12:15:58.306986+0000","last_clean":"2024-08-31T12:15:58.306986+0000","last_became_active":"2024-08-31T12:15:57.312145+0000","last_became_peered":"2024-08-31T12:15:57.312145+0000","last_unstale":"2024-08-31T12:15:58.306986+0000","last_undegraded":"2024-08-31T12:15:58.306986+0000","last_fullsized":"2024-08-31T12:15:58.306986+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:09:31.904512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300760+0000","last_change":"2024-08-31T12:15:56.281809+0000","last_active":"2024-08-31T12:15:58.300760+0000","last_peered":"2024-08-31T12:15:58.300760+0000","last_clean":"2024-08-31T12:15:58.300760+0000","last_became_active":"2024-08-31T12:15:56.281368+0000","last_became_peered":"2024-08-31T12:15:56.281368+0000","last_unstale":"2024-08-31T12:15:58.300760+0000","last_undegraded":"2024-08-31T12:15:58.300760+0000","last_fullsized":"2024-08-31T12:15:58.300760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:17:33.687196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303702+0000","last_change":"2024-08-31T12:15:57.305874+0000","last_active":"2024-08-31T12:15:58.303702+0000","last_peered":"2024-08-31T12:15:58.303702+0000","last_clean":"2024-08-31T12:15:58.303702+0000","last_became_active":"2024-08-31T12:15:57.305507+0000","last_became_peered":"2024-08-31T12:15:57.305507+0000","last_unstale":"2024-08-31T12:15:58.303702+0000","last_undegraded":"2024-08-31T12:15:58.303702+0000","last_fullsized":"2024-08-31T12:15:58.303702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:22:02.569106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-08-31T12:15:57.293819+0000","last_change":"2024-08-31T12:15:56.291851+0000","last_active":"2024-08-31T12:15:57.293819+0000","last_peered":"2024-08-31T12:15:57.293819+0000","last_clean":"2024-08-31T12:15:57.293819+0000","last_became_active":"2024-08-31T12:15:56.291446+0000","last_became_peered":"2024-08-31T12:15:56.291446+0000","last_unstale":"2024-08-31T12:15:57.293819+0000","last_undegraded":"2024-08-31T12:15:57.293819+0000","last_fullsized":"2024-08-31T12:15:57.293819+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:54:29.988079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306538+0000","last_change":"2024-08-31T12:15:57.316154+0000","last_active":"2024-08-31T12:15:58.306538+0000","last_peered":"2024-08-31T12:15:58.306538+0000","last_clean":"2024-08-31T12:15:58.306538+0000","last_became_active":"2024-08-31T12:15:57.315830+0000","last_became_peered":"2024-08-31T12:15:57.315830+0000","last_unstale":"2024-08-31T12:15:58.306538+0000","last_undegraded":"2024-08-31T12:15:58.306538+0000","last_fullsized":"2024-08-31T12:15:58.306538+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:05:53.039520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":111,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306756+0000","last_change":"2024-08-31T12:15:47.084912+0000","last_active":"2024-08-31T12:15:58.306756+0000","last_peered":"2024-08-31T12:15:58.306756+0000","last_clean":"2024-08-31T12:15:58.306756+0000","last_became_active":"2024-08-31T12:15:47.083771+0000","last_became_peered":"2024-08-31T12:15:47.083771+0000","last_unstale":"2024-08-31T12:15:58.306756+0000","last_undegraded":"2024-08-31T12:15:58.306756+0000","last_fullsized":"2024-08-31T12:15:58.306756+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:45.213688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:45.213688+0000","last_clean_scrub_stamp":"2024-08-31T12:15:45.213688+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:25:24.827530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.301865+0000","last_change":"2024-08-31T12:15:56.285355+0000","last_active":"2024-08-31T12:15:58.301865+0000","last_peered":"2024-08-31T12:15:58.301865+0000","last_clean":"2024-08-31T12:15:58.301865+0000","last_became_active":"2024-08-31T12:15:56.284470+0000","last_became_peered":"2024-08-31T12:15:56.284470+0000","last_unstale":"2024-08-31T12:15:58.301865+0000","last_undegraded":"2024-08-31T12:15:58.301865+0000","last_fullsized":"2024-08-31T12:15:58.301865+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:52:20.055757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936697+0000","last_change":"2024-08-31T12:15:57.309039+0000","last_active":"2024-08-31T12:15:58.936697+0000","last_peered":"2024-08-31T12:15:58.936697+0000","last_clean":"2024-08-31T12:15:58.936697+0000","last_became_active":"2024-08-31T12:15:57.308721+0000","last_became_peered":"2024-08-31T12:15:57.308721+0000","last_unstale":"2024-08-31T12:15:58.936697+0000","last_undegraded":"2024-08-31T12:15:58.936697+0000","last_fullsized":"2024-08-31T12:15:58.936697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:43:12.703971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306092+0000","last_change":"2024-08-31T12:15:56.293221+0000","last_active":"2024-08-31T12:15:58.306092+0000","last_peered":"2024-08-31T12:15:58.306092+0000","last_clean":"2024-08-31T12:15:58.306092+0000","last_became_active":"2024-08-31T12:15:56.292910+0000","last_became_peered":"2024-08-31T12:15:56.292910+0000","last_unstale":"2024-08-31T12:15:58.306092+0000","last_undegraded":"2024-08-31T12:15:58.306092+0000","last_fullsized":"2024-08-31T12:15:58.306092+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:14:33.483002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937638+0000","last_change":"2024-08-31T12:15:57.307266+0000","last_active":"2024-08-31T12:15:58.937638+0000","last_peered":"2024-08-31T12:15:58.937638+0000","last_clean":"2024-08-31T12:15:58.937638+0000","last_became_active":"2024-08-31T12:15:57.306529+0000","last_became_peered":"2024-08-31T12:15:57.306529+0000","last_unstale":"2024-08-31T12:15:58.937638+0000","last_undegraded":"2024-08-31T12:15:58.937638+0000","last_fullsized":"2024-08-31T12:15:58.937638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:42:39.731742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-08-31T12:15:57.294005+0000","last_change":"2024-08-31T12:15:56.291822+0000","last_active":"2024-08-31T12:15:57.294005+0000","last_peered":"2024-08-31T12:15:57.294005+0000","last_clean":"2024-08-31T12:15:57.294005+0000","last_became_active":"2024-08-31T12:15:56.291064+0000","last_became_peered":"2024-08-31T12:15:56.291064+0000","last_unstale":"2024-08-31T12:15:57.294005+0000","last_undegraded":"2024-08-31T12:15:57.294005+0000","last_fullsized":"2024-08-31T12:15:57.294005+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:53:08.929860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300606+0000","last_change":"2024-08-31T12:15:57.299212+0000","last_active":"2024-08-31T12:15:58.300606+0000","last_peered":"2024-08-31T12:15:58.300606+0000","last_clean":"2024-08-31T12:15:58.300606+0000","last_became_active":"2024-08-31T12:15:57.298892+0000","last_became_peered":"2024-08-31T12:15:57.298892+0000","last_unstale":"2024-08-31T12:15:58.300606+0000","last_undegraded":"2024-08-31T12:15:58.300606+0000","last_fullsized":"2024-08-31T12:15:58.300606+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:53:19.523811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303164+0000","last_change":"2024-08-31T12:15:56.286720+0000","last_active":"2024-08-31T12:15:58.303164+0000","last_peered":"2024-08-31T12:15:58.303164+0000","last_clean":"2024-08-31T12:15:58.303164+0000","last_became_active":"2024-08-31T12:15:56.285648+0000","last_became_peered":"2024-08-31T12:15:56.285648+0000","last_unstale":"2024-08-31T12:15:58.303164+0000","last_undegraded":"2024-08-31T12:15:58.303164+0000","last_fullsized":"2024-08-31T12:15:58.303164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:39:31.946792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936342+0000","last_change":"2024-08-31T12:15:57.311785+0000","last_active":"2024-08-31T12:15:58.936342+0000","last_peered":"2024-08-31T12:15:58.936342+0000","last_clean":"2024-08-31T12:15:58.936342+0000","last_became_active":"2024-08-31T12:15:57.311469+0000","last_became_peered":"2024-08-31T12:15:57.311469+0000","last_unstale":"2024-08-31T12:15:58.936342+0000","last_undegraded":"2024-08-31T12:15:58.936342+0000","last_fullsized":"2024-08-31T12:15:58.936342+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:29:54.122825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300691+0000","last_change":"2024-08-31T12:15:56.284302+0000","last_active":"2024-08-31T12:15:58.300691+0000","last_peered":"2024-08-31T12:15:58.300691+0000","last_clean":"2024-08-31T12:15:58.300691+0000","last_became_active":"2024-08-31T12:15:56.283979+0000","last_became_peered":"2024-08-31T12:15:56.283979+0000","last_unstale":"2024-08-31T12:15:58.300691+0000","last_undegraded":"2024-08-31T12:15:58.300691+0000","last_fullsized":"2024-08-31T12:15:58.300691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:51:59.306912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303594+0000","last_change":"2024-08-31T12:15:57.305383+0000","last_active":"2024-08-31T12:15:58.303594+0000","last_peered":"2024-08-31T12:15:58.303594+0000","last_clean":"2024-08-31T12:15:58.303594+0000","last_became_active":"2024-08-31T12:15:57.305048+0000","last_became_peered":"2024-08-31T12:15:57.305048+0000","last_unstale":"2024-08-31T12:15:58.303594+0000","last_undegraded":"2024-08-31T12:15:58.303594+0000","last_fullsized":"2024-08-31T12:15:58.303594+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:34:56.946695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306963+0000","last_change":"2024-08-31T12:15:56.297648+0000","last_active":"2024-08-31T12:15:58.306963+0000","last_peered":"2024-08-31T12:15:58.306963+0000","last_clean":"2024-08-31T12:15:58.306963+0000","last_became_active":"2024-08-31T12:15:56.297306+0000","last_became_peered":"2024-08-31T12:15:56.297306+0000","last_unstale":"2024-08-31T12:15:58.306963+0000","last_undegraded":"2024-08-31T12:15:58.306963+0000","last_fullsized":"2024-08-31T12:15:58.306963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:29:54.693819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300681+0000","last_change":"2024-08-31T12:15:57.297230+0000","last_active":"2024-08-31T12:15:58.300681+0000","last_peered":"2024-08-31T12:15:58.300681+0000","last_clean":"2024-08-31T12:15:58.300681+0000","last_became_active":"2024-08-31T12:15:57.296892+0000","last_became_peered":"2024-08-31T12:15:57.296892+0000","last_unstale":"2024-08-31T12:15:58.300681+0000","last_undegraded":"2024-08-31T12:15:58.300681+0000","last_fullsized":"2024-08-31T12:15:58.300681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:42:36.033586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300841+0000","last_change":"2024-08-31T12:15:56.286504+0000","last_active":"2024-08-31T12:15:58.300841+0000","last_peered":"2024-08-31T12:15:58.300841+0000","last_clean":"2024-08-31T12:15:58.300841+0000","last_became_active":"2024-08-31T12:15:56.286173+0000","last_became_peered":"2024-08-31T12:15:56.286173+0000","last_unstale":"2024-08-31T12:15:58.300841+0000","last_undegraded":"2024-08-31T12:15:58.300841+0000","last_fullsized":"2024-08-31T12:15:58.300841+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:06:34.313772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.938135+0000","last_change":"2024-08-31T12:15:57.308013+0000","last_active":"2024-08-31T12:15:58.938135+0000","last_peered":"2024-08-31T12:15:58.938135+0000","last_clean":"2024-08-31T12:15:58.938135+0000","last_became_active":"2024-08-31T12:15:57.307701+0000","last_became_peered":"2024-08-31T12:15:57.307701+0000","last_unstale":"2024-08-31T12:15:58.938135+0000","last_undegraded":"2024-08-31T12:15:58.938135+0000","last_fullsized":"2024-08-31T12:15:58.938135+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:06:15.161589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302555+0000","last_change":"2024-08-31T12:15:56.294852+0000","last_active":"2024-08-31T12:15:58.302555+0000","last_peered":"2024-08-31T12:15:58.302555+0000","last_clean":"2024-08-31T12:15:58.302555+0000","last_became_active":"2024-08-31T12:15:56.294496+0000","last_became_peered":"2024-08-31T12:15:56.294496+0000","last_unstale":"2024-08-31T12:15:58.302555+0000","last_undegraded":"2024-08-31T12:15:58.302555+0000","last_fullsized":"2024-08-31T12:15:58.302555+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:49:17.488528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937973+0000","last_change":"2024-08-31T12:15:57.305202+0000","last_active":"2024-08-31T12:15:58.937973+0000","last_peered":"2024-08-31T12:15:58.937973+0000","last_clean":"2024-08-31T12:15:58.937973+0000","last_became_active":"2024-08-31T12:15:57.304602+0000","last_became_peered":"2024-08-31T12:15:57.304602+0000","last_unstale":"2024-08-31T12:15:58.937973+0000","last_undegraded":"2024-08-31T12:15:58.937973+0000","last_fullsized":"2024-08-31T12:15:58.937973+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:56:03.280339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306090+0000","last_change":"2024-08-31T12:15:56.297542+0000","last_active":"2024-08-31T12:15:58.306090+0000","last_peered":"2024-08-31T12:15:58.306090+0000","last_clean":"2024-08-31T12:15:58.306090+0000","last_became_active":"2024-08-31T12:15:56.297168+0000","last_became_peered":"2024-08-31T12:15:56.297168+0000","last_unstale":"2024-08-31T12:15:58.306090+0000","last_undegraded":"2024-08-31T12:15:58.306090+0000","last_fullsized":"2024-08-31T12:15:58.306090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:56:41.924772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936311+0000","last_change":"2024-08-31T12:15:57.298845+0000","last_active":"2024-08-31T12:15:58.936311+0000","last_peered":"2024-08-31T12:15:58.936311+0000","last_clean":"2024-08-31T12:15:58.936311+0000","last_became_active":"2024-08-31T12:15:57.298520+0000","last_became_peered":"2024-08-31T12:15:57.298520+0000","last_unstale":"2024-08-31T12:15:58.936311+0000","last_undegraded":"2024-08-31T12:15:58.936311+0000","last_fullsized":"2024-08-31T12:15:58.936311+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:06:35.318873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300879+0000","last_change":"2024-08-31T12:15:56.290475+0000","last_active":"2024-08-31T12:15:58.300879+0000","last_peered":"2024-08-31T12:15:58.300879+0000","last_clean":"2024-08-31T12:15:58.300879+0000","last_became_active":"2024-08-31T12:15:56.289960+0000","last_became_peered":"2024-08-31T12:15:56.289960+0000","last_unstale":"2024-08-31T12:15:58.300879+0000","last_undegraded":"2024-08-31T12:15:58.300879+0000","last_fullsized":"2024-08-31T12:15:58.300879+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:29:53.117724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.938261+0000","last_change":"2024-08-31T12:15:57.301460+0000","last_active":"2024-08-31T12:15:58.938261+0000","last_peered":"2024-08-31T12:15:58.938261+0000","last_clean":"2024-08-31T12:15:58.938261+0000","last_became_active":"2024-08-31T12:15:57.301157+0000","last_became_peered":"2024-08-31T12:15:57.301157+0000","last_unstale":"2024-08-31T12:15:58.938261+0000","last_undegraded":"2024-08-31T12:15:58.938261+0000","last_fullsized":"2024-08-31T12:15:58.938261+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:22:52.173578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.301007+0000","last_change":"2024-08-31T12:15:56.289720+0000","last_active":"2024-08-31T12:15:58.301007+0000","last_peered":"2024-08-31T12:15:58.301007+0000","last_clean":"2024-08-31T12:15:58.301007+0000","last_became_active":"2024-08-31T12:15:56.289332+0000","last_became_peered":"2024-08-31T12:15:56.289332+0000","last_unstale":"2024-08-31T12:15:58.301007+0000","last_undegraded":"2024-08-31T12:15:58.301007+0000","last_fullsized":"2024-08-31T12:15:58.301007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:15:51.561831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937371+0000","last_change":"2024-08-31T12:15:57.313802+0000","last_active":"2024-08-31T12:15:58.937371+0000","last_peered":"2024-08-31T12:15:58.937371+0000","last_clean":"2024-08-31T12:15:58.937371+0000","last_became_active":"2024-08-31T12:15:57.313428+0000","last_became_peered":"2024-08-31T12:15:57.313428+0000","last_unstale":"2024-08-31T12:15:58.937371+0000","last_undegraded":"2024-08-31T12:15:58.937371+0000","last_fullsized":"2024-08-31T12:15:58.937371+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:22:13.435300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306928+0000","last_change":"2024-08-31T12:15:56.296105+0000","last_active":"2024-08-31T12:15:58.306928+0000","last_peered":"2024-08-31T12:15:58.306928+0000","last_clean":"2024-08-31T12:15:58.306928+0000","last_became_active":"2024-08-31T12:15:56.295524+0000","last_became_peered":"2024-08-31T12:15:56.295524+0000","last_unstale":"2024-08-31T12:15:58.306928+0000","last_undegraded":"2024-08-31T12:15:58.306928+0000","last_fullsized":"2024-08-31T12:15:58.306928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:41:55.482044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.301683+0000","last_change":"2024-08-31T12:15:57.300155+0000","last_active":"2024-08-31T12:15:58.301683+0000","last_peered":"2024-08-31T12:15:58.301683+0000","last_clean":"2024-08-31T12:15:58.301683+0000","last_became_active":"2024-08-31T12:15:57.299436+0000","last_became_peered":"2024-08-31T12:15:57.299436+0000","last_unstale":"2024-08-31T12:15:58.301683+0000","last_undegraded":"2024-08-31T12:15:58.301683+0000","last_fullsized":"2024-08-31T12:15:58.301683+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:37:45.891394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303455+0000","last_change":"2024-08-31T12:15:57.300529+0000","last_active":"2024-08-31T12:15:58.303455+0000","last_peered":"2024-08-31T12:15:58.303455+0000","last_clean":"2024-08-31T12:15:58.303455+0000","last_became_active":"2024-08-31T12:15:57.300159+0000","last_became_peered":"2024-08-31T12:15:57.300159+0000","last_unstale":"2024-08-31T12:15:58.303455+0000","last_undegraded":"2024-08-31T12:15:58.303455+0000","last_fullsized":"2024-08-31T12:15:58.303455+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:35:53.831742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306371+0000","last_change":"2024-08-31T12:15:56.297791+0000","last_active":"2024-08-31T12:15:58.306371+0000","last_peered":"2024-08-31T12:15:58.306371+0000","last_clean":"2024-08-31T12:15:58.306371+0000","last_became_active":"2024-08-31T12:15:56.296976+0000","last_became_peered":"2024-08-31T12:15:56.296976+0000","last_unstale":"2024-08-31T12:15:58.306371+0000","last_undegraded":"2024-08-31T12:15:58.306371+0000","last_fullsized":"2024-08-31T12:15:58.306371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:07:54.963050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303405+0000","last_change":"2024-08-31T12:15:57.301167+0000","last_active":"2024-08-31T12:15:58.303405+0000","last_peered":"2024-08-31T12:15:58.303405+0000","last_clean":"2024-08-31T12:15:58.303405+0000","last_became_active":"2024-08-31T12:15:57.300646+0000","last_became_peered":"2024-08-31T12:15:57.300646+0000","last_unstale":"2024-08-31T12:15:58.303405+0000","last_undegraded":"2024-08-31T12:15:58.303405+0000","last_fullsized":"2024-08-31T12:15:58.303405+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:05:53.039520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306722+0000","last_change":"2024-08-31T12:15:56.295139+0000","last_active":"2024-08-31T12:15:58.306722+0000","last_peered":"2024-08-31T12:15:58.306722+0000","last_clean":"2024-08-31T12:15:58.306722+0000","last_became_active":"2024-08-31T12:15:56.294560+0000","last_became_peered":"2024-08-31T12:15:56.294560+0000","last_unstale":"2024-08-31T12:15:58.306722+0000","last_undegraded":"2024-08-31T12:15:58.306722+0000","last_fullsized":"2024-08-31T12:15:58.306722+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:52:16.999507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937898+0000","last_change":"2024-08-31T12:15:57.311074+0000","last_active":"2024-08-31T12:15:58.937898+0000","last_peered":"2024-08-31T12:15:58.937898+0000","last_clean":"2024-08-31T12:15:58.937898+0000","last_became_active":"2024-08-31T12:15:57.310262+0000","last_became_peered":"2024-08-31T12:15:57.310262+0000","last_unstale":"2024-08-31T12:15:58.937898+0000","last_undegraded":"2024-08-31T12:15:58.937898+0000","last_fullsized":"2024-08-31T12:15:58.937898+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:47:51.267409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937288+0000","last_change":"2024-08-31T12:15:56.285428+0000","last_active":"2024-08-31T12:15:58.937288+0000","last_peered":"2024-08-31T12:15:58.937288+0000","last_clean":"2024-08-31T12:15:58.937288+0000","last_became_active":"2024-08-31T12:15:56.284884+0000","last_became_peered":"2024-08-31T12:15:56.284884+0000","last_unstale":"2024-08-31T12:15:58.937288+0000","last_undegraded":"2024-08-31T12:15:58.937288+0000","last_fullsized":"2024-08-31T12:15:58.937288+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:49:25.181432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937174+0000","last_change":"2024-08-31T12:15:57.305772+0000","last_active":"2024-08-31T12:15:58.937174+0000","last_peered":"2024-08-31T12:15:58.937174+0000","last_clean":"2024-08-31T12:15:58.937174+0000","last_became_active":"2024-08-31T12:15:57.305409+0000","last_became_peered":"2024-08-31T12:15:57.305409+0000","last_unstale":"2024-08-31T12:15:58.937174+0000","last_undegraded":"2024-08-31T12:15:58.937174+0000","last_fullsized":"2024-08-31T12:15:58.937174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:53:19.523811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937989+0000","last_change":"2024-08-31T12:15:56.294197+0000","last_active":"2024-08-31T12:15:58.937989+0000","last_peered":"2024-08-31T12:15:58.937989+0000","last_clean":"2024-08-31T12:15:58.937989+0000","last_became_active":"2024-08-31T12:15:56.293864+0000","last_became_peered":"2024-08-31T12:15:56.293864+0000","last_unstale":"2024-08-31T12:15:58.937989+0000","last_undegraded":"2024-08-31T12:15:58.937989+0000","last_fullsized":"2024-08-31T12:15:58.937989+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:43:17.443433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509446,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35560,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336280,"statfs":{"total":96636764160,"available":96600350720,"internally_reserved":0,"allocated":208896,"data_stored":73101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509445,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":73101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509446,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":208896,"data_stored":73101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509446,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":802816,"data_stored":663469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36136,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335704,"statfs":{"total":96636764160,"available":96599760896,"internally_reserved":0,"allocated":802816,"data_stored":663469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542151,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35552,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336288,"statfs":{"total":96636764160,"available":96600358912,"internally_reserved":0,"allocated":208896,"data_stored":73101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542149,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35552,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336288,"statfs":{"total":96636764160,"available":96600358912,"internally_reserved":0,"allocated":208896,"data_stored":73101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":196608,"data_stored":66801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-31T12:16:03.353 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-31T12:16:03.503 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.502+0000 7f1fdb59d640 1 -- 172.21.15.19:0/1969453245 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1fc0004800 msgr2=0x7f1fc0004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:03.503 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.502+0000 7f1fdb59d640 1 --2- 172.21.15.19:0/1969453245 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1fc0004800 0x7f1fc0004c10 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1fd40cbc20 tx=0x7f1fc0005830 comp rx=0 tx=0).stop 2024-08-31T12:16:03.503 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.502+0000 7f1fdad9c640 1 -- 172.21.15.19:0/1969453245 reap_dead start 2024-08-31T12:16:03.503 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.502+0000 7f1fdb59d640 1 -- 172.21.15.19:0/1969453245 shutdown_connections 2024-08-31T12:16:03.503 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.502+0000 7f1fdb59d640 1 -- 172.21.15.19:0/1969453245 >> 172.21.15.19:0/1969453245 conn(0x7f1fd406d6c0 msgr2=0x7f1fd406dad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:03.503 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.502+0000 7f1fdb59d640 1 -- 172.21.15.19:0/1969453245 shutdown_connections 2024-08-31T12:16:03.503 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.502+0000 7f1fdb59d640 1 -- 172.21.15.19:0/1969453245 wait complete. 2024-08-31T12:16:03.504 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.504+0000 7f1fdb59d640 1 Processor -- start 2024-08-31T12:16:03.504 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.504+0000 7f1fdb59d640 1 -- start start 2024-08-31T12:16:03.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.505+0000 7f1fdb59d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1fc0004800 0x7f1fd4151a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:03.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.505+0000 7f1fdb59d640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1fd41e4ab0 0x7f1fd4150970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:03.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.505+0000 7f1fdb59d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1fd41e4ec0 0x7f1fd4150eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:03.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.505+0000 7f1fdb59d640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd40cc3a0 con 0x7f1fc0004800 2024-08-31T12:16:03.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.505+0000 7f1fdb59d640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd415f080 con 0x7f1fd41e4ab0 2024-08-31T12:16:03.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.505+0000 7f1fdb59d640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f1fd40cc520 con 0x7f1fd41e4ec0 2024-08-31T12:16:03.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.505+0000 7f1fda59b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1fc0004800 0x7f1fd4151a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:03.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.505+0000 7f1fda59b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1fc0004800 0x7f1fd4151a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:56494/0 (socket says 172.21.15.19:56494) 2024-08-31T12:16:03.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.505+0000 7f1fda59b640 1 -- 172.21.15.19:0/2582664293 learned_addr learned my addr 172.21.15.19:0/2582664293 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:03.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.505+0000 7f1fd9d9a640 1 --2- 172.21.15.19:0/2582664293 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1fd41e4ab0 0x7f1fd4150970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:03.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.505+0000 7f1fdad9c640 1 --2- 172.21.15.19:0/2582664293 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1fd41e4ec0 0x7f1fd4150eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:03.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.505+0000 7f1fda59b640 1 -- 172.21.15.19:0/2582664293 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1fd41e4ec0 msgr2=0x7f1fd4150eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:03.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.505+0000 7f1fda59b640 1 --2- 172.21.15.19:0/2582664293 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1fd41e4ec0 0x7f1fd4150eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:03.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.505+0000 7f1fda59b640 1 -- 172.21.15.19:0/2582664293 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1fd41e4ab0 msgr2=0x7f1fd4150970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:03.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.505+0000 7f1fda59b640 1 --2- 172.21.15.19:0/2582664293 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1fd41e4ab0 0x7f1fd4150970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:03.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.505+0000 7f1fda59b640 1 -- 172.21.15.19:0/2582664293 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fd41513f0 con 0x7f1fc0004800 2024-08-31T12:16:03.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.505+0000 7f1fd9d9a640 1 --2- 172.21.15.19:0/2582664293 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1fd41e4ab0 0x7f1fd4150970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-31T12:16:03.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.505+0000 7f1fda59b640 1 --2- 172.21.15.19:0/2582664293 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1fc0004800 0x7f1fd4151a50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0011b10 tx=0x7f1fd0009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:03.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.505+0000 7f1fcaffd640 1 -- 172.21.15.19:0/2582664293 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd001e070 con 0x7f1fc0004800 2024-08-31T12:16:03.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.506+0000 7f1fcaffd640 1 -- 172.21.15.19:0/2582664293 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1fd0004050 con 0x7f1fc0004800 2024-08-31T12:16:03.509 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.506+0000 7f1fcaffd640 1 -- 172.21.15.19:0/2582664293 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd000fea0 con 0x7f1fc0004800 2024-08-31T12:16:03.509 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.506+0000 7f1fdb59d640 1 -- 172.21.15.19:0/2582664293 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fd4151f90 con 0x7f1fc0004800 2024-08-31T12:16:03.509 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.506+0000 7f1fdb59d640 1 -- 172.21.15.19:0/2582664293 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fd41540f0 con 0x7f1fc0004800 2024-08-31T12:16:03.509 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.507+0000 7f1fdb59d640 1 -- 172.21.15.19:0/2582664293 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fd40ca910 con 0x7f1fc0004800 2024-08-31T12:16:03.510 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.507+0000 7f1fcaffd640 1 -- 172.21.15.19:0/2582664293 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f1fd0021070 con 0x7f1fc0004800 2024-08-31T12:16:03.510 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.508+0000 7f1fcaffd640 1 --2- 172.21.15.19:0/2582664293 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1fb8077a80 0x7f1fb8079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:03.510 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.508+0000 7f1fcaffd640 1 -- 172.21.15.19:0/2582664293 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f1fd009a000 con 0x7f1fc0004800 2024-08-31T12:16:03.510 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.508+0000 7f1fd9d9a640 1 --2- 172.21.15.19:0/2582664293 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1fb8077a80 0x7f1fb8079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:03.510 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.508+0000 7f1fd9d9a640 1 --2- 172.21.15.19:0/2582664293 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1fb8077a80 0x7f1fb8079f90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4032000 tx=0x7f1fc4001190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:03.513 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.512+0000 7f1fcaffd640 1 -- 172.21.15.19:0/2582664293 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f1fd0062870 con 0x7f1fc0004800 2024-08-31T12:16:03.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.691+0000 7f1fdb59d640 1 -- 172.21.15.19:0/2582664293 --> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1fd40b2f00 con 0x7f1fb8077a80 2024-08-31T12:16:03.696 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.696+0000 7f1fcaffd640 1 -- 172.21.15.19:0/2582664293 <== mgr.4100 v2:172.21.15.43:6800/2824336655 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251838 (secure 0 0 0) 0x7f1fc40807d0 con 0x7f1fb8077a80 2024-08-31T12:16:03.699 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:16:03.700 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-08-31T12:16:03.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.699+0000 7f1fdb59d640 1 -- 172.21.15.19:0/2582664293 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1fb8077a80 msgr2=0x7f1fb8079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:03.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.699+0000 7f1fdb59d640 1 --2- 172.21.15.19:0/2582664293 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1fb8077a80 0x7f1fb8079f90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4032000 tx=0x7f1fc4001190 comp rx=0 tx=0).stop 2024-08-31T12:16:03.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.699+0000 7f1fdb59d640 1 -- 172.21.15.19:0/2582664293 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1fc0004800 msgr2=0x7f1fd4151a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:03.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.699+0000 7f1fdb59d640 1 --2- 172.21.15.19:0/2582664293 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1fc0004800 0x7f1fd4151a50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0011b10 tx=0x7f1fd0009e90 comp rx=0 tx=0).stop 2024-08-31T12:16:03.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.699+0000 7f1fdb59d640 1 -- 172.21.15.19:0/2582664293 shutdown_connections 2024-08-31T12:16:03.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.699+0000 7f1fdb59d640 1 --2- 172.21.15.19:0/2582664293 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1fb8077a80 0x7f1fb8079f90 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:03.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.699+0000 7f1fdb59d640 1 --2- 172.21.15.19:0/2582664293 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1fd41e4ec0 0x7f1fd4150eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:03.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.699+0000 7f1fdb59d640 1 --2- 172.21.15.19:0/2582664293 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1fd41e4ab0 0x7f1fd4150970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:03.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.699+0000 7f1fdb59d640 1 --2- 172.21.15.19:0/2582664293 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1fc0004800 0x7f1fd4151a50 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:03.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.699+0000 7f1fdb59d640 1 -- 172.21.15.19:0/2582664293 >> 172.21.15.19:0/2582664293 conn(0x7f1fd406d6c0 msgr2=0x7f1fd40b3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:03.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.699+0000 7f1fdb59d640 1 -- 172.21.15.19:0/2582664293 shutdown_connections 2024-08-31T12:16:03.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.699+0000 7f1fdb59d640 1 -- 172.21.15.19:0/2582664293 wait complete. 2024-08-31T12:16:03.710 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-08-31T12:16:02.492207+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":176,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":252316,"kb_used_data":2780,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754722404,"statfs":{"total":773094113280,"available":772835741696,"internally_reserved":0,"allocated":2846720,"data_stored":1759244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.223387"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306730+0000","last_change":"2024-08-31T12:15:57.306560+0000","last_active":"2024-08-31T12:15:58.306730+0000","last_peered":"2024-08-31T12:15:58.306730+0000","last_clean":"2024-08-31T12:15:58.306730+0000","last_became_active":"2024-08-31T12:15:57.306148+0000","last_became_peered":"2024-08-31T12:15:57.306148+0000","last_unstale":"2024-08-31T12:15:58.306730+0000","last_undegraded":"2024-08-31T12:15:58.306730+0000","last_fullsized":"2024-08-31T12:15:58.306730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:35:53.831742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300935+0000","last_change":"2024-08-31T12:15:57.307314+0000","last_active":"2024-08-31T12:15:58.300935+0000","last_peered":"2024-08-31T12:15:58.300935+0000","last_clean":"2024-08-31T12:15:58.300935+0000","last_became_active":"2024-08-31T12:15:57.306940+0000","last_became_peered":"2024-08-31T12:15:57.306940+0000","last_unstale":"2024-08-31T12:15:58.300935+0000","last_undegraded":"2024-08-31T12:15:58.300935+0000","last_fullsized":"2024-08-31T12:15:58.300935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:38:30.321214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937267+0000","last_change":"2024-08-31T12:15:57.309372+0000","last_active":"2024-08-31T12:15:58.937267+0000","last_peered":"2024-08-31T12:15:58.937267+0000","last_clean":"2024-08-31T12:15:58.937267+0000","last_became_active":"2024-08-31T12:15:57.305795+0000","last_became_peered":"2024-08-31T12:15:57.305795+0000","last_unstale":"2024-08-31T12:15:58.937267+0000","last_undegraded":"2024-08-31T12:15:58.937267+0000","last_fullsized":"2024-08-31T12:15:58.937267+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:37:45.891394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302363+0000","last_change":"2024-08-31T12:15:57.307339+0000","last_active":"2024-08-31T12:15:58.302363+0000","last_peered":"2024-08-31T12:15:58.302363+0000","last_clean":"2024-08-31T12:15:58.302363+0000","last_became_active":"2024-08-31T12:15:57.306957+0000","last_became_peered":"2024-08-31T12:15:57.306957+0000","last_unstale":"2024-08-31T12:15:58.302363+0000","last_undegraded":"2024-08-31T12:15:58.302363+0000","last_fullsized":"2024-08-31T12:15:58.302363+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:43:33.705099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306757+0000","last_change":"2024-08-31T12:15:57.317492+0000","last_active":"2024-08-31T12:15:58.306757+0000","last_peered":"2024-08-31T12:15:58.306757+0000","last_clean":"2024-08-31T12:15:58.306757+0000","last_became_active":"2024-08-31T12:15:57.317079+0000","last_became_peered":"2024-08-31T12:15:57.317079+0000","last_unstale":"2024-08-31T12:15:58.306757+0000","last_undegraded":"2024-08-31T12:15:58.306757+0000","last_fullsized":"2024-08-31T12:15:58.306757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:44:40.712699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302342+0000","last_change":"2024-08-31T12:15:57.307505+0000","last_active":"2024-08-31T12:15:58.302342+0000","last_peered":"2024-08-31T12:15:58.302342+0000","last_clean":"2024-08-31T12:15:58.302342+0000","last_became_active":"2024-08-31T12:15:57.307010+0000","last_became_peered":"2024-08-31T12:15:57.307010+0000","last_unstale":"2024-08-31T12:15:58.302342+0000","last_undegraded":"2024-08-31T12:15:58.302342+0000","last_fullsized":"2024-08-31T12:15:58.302342+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:39:31.100565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306247+0000","last_change":"2024-08-31T12:15:57.316892+0000","last_active":"2024-08-31T12:15:58.306247+0000","last_peered":"2024-08-31T12:15:58.306247+0000","last_clean":"2024-08-31T12:15:58.306247+0000","last_became_active":"2024-08-31T12:15:57.316504+0000","last_became_peered":"2024-08-31T12:15:57.316504+0000","last_unstale":"2024-08-31T12:15:58.306247+0000","last_undegraded":"2024-08-31T12:15:58.306247+0000","last_fullsized":"2024-08-31T12:15:58.306247+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:05:48.614918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937976+0000","last_change":"2024-08-31T12:15:57.309224+0000","last_active":"2024-08-31T12:15:58.937976+0000","last_peered":"2024-08-31T12:15:58.937976+0000","last_clean":"2024-08-31T12:15:58.937976+0000","last_became_active":"2024-08-31T12:15:57.308914+0000","last_became_peered":"2024-08-31T12:15:57.308914+0000","last_unstale":"2024-08-31T12:15:58.937976+0000","last_undegraded":"2024-08-31T12:15:58.937976+0000","last_fullsized":"2024-08-31T12:15:58.937976+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:37:02.435678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306762+0000","last_change":"2024-08-31T12:15:57.316723+0000","last_active":"2024-08-31T12:15:58.306762+0000","last_peered":"2024-08-31T12:15:58.306762+0000","last_clean":"2024-08-31T12:15:58.306762+0000","last_became_active":"2024-08-31T12:15:57.316354+0000","last_became_peered":"2024-08-31T12:15:57.316354+0000","last_unstale":"2024-08-31T12:15:58.306762+0000","last_undegraded":"2024-08-31T12:15:58.306762+0000","last_fullsized":"2024-08-31T12:15:58.306762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:31:19.860950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302888+0000","last_change":"2024-08-31T12:15:57.311161+0000","last_active":"2024-08-31T12:15:58.302888+0000","last_peered":"2024-08-31T12:15:58.302888+0000","last_clean":"2024-08-31T12:15:58.302888+0000","last_became_active":"2024-08-31T12:15:57.310807+0000","last_became_peered":"2024-08-31T12:15:57.310807+0000","last_unstale":"2024-08-31T12:15:58.302888+0000","last_undegraded":"2024-08-31T12:15:58.302888+0000","last_fullsized":"2024-08-31T12:15:58.302888+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:56:03.280339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302118+0000","last_change":"2024-08-31T12:15:57.300930+0000","last_active":"2024-08-31T12:15:58.302118+0000","last_peered":"2024-08-31T12:15:58.302118+0000","last_clean":"2024-08-31T12:15:58.302118+0000","last_became_active":"2024-08-31T12:15:57.300610+0000","last_became_peered":"2024-08-31T12:15:57.300610+0000","last_unstale":"2024-08-31T12:15:58.302118+0000","last_undegraded":"2024-08-31T12:15:58.302118+0000","last_fullsized":"2024-08-31T12:15:58.302118+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:42:39.731742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936435+0000","last_change":"2024-08-31T12:15:57.303151+0000","last_active":"2024-08-31T12:15:58.936435+0000","last_peered":"2024-08-31T12:15:58.936435+0000","last_clean":"2024-08-31T12:15:58.936435+0000","last_became_active":"2024-08-31T12:15:57.302622+0000","last_became_peered":"2024-08-31T12:15:57.302622+0000","last_unstale":"2024-08-31T12:15:58.936435+0000","last_undegraded":"2024-08-31T12:15:58.936435+0000","last_fullsized":"2024-08-31T12:15:58.936435+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:49:18.493629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937309+0000","last_change":"2024-08-31T12:15:57.305676+0000","last_active":"2024-08-31T12:15:58.937309+0000","last_peered":"2024-08-31T12:15:58.937309+0000","last_clean":"2024-08-31T12:15:58.937309+0000","last_became_active":"2024-08-31T12:15:57.305321+0000","last_became_peered":"2024-08-31T12:15:57.305321+0000","last_unstale":"2024-08-31T12:15:58.937309+0000","last_undegraded":"2024-08-31T12:15:58.937309+0000","last_fullsized":"2024-08-31T12:15:58.937309+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:42:39.731742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936688+0000","last_change":"2024-08-31T12:15:57.309671+0000","last_active":"2024-08-31T12:15:58.936688+0000","last_peered":"2024-08-31T12:15:58.936688+0000","last_clean":"2024-08-31T12:15:58.936688+0000","last_became_active":"2024-08-31T12:15:57.307013+0000","last_became_peered":"2024-08-31T12:15:57.307013+0000","last_unstale":"2024-08-31T12:15:58.936688+0000","last_undegraded":"2024-08-31T12:15:58.936688+0000","last_fullsized":"2024-08-31T12:15:58.936688+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:18:55.451164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300929+0000","last_change":"2024-08-31T12:15:57.298150+0000","last_active":"2024-08-31T12:15:58.300929+0000","last_peered":"2024-08-31T12:15:58.300929+0000","last_clean":"2024-08-31T12:15:58.300929+0000","last_became_active":"2024-08-31T12:15:57.297182+0000","last_became_peered":"2024-08-31T12:15:57.297182+0000","last_unstale":"2024-08-31T12:15:58.300929+0000","last_undegraded":"2024-08-31T12:15:58.300929+0000","last_fullsized":"2024-08-31T12:15:58.300929+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:16:42.625396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302762+0000","last_change":"2024-08-31T12:15:57.314224+0000","last_active":"2024-08-31T12:15:58.302762+0000","last_peered":"2024-08-31T12:15:58.302762+0000","last_clean":"2024-08-31T12:15:58.302762+0000","last_became_active":"2024-08-31T12:15:57.313887+0000","last_became_peered":"2024-08-31T12:15:57.313887+0000","last_unstale":"2024-08-31T12:15:58.302762+0000","last_undegraded":"2024-08-31T12:15:58.302762+0000","last_fullsized":"2024-08-31T12:15:58.302762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:54:45.720621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303640+0000","last_change":"2024-08-31T12:15:57.308899+0000","last_active":"2024-08-31T12:15:58.303640+0000","last_peered":"2024-08-31T12:15:58.303640+0000","last_clean":"2024-08-31T12:15:58.303640+0000","last_became_active":"2024-08-31T12:15:57.308388+0000","last_became_peered":"2024-08-31T12:15:57.308388+0000","last_unstale":"2024-08-31T12:15:58.303640+0000","last_undegraded":"2024-08-31T12:15:58.303640+0000","last_fullsized":"2024-08-31T12:15:58.303640+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:54:03.871840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-08-31T12:15:57.302751+0000","last_change":"2024-08-31T12:15:57.300201+0000","last_active":"2024-08-31T12:15:57.302751+0000","last_peered":"2024-08-31T12:15:57.302751+0000","last_clean":"2024-08-31T12:15:57.302751+0000","last_became_active":"2024-08-31T12:15:57.299826+0000","last_became_peered":"2024-08-31T12:15:57.299826+0000","last_unstale":"2024-08-31T12:15:57.302751+0000","last_undegraded":"2024-08-31T12:15:57.302751+0000","last_fullsized":"2024-08-31T12:15:57.302751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:50:24.220339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.307053+0000","last_change":"2024-08-31T12:15:57.317606+0000","last_active":"2024-08-31T12:15:58.307053+0000","last_peered":"2024-08-31T12:15:58.307053+0000","last_clean":"2024-08-31T12:15:58.307053+0000","last_became_active":"2024-08-31T12:15:57.317185+0000","last_became_peered":"2024-08-31T12:15:57.317185+0000","last_unstale":"2024-08-31T12:15:58.307053+0000","last_undegraded":"2024-08-31T12:15:58.307053+0000","last_fullsized":"2024-08-31T12:15:58.307053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:25:46.346490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302171+0000","last_change":"2024-08-31T12:15:57.298778+0000","last_active":"2024-08-31T12:15:58.302171+0000","last_peered":"2024-08-31T12:15:58.302171+0000","last_clean":"2024-08-31T12:15:58.302171+0000","last_became_active":"2024-08-31T12:15:57.298430+0000","last_became_peered":"2024-08-31T12:15:57.298430+0000","last_unstale":"2024-08-31T12:15:58.302171+0000","last_undegraded":"2024-08-31T12:15:58.302171+0000","last_fullsized":"2024-08-31T12:15:58.302171+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:22:13.435300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937190+0000","last_change":"2024-08-31T12:15:57.317176+0000","last_active":"2024-08-31T12:15:58.937190+0000","last_peered":"2024-08-31T12:15:58.937190+0000","last_clean":"2024-08-31T12:15:58.937190+0000","last_became_active":"2024-08-31T12:15:57.316870+0000","last_became_peered":"2024-08-31T12:15:57.316870+0000","last_unstale":"2024-08-31T12:15:58.937190+0000","last_undegraded":"2024-08-31T12:15:58.937190+0000","last_fullsized":"2024-08-31T12:15:58.937190+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:43:12.703971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302636+0000","last_change":"2024-08-31T12:15:57.308907+0000","last_active":"2024-08-31T12:15:58.302636+0000","last_peered":"2024-08-31T12:15:58.302636+0000","last_clean":"2024-08-31T12:15:58.302636+0000","last_became_active":"2024-08-31T12:15:57.308459+0000","last_became_peered":"2024-08-31T12:15:57.308459+0000","last_unstale":"2024-08-31T12:15:58.302636+0000","last_undegraded":"2024-08-31T12:15:58.302636+0000","last_fullsized":"2024-08-31T12:15:58.302636+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:16:46.648912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937760+0000","last_change":"2024-08-31T12:15:57.304032+0000","last_active":"2024-08-31T12:15:58.937760+0000","last_peered":"2024-08-31T12:15:58.937760+0000","last_clean":"2024-08-31T12:15:58.937760+0000","last_became_active":"2024-08-31T12:15:57.303401+0000","last_became_peered":"2024-08-31T12:15:57.303401+0000","last_unstale":"2024-08-31T12:15:58.937760+0000","last_undegraded":"2024-08-31T12:15:58.937760+0000","last_fullsized":"2024-08-31T12:15:58.937760+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:56:03.280339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303013+0000","last_change":"2024-08-31T12:15:57.305412+0000","last_active":"2024-08-31T12:15:58.303013+0000","last_peered":"2024-08-31T12:15:58.303013+0000","last_clean":"2024-08-31T12:15:58.303013+0000","last_became_active":"2024-08-31T12:15:57.305075+0000","last_became_peered":"2024-08-31T12:15:57.305075+0000","last_unstale":"2024-08-31T12:15:58.303013+0000","last_undegraded":"2024-08-31T12:15:58.303013+0000","last_fullsized":"2024-08-31T12:15:58.303013+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:30:13.715505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937828+0000","last_change":"2024-08-31T12:15:57.300179+0000","last_active":"2024-08-31T12:15:58.937828+0000","last_peered":"2024-08-31T12:15:58.937828+0000","last_clean":"2024-08-31T12:15:58.937828+0000","last_became_active":"2024-08-31T12:15:57.299861+0000","last_became_peered":"2024-08-31T12:15:57.299861+0000","last_unstale":"2024-08-31T12:15:58.937828+0000","last_undegraded":"2024-08-31T12:15:58.937828+0000","last_fullsized":"2024-08-31T12:15:58.937828+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:47:51.267409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303001+0000","last_change":"2024-08-31T12:15:57.308519+0000","last_active":"2024-08-31T12:15:58.303001+0000","last_peered":"2024-08-31T12:15:58.303001+0000","last_clean":"2024-08-31T12:15:58.303001+0000","last_became_active":"2024-08-31T12:15:57.308189+0000","last_became_peered":"2024-08-31T12:15:57.308189+0000","last_unstale":"2024-08-31T12:15:58.303001+0000","last_undegraded":"2024-08-31T12:15:58.303001+0000","last_fullsized":"2024-08-31T12:15:58.303001+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:37:02.435678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300666+0000","last_change":"2024-08-31T12:15:57.308321+0000","last_active":"2024-08-31T12:15:58.300666+0000","last_peered":"2024-08-31T12:15:58.300666+0000","last_clean":"2024-08-31T12:15:58.300666+0000","last_became_active":"2024-08-31T12:15:57.304514+0000","last_became_peered":"2024-08-31T12:15:57.304514+0000","last_unstale":"2024-08-31T12:15:58.300666+0000","last_undegraded":"2024-08-31T12:15:58.300666+0000","last_fullsized":"2024-08-31T12:15:58.300666+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:09:49.617826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936830+0000","last_change":"2024-08-31T12:15:57.299268+0000","last_active":"2024-08-31T12:15:58.936830+0000","last_peered":"2024-08-31T12:15:58.936830+0000","last_clean":"2024-08-31T12:15:58.936830+0000","last_became_active":"2024-08-31T12:15:57.298766+0000","last_became_peered":"2024-08-31T12:15:57.298766+0000","last_unstale":"2024-08-31T12:15:58.936830+0000","last_undegraded":"2024-08-31T12:15:58.936830+0000","last_fullsized":"2024-08-31T12:15:58.936830+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:37:45.891394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937525+0000","last_change":"2024-08-31T12:15:57.310121+0000","last_active":"2024-08-31T12:15:58.937525+0000","last_peered":"2024-08-31T12:15:58.937525+0000","last_clean":"2024-08-31T12:15:58.937525+0000","last_became_active":"2024-08-31T12:15:57.309440+0000","last_became_peered":"2024-08-31T12:15:57.309440+0000","last_unstale":"2024-08-31T12:15:58.937525+0000","last_undegraded":"2024-08-31T12:15:58.937525+0000","last_fullsized":"2024-08-31T12:15:58.937525+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:53:19.523811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302889+0000","last_change":"2024-08-31T12:15:57.306214+0000","last_active":"2024-08-31T12:15:58.302889+0000","last_peered":"2024-08-31T12:15:58.302889+0000","last_clean":"2024-08-31T12:15:58.302889+0000","last_became_active":"2024-08-31T12:15:57.305878+0000","last_became_peered":"2024-08-31T12:15:57.305878+0000","last_unstale":"2024-08-31T12:15:58.302889+0000","last_undegraded":"2024-08-31T12:15:58.302889+0000","last_fullsized":"2024-08-31T12:15:58.302889+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:05:48.614918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-08-31T12:15:57.312611+0000","last_change":"2024-08-31T12:15:57.307376+0000","last_active":"2024-08-31T12:15:57.312611+0000","last_peered":"2024-08-31T12:15:57.312611+0000","last_clean":"2024-08-31T12:15:57.312611+0000","last_became_active":"2024-08-31T12:15:57.307056+0000","last_became_peered":"2024-08-31T12:15:57.307056+0000","last_unstale":"2024-08-31T12:15:57.312611+0000","last_undegraded":"2024-08-31T12:15:57.312611+0000","last_fullsized":"2024-08-31T12:15:57.312611+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:39:32.951892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302319+0000","last_change":"2024-08-31T12:15:57.313798+0000","last_active":"2024-08-31T12:15:58.302319+0000","last_peered":"2024-08-31T12:15:58.302319+0000","last_clean":"2024-08-31T12:15:58.302319+0000","last_became_active":"2024-08-31T12:15:57.313463+0000","last_became_peered":"2024-08-31T12:15:57.313463+0000","last_unstale":"2024-08-31T12:15:58.302319+0000","last_undegraded":"2024-08-31T12:15:58.302319+0000","last_fullsized":"2024-08-31T12:15:58.302319+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:47:51.267409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936240+0000","last_change":"2024-08-31T12:15:57.293350+0000","last_active":"2024-08-31T12:15:58.936240+0000","last_peered":"2024-08-31T12:15:58.936240+0000","last_clean":"2024-08-31T12:15:58.936240+0000","last_became_active":"2024-08-31T12:15:57.292981+0000","last_became_peered":"2024-08-31T12:15:57.292981+0000","last_unstale":"2024-08-31T12:15:58.936240+0000","last_undegraded":"2024-08-31T12:15:58.936240+0000","last_fullsized":"2024-08-31T12:15:58.936240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:52:00.312013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937396+0000","last_change":"2024-08-31T12:15:56.293497+0000","last_active":"2024-08-31T12:15:58.937396+0000","last_peered":"2024-08-31T12:15:58.937396+0000","last_clean":"2024-08-31T12:15:58.937396+0000","last_became_active":"2024-08-31T12:15:56.293164+0000","last_became_peered":"2024-08-31T12:15:56.293164+0000","last_unstale":"2024-08-31T12:15:58.937396+0000","last_undegraded":"2024-08-31T12:15:58.937396+0000","last_fullsized":"2024-08-31T12:15:58.937396+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:43:17.443433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937814+0000","last_change":"2024-08-31T12:15:57.312738+0000","last_active":"2024-08-31T12:15:58.937814+0000","last_peered":"2024-08-31T12:15:58.937814+0000","last_clean":"2024-08-31T12:15:58.937814+0000","last_became_active":"2024-08-31T12:15:57.312360+0000","last_became_peered":"2024-08-31T12:15:57.312360+0000","last_unstale":"2024-08-31T12:15:58.937814+0000","last_undegraded":"2024-08-31T12:15:58.937814+0000","last_fullsized":"2024-08-31T12:15:58.937814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:22:13.435300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.301762+0000","last_change":"2024-08-31T12:15:56.283526+0000","last_active":"2024-08-31T12:15:58.301762+0000","last_peered":"2024-08-31T12:15:58.301762+0000","last_clean":"2024-08-31T12:15:58.301762+0000","last_became_active":"2024-08-31T12:15:56.283147+0000","last_became_peered":"2024-08-31T12:15:56.283147+0000","last_unstale":"2024-08-31T12:15:58.301762+0000","last_undegraded":"2024-08-31T12:15:58.301762+0000","last_fullsized":"2024-08-31T12:15:58.301762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:51:59.306912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937727+0000","last_change":"2024-08-31T12:15:56.291170+0000","last_active":"2024-08-31T12:15:58.937727+0000","last_peered":"2024-08-31T12:15:58.937727+0000","last_clean":"2024-08-31T12:15:58.937727+0000","last_became_active":"2024-08-31T12:15:56.290835+0000","last_became_peered":"2024-08-31T12:15:56.290835+0000","last_unstale":"2024-08-31T12:15:58.937727+0000","last_undegraded":"2024-08-31T12:15:58.937727+0000","last_fullsized":"2024-08-31T12:15:58.937727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:15:51.561831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.938053+0000","last_change":"2024-08-31T12:15:57.308183+0000","last_active":"2024-08-31T12:15:58.938053+0000","last_peered":"2024-08-31T12:15:58.938053+0000","last_clean":"2024-08-31T12:15:58.938053+0000","last_became_active":"2024-08-31T12:15:57.306962+0000","last_became_peered":"2024-08-31T12:15:57.306962+0000","last_unstale":"2024-08-31T12:15:58.938053+0000","last_undegraded":"2024-08-31T12:15:58.938053+0000","last_fullsized":"2024-08-31T12:15:58.938053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:01:18.188264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303216+0000","last_change":"2024-08-31T12:15:56.288126+0000","last_active":"2024-08-31T12:15:58.303216+0000","last_peered":"2024-08-31T12:15:58.303216+0000","last_clean":"2024-08-31T12:15:58.303216+0000","last_became_active":"2024-08-31T12:15:56.287552+0000","last_became_peered":"2024-08-31T12:15:56.287552+0000","last_unstale":"2024-08-31T12:15:58.303216+0000","last_undegraded":"2024-08-31T12:15:58.303216+0000","last_fullsized":"2024-08-31T12:15:58.303216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:54:29.988079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300810+0000","last_change":"2024-08-31T12:15:57.305029+0000","last_active":"2024-08-31T12:15:58.300810+0000","last_peered":"2024-08-31T12:15:58.300810+0000","last_clean":"2024-08-31T12:15:58.300810+0000","last_became_active":"2024-08-31T12:15:57.304700+0000","last_became_peered":"2024-08-31T12:15:57.304700+0000","last_unstale":"2024-08-31T12:15:58.300810+0000","last_undegraded":"2024-08-31T12:15:58.300810+0000","last_fullsized":"2024-08-31T12:15:58.300810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:58:17.482204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302528+0000","last_change":"2024-08-31T12:15:56.284907+0000","last_active":"2024-08-31T12:15:58.302528+0000","last_peered":"2024-08-31T12:15:58.302528+0000","last_clean":"2024-08-31T12:15:58.302528+0000","last_became_active":"2024-08-31T12:15:56.284558+0000","last_became_peered":"2024-08-31T12:15:56.284558+0000","last_unstale":"2024-08-31T12:15:58.302528+0000","last_undegraded":"2024-08-31T12:15:58.302528+0000","last_fullsized":"2024-08-31T12:15:58.302528+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:29:53.117724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303538+0000","last_change":"2024-08-31T12:15:57.308679+0000","last_active":"2024-08-31T12:15:58.303538+0000","last_peered":"2024-08-31T12:15:58.303538+0000","last_clean":"2024-08-31T12:15:58.303538+0000","last_became_active":"2024-08-31T12:15:57.308260+0000","last_became_peered":"2024-08-31T12:15:57.308260+0000","last_unstale":"2024-08-31T12:15:58.303538+0000","last_undegraded":"2024-08-31T12:15:58.303538+0000","last_fullsized":"2024-08-31T12:15:58.303538+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:31:19.860950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303564+0000","last_change":"2024-08-31T12:15:56.290763+0000","last_active":"2024-08-31T12:15:58.303564+0000","last_peered":"2024-08-31T12:15:58.303564+0000","last_clean":"2024-08-31T12:15:58.303564+0000","last_became_active":"2024-08-31T12:15:56.290400+0000","last_became_peered":"2024-08-31T12:15:56.290400+0000","last_unstale":"2024-08-31T12:15:58.303564+0000","last_undegraded":"2024-08-31T12:15:58.303564+0000","last_fullsized":"2024-08-31T12:15:58.303564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:53:08.929860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.938122+0000","last_change":"2024-08-31T12:15:57.311657+0000","last_active":"2024-08-31T12:15:58.938122+0000","last_peered":"2024-08-31T12:15:58.938122+0000","last_clean":"2024-08-31T12:15:58.938122+0000","last_became_active":"2024-08-31T12:15:57.310564+0000","last_became_peered":"2024-08-31T12:15:57.310564+0000","last_unstale":"2024-08-31T12:15:58.938122+0000","last_undegraded":"2024-08-31T12:15:58.938122+0000","last_fullsized":"2024-08-31T12:15:58.938122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:37:02.435678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300755+0000","last_change":"2024-08-31T12:15:56.286792+0000","last_active":"2024-08-31T12:15:58.300755+0000","last_peered":"2024-08-31T12:15:58.300755+0000","last_clean":"2024-08-31T12:15:58.300755+0000","last_became_active":"2024-08-31T12:15:56.286458+0000","last_became_peered":"2024-08-31T12:15:56.286458+0000","last_unstale":"2024-08-31T12:15:58.300755+0000","last_undegraded":"2024-08-31T12:15:58.300755+0000","last_fullsized":"2024-08-31T12:15:58.300755+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:49:17.488528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936603+0000","last_change":"2024-08-31T12:15:57.309485+0000","last_active":"2024-08-31T12:15:58.936603+0000","last_peered":"2024-08-31T12:15:58.936603+0000","last_clean":"2024-08-31T12:15:58.936603+0000","last_became_active":"2024-08-31T12:15:57.309102+0000","last_became_peered":"2024-08-31T12:15:57.309102+0000","last_unstale":"2024-08-31T12:15:58.936603+0000","last_undegraded":"2024-08-31T12:15:58.936603+0000","last_fullsized":"2024-08-31T12:15:58.936603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:17:34.692296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302279+0000","last_change":"2024-08-31T12:15:56.289460+0000","last_active":"2024-08-31T12:15:58.302279+0000","last_peered":"2024-08-31T12:15:58.302279+0000","last_clean":"2024-08-31T12:15:58.302279+0000","last_became_active":"2024-08-31T12:15:56.289096+0000","last_became_peered":"2024-08-31T12:15:56.289096+0000","last_unstale":"2024-08-31T12:15:58.302279+0000","last_undegraded":"2024-08-31T12:15:58.302279+0000","last_fullsized":"2024-08-31T12:15:58.302279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:06:34.313772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306865+0000","last_change":"2024-08-31T12:15:57.317968+0000","last_active":"2024-08-31T12:15:58.306865+0000","last_peered":"2024-08-31T12:15:58.306865+0000","last_clean":"2024-08-31T12:15:58.306865+0000","last_became_active":"2024-08-31T12:15:57.317632+0000","last_became_peered":"2024-08-31T12:15:57.317632+0000","last_unstale":"2024-08-31T12:15:58.306865+0000","last_undegraded":"2024-08-31T12:15:58.306865+0000","last_fullsized":"2024-08-31T12:15:58.306865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:22:02.569106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937785+0000","last_change":"2024-08-31T12:15:56.284014+0000","last_active":"2024-08-31T12:15:58.937785+0000","last_peered":"2024-08-31T12:15:58.937785+0000","last_clean":"2024-08-31T12:15:58.937785+0000","last_became_active":"2024-08-31T12:15:56.283675+0000","last_became_peered":"2024-08-31T12:15:56.283675+0000","last_unstale":"2024-08-31T12:15:58.937785+0000","last_undegraded":"2024-08-31T12:15:58.937785+0000","last_fullsized":"2024-08-31T12:15:58.937785+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:51:59.306912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937018+0000","last_change":"2024-08-31T12:15:57.312440+0000","last_active":"2024-08-31T12:15:58.937018+0000","last_peered":"2024-08-31T12:15:58.937018+0000","last_clean":"2024-08-31T12:15:58.937018+0000","last_became_active":"2024-08-31T12:15:57.312007+0000","last_became_peered":"2024-08-31T12:15:57.312007+0000","last_unstale":"2024-08-31T12:15:58.937018+0000","last_undegraded":"2024-08-31T12:15:58.937018+0000","last_fullsized":"2024-08-31T12:15:58.937018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:23:43.553625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303030+0000","last_change":"2024-08-31T12:15:56.290968+0000","last_active":"2024-08-31T12:15:58.303030+0000","last_peered":"2024-08-31T12:15:58.303030+0000","last_clean":"2024-08-31T12:15:58.303030+0000","last_became_active":"2024-08-31T12:15:56.290603+0000","last_became_peered":"2024-08-31T12:15:56.290603+0000","last_unstale":"2024-08-31T12:15:58.303030+0000","last_undegraded":"2024-08-31T12:15:58.303030+0000","last_fullsized":"2024-08-31T12:15:58.303030+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:50:23.215239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.307024+0000","last_change":"2024-08-31T12:15:57.312790+0000","last_active":"2024-08-31T12:15:58.307024+0000","last_peered":"2024-08-31T12:15:58.307024+0000","last_clean":"2024-08-31T12:15:58.307024+0000","last_became_active":"2024-08-31T12:15:57.312383+0000","last_became_peered":"2024-08-31T12:15:57.312383+0000","last_unstale":"2024-08-31T12:15:58.307024+0000","last_undegraded":"2024-08-31T12:15:58.307024+0000","last_fullsized":"2024-08-31T12:15:58.307024+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:20:41.042375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.301697+0000","last_change":"2024-08-31T12:15:56.289795+0000","last_active":"2024-08-31T12:15:58.301697+0000","last_peered":"2024-08-31T12:15:58.301697+0000","last_clean":"2024-08-31T12:15:58.301697+0000","last_became_active":"2024-08-31T12:15:56.289477+0000","last_became_peered":"2024-08-31T12:15:56.289477+0000","last_unstale":"2024-08-31T12:15:58.301697+0000","last_undegraded":"2024-08-31T12:15:58.301697+0000","last_fullsized":"2024-08-31T12:15:58.301697+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:15:51.561831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302374+0000","last_change":"2024-08-31T12:15:57.312865+0000","last_active":"2024-08-31T12:15:58.302374+0000","last_peered":"2024-08-31T12:15:58.302374+0000","last_clean":"2024-08-31T12:15:58.302374+0000","last_became_active":"2024-08-31T12:15:57.312462+0000","last_became_peered":"2024-08-31T12:15:57.312462+0000","last_unstale":"2024-08-31T12:15:58.302374+0000","last_undegraded":"2024-08-31T12:15:58.302374+0000","last_fullsized":"2024-08-31T12:15:58.302374+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:16:42.625396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-08-31T12:15:57.292892+0000","last_change":"2024-08-31T12:15:56.285412+0000","last_active":"2024-08-31T12:15:57.292892+0000","last_peered":"2024-08-31T12:15:57.292892+0000","last_clean":"2024-08-31T12:15:57.292892+0000","last_became_active":"2024-08-31T12:15:56.285108+0000","last_became_peered":"2024-08-31T12:15:56.285108+0000","last_unstale":"2024-08-31T12:15:57.292892+0000","last_undegraded":"2024-08-31T12:15:57.292892+0000","last_fullsized":"2024-08-31T12:15:57.292892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:29:45.580957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306407+0000","last_change":"2024-08-31T12:15:57.311701+0000","last_active":"2024-08-31T12:15:58.306407+0000","last_peered":"2024-08-31T12:15:58.306407+0000","last_clean":"2024-08-31T12:15:58.306407+0000","last_became_active":"2024-08-31T12:15:57.311368+0000","last_became_peered":"2024-08-31T12:15:57.311368+0000","last_unstale":"2024-08-31T12:15:58.306407+0000","last_undegraded":"2024-08-31T12:15:58.306407+0000","last_fullsized":"2024-08-31T12:15:58.306407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:30:13.715505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936411+0000","last_change":"2024-08-31T12:15:56.289979+0000","last_active":"2024-08-31T12:15:58.936411+0000","last_peered":"2024-08-31T12:15:58.936411+0000","last_clean":"2024-08-31T12:15:58.936411+0000","last_became_active":"2024-08-31T12:15:56.288289+0000","last_became_peered":"2024-08-31T12:15:56.288289+0000","last_unstale":"2024-08-31T12:15:58.936411+0000","last_undegraded":"2024-08-31T12:15:58.936411+0000","last_fullsized":"2024-08-31T12:15:58.936411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:46:55.057087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306825+0000","last_change":"2024-08-31T12:15:57.311580+0000","last_active":"2024-08-31T12:15:58.306825+0000","last_peered":"2024-08-31T12:15:58.306825+0000","last_clean":"2024-08-31T12:15:58.306825+0000","last_became_active":"2024-08-31T12:15:57.311323+0000","last_became_peered":"2024-08-31T12:15:57.311323+0000","last_unstale":"2024-08-31T12:15:58.306825+0000","last_undegraded":"2024-08-31T12:15:58.306825+0000","last_fullsized":"2024-08-31T12:15:58.306825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:54:03.871840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-08-31T12:15:57.293295+0000","last_change":"2024-08-31T12:15:56.290496+0000","last_active":"2024-08-31T12:15:57.293295+0000","last_peered":"2024-08-31T12:15:57.293295+0000","last_clean":"2024-08-31T12:15:57.293295+0000","last_became_active":"2024-08-31T12:15:56.290196+0000","last_became_peered":"2024-08-31T12:15:56.290196+0000","last_unstale":"2024-08-31T12:15:57.293295+0000","last_undegraded":"2024-08-31T12:15:57.293295+0000","last_fullsized":"2024-08-31T12:15:57.293295+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:52:20.055757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306986+0000","last_change":"2024-08-31T12:15:57.313317+0000","last_active":"2024-08-31T12:15:58.306986+0000","last_peered":"2024-08-31T12:15:58.306986+0000","last_clean":"2024-08-31T12:15:58.306986+0000","last_became_active":"2024-08-31T12:15:57.312145+0000","last_became_peered":"2024-08-31T12:15:57.312145+0000","last_unstale":"2024-08-31T12:15:58.306986+0000","last_undegraded":"2024-08-31T12:15:58.306986+0000","last_fullsized":"2024-08-31T12:15:58.306986+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:09:31.904512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300760+0000","last_change":"2024-08-31T12:15:56.281809+0000","last_active":"2024-08-31T12:15:58.300760+0000","last_peered":"2024-08-31T12:15:58.300760+0000","last_clean":"2024-08-31T12:15:58.300760+0000","last_became_active":"2024-08-31T12:15:56.281368+0000","last_became_peered":"2024-08-31T12:15:56.281368+0000","last_unstale":"2024-08-31T12:15:58.300760+0000","last_undegraded":"2024-08-31T12:15:58.300760+0000","last_fullsized":"2024-08-31T12:15:58.300760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:17:33.687196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303702+0000","last_change":"2024-08-31T12:15:57.305874+0000","last_active":"2024-08-31T12:15:58.303702+0000","last_peered":"2024-08-31T12:15:58.303702+0000","last_clean":"2024-08-31T12:15:58.303702+0000","last_became_active":"2024-08-31T12:15:57.305507+0000","last_became_peered":"2024-08-31T12:15:57.305507+0000","last_unstale":"2024-08-31T12:15:58.303702+0000","last_undegraded":"2024-08-31T12:15:58.303702+0000","last_fullsized":"2024-08-31T12:15:58.303702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:22:02.569106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-08-31T12:15:57.293819+0000","last_change":"2024-08-31T12:15:56.291851+0000","last_active":"2024-08-31T12:15:57.293819+0000","last_peered":"2024-08-31T12:15:57.293819+0000","last_clean":"2024-08-31T12:15:57.293819+0000","last_became_active":"2024-08-31T12:15:56.291446+0000","last_became_peered":"2024-08-31T12:15:56.291446+0000","last_unstale":"2024-08-31T12:15:57.293819+0000","last_undegraded":"2024-08-31T12:15:57.293819+0000","last_fullsized":"2024-08-31T12:15:57.293819+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:54:29.988079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306538+0000","last_change":"2024-08-31T12:15:57.316154+0000","last_active":"2024-08-31T12:15:58.306538+0000","last_peered":"2024-08-31T12:15:58.306538+0000","last_clean":"2024-08-31T12:15:58.306538+0000","last_became_active":"2024-08-31T12:15:57.315830+0000","last_became_peered":"2024-08-31T12:15:57.315830+0000","last_unstale":"2024-08-31T12:15:58.306538+0000","last_undegraded":"2024-08-31T12:15:58.306538+0000","last_fullsized":"2024-08-31T12:15:58.306538+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:05:53.039520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":111,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306756+0000","last_change":"2024-08-31T12:15:47.084912+0000","last_active":"2024-08-31T12:15:58.306756+0000","last_peered":"2024-08-31T12:15:58.306756+0000","last_clean":"2024-08-31T12:15:58.306756+0000","last_became_active":"2024-08-31T12:15:47.083771+0000","last_became_peered":"2024-08-31T12:15:47.083771+0000","last_unstale":"2024-08-31T12:15:58.306756+0000","last_undegraded":"2024-08-31T12:15:58.306756+0000","last_fullsized":"2024-08-31T12:15:58.306756+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:45.213688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:45.213688+0000","last_clean_scrub_stamp":"2024-08-31T12:15:45.213688+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:25:24.827530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.301865+0000","last_change":"2024-08-31T12:15:56.285355+0000","last_active":"2024-08-31T12:15:58.301865+0000","last_peered":"2024-08-31T12:15:58.301865+0000","last_clean":"2024-08-31T12:15:58.301865+0000","last_became_active":"2024-08-31T12:15:56.284470+0000","last_became_peered":"2024-08-31T12:15:56.284470+0000","last_unstale":"2024-08-31T12:15:58.301865+0000","last_undegraded":"2024-08-31T12:15:58.301865+0000","last_fullsized":"2024-08-31T12:15:58.301865+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:52:20.055757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936697+0000","last_change":"2024-08-31T12:15:57.309039+0000","last_active":"2024-08-31T12:15:58.936697+0000","last_peered":"2024-08-31T12:15:58.936697+0000","last_clean":"2024-08-31T12:15:58.936697+0000","last_became_active":"2024-08-31T12:15:57.308721+0000","last_became_peered":"2024-08-31T12:15:57.308721+0000","last_unstale":"2024-08-31T12:15:58.936697+0000","last_undegraded":"2024-08-31T12:15:58.936697+0000","last_fullsized":"2024-08-31T12:15:58.936697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:43:12.703971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306092+0000","last_change":"2024-08-31T12:15:56.293221+0000","last_active":"2024-08-31T12:15:58.306092+0000","last_peered":"2024-08-31T12:15:58.306092+0000","last_clean":"2024-08-31T12:15:58.306092+0000","last_became_active":"2024-08-31T12:15:56.292910+0000","last_became_peered":"2024-08-31T12:15:56.292910+0000","last_unstale":"2024-08-31T12:15:58.306092+0000","last_undegraded":"2024-08-31T12:15:58.306092+0000","last_fullsized":"2024-08-31T12:15:58.306092+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:14:33.483002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937638+0000","last_change":"2024-08-31T12:15:57.307266+0000","last_active":"2024-08-31T12:15:58.937638+0000","last_peered":"2024-08-31T12:15:58.937638+0000","last_clean":"2024-08-31T12:15:58.937638+0000","last_became_active":"2024-08-31T12:15:57.306529+0000","last_became_peered":"2024-08-31T12:15:57.306529+0000","last_unstale":"2024-08-31T12:15:58.937638+0000","last_undegraded":"2024-08-31T12:15:58.937638+0000","last_fullsized":"2024-08-31T12:15:58.937638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:42:39.731742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-08-31T12:15:57.294005+0000","last_change":"2024-08-31T12:15:56.291822+0000","last_active":"2024-08-31T12:15:57.294005+0000","last_peered":"2024-08-31T12:15:57.294005+0000","last_clean":"2024-08-31T12:15:57.294005+0000","last_became_active":"2024-08-31T12:15:56.291064+0000","last_became_peered":"2024-08-31T12:15:56.291064+0000","last_unstale":"2024-08-31T12:15:57.294005+0000","last_undegraded":"2024-08-31T12:15:57.294005+0000","last_fullsized":"2024-08-31T12:15:57.294005+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:53:08.929860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300606+0000","last_change":"2024-08-31T12:15:57.299212+0000","last_active":"2024-08-31T12:15:58.300606+0000","last_peered":"2024-08-31T12:15:58.300606+0000","last_clean":"2024-08-31T12:15:58.300606+0000","last_became_active":"2024-08-31T12:15:57.298892+0000","last_became_peered":"2024-08-31T12:15:57.298892+0000","last_unstale":"2024-08-31T12:15:58.300606+0000","last_undegraded":"2024-08-31T12:15:58.300606+0000","last_fullsized":"2024-08-31T12:15:58.300606+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:53:19.523811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303164+0000","last_change":"2024-08-31T12:15:56.286720+0000","last_active":"2024-08-31T12:15:58.303164+0000","last_peered":"2024-08-31T12:15:58.303164+0000","last_clean":"2024-08-31T12:15:58.303164+0000","last_became_active":"2024-08-31T12:15:56.285648+0000","last_became_peered":"2024-08-31T12:15:56.285648+0000","last_unstale":"2024-08-31T12:15:58.303164+0000","last_undegraded":"2024-08-31T12:15:58.303164+0000","last_fullsized":"2024-08-31T12:15:58.303164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:39:31.946792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936342+0000","last_change":"2024-08-31T12:15:57.311785+0000","last_active":"2024-08-31T12:15:58.936342+0000","last_peered":"2024-08-31T12:15:58.936342+0000","last_clean":"2024-08-31T12:15:58.936342+0000","last_became_active":"2024-08-31T12:15:57.311469+0000","last_became_peered":"2024-08-31T12:15:57.311469+0000","last_unstale":"2024-08-31T12:15:58.936342+0000","last_undegraded":"2024-08-31T12:15:58.936342+0000","last_fullsized":"2024-08-31T12:15:58.936342+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:29:54.122825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300691+0000","last_change":"2024-08-31T12:15:56.284302+0000","last_active":"2024-08-31T12:15:58.300691+0000","last_peered":"2024-08-31T12:15:58.300691+0000","last_clean":"2024-08-31T12:15:58.300691+0000","last_became_active":"2024-08-31T12:15:56.283979+0000","last_became_peered":"2024-08-31T12:15:56.283979+0000","last_unstale":"2024-08-31T12:15:58.300691+0000","last_undegraded":"2024-08-31T12:15:58.300691+0000","last_fullsized":"2024-08-31T12:15:58.300691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:51:59.306912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303594+0000","last_change":"2024-08-31T12:15:57.305383+0000","last_active":"2024-08-31T12:15:58.303594+0000","last_peered":"2024-08-31T12:15:58.303594+0000","last_clean":"2024-08-31T12:15:58.303594+0000","last_became_active":"2024-08-31T12:15:57.305048+0000","last_became_peered":"2024-08-31T12:15:57.305048+0000","last_unstale":"2024-08-31T12:15:58.303594+0000","last_undegraded":"2024-08-31T12:15:58.303594+0000","last_fullsized":"2024-08-31T12:15:58.303594+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:34:56.946695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306963+0000","last_change":"2024-08-31T12:15:56.297648+0000","last_active":"2024-08-31T12:15:58.306963+0000","last_peered":"2024-08-31T12:15:58.306963+0000","last_clean":"2024-08-31T12:15:58.306963+0000","last_became_active":"2024-08-31T12:15:56.297306+0000","last_became_peered":"2024-08-31T12:15:56.297306+0000","last_unstale":"2024-08-31T12:15:58.306963+0000","last_undegraded":"2024-08-31T12:15:58.306963+0000","last_fullsized":"2024-08-31T12:15:58.306963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:29:54.693819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300681+0000","last_change":"2024-08-31T12:15:57.297230+0000","last_active":"2024-08-31T12:15:58.300681+0000","last_peered":"2024-08-31T12:15:58.300681+0000","last_clean":"2024-08-31T12:15:58.300681+0000","last_became_active":"2024-08-31T12:15:57.296892+0000","last_became_peered":"2024-08-31T12:15:57.296892+0000","last_unstale":"2024-08-31T12:15:58.300681+0000","last_undegraded":"2024-08-31T12:15:58.300681+0000","last_fullsized":"2024-08-31T12:15:58.300681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:42:36.033586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300841+0000","last_change":"2024-08-31T12:15:56.286504+0000","last_active":"2024-08-31T12:15:58.300841+0000","last_peered":"2024-08-31T12:15:58.300841+0000","last_clean":"2024-08-31T12:15:58.300841+0000","last_became_active":"2024-08-31T12:15:56.286173+0000","last_became_peered":"2024-08-31T12:15:56.286173+0000","last_unstale":"2024-08-31T12:15:58.300841+0000","last_undegraded":"2024-08-31T12:15:58.300841+0000","last_fullsized":"2024-08-31T12:15:58.300841+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:06:34.313772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.938135+0000","last_change":"2024-08-31T12:15:57.308013+0000","last_active":"2024-08-31T12:15:58.938135+0000","last_peered":"2024-08-31T12:15:58.938135+0000","last_clean":"2024-08-31T12:15:58.938135+0000","last_became_active":"2024-08-31T12:15:57.307701+0000","last_became_peered":"2024-08-31T12:15:57.307701+0000","last_unstale":"2024-08-31T12:15:58.938135+0000","last_undegraded":"2024-08-31T12:15:58.938135+0000","last_fullsized":"2024-08-31T12:15:58.938135+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:06:15.161589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302555+0000","last_change":"2024-08-31T12:15:56.294852+0000","last_active":"2024-08-31T12:15:58.302555+0000","last_peered":"2024-08-31T12:15:58.302555+0000","last_clean":"2024-08-31T12:15:58.302555+0000","last_became_active":"2024-08-31T12:15:56.294496+0000","last_became_peered":"2024-08-31T12:15:56.294496+0000","last_unstale":"2024-08-31T12:15:58.302555+0000","last_undegraded":"2024-08-31T12:15:58.302555+0000","last_fullsized":"2024-08-31T12:15:58.302555+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:49:17.488528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937973+0000","last_change":"2024-08-31T12:15:57.305202+0000","last_active":"2024-08-31T12:15:58.937973+0000","last_peered":"2024-08-31T12:15:58.937973+0000","last_clean":"2024-08-31T12:15:58.937973+0000","last_became_active":"2024-08-31T12:15:57.304602+0000","last_became_peered":"2024-08-31T12:15:57.304602+0000","last_unstale":"2024-08-31T12:15:58.937973+0000","last_undegraded":"2024-08-31T12:15:58.937973+0000","last_fullsized":"2024-08-31T12:15:58.937973+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:56:03.280339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306090+0000","last_change":"2024-08-31T12:15:56.297542+0000","last_active":"2024-08-31T12:15:58.306090+0000","last_peered":"2024-08-31T12:15:58.306090+0000","last_clean":"2024-08-31T12:15:58.306090+0000","last_became_active":"2024-08-31T12:15:56.297168+0000","last_became_peered":"2024-08-31T12:15:56.297168+0000","last_unstale":"2024-08-31T12:15:58.306090+0000","last_undegraded":"2024-08-31T12:15:58.306090+0000","last_fullsized":"2024-08-31T12:15:58.306090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:56:41.924772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936311+0000","last_change":"2024-08-31T12:15:57.298845+0000","last_active":"2024-08-31T12:15:58.936311+0000","last_peered":"2024-08-31T12:15:58.936311+0000","last_clean":"2024-08-31T12:15:58.936311+0000","last_became_active":"2024-08-31T12:15:57.298520+0000","last_became_peered":"2024-08-31T12:15:57.298520+0000","last_unstale":"2024-08-31T12:15:58.936311+0000","last_undegraded":"2024-08-31T12:15:58.936311+0000","last_fullsized":"2024-08-31T12:15:58.936311+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:06:35.318873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300879+0000","last_change":"2024-08-31T12:15:56.290475+0000","last_active":"2024-08-31T12:15:58.300879+0000","last_peered":"2024-08-31T12:15:58.300879+0000","last_clean":"2024-08-31T12:15:58.300879+0000","last_became_active":"2024-08-31T12:15:56.289960+0000","last_became_peered":"2024-08-31T12:15:56.289960+0000","last_unstale":"2024-08-31T12:15:58.300879+0000","last_undegraded":"2024-08-31T12:15:58.300879+0000","last_fullsized":"2024-08-31T12:15:58.300879+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:29:53.117724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.938261+0000","last_change":"2024-08-31T12:15:57.301460+0000","last_active":"2024-08-31T12:15:58.938261+0000","last_peered":"2024-08-31T12:15:58.938261+0000","last_clean":"2024-08-31T12:15:58.938261+0000","last_became_active":"2024-08-31T12:15:57.301157+0000","last_became_peered":"2024-08-31T12:15:57.301157+0000","last_unstale":"2024-08-31T12:15:58.938261+0000","last_undegraded":"2024-08-31T12:15:58.938261+0000","last_fullsized":"2024-08-31T12:15:58.938261+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:22:52.173578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.301007+0000","last_change":"2024-08-31T12:15:56.289720+0000","last_active":"2024-08-31T12:15:58.301007+0000","last_peered":"2024-08-31T12:15:58.301007+0000","last_clean":"2024-08-31T12:15:58.301007+0000","last_became_active":"2024-08-31T12:15:56.289332+0000","last_became_peered":"2024-08-31T12:15:56.289332+0000","last_unstale":"2024-08-31T12:15:58.301007+0000","last_undegraded":"2024-08-31T12:15:58.301007+0000","last_fullsized":"2024-08-31T12:15:58.301007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:15:51.561831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937371+0000","last_change":"2024-08-31T12:15:57.313802+0000","last_active":"2024-08-31T12:15:58.937371+0000","last_peered":"2024-08-31T12:15:58.937371+0000","last_clean":"2024-08-31T12:15:58.937371+0000","last_became_active":"2024-08-31T12:15:57.313428+0000","last_became_peered":"2024-08-31T12:15:57.313428+0000","last_unstale":"2024-08-31T12:15:58.937371+0000","last_undegraded":"2024-08-31T12:15:58.937371+0000","last_fullsized":"2024-08-31T12:15:58.937371+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:22:13.435300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306928+0000","last_change":"2024-08-31T12:15:56.296105+0000","last_active":"2024-08-31T12:15:58.306928+0000","last_peered":"2024-08-31T12:15:58.306928+0000","last_clean":"2024-08-31T12:15:58.306928+0000","last_became_active":"2024-08-31T12:15:56.295524+0000","last_became_peered":"2024-08-31T12:15:56.295524+0000","last_unstale":"2024-08-31T12:15:58.306928+0000","last_undegraded":"2024-08-31T12:15:58.306928+0000","last_fullsized":"2024-08-31T12:15:58.306928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:41:55.482044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.301683+0000","last_change":"2024-08-31T12:15:57.300155+0000","last_active":"2024-08-31T12:15:58.301683+0000","last_peered":"2024-08-31T12:15:58.301683+0000","last_clean":"2024-08-31T12:15:58.301683+0000","last_became_active":"2024-08-31T12:15:57.299436+0000","last_became_peered":"2024-08-31T12:15:57.299436+0000","last_unstale":"2024-08-31T12:15:58.301683+0000","last_undegraded":"2024-08-31T12:15:58.301683+0000","last_fullsized":"2024-08-31T12:15:58.301683+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:37:45.891394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303455+0000","last_change":"2024-08-31T12:15:57.300529+0000","last_active":"2024-08-31T12:15:58.303455+0000","last_peered":"2024-08-31T12:15:58.303455+0000","last_clean":"2024-08-31T12:15:58.303455+0000","last_became_active":"2024-08-31T12:15:57.300159+0000","last_became_peered":"2024-08-31T12:15:57.300159+0000","last_unstale":"2024-08-31T12:15:58.303455+0000","last_undegraded":"2024-08-31T12:15:58.303455+0000","last_fullsized":"2024-08-31T12:15:58.303455+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:35:53.831742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306371+0000","last_change":"2024-08-31T12:15:56.297791+0000","last_active":"2024-08-31T12:15:58.306371+0000","last_peered":"2024-08-31T12:15:58.306371+0000","last_clean":"2024-08-31T12:15:58.306371+0000","last_became_active":"2024-08-31T12:15:56.296976+0000","last_became_peered":"2024-08-31T12:15:56.296976+0000","last_unstale":"2024-08-31T12:15:58.306371+0000","last_undegraded":"2024-08-31T12:15:58.306371+0000","last_fullsized":"2024-08-31T12:15:58.306371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:07:54.963050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303405+0000","last_change":"2024-08-31T12:15:57.301167+0000","last_active":"2024-08-31T12:15:58.303405+0000","last_peered":"2024-08-31T12:15:58.303405+0000","last_clean":"2024-08-31T12:15:58.303405+0000","last_became_active":"2024-08-31T12:15:57.300646+0000","last_became_peered":"2024-08-31T12:15:57.300646+0000","last_unstale":"2024-08-31T12:15:58.303405+0000","last_undegraded":"2024-08-31T12:15:58.303405+0000","last_fullsized":"2024-08-31T12:15:58.303405+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:05:53.039520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306722+0000","last_change":"2024-08-31T12:15:56.295139+0000","last_active":"2024-08-31T12:15:58.306722+0000","last_peered":"2024-08-31T12:15:58.306722+0000","last_clean":"2024-08-31T12:15:58.306722+0000","last_became_active":"2024-08-31T12:15:56.294560+0000","last_became_peered":"2024-08-31T12:15:56.294560+0000","last_unstale":"2024-08-31T12:15:58.306722+0000","last_undegraded":"2024-08-31T12:15:58.306722+0000","last_fullsized":"2024-08-31T12:15:58.306722+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:52:16.999507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937898+0000","last_change":"2024-08-31T12:15:57.311074+0000","last_active":"2024-08-31T12:15:58.937898+0000","last_peered":"2024-08-31T12:15:58.937898+0000","last_clean":"2024-08-31T12:15:58.937898+0000","last_became_active":"2024-08-31T12:15:57.310262+0000","last_became_peered":"2024-08-31T12:15:57.310262+0000","last_unstale":"2024-08-31T12:15:58.937898+0000","last_undegraded":"2024-08-31T12:15:58.937898+0000","last_fullsized":"2024-08-31T12:15:58.937898+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:47:51.267409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937288+0000","last_change":"2024-08-31T12:15:56.285428+0000","last_active":"2024-08-31T12:15:58.937288+0000","last_peered":"2024-08-31T12:15:58.937288+0000","last_clean":"2024-08-31T12:15:58.937288+0000","last_became_active":"2024-08-31T12:15:56.284884+0000","last_became_peered":"2024-08-31T12:15:56.284884+0000","last_unstale":"2024-08-31T12:15:58.937288+0000","last_undegraded":"2024-08-31T12:15:58.937288+0000","last_fullsized":"2024-08-31T12:15:58.937288+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:49:25.181432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937174+0000","last_change":"2024-08-31T12:15:57.305772+0000","last_active":"2024-08-31T12:15:58.937174+0000","last_peered":"2024-08-31T12:15:58.937174+0000","last_clean":"2024-08-31T12:15:58.937174+0000","last_became_active":"2024-08-31T12:15:57.305409+0000","last_became_peered":"2024-08-31T12:15:57.305409+0000","last_unstale":"2024-08-31T12:15:58.937174+0000","last_undegraded":"2024-08-31T12:15:58.937174+0000","last_fullsized":"2024-08-31T12:15:58.937174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:53:19.523811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937989+0000","last_change":"2024-08-31T12:15:56.294197+0000","last_active":"2024-08-31T12:15:58.937989+0000","last_peered":"2024-08-31T12:15:58.937989+0000","last_clean":"2024-08-31T12:15:58.937989+0000","last_became_active":"2024-08-31T12:15:56.293864+0000","last_became_peered":"2024-08-31T12:15:56.293864+0000","last_unstale":"2024-08-31T12:15:58.937989+0000","last_undegraded":"2024-08-31T12:15:58.937989+0000","last_fullsized":"2024-08-31T12:15:58.937989+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:43:17.443433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509446,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35560,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336280,"statfs":{"total":96636764160,"available":96600350720,"internally_reserved":0,"allocated":208896,"data_stored":73101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509445,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":73101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509446,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":208896,"data_stored":73101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509446,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":802816,"data_stored":663469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36136,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335704,"statfs":{"total":96636764160,"available":96599760896,"internally_reserved":0,"allocated":802816,"data_stored":663469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542151,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35552,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336288,"statfs":{"total":96636764160,"available":96600358912,"internally_reserved":0,"allocated":208896,"data_stored":73101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542149,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35552,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336288,"statfs":{"total":96636764160,"available":96600358912,"internally_reserved":0,"allocated":208896,"data_stored":73101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":196608,"data_stored":66801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-31T12:16:03.714 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-08-31T12:16:03.715 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-08-31T12:16:03.715 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-08-31T12:16:03.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.865+0000 7f5c2de27640 1 -- 172.21.15.19:0/111507908 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c280c8a40 msgr2=0x7f5c280c9850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:03.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.865+0000 7f5c2de27640 1 --2- 172.21.15.19:0/111507908 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c280c8a40 0x7f5c280c9850 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5c18009830 tx=0x7f5c1802a640 comp rx=0 tx=0).stop 2024-08-31T12:16:03.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.865+0000 7f5c2de27640 1 -- 172.21.15.19:0/111507908 shutdown_connections 2024-08-31T12:16:03.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.865+0000 7f5c2de27640 1 --2- 172.21.15.19:0/111507908 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c280c8a40 0x7f5c280c9850 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:03.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.865+0000 7f5c2de27640 1 -- 172.21.15.19:0/111507908 >> 172.21.15.19:0/111507908 conn(0x7f5c280690d0 msgr2=0x7f5c280d0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:03.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.865+0000 7f5c2de27640 1 -- 172.21.15.19:0/111507908 shutdown_connections 2024-08-31T12:16:03.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.865+0000 7f5c2de27640 1 -- 172.21.15.19:0/111507908 wait complete. 2024-08-31T12:16:03.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.866+0000 7f5c2de27640 1 Processor -- start 2024-08-31T12:16:03.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.866+0000 7f5c2de27640 1 -- start start 2024-08-31T12:16:03.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.868+0000 7f5c2de27640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5c2806e0e0 0x7f5c2814c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:03.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.868+0000 7f5c2de27640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5c280c8a40 0x7f5c2814b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:03.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.868+0000 7f5c2de27640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c28155300 0x7f5c2814bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:03.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.868+0000 7f5c2de27640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f5c2815b250 con 0x7f5c2806e0e0 2024-08-31T12:16:03.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.868+0000 7f5c2de27640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5c2815af70 con 0x7f5c28155300 2024-08-31T12:16:03.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.868+0000 7f5c2de27640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f5c280c8610 con 0x7f5c280c8a40 2024-08-31T12:16:03.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.868+0000 7f5c277fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5c2806e0e0 0x7f5c2814c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:03.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.868+0000 7f5c277fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5c2806e0e0 0x7f5c2814c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:56510/0 (socket says 172.21.15.19:56510) 2024-08-31T12:16:03.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.868+0000 7f5c277fe640 1 -- 172.21.15.19:0/4238289870 learned_addr learned my addr 172.21.15.19:0/4238289870 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:03.871 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.868+0000 7f5c26ffd640 1 --2- 172.21.15.19:0/4238289870 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5c280c8a40 0x7f5c2814b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:03.871 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.868+0000 7f5c27fff640 1 --2- 172.21.15.19:0/4238289870 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c28155300 0x7f5c2814bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:03.871 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.868+0000 7f5c277fe640 1 -- 172.21.15.19:0/4238289870 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5c280c8a40 msgr2=0x7f5c2814b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:03.871 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.868+0000 7f5c277fe640 1 --2- 172.21.15.19:0/4238289870 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5c280c8a40 0x7f5c2814b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:03.871 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.868+0000 7f5c277fe640 1 -- 172.21.15.19:0/4238289870 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c28155300 msgr2=0x7f5c2814bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:03.871 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.868+0000 7f5c277fe640 1 --2- 172.21.15.19:0/4238289870 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c28155300 0x7f5c2814bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:03.871 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.868+0000 7f5c277fe640 1 -- 172.21.15.19:0/4238289870 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c281f4d40 con 0x7f5c2806e0e0 2024-08-31T12:16:03.871 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.868+0000 7f5c277fe640 1 --2- 172.21.15.19:0/4238289870 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5c2806e0e0 0x7f5c2814c8f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5c1802ac30 tx=0x7f5c18033df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:03.871 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.868+0000 7f5c0ffff640 1 -- 172.21.15.19:0/4238289870 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c1802ae60 con 0x7f5c2806e0e0 2024-08-31T12:16:03.871 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.869+0000 7f5c2de27640 1 -- 172.21.15.19:0/4238289870 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c280d0cc0 con 0x7f5c2806e0e0 2024-08-31T12:16:03.871 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.869+0000 7f5c0ffff640 1 -- 172.21.15.19:0/4238289870 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5c180333d0 con 0x7f5c2806e0e0 2024-08-31T12:16:03.871 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.869+0000 7f5c0ffff640 1 -- 172.21.15.19:0/4238289870 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c18005020 con 0x7f5c2806e0e0 2024-08-31T12:16:03.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.869+0000 7f5c2de27640 1 -- 172.21.15.19:0/4238289870 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c281f4fe0 con 0x7f5c2806e0e0 2024-08-31T12:16:03.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.869+0000 7f5c2de27640 1 -- 172.21.15.19:0/4238289870 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c280c9850 con 0x7f5c2806e0e0 2024-08-31T12:16:03.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.870+0000 7f5c0ffff640 1 -- 172.21.15.19:0/4238289870 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f5c180051c0 con 0x7f5c2806e0e0 2024-08-31T12:16:03.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.871+0000 7f5c0ffff640 1 --2- 172.21.15.19:0/4238289870 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5c14077a50 0x7f5c14079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:03.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.871+0000 7f5c0ffff640 1 -- 172.21.15.19:0/4238289870 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5c180ba6a0 con 0x7f5c2806e0e0 2024-08-31T12:16:03.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.871+0000 7f5c26ffd640 1 --2- 172.21.15.19:0/4238289870 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5c14077a50 0x7f5c14079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:03.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.871+0000 7f5c26ffd640 1 --2- 172.21.15.19:0/4238289870 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5c14077a50 0x7f5c14079f60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5c280d0c50 tx=0x7f5c1c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:03.876 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:03.875+0000 7f5c0ffff640 1 -- 172.21.15.19:0/4238289870 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f5c1803f030 con 0x7f5c2806e0e0 2024-08-31T12:16:04.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.058+0000 7f5c2de27640 1 -- 172.21.15.19:0/4238289870 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7f5c280ccf20 con 0x7f5c2806e0e0 2024-08-31T12:16:04.059 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.058+0000 7f5c0ffff640 1 -- 172.21.15.19:0/4238289870 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v32) ==== 86+0+160 (secure 0 0 0) 0x7f5c18083380 con 0x7f5c2806e0e0 2024-08-31T12:16:04.059 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:16:04.061 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.060+0000 7f5c2de27640 1 -- 172.21.15.19:0/4238289870 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5c14077a50 msgr2=0x7f5c14079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:04.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.060+0000 7f5c2de27640 1 --2- 172.21.15.19:0/4238289870 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5c14077a50 0x7f5c14079f60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5c280d0c50 tx=0x7f5c1c009040 comp rx=0 tx=0).stop 2024-08-31T12:16:04.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.060+0000 7f5c2de27640 1 -- 172.21.15.19:0/4238289870 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5c2806e0e0 msgr2=0x7f5c2814c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:04.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.060+0000 7f5c2de27640 1 --2- 172.21.15.19:0/4238289870 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5c2806e0e0 0x7f5c2814c8f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5c1802ac30 tx=0x7f5c18033df0 comp rx=0 tx=0).stop 2024-08-31T12:16:04.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.060+0000 7f5c2de27640 1 -- 172.21.15.19:0/4238289870 shutdown_connections 2024-08-31T12:16:04.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.060+0000 7f5c2de27640 1 --2- 172.21.15.19:0/4238289870 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5c14077a50 0x7f5c14079f60 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:04.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.060+0000 7f5c2de27640 1 --2- 172.21.15.19:0/4238289870 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c28155300 0x7f5c2814bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:04.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.060+0000 7f5c2de27640 1 --2- 172.21.15.19:0/4238289870 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5c280c8a40 0x7f5c2814b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:04.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.060+0000 7f5c2de27640 1 --2- 172.21.15.19:0/4238289870 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5c2806e0e0 0x7f5c2814c8f0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:04.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.060+0000 7f5c2de27640 1 -- 172.21.15.19:0/4238289870 >> 172.21.15.19:0/4238289870 conn(0x7f5c280690d0 msgr2=0x7f5c2815a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:04.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.060+0000 7f5c2de27640 1 -- 172.21.15.19:0/4238289870 shutdown_connections 2024-08-31T12:16:04.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.061+0000 7f5c2de27640 1 -- 172.21.15.19:0/4238289870 wait complete. 2024-08-31T12:16:04.072 INFO:teuthology.orchestra.run.smithi019.stdout:[{"version":32,"timestamp":"2024-08-31T12:15:44.924100+0000","name":"","changes":[{"name":"osd.3/osd_mclock_max_capacity_iops_ssd","new_value":"4961.770608"}]}] 2024-08-31T12:16:04.072 INFO:tasks.ceph_manager:config epoch is 32 2024-08-31T12:16:04.073 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-08-31T12:16:04.073 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-08-31T12:16:04.073 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-08-31T12:16:04.186 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.185+0000 7f64c27f8640 1 -- 172.21.15.19:0/240155383 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f64bc0c8a40 msgr2=0x7f64bc0c9850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:04.187 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.185+0000 7f64c27f8640 1 --2- 172.21.15.19:0/240155383 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f64bc0c8a40 0x7f64bc0c9850 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f64ac005fd0 tx=0x7f64ac02a670 comp rx=0 tx=0).stop 2024-08-31T12:16:04.187 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.186+0000 7f64c27f8640 1 -- 172.21.15.19:0/240155383 shutdown_connections 2024-08-31T12:16:04.187 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.186+0000 7f64c27f8640 1 --2- 172.21.15.19:0/240155383 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64bc155300 0x7f64bc06d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:04.187 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.186+0000 7f64c27f8640 1 --2- 172.21.15.19:0/240155383 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f64bc0c8a40 0x7f64bc0c9850 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:04.187 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.186+0000 7f64c27f8640 1 -- 172.21.15.19:0/240155383 >> 172.21.15.19:0/240155383 conn(0x7f64bc0690d0 msgr2=0x7f64bc0d0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:04.187 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.186+0000 7f64c27f8640 1 -- 172.21.15.19:0/240155383 shutdown_connections 2024-08-31T12:16:04.187 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.186+0000 7f64c27f8640 1 -- 172.21.15.19:0/240155383 wait complete. 2024-08-31T12:16:04.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.187+0000 7f64c27f8640 1 Processor -- start 2024-08-31T12:16:04.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.188+0000 7f64c27f8640 1 -- start start 2024-08-31T12:16:04.191 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.189+0000 7f64c27f8640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f64bc06e0e0 0x7f64bc1f1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:04.191 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.189+0000 7f64c27f8640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64bc0c8a40 0x7f64bc0bf440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:04.191 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.189+0000 7f64c27f8640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64bc155300 0x7f64bc1e6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:04.191 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.189+0000 7f64c27f8640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f64bc0c8330 con 0x7f64bc06e0e0 2024-08-31T12:16:04.191 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.189+0000 7f64c27f8640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f64bc15b250 con 0x7f64bc0c8a40 2024-08-31T12:16:04.191 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.189+0000 7f64c27f8640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f64bc0c8610 con 0x7f64bc155300 2024-08-31T12:16:04.191 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.189+0000 7f64bbfff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f64bc06e0e0 0x7f64bc1f1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:04.191 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.189+0000 7f64bbfff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f64bc06e0e0 0x7f64bc1f1dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:56528/0 (socket says 172.21.15.19:56528) 2024-08-31T12:16:04.191 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.189+0000 7f64bbfff640 1 -- 172.21.15.19:0/80264270 learned_addr learned my addr 172.21.15.19:0/80264270 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:04.191 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.189+0000 7f64bb7fe640 1 --2- 172.21.15.19:0/80264270 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64bc0c8a40 0x7f64bc0bf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:04.191 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.189+0000 7f64c0d95640 1 --2- 172.21.15.19:0/80264270 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64bc155300 0x7f64bc1e6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:04.192 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.189+0000 7f64bbfff640 1 -- 172.21.15.19:0/80264270 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64bc155300 msgr2=0x7f64bc1e6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:04.192 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.189+0000 7f64bbfff640 1 --2- 172.21.15.19:0/80264270 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64bc155300 0x7f64bc1e6160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:04.192 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.189+0000 7f64bbfff640 1 -- 172.21.15.19:0/80264270 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64bc0c8a40 msgr2=0x7f64bc0bf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:04.192 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.189+0000 7f64bbfff640 1 --2- 172.21.15.19:0/80264270 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64bc0c8a40 0x7f64bc0bf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:04.192 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.189+0000 7f64bbfff640 1 -- 172.21.15.19:0/80264270 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64bc0cead0 con 0x7f64bc06e0e0 2024-08-31T12:16:04.192 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.189+0000 7f64bbfff640 1 --2- 172.21.15.19:0/80264270 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f64bc06e0e0 0x7f64bc1f1dc0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f64ac006010 tx=0x7f64ac0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:04.192 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.189+0000 7f64b8ff9640 1 -- 172.21.15.19:0/80264270 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ac03e030 con 0x7f64bc06e0e0 2024-08-31T12:16:04.192 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.190+0000 7f64b8ff9640 1 -- 172.21.15.19:0/80264270 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f64ac003d90 con 0x7f64bc06e0e0 2024-08-31T12:16:04.192 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.190+0000 7f64c27f8640 1 -- 172.21.15.19:0/80264270 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64bc0cfe50 con 0x7f64bc06e0e0 2024-08-31T12:16:04.192 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.190+0000 7f64b8ff9640 1 -- 172.21.15.19:0/80264270 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ac035040 con 0x7f64bc06e0e0 2024-08-31T12:16:04.192 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.190+0000 7f64c27f8640 1 -- 172.21.15.19:0/80264270 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64bc0cf090 con 0x7f64bc06e0e0 2024-08-31T12:16:04.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.191+0000 7f64c27f8640 1 -- 172.21.15.19:0/80264270 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64bc0c9850 con 0x7f64bc06e0e0 2024-08-31T12:16:04.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.191+0000 7f64b8ff9640 1 -- 172.21.15.19:0/80264270 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f64ac041030 con 0x7f64bc06e0e0 2024-08-31T12:16:04.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.192+0000 7f64b8ff9640 1 --2- 172.21.15.19:0/80264270 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f64a8077a80 0x7f64a8079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:04.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.192+0000 7f64b8ff9640 1 -- 172.21.15.19:0/80264270 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f64ac0b9e10 con 0x7f64bc06e0e0 2024-08-31T12:16:04.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.192+0000 7f64bb7fe640 1 --2- 172.21.15.19:0/80264270 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f64a8077a80 0x7f64a8079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:04.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.193+0000 7f64bb7fe640 1 --2- 172.21.15.19:0/80264270 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f64a8077a80 0x7f64a8079f90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f64bc1ef9f0 tx=0x7f64b0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:04.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.197+0000 7f64b8ff9640 1 -- 172.21.15.19:0/80264270 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f64ac0824a0 con 0x7f64bc06e0e0 2024-08-31T12:16:04.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.434+0000 7f64c27f8640 1 -- 172.21.15.19:0/80264270 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f64bc15a820 con 0x7f64bc06e0e0 2024-08-31T12:16:04.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.437+0000 7f64b8ff9640 1 -- 172.21.15.19:0/80264270 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+192551 (secure 0 0 0) 0x7f64ac086c60 con 0x7f64bc06e0e0 2024-08-31T12:16:04.440 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:16:04.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.440+0000 7f64c27f8640 1 -- 172.21.15.19:0/80264270 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f64a8077a80 msgr2=0x7f64a8079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:04.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.440+0000 7f64c27f8640 1 --2- 172.21.15.19:0/80264270 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f64a8077a80 0x7f64a8079f90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f64bc1ef9f0 tx=0x7f64b0009040 comp rx=0 tx=0).stop 2024-08-31T12:16:04.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.440+0000 7f64c27f8640 1 -- 172.21.15.19:0/80264270 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f64bc06e0e0 msgr2=0x7f64bc1f1dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:04.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.440+0000 7f64c27f8640 1 --2- 172.21.15.19:0/80264270 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f64bc06e0e0 0x7f64bc1f1dc0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f64ac006010 tx=0x7f64ac0038a0 comp rx=0 tx=0).stop 2024-08-31T12:16:04.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.440+0000 7f64c27f8640 1 -- 172.21.15.19:0/80264270 shutdown_connections 2024-08-31T12:16:04.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.440+0000 7f64c27f8640 1 --2- 172.21.15.19:0/80264270 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f64a8077a80 0x7f64a8079f90 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:04.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.440+0000 7f64c27f8640 1 --2- 172.21.15.19:0/80264270 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64bc155300 0x7f64bc1e6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:04.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.440+0000 7f64c27f8640 1 --2- 172.21.15.19:0/80264270 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64bc0c8a40 0x7f64bc0bf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:04.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.440+0000 7f64c27f8640 1 --2- 172.21.15.19:0/80264270 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f64bc06e0e0 0x7f64bc1f1dc0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:04.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.440+0000 7f64c27f8640 1 -- 172.21.15.19:0/80264270 >> 172.21.15.19:0/80264270 conn(0x7f64bc0690d0 msgr2=0x7f64bc158c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:04.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.440+0000 7f64c27f8640 1 -- 172.21.15.19:0/80264270 shutdown_connections 2024-08-31T12:16:04.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.440+0000 7f64c27f8640 1 -- 172.21.15.19:0/80264270 wait complete. 2024-08-31T12:16:04.452 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":5,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":2824336655},{"type":"v1","addr":"172.21.15.43:6801","nonce":2824336655}]},"active_addr":"172.21.15.43:6801/2824336655","active_change":"2024-08-31T12:15:36.482496+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4114,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.43:0","nonce":1477498513}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.43:0","nonce":3123341599}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.43:0","nonce":1041184017}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.43:0","nonce":3271165271}]}]} 2024-08-31T12:16:04.457 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-08-31T12:16:04.457 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-08-31T12:16:04.457 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-31T12:16:04.572 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.571+0000 7f15e7d00640 1 -- 172.21.15.19:0/1956804401 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f15cc003a00 msgr2=0x7f15cc003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:04.572 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.571+0000 7f15e7d00640 1 --2- 172.21.15.19:0/1956804401 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f15cc003a00 0x7f15cc003e70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f15dc009b50 tx=0x7f15cc005830 comp rx=0 tx=0).stop 2024-08-31T12:16:04.572 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.571+0000 7f15e629d640 1 -- 172.21.15.19:0/1956804401 reap_dead start 2024-08-31T12:16:04.572 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.571+0000 7f15e7d00640 1 -- 172.21.15.19:0/1956804401 shutdown_connections 2024-08-31T12:16:04.572 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.571+0000 7f15e7d00640 1 -- 172.21.15.19:0/1956804401 >> 172.21.15.19:0/1956804401 conn(0x7f15e00ca670 msgr2=0x7f15e00cd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:04.572 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.571+0000 7f15e7d00640 1 -- 172.21.15.19:0/1956804401 shutdown_connections 2024-08-31T12:16:04.572 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.571+0000 7f15e7d00640 1 -- 172.21.15.19:0/1956804401 wait complete. 2024-08-31T12:16:04.573 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.573+0000 7f15e7d00640 1 Processor -- start 2024-08-31T12:16:04.573 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.573+0000 7f15e7d00640 1 -- start start 2024-08-31T12:16:04.576 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.574+0000 7f15e7d00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15cc003a00 0x7f15e014fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:04.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.574+0000 7f15e7d00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f15e0151b60 0x7f15e01511a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:04.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.574+0000 7f15e7d00640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f15e00c83c0 0x7f15e0148d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:04.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.574+0000 7f15e7d00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f15e00cd130 con 0x7f15e0151b60 2024-08-31T12:16:04.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.574+0000 7f15e7d00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f15e0073d70 con 0x7f15cc003a00 2024-08-31T12:16:04.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.574+0000 7f15e7d00640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f15e006eb40 con 0x7f15e00c83c0 2024-08-31T12:16:04.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.574+0000 7f15e529b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f15e0151b60 0x7f15e01511a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:04.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.574+0000 7f15e529b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f15e0151b60 0x7f15e01511a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:56552/0 (socket says 172.21.15.19:56552) 2024-08-31T12:16:04.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.574+0000 7f15e529b640 1 -- 172.21.15.19:0/3486328113 learned_addr learned my addr 172.21.15.19:0/3486328113 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:04.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.574+0000 7f15e5a9c640 1 --2- 172.21.15.19:0/3486328113 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15cc003a00 0x7f15e014fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:04.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.574+0000 7f15e629d640 1 --2- 172.21.15.19:0/3486328113 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f15e00c83c0 0x7f15e0148d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:04.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.574+0000 7f15e529b640 1 -- 172.21.15.19:0/3486328113 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f15e00c83c0 msgr2=0x7f15e0148d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:04.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.574+0000 7f15e529b640 1 --2- 172.21.15.19:0/3486328113 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f15e00c83c0 0x7f15e0148d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:04.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.574+0000 7f15e529b640 1 -- 172.21.15.19:0/3486328113 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15cc003a00 msgr2=0x7f15e014fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:04.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.574+0000 7f15e529b640 1 --2- 172.21.15.19:0/3486328113 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15cc003a00 0x7f15e014fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:04.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.574+0000 7f15e529b640 1 -- 172.21.15.19:0/3486328113 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15e0150370 con 0x7f15e0151b60 2024-08-31T12:16:04.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.574+0000 7f15e629d640 1 --2- 172.21.15.19:0/3486328113 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f15e00c83c0 0x7f15e0148d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:16:04.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.574+0000 7f15e529b640 1 --2- 172.21.15.19:0/3486328113 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f15e0151b60 0x7f15e01511a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f15d000e5f0 tx=0x7f15d000e9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:04.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.575+0000 7f15d67fc640 1 -- 172.21.15.19:0/3486328113 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d0015070 con 0x7f15e0151b60 2024-08-31T12:16:04.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.575+0000 7f15d67fc640 1 -- 172.21.15.19:0/3486328113 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f15d0010980 con 0x7f15e0151b60 2024-08-31T12:16:04.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.575+0000 7f15e7d00640 1 -- 172.21.15.19:0/3486328113 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15e0155710 con 0x7f15e0151b60 2024-08-31T12:16:04.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.575+0000 7f15d67fc640 1 -- 172.21.15.19:0/3486328113 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d0010dd0 con 0x7f15e0151b60 2024-08-31T12:16:04.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.575+0000 7f15e7d00640 1 -- 172.21.15.19:0/3486328113 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15e0155f70 con 0x7f15e0151b60 2024-08-31T12:16:04.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.576+0000 7f15e7d00640 1 -- 172.21.15.19:0/3486328113 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15e0155a00 con 0x7f15e0151b60 2024-08-31T12:16:04.579 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.577+0000 7f15d67fc640 1 -- 172.21.15.19:0/3486328113 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f15d0018070 con 0x7f15e0151b60 2024-08-31T12:16:04.579 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.577+0000 7f15d67fc640 1 --2- 172.21.15.19:0/3486328113 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f15c4077a50 0x7f15c4079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:04.579 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.577+0000 7f15d67fc640 1 -- 172.21.15.19:0/3486328113 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f15d0099a60 con 0x7f15e0151b60 2024-08-31T12:16:04.579 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.577+0000 7f15e5a9c640 1 --2- 172.21.15.19:0/3486328113 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f15c4077a50 0x7f15c4079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:04.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.578+0000 7f15e5a9c640 1 --2- 172.21.15.19:0/3486328113 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f15c4077a50 0x7f15c4079f60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f15dc009b50 tx=0x7f15dc02d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:04.582 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.581+0000 7f15d67fc640 1 -- 172.21.15.19:0/3486328113 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f15d00620f0 con 0x7f15e0151b60 2024-08-31T12:16:04.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.767+0000 7f15e7d00640 1 -- 172.21.15.19:0/3486328113 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f15e0158f70 con 0x7f15e0151b60 2024-08-31T12:16:04.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.768+0000 7f15d67fc640 1 -- 172.21.15.19:0/3486328113 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16617 (secure 0 0 0) 0x7f15d00668b0 con 0x7f15e0151b60 2024-08-31T12:16:04.769 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:16:04.769 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":21,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","created":"2024-08-31T12:15:32.084309+0000","modified":"2024-08-31T12:15:58.292413+0000","last_up_change":"2024-08-31T12:15:45.213688+0000","last_in_change":"2024-08-31T12:15:35.545202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-31T12:15:44.508197+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-31T12:15:54.512584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-31T12:15:55.618251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b80d6a3f-7e4e-4d4d-b1b4-0df71c9176ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6800","nonce":680447146},{"type":"v1","addr":"172.21.15.19:6801","nonce":680447146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":680447146},{"type":"v1","addr":"172.21.15.19:6803","nonce":680447146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":680447146},{"type":"v1","addr":"172.21.15.19:6807","nonce":680447146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":680447146},{"type":"v1","addr":"172.21.15.19:6805","nonce":680447146}]},"public_addr":"172.21.15.19:6801/680447146","cluster_addr":"172.21.15.19:6803/680447146","heartbeat_back_addr":"172.21.15.19:6807/680447146","heartbeat_front_addr":"172.21.15.19:6805/680447146","state":["exists","up"]},{"osd":1,"uuid":"6f986366-82a6-4133-9f2e-43f84c7b03bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":1522610034},{"type":"v1","addr":"172.21.15.19:6817","nonce":1522610034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":1522610034},{"type":"v1","addr":"172.21.15.19:6819","nonce":1522610034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":1522610034},{"type":"v1","addr":"172.21.15.19:6823","nonce":1522610034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":1522610034},{"type":"v1","addr":"172.21.15.19:6821","nonce":1522610034}]},"public_addr":"172.21.15.19:6817/1522610034","cluster_addr":"172.21.15.19:6819/1522610034","heartbeat_back_addr":"172.21.15.19:6823/1522610034","heartbeat_front_addr":"172.21.15.19:6821/1522610034","state":["exists","up"]},{"osd":2,"uuid":"ce8c2068-845c-4cbb-a4d1-077375d605ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6809","nonce":2332368824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6811","nonce":2332368824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6815","nonce":2332368824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6813","nonce":2332368824}]},"public_addr":"172.21.15.19:6809/2332368824","cluster_addr":"172.21.15.19:6811/2332368824","heartbeat_back_addr":"172.21.15.19:6815/2332368824","heartbeat_front_addr":"172.21.15.19:6813/2332368824","state":["exists","up"]},{"osd":3,"uuid":"b1cb0ef4-aa9b-421b-a94e-82de0c46e5cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":3785550598},{"type":"v1","addr":"172.21.15.19:6825","nonce":3785550598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":3785550598},{"type":"v1","addr":"172.21.15.19:6827","nonce":3785550598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":3785550598},{"type":"v1","addr":"172.21.15.19:6831","nonce":3785550598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":3785550598},{"type":"v1","addr":"172.21.15.19:6829","nonce":3785550598}]},"public_addr":"172.21.15.19:6825/3785550598","cluster_addr":"172.21.15.19:6827/3785550598","heartbeat_back_addr":"172.21.15.19:6831/3785550598","heartbeat_front_addr":"172.21.15.19:6829/3785550598","state":["exists","up"]},{"osd":4,"uuid":"825b58ec-a6da-4515-b6a5-81debe327011","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":1985519349},{"type":"v1","addr":"172.21.15.43:6819","nonce":1985519349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":1985519349},{"type":"v1","addr":"172.21.15.43:6821","nonce":1985519349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":1985519349},{"type":"v1","addr":"172.21.15.43:6825","nonce":1985519349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":1985519349},{"type":"v1","addr":"172.21.15.43:6823","nonce":1985519349}]},"public_addr":"172.21.15.43:6819/1985519349","cluster_addr":"172.21.15.43:6821/1985519349","heartbeat_back_addr":"172.21.15.43:6825/1985519349","heartbeat_front_addr":"172.21.15.43:6823/1985519349","state":["exists","up"]},{"osd":5,"uuid":"7f34e537-5997-40e4-92e7-2570507160c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":4271971521},{"type":"v1","addr":"172.21.15.43:6803","nonce":4271971521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":4271971521},{"type":"v1","addr":"172.21.15.43:6805","nonce":4271971521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":4271971521},{"type":"v1","addr":"172.21.15.43:6809","nonce":4271971521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":4271971521},{"type":"v1","addr":"172.21.15.43:6807","nonce":4271971521}]},"public_addr":"172.21.15.43:6803/4271971521","cluster_addr":"172.21.15.43:6805/4271971521","heartbeat_back_addr":"172.21.15.43:6809/4271971521","heartbeat_front_addr":"172.21.15.43:6807/4271971521","state":["exists","up"]},{"osd":6,"uuid":"195dd8ad-640a-4fa3-9ee4-06e4172e0464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2863384694},{"type":"v1","addr":"172.21.15.43:6827","nonce":2863384694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2863384694},{"type":"v1","addr":"172.21.15.43:6829","nonce":2863384694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":2863384694},{"type":"v1","addr":"172.21.15.43:6833","nonce":2863384694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2863384694},{"type":"v1","addr":"172.21.15.43:6831","nonce":2863384694}]},"public_addr":"172.21.15.43:6827/2863384694","cluster_addr":"172.21.15.43:6829/2863384694","heartbeat_back_addr":"172.21.15.43:6833/2863384694","heartbeat_front_addr":"172.21.15.43:6831/2863384694","state":["exists","up"]},{"osd":7,"uuid":"4645661f-f430-4f05-bdd1-69138dcd1bbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":792572234},{"type":"v1","addr":"172.21.15.43:6811","nonce":792572234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":792572234},{"type":"v1","addr":"172.21.15.43:6813","nonce":792572234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":792572234},{"type":"v1","addr":"172.21.15.43:6817","nonce":792572234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":792572234},{"type":"v1","addr":"172.21.15.43:6815","nonce":792572234}]},"public_addr":"172.21.15.43:6811/792572234","cluster_addr":"172.21.15.43:6813/792572234","heartbeat_back_addr":"172.21.15.43:6817/792572234","heartbeat_front_addr":"172.21.15.43:6815/792572234","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.156485+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.578126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.075440+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.639867+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.543602+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:41.919592+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.615557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:41.813337+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-31T12:16:04.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.770+0000 7f15e7d00640 1 -- 172.21.15.19:0/3486328113 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f15c4077a50 msgr2=0x7f15c4079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:04.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.770+0000 7f15e7d00640 1 --2- 172.21.15.19:0/3486328113 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f15c4077a50 0x7f15c4079f60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f15dc009b50 tx=0x7f15dc02d040 comp rx=0 tx=0).stop 2024-08-31T12:16:04.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.770+0000 7f15e7d00640 1 -- 172.21.15.19:0/3486328113 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f15e0151b60 msgr2=0x7f15e01511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:04.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.770+0000 7f15e7d00640 1 --2- 172.21.15.19:0/3486328113 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f15e0151b60 0x7f15e01511a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f15d000e5f0 tx=0x7f15d000e9f0 comp rx=0 tx=0).stop 2024-08-31T12:16:04.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.770+0000 7f15e7d00640 1 -- 172.21.15.19:0/3486328113 shutdown_connections 2024-08-31T12:16:04.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.770+0000 7f15e7d00640 1 --2- 172.21.15.19:0/3486328113 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f15c4077a50 0x7f15c4079f60 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:04.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.770+0000 7f15e7d00640 1 --2- 172.21.15.19:0/3486328113 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f15e00c83c0 0x7f15e0148d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:04.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.770+0000 7f15e7d00640 1 --2- 172.21.15.19:0/3486328113 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f15e0151b60 0x7f15e01511a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:04.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.770+0000 7f15e7d00640 1 --2- 172.21.15.19:0/3486328113 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15cc003a00 0x7f15e014fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:04.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.770+0000 7f15e7d00640 1 -- 172.21.15.19:0/3486328113 >> 172.21.15.19:0/3486328113 conn(0x7f15e00ca670 msgr2=0x7f15e00cd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:04.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.770+0000 7f15e7d00640 1 -- 172.21.15.19:0/3486328113 shutdown_connections 2024-08-31T12:16:04.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.770+0000 7f15e7d00640 1 -- 172.21.15.19:0/3486328113 wait complete. 2024-08-31T12:16:04.782 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-08-31T12:16:04.782 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-31T12:16:04.895 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.894+0000 7f3c372ec640 1 -- 172.21.15.19:0/3696972940 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3c3014e700 msgr2=0x7f3c3006dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:04.895 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.894+0000 7f3c372ec640 1 --2- 172.21.15.19:0/3696972940 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3c3014e700 0x7f3c3006dbf0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3c28005fd0 tx=0x7f3c2802a540 comp rx=0 tx=0).stop 2024-08-31T12:16:04.896 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.894+0000 7f3c372ec640 1 -- 172.21.15.19:0/3696972940 shutdown_connections 2024-08-31T12:16:04.896 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.894+0000 7f3c372ec640 1 --2- 172.21.15.19:0/3696972940 >> v2:172.21.15.43:3300/0 conn(0x7f3c30160610 0x7f3c30162a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:04.896 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.894+0000 7f3c372ec640 1 --2- 172.21.15.19:0/3696972940 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3c3014e700 0x7f3c3006dbf0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:04.896 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.894+0000 7f3c372ec640 1 -- 172.21.15.19:0/3696972940 >> 172.21.15.19:0/3696972940 conn(0x7f3c300cda10 msgr2=0x7f3c300c9e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:04.896 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.894+0000 7f3c372ec640 1 -- 172.21.15.19:0/3696972940 shutdown_connections 2024-08-31T12:16:04.896 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.894+0000 7f3c372ec640 1 -- 172.21.15.19:0/3696972940 wait complete. 2024-08-31T12:16:04.897 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.896+0000 7f3c372ec640 1 Processor -- start 2024-08-31T12:16:04.897 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.896+0000 7f3c372ec640 1 -- start start 2024-08-31T12:16:04.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.897+0000 7f3c372ec640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3c3006e130 0x7f3c30144080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:04.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.897+0000 7f3c372ec640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3c3014e700 0x7f3c301458b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:04.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.897+0000 7f3c372ec640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c30160610 0x7f3c30145df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:04.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.897+0000 7f3c372ec640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f3c300c49c0 con 0x7f3c3014e700 2024-08-31T12:16:04.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.897+0000 7f3c372ec640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3c30158970 con 0x7f3c30160610 2024-08-31T12:16:04.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.897+0000 7f3c372ec640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f3c300c4ca0 con 0x7f3c3006e130 2024-08-31T12:16:04.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.897+0000 7f3c34887640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3c3014e700 0x7f3c301458b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:04.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.897+0000 7f3c34887640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3c3014e700 0x7f3c301458b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:56556/0 (socket says 172.21.15.19:56556) 2024-08-31T12:16:04.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.897+0000 7f3c34887640 1 -- 172.21.15.19:0/4072798811 learned_addr learned my addr 172.21.15.19:0/4072798811 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:04.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.897+0000 7f3c35889640 1 --2- 172.21.15.19:0/4072798811 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c30160610 0x7f3c30145df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:04.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.897+0000 7f3c35088640 1 --2- 172.21.15.19:0/4072798811 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3c3006e130 0x7f3c30144080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:04.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.898+0000 7f3c34887640 1 -- 172.21.15.19:0/4072798811 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3c3006e130 msgr2=0x7f3c30144080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:04.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.898+0000 7f3c34887640 1 --2- 172.21.15.19:0/4072798811 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3c3006e130 0x7f3c30144080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:04.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.898+0000 7f3c34887640 1 -- 172.21.15.19:0/4072798811 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c30160610 msgr2=0x7f3c30145df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:04.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.898+0000 7f3c34887640 1 --2- 172.21.15.19:0/4072798811 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c30160610 0x7f3c30145df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:04.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.898+0000 7f3c34887640 1 -- 172.21.15.19:0/4072798811 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c30149e10 con 0x7f3c3014e700 2024-08-31T12:16:04.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.898+0000 7f3c35889640 1 --2- 172.21.15.19:0/4072798811 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c30160610 0x7f3c30145df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:16:04.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.898+0000 7f3c34887640 1 --2- 172.21.15.19:0/4072798811 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3c3014e700 0x7f3c301458b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3c2400c3a0 tx=0x7f3c2400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:04.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.898+0000 7f3c2dffb640 1 -- 172.21.15.19:0/4072798811 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c240052d0 con 0x7f3c3014e700 2024-08-31T12:16:04.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.898+0000 7f3c2dffb640 1 -- 172.21.15.19:0/4072798811 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3c24007020 con 0x7f3c3014e700 2024-08-31T12:16:04.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.898+0000 7f3c2dffb640 1 -- 172.21.15.19:0/4072798811 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c24007470 con 0x7f3c3014e700 2024-08-31T12:16:04.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.898+0000 7f3c372ec640 1 -- 172.21.15.19:0/4072798811 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c30154810 con 0x7f3c3014e700 2024-08-31T12:16:04.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.898+0000 7f3c372ec640 1 -- 172.21.15.19:0/4072798811 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c301495a0 con 0x7f3c3014e700 2024-08-31T12:16:04.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.899+0000 7f3c372ec640 1 -- 172.21.15.19:0/4072798811 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c300ca7e0 con 0x7f3c3014e700 2024-08-31T12:16:04.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.900+0000 7f3c2dffb640 1 -- 172.21.15.19:0/4072798811 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f3c240067d0 con 0x7f3c3014e700 2024-08-31T12:16:04.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.901+0000 7f3c2dffb640 1 --2- 172.21.15.19:0/4072798811 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3c1c077a80 0x7f3c1c079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:04.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.901+0000 7f3c2dffb640 1 -- 172.21.15.19:0/4072798811 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3c24098b30 con 0x7f3c3014e700 2024-08-31T12:16:04.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.901+0000 7f3c35088640 1 --2- 172.21.15.19:0/4072798811 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3c1c077a80 0x7f3c1c079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:04.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.902+0000 7f3c35088640 1 --2- 172.21.15.19:0/4072798811 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3c1c077a80 0x7f3c1c079f90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3c28033000 tx=0x7f3c28004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:04.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:04.905+0000 7f3c2dffb640 1 -- 172.21.15.19:0/4072798811 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f3c24061120 con 0x7f3c3014e700 2024-08-31T12:16:05.090 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.089+0000 7f3c372ec640 1 -- 172.21.15.19:0/4072798811 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3c30157d10 con 0x7f3c3014e700 2024-08-31T12:16:05.091 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.090+0000 7f3c2dffb640 1 -- 172.21.15.19:0/4072798811 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16617 (secure 0 0 0) 0x7f3c240658e0 con 0x7f3c3014e700 2024-08-31T12:16:05.092 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:16:05.092 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":21,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","created":"2024-08-31T12:15:32.084309+0000","modified":"2024-08-31T12:15:58.292413+0000","last_up_change":"2024-08-31T12:15:45.213688+0000","last_in_change":"2024-08-31T12:15:35.545202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-31T12:15:44.508197+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-31T12:15:54.512584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-31T12:15:55.618251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b80d6a3f-7e4e-4d4d-b1b4-0df71c9176ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6800","nonce":680447146},{"type":"v1","addr":"172.21.15.19:6801","nonce":680447146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":680447146},{"type":"v1","addr":"172.21.15.19:6803","nonce":680447146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":680447146},{"type":"v1","addr":"172.21.15.19:6807","nonce":680447146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":680447146},{"type":"v1","addr":"172.21.15.19:6805","nonce":680447146}]},"public_addr":"172.21.15.19:6801/680447146","cluster_addr":"172.21.15.19:6803/680447146","heartbeat_back_addr":"172.21.15.19:6807/680447146","heartbeat_front_addr":"172.21.15.19:6805/680447146","state":["exists","up"]},{"osd":1,"uuid":"6f986366-82a6-4133-9f2e-43f84c7b03bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":1522610034},{"type":"v1","addr":"172.21.15.19:6817","nonce":1522610034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":1522610034},{"type":"v1","addr":"172.21.15.19:6819","nonce":1522610034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":1522610034},{"type":"v1","addr":"172.21.15.19:6823","nonce":1522610034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":1522610034},{"type":"v1","addr":"172.21.15.19:6821","nonce":1522610034}]},"public_addr":"172.21.15.19:6817/1522610034","cluster_addr":"172.21.15.19:6819/1522610034","heartbeat_back_addr":"172.21.15.19:6823/1522610034","heartbeat_front_addr":"172.21.15.19:6821/1522610034","state":["exists","up"]},{"osd":2,"uuid":"ce8c2068-845c-4cbb-a4d1-077375d605ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6809","nonce":2332368824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6811","nonce":2332368824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6815","nonce":2332368824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6813","nonce":2332368824}]},"public_addr":"172.21.15.19:6809/2332368824","cluster_addr":"172.21.15.19:6811/2332368824","heartbeat_back_addr":"172.21.15.19:6815/2332368824","heartbeat_front_addr":"172.21.15.19:6813/2332368824","state":["exists","up"]},{"osd":3,"uuid":"b1cb0ef4-aa9b-421b-a94e-82de0c46e5cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":3785550598},{"type":"v1","addr":"172.21.15.19:6825","nonce":3785550598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":3785550598},{"type":"v1","addr":"172.21.15.19:6827","nonce":3785550598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":3785550598},{"type":"v1","addr":"172.21.15.19:6831","nonce":3785550598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":3785550598},{"type":"v1","addr":"172.21.15.19:6829","nonce":3785550598}]},"public_addr":"172.21.15.19:6825/3785550598","cluster_addr":"172.21.15.19:6827/3785550598","heartbeat_back_addr":"172.21.15.19:6831/3785550598","heartbeat_front_addr":"172.21.15.19:6829/3785550598","state":["exists","up"]},{"osd":4,"uuid":"825b58ec-a6da-4515-b6a5-81debe327011","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":1985519349},{"type":"v1","addr":"172.21.15.43:6819","nonce":1985519349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":1985519349},{"type":"v1","addr":"172.21.15.43:6821","nonce":1985519349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":1985519349},{"type":"v1","addr":"172.21.15.43:6825","nonce":1985519349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":1985519349},{"type":"v1","addr":"172.21.15.43:6823","nonce":1985519349}]},"public_addr":"172.21.15.43:6819/1985519349","cluster_addr":"172.21.15.43:6821/1985519349","heartbeat_back_addr":"172.21.15.43:6825/1985519349","heartbeat_front_addr":"172.21.15.43:6823/1985519349","state":["exists","up"]},{"osd":5,"uuid":"7f34e537-5997-40e4-92e7-2570507160c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":4271971521},{"type":"v1","addr":"172.21.15.43:6803","nonce":4271971521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":4271971521},{"type":"v1","addr":"172.21.15.43:6805","nonce":4271971521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":4271971521},{"type":"v1","addr":"172.21.15.43:6809","nonce":4271971521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":4271971521},{"type":"v1","addr":"172.21.15.43:6807","nonce":4271971521}]},"public_addr":"172.21.15.43:6803/4271971521","cluster_addr":"172.21.15.43:6805/4271971521","heartbeat_back_addr":"172.21.15.43:6809/4271971521","heartbeat_front_addr":"172.21.15.43:6807/4271971521","state":["exists","up"]},{"osd":6,"uuid":"195dd8ad-640a-4fa3-9ee4-06e4172e0464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2863384694},{"type":"v1","addr":"172.21.15.43:6827","nonce":2863384694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2863384694},{"type":"v1","addr":"172.21.15.43:6829","nonce":2863384694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":2863384694},{"type":"v1","addr":"172.21.15.43:6833","nonce":2863384694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2863384694},{"type":"v1","addr":"172.21.15.43:6831","nonce":2863384694}]},"public_addr":"172.21.15.43:6827/2863384694","cluster_addr":"172.21.15.43:6829/2863384694","heartbeat_back_addr":"172.21.15.43:6833/2863384694","heartbeat_front_addr":"172.21.15.43:6831/2863384694","state":["exists","up"]},{"osd":7,"uuid":"4645661f-f430-4f05-bdd1-69138dcd1bbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":792572234},{"type":"v1","addr":"172.21.15.43:6811","nonce":792572234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":792572234},{"type":"v1","addr":"172.21.15.43:6813","nonce":792572234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":792572234},{"type":"v1","addr":"172.21.15.43:6817","nonce":792572234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":792572234},{"type":"v1","addr":"172.21.15.43:6815","nonce":792572234}]},"public_addr":"172.21.15.43:6811/792572234","cluster_addr":"172.21.15.43:6813/792572234","heartbeat_back_addr":"172.21.15.43:6817/792572234","heartbeat_front_addr":"172.21.15.43:6815/792572234","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.156485+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.578126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.075440+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.639867+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.543602+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:41.919592+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.615557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:41.813337+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-31T12:16:05.095 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.093+0000 7f3c372ec640 1 -- 172.21.15.19:0/4072798811 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3c1c077a80 msgr2=0x7f3c1c079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.095 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.093+0000 7f3c372ec640 1 --2- 172.21.15.19:0/4072798811 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3c1c077a80 0x7f3c1c079f90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3c28033000 tx=0x7f3c28004730 comp rx=0 tx=0).stop 2024-08-31T12:16:05.095 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.093+0000 7f3c372ec640 1 -- 172.21.15.19:0/4072798811 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3c3014e700 msgr2=0x7f3c301458b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.095 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.093+0000 7f3c372ec640 1 --2- 172.21.15.19:0/4072798811 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3c3014e700 0x7f3c301458b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3c2400c3a0 tx=0x7f3c2400c860 comp rx=0 tx=0).stop 2024-08-31T12:16:05.095 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.093+0000 7f3c372ec640 1 -- 172.21.15.19:0/4072798811 shutdown_connections 2024-08-31T12:16:05.095 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.093+0000 7f3c372ec640 1 --2- 172.21.15.19:0/4072798811 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3c1c077a80 0x7f3c1c079f90 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.095 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.093+0000 7f3c372ec640 1 --2- 172.21.15.19:0/4072798811 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c30160610 0x7f3c30145df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.095 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.093+0000 7f3c372ec640 1 --2- 172.21.15.19:0/4072798811 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3c3014e700 0x7f3c301458b0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.095 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.093+0000 7f3c372ec640 1 --2- 172.21.15.19:0/4072798811 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3c3006e130 0x7f3c30144080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.095 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.093+0000 7f3c372ec640 1 -- 172.21.15.19:0/4072798811 >> 172.21.15.19:0/4072798811 conn(0x7f3c300cda10 msgr2=0x7f3c300c94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:05.095 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.093+0000 7f3c372ec640 1 -- 172.21.15.19:0/4072798811 shutdown_connections 2024-08-31T12:16:05.095 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.093+0000 7f3c372ec640 1 -- 172.21.15.19:0/4072798811 wait complete. 2024-08-31T12:16:05.105 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-08-31T12:16:05.106 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-08-31T12:16:05.106 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-08-31T12:16:05.107 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-08-31T12:16:05.107 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-08-31T12:16:05.108 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-08-31T12:16:05.108 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-08-31T12:16:05.108 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-08-31T12:16:05.335 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.334+0000 7f1b0b364640 1 -- 172.21.15.19:0/2794995541 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1afc04c5b0 msgr2=0x7f1afc0dac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.335 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.334+0000 7f1b0b364640 1 --2- 172.21.15.19:0/2794995541 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1afc04c5b0 0x7f1afc0dac50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1b04075480 tx=0x7f1b040a6e60 comp rx=0 tx=0).stop 2024-08-31T12:16:05.336 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.334+0000 7f0e2eaf2640 1 -- 172.21.15.19:0/2053915820 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0e28157920 msgr2=0x7f0e28157d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.336 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.334+0000 7f0e2eaf2640 1 --2- 172.21.15.19:0/2053915820 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0e28157920 0x7f0e28157d30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0e20005fd0 tx=0x7f0e2002bac0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.336 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.334+0000 7f1b0b364640 1 -- 172.21.15.19:0/2794995541 shutdown_connections 2024-08-31T12:16:05.336 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.334+0000 7f1b0b364640 1 --2- 172.21.15.19:0/2794995541 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1afc04c5b0 0x7f1afc0dac50 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.336 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.334+0000 7f1b0b364640 1 -- 172.21.15.19:0/2794995541 >> 172.21.15.19:0/2794995541 conn(0x7f1afc050280 msgr2=0x7f1afc050690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:05.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.334+0000 7fe20e47e640 1 -- 172.21.15.19:0/1602330685 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe2080b2ff0 msgr2=0x7fe20806f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.334+0000 7fe20e47e640 1 --2- 172.21.15.19:0/1602330685 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe2080b2ff0 0x7fe20806f080 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe1f8005fd0 tx=0x7fe1f802a560 comp rx=0 tx=0).stop 2024-08-31T12:16:05.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.334+0000 7f0e2eaf2640 1 -- 172.21.15.19:0/2053915820 shutdown_connections 2024-08-31T12:16:05.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.334+0000 7f0e2eaf2640 1 --2- 172.21.15.19:0/2053915820 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0e28157920 0x7f0e28157d30 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.334+0000 7f1b0b364640 1 -- 172.21.15.19:0/2794995541 shutdown_connections 2024-08-31T12:16:05.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.335+0000 7f0e2eaf2640 1 --2- 172.21.15.19:0/2053915820 >> v2:172.21.15.19:3300/0 conn(0x7f0e280c3640 0x7f0e280c3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.335+0000 7f0e2eaf2640 1 -- 172.21.15.19:0/2053915820 >> 172.21.15.19:0/2053915820 conn(0x7f0e2806cae0 msgr2=0x7f0e2806cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:05.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.335+0000 7f1b0b364640 1 -- 172.21.15.19:0/2794995541 wait complete. 2024-08-31T12:16:05.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.335+0000 7fe20e47e640 1 -- 172.21.15.19:0/1602330685 shutdown_connections 2024-08-31T12:16:05.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.335+0000 7fe20e47e640 1 --2- 172.21.15.19:0/1602330685 >> v2:172.21.15.43:3300/0 conn(0x7fe20806f5c0 0x7fe20815e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.335+0000 7fe20e47e640 1 --2- 172.21.15.19:0/1602330685 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe2080b2ff0 0x7fe20806f080 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.335+0000 7fe20e47e640 1 -- 172.21.15.19:0/1602330685 >> 172.21.15.19:0/1602330685 conn(0x7fe20806cae0 msgr2=0x7fe20806cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:05.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.335+0000 7fe20e47e640 1 -- 172.21.15.19:0/1602330685 shutdown_connections 2024-08-31T12:16:05.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.335+0000 7fc2d56e8640 1 -- 172.21.15.19:0/2851426689 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc2c80e2330 msgr2=0x7fc2c80e2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.335+0000 7fc2d56e8640 1 --2- 172.21.15.19:0/2851426689 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc2c80e2330 0x7fc2c80e2740 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc2d006b8f0 tx=0x7fc2d00a6db0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.336+0000 7f0e2eaf2640 1 -- 172.21.15.19:0/2053915820 shutdown_connections 2024-08-31T12:16:05.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.336+0000 7fc2d56e8640 1 -- 172.21.15.19:0/2851426689 shutdown_connections 2024-08-31T12:16:05.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.336+0000 7fc2d56e8640 1 --2- 172.21.15.19:0/2851426689 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc2c80e2330 0x7fc2c80e2740 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.336+0000 7fc2d56e8640 1 --2- 172.21.15.19:0/2851426689 >> v2:172.21.15.19:3300/0 conn(0x7fc2c80483a0 0x7fc2c8007230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.336+0000 7fc2d56e8640 1 -- 172.21.15.19:0/2851426689 >> 172.21.15.19:0/2851426689 conn(0x7fc2c8050fa0 msgr2=0x7fc2c80513b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:05.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.336+0000 7f0e2eaf2640 1 -- 172.21.15.19:0/2053915820 wait complete. 2024-08-31T12:16:05.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.336+0000 7fc2d56e8640 1 -- 172.21.15.19:0/2851426689 shutdown_connections 2024-08-31T12:16:05.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.336+0000 7fc2d56e8640 1 -- 172.21.15.19:0/2851426689 wait complete. 2024-08-31T12:16:05.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.335+0000 7fe20e47e640 1 -- 172.21.15.19:0/1602330685 wait complete. 2024-08-31T12:16:05.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.338+0000 7fc2d56e8640 1 Processor -- start 2024-08-31T12:16:05.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.338+0000 7fc2d56e8640 1 -- start start 2024-08-31T12:16:05.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.338+0000 7f1b0b364640 1 Processor -- start 2024-08-31T12:16:05.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.338+0000 7f1b0b364640 1 -- start start 2024-08-31T12:16:05.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.338+0000 7fe20e47e640 1 Processor -- start 2024-08-31T12:16:05.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.338+0000 7fe20e47e640 1 -- start start 2024-08-31T12:16:05.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.339+0000 7f0e2eaf2640 1 Processor -- start 2024-08-31T12:16:05.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.339+0000 7f8847f1a640 1 -- 172.21.15.19:0/1013448331 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f88400738a0 msgr2=0x7f8840161930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.339+0000 7f0e2eaf2640 1 -- start start 2024-08-31T12:16:05.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.339+0000 7f8847f1a640 1 --2- 172.21.15.19:0/1013448331 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f88400738a0 0x7f8840161930 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8838005fd0 tx=0x7f883802a930 comp rx=0 tx=0).stop 2024-08-31T12:16:05.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.339+0000 7f8847f1a640 1 -- 172.21.15.19:0/1013448331 shutdown_connections 2024-08-31T12:16:05.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.339+0000 7f8847f1a640 1 --2- 172.21.15.19:0/1013448331 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f88400738a0 0x7f8840161930 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.339+0000 7f8847f1a640 1 --2- 172.21.15.19:0/1013448331 >> v2:172.21.15.19:3300/0 conn(0x7f88400c7110 0x7f884015b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.339+0000 7f8847f1a640 1 -- 172.21.15.19:0/1013448331 >> 172.21.15.19:0/1013448331 conn(0x7f884006cae0 msgr2=0x7f884006cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:05.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.340+0000 7fc2d56e8640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc2c80483a0 0x7fc2c8169af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.340+0000 7fd5fd2f0640 1 -- 172.21.15.19:0/703211412 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd5f00d5ac0 msgr2=0x7fd5f0008760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.340+0000 7fd5fd2f0640 1 --2- 172.21.15.19:0/703211412 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd5f00d5ac0 0x7fd5f0008760 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec005fd0 tx=0x7fd5ec02a720 comp rx=0 tx=0).stop 2024-08-31T12:16:05.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.340+0000 7fc2d56e8640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc2c80dfa30 0x7fc2c80d7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.340+0000 7fc2d56e8640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2c80e2330 0x7fc2c80d7860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.340+0000 7fd5fd2f0640 1 -- 172.21.15.19:0/703211412 shutdown_connections 2024-08-31T12:16:05.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.340+0000 7fd5fd2f0640 1 --2- 172.21.15.19:0/703211412 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd5f00d5ac0 0x7fd5f0008760 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.340+0000 7fc2d56e8640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c804a9f0 con 0x7fc2c80dfa30 2024-08-31T12:16:05.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.340+0000 7fc2d56e8640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c80df510 con 0x7fc2c80e2330 2024-08-31T12:16:05.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.340+0000 7fc2d56e8640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fc2c804ab70 con 0x7fc2c80483a0 2024-08-31T12:16:05.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.340+0000 7fd5fd2f0640 1 -- 172.21.15.19:0/703211412 >> 172.21.15.19:0/703211412 conn(0x7fd5f0050f40 msgr2=0x7fd5f0051350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:05.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.340+0000 7fd5fd2f0640 1 -- 172.21.15.19:0/703211412 shutdown_connections 2024-08-31T12:16:05.349 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.340+0000 7f1b0b364640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1afc004a70 0x7f1afc16ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.349 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.340+0000 7fc2cffff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc2c80483a0 0x7fc2c8169af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.349 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.340+0000 7f1b0b364640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1afc04c5b0 0x7f1afc0d9d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.349 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.340+0000 7fd5fd2f0640 1 -- 172.21.15.19:0/703211412 wait complete. 2024-08-31T12:16:05.350 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.340+0000 7fc2cf7fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc2c80dfa30 0x7fc2c80d7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.350 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.340+0000 7fc2cf7fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc2c80dfa30 0x7fc2c80d7320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:56598/0 (socket says 172.21.15.19:56598) 2024-08-31T12:16:05.350 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.341+0000 7fe20e47e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe20806f5c0 0x7fe2080c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.350 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.340+0000 7fc2cf7fe640 1 -- 172.21.15.19:0/1553236222 learned_addr learned my addr 172.21.15.19:0/1553236222 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:05.351 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.341+0000 7fe20e47e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe2080b2ff0 0x7fe2080c4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.351 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.341+0000 7f8847f1a640 1 -- 172.21.15.19:0/1013448331 shutdown_connections 2024-08-31T12:16:05.351 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.341+0000 7fc2cf7fe640 1 -- 172.21.15.19:0/1553236222 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc2c80483a0 msgr2=0x7fc2c8169af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.351 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.341+0000 7fc2cf7fe640 1 --2- 172.21.15.19:0/1553236222 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc2c80483a0 0x7fc2c8169af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.351 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.341+0000 7fc2d4ee7640 1 --2- 172.21.15.19:0/1553236222 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2c80e2330 0x7fc2c80d7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.352 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.341+0000 7fc2cf7fe640 1 -- 172.21.15.19:0/1553236222 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2c80e2330 msgr2=0x7fc2c80d7860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.352 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.341+0000 7fc2cf7fe640 1 --2- 172.21.15.19:0/1553236222 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2c80e2330 0x7fc2c80d7860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.352 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.341+0000 7fc2cf7fe640 1 -- 172.21.15.19:0/1553236222 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2c816a030 con 0x7fc2c80dfa30 2024-08-31T12:16:05.352 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.341+0000 7fc49159a640 1 -- 172.21.15.19:0/3253747016 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc488035020 con 0x7fc494154a30 2024-08-31T12:16:05.352 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.341+0000 7feae5964640 1 -- 172.21.15.19:0/2265362713 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fead804fba0 msgr2=0x7fead804e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.352 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.341+0000 7feae5964640 1 --2- 172.21.15.19:0/2265362713 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fead804fba0 0x7fead804e290 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7feae006c260 tx=0x7feae00a6ec0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.353 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.341+0000 7f8847f1a640 1 -- 172.21.15.19:0/1013448331 wait complete. 2024-08-31T12:16:05.353 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.341+0000 7f1b0b364640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1afc0d5070 0x7f1afc0da240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.353 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.341+0000 7f1b0b364640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f1afc0dbb80 con 0x7f1afc04c5b0 2024-08-31T12:16:05.353 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.341+0000 7f1b0b364640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1afc0db640 con 0x7f1afc004a70 2024-08-31T12:16:05.353 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.341+0000 7f1b0b364640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f1afc0461f0 con 0x7f1afc0d5070 2024-08-31T12:16:05.354 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.341+0000 7f1b0a362640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1afc004a70 0x7f1afc16ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.354 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.341+0000 7f1b0a362640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1afc004a70 0x7f1afc16ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:47032/0 (socket says 172.21.15.19:47032) 2024-08-31T12:16:05.354 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.341+0000 7f1b0a362640 1 -- 172.21.15.19:0/490765926 learned_addr learned my addr 172.21.15.19:0/490765926 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:05.354 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.341+0000 7fc498dad640 1 -- 172.21.15.19:0/3253747016 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc494154a30 msgr2=0x7fc49406d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.354 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.341+0000 7f1b0ab63640 1 --2- 172.21.15.19:0/490765926 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1afc0d5070 0x7f1afc0da240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.355 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.341+0000 7f1b09b61640 1 --2- 172.21.15.19:0/490765926 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1afc04c5b0 0x7f1afc0d9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.355 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.341+0000 7fe20e47e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe20815ee30 0x7fe20815b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.355 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.341+0000 7fe20e47e640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fe2080c3c80 con 0x7fe20806f5c0 2024-08-31T12:16:05.355 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.341+0000 7fe20e47e640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe208074970 con 0x7fe20815ee30 2024-08-31T12:16:05.355 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.341+0000 7fe20e47e640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fe208074c50 con 0x7fe2080b2ff0 2024-08-31T12:16:05.355 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.342+0000 7fe207fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe20806f5c0 0x7fe2080c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.356 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.342+0000 7fc498dad640 1 --2- 172.21.15.19:0/3253747016 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc494154a30 0x7fc49406d1f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc494072420 tx=0x7fc48802a530 comp rx=0 tx=0).stop 2024-08-31T12:16:05.356 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.342+0000 7f0e2eaf2640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e28070310 0x7f0e28145f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.342+0000 7fe207fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe20806f5c0 0x7fe2080c5eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:56626/0 (socket says 172.21.15.19:56626) 2024-08-31T12:16:05.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.342+0000 7fe207fff640 1 -- 172.21.15.19:0/1984143439 learned_addr learned my addr 172.21.15.19:0/1984143439 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:05.358 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.342+0000 7fc2cf7fe640 1 --2- 172.21.15.19:0/1553236222 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc2c80dfa30 0x7fc2c80d7320 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc00d700 tx=0x7fc2bc00dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:05.358 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.342+0000 7f0e2eaf2640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0e280c3640 0x7f0e2814cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.358 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.342+0000 7feae5964640 1 -- 172.21.15.19:0/2265362713 shutdown_connections 2024-08-31T12:16:05.358 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.342+0000 7feae5964640 1 --2- 172.21.15.19:0/2265362713 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fead80e7040 0x7fead80e7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.359 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.342+0000 7feae5964640 1 --2- 172.21.15.19:0/2265362713 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fead804fba0 0x7fead804e290 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.359 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.342+0000 7feae5964640 1 -- 172.21.15.19:0/2265362713 >> 172.21.15.19:0/2265362713 conn(0x7fead80543b0 msgr2=0x7fead80547c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:05.359 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.342+0000 7f1b0a362640 1 -- 172.21.15.19:0/490765926 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1afc0d5070 msgr2=0x7f1afc0da240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.359 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.342+0000 7f1b0a362640 1 --2- 172.21.15.19:0/490765926 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1afc0d5070 0x7f1afc0da240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.359 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.342+0000 7fe207fff640 1 -- 172.21.15.19:0/1984143439 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe2080b2ff0 msgr2=0x7fe2080c4da0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-08-31T12:16:05.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.342+0000 7fe207fff640 1 --2- 172.21.15.19:0/1984143439 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe2080b2ff0 0x7fe2080c4da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.342+0000 7fe207fff640 1 -- 172.21.15.19:0/1984143439 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe20815ee30 msgr2=0x7fe20815b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.342+0000 7fe207fff640 1 --2- 172.21.15.19:0/1984143439 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe20815ee30 0x7fe20815b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.342+0000 7f0e2c88e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e28070310 0x7f0e28145f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.342+0000 7f0e2c88e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e28070310 0x7f0e28145f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:47038/0 (socket says 172.21.15.19:47038) 2024-08-31T12:16:05.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.342+0000 7f0e2c88e640 1 -- 172.21.15.19:0/856725592 learned_addr learned my addr 172.21.15.19:0/856725592 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:05.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.342+0000 7f0e2eaf2640 1 --2- 172.21.15.19:0/856725592 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0e28157920 0x7f0e281e2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.342+0000 7f1b0a362640 1 -- 172.21.15.19:0/490765926 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1afc04c5b0 msgr2=0x7f1afc0d9d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.342+0000 7fe207fff640 1 -- 172.21.15.19:0/1984143439 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2080c63f0 con 0x7fe20806f5c0 2024-08-31T12:16:05.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.342+0000 7f1b0a362640 1 --2- 172.21.15.19:0/490765926 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1afc04c5b0 0x7f1afc0d9d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.342+0000 7f0e27fff640 1 --2- 172.21.15.19:0/856725592 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0e280c3640 0x7f0e2814cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.342+0000 7f0e2eaf2640 1 -- 172.21.15.19:0/856725592 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f0e2806f5d0 con 0x7f0e280c3640 2024-08-31T12:16:05.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.342+0000 7f0e2eaf2640 1 -- 172.21.15.19:0/856725592 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0e2806f2f0 con 0x7f0e28070310 2024-08-31T12:16:05.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.342+0000 7f0e2eaf2640 1 -- 172.21.15.19:0/856725592 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f0e2806f8b0 con 0x7f0e28157920 2024-08-31T12:16:05.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.342+0000 7f1b0a362640 1 -- 172.21.15.19:0/490765926 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1afc16ef50 con 0x7f1afc004a70 2024-08-31T12:16:05.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.342+0000 7fc2ccff9640 1 -- 172.21.15.19:0/1553236222 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc017070 con 0x7fc2c80dfa30 2024-08-31T12:16:05.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.342+0000 7f0e2d08f640 1 --2- 172.21.15.19:0/856725592 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0e28157920 0x7f0e281e2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.342+0000 7f1b09b61640 1 --2- 172.21.15.19:0/490765926 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1afc04c5b0 0x7f1afc0d9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-31T12:16:05.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.342+0000 7feae5964640 1 -- 172.21.15.19:0/2265362713 shutdown_connections 2024-08-31T12:16:05.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.343+0000 7feae5964640 1 -- 172.21.15.19:0/2265362713 wait complete. 2024-08-31T12:16:05.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.342+0000 7f1b0a362640 1 --2- 172.21.15.19:0/490765926 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1afc004a70 0x7f1afc16ea10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1b0000c3d0 tx=0x7f1b0000c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:05.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.342+0000 7f1afaffd640 1 -- 172.21.15.19:0/490765926 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b00017070 con 0x7f1afc004a70 2024-08-31T12:16:05.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.342+0000 7f1afaffd640 1 -- 172.21.15.19:0/490765926 <== mon.1 v2:172.21.15.43:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1b000053c0 con 0x7f1afc004a70 2024-08-31T12:16:05.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.343+0000 7f1b0b364640 1 -- 172.21.15.19:0/490765926 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1afc034930 con 0x7f1afc004a70 2024-08-31T12:16:05.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.343+0000 7f1b0b364640 1 -- 172.21.15.19:0/490765926 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1afc030500 con 0x7f1afc004a70 2024-08-31T12:16:05.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.343+0000 7fc498dad640 1 -- 172.21.15.19:0/3253747016 shutdown_connections 2024-08-31T12:16:05.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.343+0000 7f0e2c88e640 1 -- 172.21.15.19:0/856725592 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0e28157920 msgr2=0x7f0e281e2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.343+0000 7f0e2c88e640 1 --2- 172.21.15.19:0/856725592 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0e28157920 0x7f0e281e2ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.343+0000 7f0e2c88e640 1 -- 172.21.15.19:0/856725592 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0e280c3640 msgr2=0x7f0e2814cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.343+0000 7fc498dad640 1 --2- 172.21.15.19:0/3253747016 >> v2:172.21.15.43:3300/0 conn(0x7fc4940c5760 0x7fc4940c5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.343+0000 7f0e2c88e640 1 --2- 172.21.15.19:0/856725592 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0e280c3640 0x7f0e2814cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.343+0000 7f0e2c88e640 1 -- 172.21.15.19:0/856725592 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e2814d4e0 con 0x7f0e28070310 2024-08-31T12:16:05.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.343+0000 7fc498dad640 1 --2- 172.21.15.19:0/3253747016 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc494154a30 0x7fc49406d1f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.343+0000 7fc498dad640 1 -- 172.21.15.19:0/3253747016 >> 172.21.15.19:0/3253747016 conn(0x7fc494071100 msgr2=0x7fc49406cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:05.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.342+0000 7fc2ccff9640 1 -- 172.21.15.19:0/1553236222 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc2bc004310 con 0x7fc2c80dfa30 2024-08-31T12:16:05.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.342+0000 7fc2ccff9640 1 -- 172.21.15.19:0/1553236222 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc009a80 con 0x7fc2c80dfa30 2024-08-31T12:16:05.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.343+0000 7f0e2c88e640 1 --2- 172.21.15.19:0/856725592 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e28070310 0x7f0e28145f30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c00c500 tx=0x7f0e1c00c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:05.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.343+0000 7fd5fd2f0640 1 Processor -- start 2024-08-31T12:16:05.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.343+0000 7fc2d56e8640 1 -- 172.21.15.19:0/1553236222 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2c8169050 con 0x7fc2c80dfa30 2024-08-31T12:16:05.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.344+0000 7fc2d56e8640 1 -- 172.21.15.19:0/1553236222 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2c8176290 con 0x7fc2c80dfa30 2024-08-31T12:16:05.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.344+0000 7fd5fd2f0640 1 -- start start 2024-08-31T12:16:05.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.344+0000 7f0e257fa640 1 -- 172.21.15.19:0/856725592 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e1c017070 con 0x7f0e28070310 2024-08-31T12:16:05.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.344+0000 7f0e257fa640 1 -- 172.21.15.19:0/856725592 <== mon.1 v2:172.21.15.43:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0e1c0035f0 con 0x7f0e28070310 2024-08-31T12:16:05.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.344+0000 7f1afaffd640 1 -- 172.21.15.19:0/490765926 <== mon.1 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b00017070 con 0x7f1afc004a70 2024-08-31T12:16:05.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.342+0000 7fe207fff640 1 --2- 172.21.15.19:0/1984143439 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe20806f5c0 0x7fe2080c5eb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe1f8006010 tx=0x7fe1f8003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:05.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.344+0000 7fc498dad640 1 -- 172.21.15.19:0/3253747016 shutdown_connections 2024-08-31T12:16:05.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.344+0000 7f0e257fa640 1 -- 172.21.15.19:0/856725592 <== mon.1 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e1c003a60 con 0x7f0e28070310 2024-08-31T12:16:05.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.344+0000 7f0e2eaf2640 1 -- 172.21.15.19:0/856725592 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e28075140 con 0x7f0e28070310 2024-08-31T12:16:05.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.345+0000 7f0e2eaf2640 1 -- 172.21.15.19:0/856725592 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e281570f0 con 0x7f0e28070310 2024-08-31T12:16:05.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.346+0000 7f0e2eaf2640 1 -- 172.21.15.19:0/856725592 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0e280cb330 con 0x7f0e28070310 2024-08-31T12:16:05.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.344+0000 7f8847f1a640 1 Processor -- start 2024-08-31T12:16:05.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.344+0000 7f8847f1a640 1 -- start start 2024-08-31T12:16:05.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.344+0000 7f1b0b364640 1 -- 172.21.15.19:0/490765926 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1afc045c00 con 0x7f1afc004a70 2024-08-31T12:16:05.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.345+0000 7fe204ff9640 1 -- 172.21.15.19:0/1984143439 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f803e030 con 0x7fe20806f5c0 2024-08-31T12:16:05.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.345+0000 7fe204ff9640 1 -- 172.21.15.19:0/1984143439 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe1f80047a0 con 0x7fe20806f5c0 2024-08-31T12:16:05.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.345+0000 7fe20e47e640 1 -- 172.21.15.19:0/1984143439 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2080708a0 con 0x7fe20806f5c0 2024-08-31T12:16:05.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.345+0000 7fe204ff9640 1 -- 172.21.15.19:0/1984143439 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f803e030 con 0x7fe20806f5c0 2024-08-31T12:16:05.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.345+0000 7fe20e47e640 1 -- 172.21.15.19:0/1984143439 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2080c4260 con 0x7fe20806f5c0 2024-08-31T12:16:05.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.347+0000 7fe1ee7fc640 1 -- 172.21.15.19:0/1984143439 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe20815e4a0 con 0x7fe20806f5c0 2024-08-31T12:16:05.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.346+0000 7fd5fd2f0640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd5f004d230 0x7fd5f0177600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.346+0000 7fd5fd2f0640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd5f00d5ac0 0x7fd5f0178e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.346+0000 7fd5fd2f0640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd5f00dee60 0x7fd5f0179360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.346+0000 7fd5f7fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd5f004d230 0x7fd5f0177600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.346+0000 7fd5fd2f0640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f0008ca0 con 0x7fd5f004d230 2024-08-31T12:16:05.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.346+0000 7fd5f77fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd5f00d5ac0 0x7fd5f0178e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.346+0000 7fd5f77fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd5f00d5ac0 0x7fd5f0178e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.19:48798/0 (socket says 172.21.15.19:48798) 2024-08-31T12:16:05.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.346+0000 7fd5f77fe640 1 -- 172.21.15.19:0/2199610839 learned_addr learned my addr 172.21.15.19:0/2199610839 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:05.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.347+0000 7fd5fd2f0640 1 -- 172.21.15.19:0/2199610839 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f00d5ed0 con 0x7fd5f00dee60 2024-08-31T12:16:05.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.347+0000 7fd5fd2f0640 1 -- 172.21.15.19:0/2199610839 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fd5f0034db0 con 0x7fd5f00d5ac0 2024-08-31T12:16:05.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.347+0000 7fd5fcaef640 1 --2- 172.21.15.19:0/2199610839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd5f00dee60 0x7fd5f0179360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.373 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.345+0000 7fc2d56e8640 1 -- 172.21.15.19:0/1553236222 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc2c81696d0 con 0x7fc2c80dfa30 2024-08-31T12:16:05.373 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.344+0000 7fc498dad640 1 -- 172.21.15.19:0/3253747016 wait complete. 2024-08-31T12:16:05.374 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.347+0000 7fd5f7fff640 1 -- 172.21.15.19:0/2199610839 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd5f00d5ac0 msgr2=0x7fd5f0178e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.374 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.347+0000 7fd5f7fff640 1 --2- 172.21.15.19:0/2199610839 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd5f00d5ac0 0x7fd5f0178e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.374 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.347+0000 7fd5f7fff640 1 -- 172.21.15.19:0/2199610839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd5f00dee60 msgr2=0x7fd5f0179360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.374 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.347+0000 7fd5f7fff640 1 --2- 172.21.15.19:0/2199610839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd5f00dee60 0x7fd5f0179360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.374 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.347+0000 7fd5f7fff640 1 -- 172.21.15.19:0/2199610839 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5f0177b40 con 0x7fd5f004d230 2024-08-31T12:16:05.374 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.347+0000 7fd5f77fe640 1 --2- 172.21.15.19:0/2199610839 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd5f00d5ac0 0x7fd5f0178e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:16:05.374 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.347+0000 7f1afaffd640 1 -- 172.21.15.19:0/490765926 <== mon.1 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f1b0001a070 con 0x7f1afc004a70 2024-08-31T12:16:05.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.348+0000 7fd5f7fff640 1 --2- 172.21.15.19:0/2199610839 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd5f004d230 0x7fd5f0177600 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd5e400c3c0 tx=0x7fd5e400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:05.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.348+0000 7fc498dad640 1 Processor -- start 2024-08-31T12:16:05.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.348+0000 7fc498dad640 1 -- start start 2024-08-31T12:16:05.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.348+0000 7fd5f4ff9640 1 -- 172.21.15.19:0/2199610839 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e4017070 con 0x7fd5f004d230 2024-08-31T12:16:05.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.348+0000 7fe204ff9640 1 -- 172.21.15.19:0/1984143439 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fe1f8041030 con 0x7fe20806f5c0 2024-08-31T12:16:05.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.348+0000 7f0e257fa640 1 -- 172.21.15.19:0/856725592 <== mon.1 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f0e1c01a070 con 0x7f0e28070310 2024-08-31T12:16:05.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.348+0000 7fd5f4ff9640 1 -- 172.21.15.19:0/2199610839 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd5e40052d0 con 0x7fd5f004d230 2024-08-31T12:16:05.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.346+0000 7feae5964640 1 Processor -- start 2024-08-31T12:16:05.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.346+0000 7feae5964640 1 -- start start 2024-08-31T12:16:05.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.348+0000 7feae5964640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fead804dd40 0x7fead804a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.348+0000 7feae5964640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fead804fba0 0x7fead800bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.348+0000 7feae5964640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fead80e7040 0x7fead8185a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.348+0000 7fd5fd2f0640 1 -- 172.21.15.19:0/2199610839 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5f0176b40 con 0x7fd5f004d230 2024-08-31T12:16:05.378 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.348+0000 7fd5fd2f0640 1 -- 172.21.15.19:0/2199610839 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5f004b180 con 0x7fd5f004d230 2024-08-31T12:16:05.378 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.348+0000 7fd5f4ff9640 1 -- 172.21.15.19:0/2199610839 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e4003850 con 0x7fd5f004d230 2024-08-31T12:16:05.378 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.348+0000 7feae5163640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fead80e7040 0x7fead8185a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.378 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.348+0000 7feae5163640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fead80e7040 0x7fead8185a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:56650/0 (socket says 172.21.15.19:56650) 2024-08-31T12:16:05.378 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.348+0000 7feae5163640 1 -- 172.21.15.19:0/751701327 learned_addr learned my addr 172.21.15.19:0/751701327 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:05.378 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.348+0000 7fc2ccff9640 1 -- 172.21.15.19:0/1553236222 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fc2bc009c20 con 0x7fc2c80dfa30 2024-08-31T12:16:05.379 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.348+0000 7feadffff640 1 --2- 172.21.15.19:0/751701327 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fead804fba0 0x7fead800bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.379 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.348+0000 7f1afaffd640 1 --2- 172.21.15.19:0/490765926 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1ae8077e50 0x7f1ae807a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.379 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.348+0000 7f1afaffd640 1 -- 172.21.15.19:0/490765926 <== mon.1 v2:172.21.15.43:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f1b000050d0 con 0x7f1afc004a70 2024-08-31T12:16:05.379 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.348+0000 7feae4962640 1 --2- 172.21.15.19:0/751701327 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fead804dd40 0x7fead804a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.380 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.349+0000 7feae5964640 1 -- 172.21.15.19:0/751701327 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fead80e6b20 con 0x7fead80e7040 2024-08-31T12:16:05.380 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.349+0000 7feae5964640 1 -- 172.21.15.19:0/751701327 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fead8050410 con 0x7fead804fba0 2024-08-31T12:16:05.380 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.349+0000 7f1afaffd640 1 --2- 172.21.15.19:0/490765926 >> [v2:172.21.15.19:6808/2332368824,v1:172.21.15.19:6809/2332368824] conn(0x7f1ae80800a0 0x7f1ae80824f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.380 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.349+0000 7f1afaffd640 1 -- 172.21.15.19:0/490765926 --> [v2:172.21.15.19:6808/2332368824,v1:172.21.15.19:6809/2332368824] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1b00009e50 con 0x7f1ae80800a0 2024-08-31T12:16:05.380 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.349+0000 7f1afaffd640 1 -- 172.21.15.19:0/490765926 <== mon.1 v2:172.21.15.43:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f1b000988d0 con 0x7f1afc004a70 2024-08-31T12:16:05.380 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.349+0000 7feae5964640 1 -- 172.21.15.19:0/751701327 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fead8039730 con 0x7fead804dd40 2024-08-31T12:16:05.381 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.349+0000 7f8847f1a640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f88400738a0 0x7f88401f3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.381 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.349+0000 7f8847f1a640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f88400c7110 0x7f8840148140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.381 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.349+0000 7f0e257fa640 1 --2- 172.21.15.19:0/856725592 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0e14077a50 0x7f0e14079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.381 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.349+0000 7f1b0ab63640 1 --2- 172.21.15.19:0/490765926 >> [v2:172.21.15.19:6808/2332368824,v1:172.21.15.19:6809/2332368824] conn(0x7f1ae80800a0 0x7f1ae80824f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.382 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.349+0000 7f8845cb6640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f88400738a0 0x7f88401f3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.382 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.349+0000 7f8845cb6640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f88400738a0 0x7f88401f3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:47084/0 (socket says 172.21.15.19:47084) 2024-08-31T12:16:05.382 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.349+0000 7f8847f1a640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8840151040 0x7f8840144ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.382 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.349+0000 7f88454b5640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f88400c7110 0x7f8840148140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.382 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.349+0000 7f88454b5640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f88400c7110 0x7f8840148140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.19:48818/0 (socket says 172.21.15.19:48818) 2024-08-31T12:16:05.382 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.349+0000 7f0e257fa640 1 -- 172.21.15.19:0/856725592 <== mon.1 v2:172.21.15.43:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0e1c098970 con 0x7f0e28070310 2024-08-31T12:16:05.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.349+0000 7f8845cb6640 1 -- 172.21.15.19:0/728413247 learned_addr learned my addr 172.21.15.19:0/728413247 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:05.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.349+0000 7f8847f1a640 1 -- 172.21.15.19:0/728413247 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f884006f150 con 0x7f8840151040 2024-08-31T12:16:05.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.349+0000 7feae5163640 1 -- 172.21.15.19:0/751701327 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fead804dd40 msgr2=0x7fead804a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.349+0000 7feae5163640 1 --2- 172.21.15.19:0/751701327 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fead804dd40 0x7fead804a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.349+0000 7feae5163640 1 -- 172.21.15.19:0/751701327 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fead804fba0 msgr2=0x7fead800bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.384 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.349+0000 7feae5163640 1 --2- 172.21.15.19:0/751701327 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fead804fba0 0x7fead800bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.384 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.349+0000 7feae5163640 1 -- 172.21.15.19:0/751701327 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fead816de90 con 0x7fead80e7040 2024-08-31T12:16:05.384 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.349+0000 7f0e257fa640 1 --2- 172.21.15.19:0/856725592 >> [v2:172.21.15.19:6800/680447146,v1:172.21.15.19:6801/680447146] conn(0x7f0e1407fca0 0x7f0e140820f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.384 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.349+0000 7f8847f1a640 1 -- 172.21.15.19:0/728413247 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8840151450 con 0x7f88400738a0 2024-08-31T12:16:05.384 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.349+0000 7f0e257fa640 1 -- 172.21.15.19:0/856725592 --> [v2:172.21.15.19:6800/680447146,v1:172.21.15.19:6801/680447146] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0e140827c0 con 0x7f0e1407fca0 2024-08-31T12:16:05.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.349+0000 7f8847f1a640 1 -- 172.21.15.19:0/728413247 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f884015a5a0 con 0x7f88400c7110 2024-08-31T12:16:05.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.349+0000 7f88464b7640 1 --2- 172.21.15.19:0/728413247 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8840151040 0x7f8840144ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.350+0000 7f1b0ab63640 1 --2- 172.21.15.19:0/490765926 >> [v2:172.21.15.19:6808/2332368824,v1:172.21.15.19:6809/2332368824] conn(0x7f1ae80800a0 0x7f1ae80824f0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:05.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.350+0000 7fe204ff9640 1 --2- 172.21.15.19:0/1984143439 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe1f4077c50 0x7fe1f407a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.350+0000 7feae5163640 1 --2- 172.21.15.19:0/751701327 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fead80e7040 0x7fead8185a90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fead000e970 tx=0x7fead000ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:05.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.350+0000 7feae4962640 1 --2- 172.21.15.19:0/751701327 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fead804dd40 0x7fead804a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-31T12:16:05.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.350+0000 7f0e257fa640 1 -- 172.21.15.19:0/856725592 <== mon.1 v2:172.21.15.43:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f0e1c05d830 con 0x7f0e28070310 2024-08-31T12:16:05.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.351+0000 7f0e27fff640 1 --2- 172.21.15.19:0/856725592 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0e14077a50 0x7f0e14079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.351+0000 7f0e2d08f640 1 --2- 172.21.15.19:0/856725592 >> [v2:172.21.15.19:6800/680447146,v1:172.21.15.19:6801/680447146] conn(0x7f0e1407fca0 0x7f0e140820f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.350+0000 7f88464b7640 1 -- 172.21.15.19:0/728413247 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f88400c7110 msgr2=0x7f8840148140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.350+0000 7f88464b7640 1 --2- 172.21.15.19:0/728413247 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f88400c7110 0x7f8840148140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.350+0000 7f88464b7640 1 -- 172.21.15.19:0/728413247 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f88400738a0 msgr2=0x7f88401f3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.350+0000 7f88464b7640 1 --2- 172.21.15.19:0/728413247 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f88400738a0 0x7f88401f3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.350+0000 7f88464b7640 1 -- 172.21.15.19:0/728413247 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8840149cf0 con 0x7f8840151040 2024-08-31T12:16:05.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.351+0000 7f88464b7640 1 --2- 172.21.15.19:0/728413247 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8840151040 0x7f8840144ce0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f884006ed70 tx=0x7f8838030650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:05.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.350+0000 7f1b09b61640 1 --2- 172.21.15.19:0/490765926 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1ae8077e50 0x7f1ae807a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.350+0000 7fe2077fe640 1 --2- 172.21.15.19:0/1984143439 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe1f4077c50 0x7fe1f407a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.350+0000 7fe204ff9640 1 -- 172.21.15.19:0/1984143439 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe1f807ece0 con 0x7fe20806f5c0 2024-08-31T12:16:05.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.351+0000 7fe2077fe640 1 --2- 172.21.15.19:0/1984143439 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe1f4077c50 0x7fe1f407a160 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc0095d0 tx=0x7fe1fc006b30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:05.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.351+0000 7fe204ff9640 1 --2- 172.21.15.19:0/1984143439 >> [v2:172.21.15.19:6824/3785550598,v1:172.21.15.19:6825/3785550598] conn(0x7fe1f40809d0 0x7fe1f4082ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.351+0000 7fe204ff9640 1 -- 172.21.15.19:0/1984143439 --> [v2:172.21.15.19:6824/3785550598,v1:172.21.15.19:6825/3785550598] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe1f4083590 con 0x7fe1f40809d0 2024-08-31T12:16:05.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.351+0000 7fe204ff9640 1 -- 172.21.15.19:0/1984143439 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fe1f807eee0 con 0x7fe20806f5c0 2024-08-31T12:16:05.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.351+0000 7fd5fd2f0640 1 -- 172.21.15.19:0/2199610839 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd5d8001030 con 0x7fd5f004d230 2024-08-31T12:16:05.389 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.351+0000 7fc498dad640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc4940c3e50 0x7fc494159f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.389 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.351+0000 7fc498dad640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc4940c5760 0x7fc494159150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.389 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.351+0000 7fc49259c640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc4940c3e50 0x7fc494159f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.389 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.351+0000 7fc49259c640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc4940c3e50 0x7fc494159f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:47096/0 (socket says 172.21.15.19:47096) 2024-08-31T12:16:05.389 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.351+0000 7feadd7fa640 1 -- 172.21.15.19:0/751701327 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead000cd80 con 0x7fead80e7040 2024-08-31T12:16:05.389 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.351+0000 7feadd7fa640 1 -- 172.21.15.19:0/751701327 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fead0004030 con 0x7fead80e7040 2024-08-31T12:16:05.389 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.351+0000 7feadd7fa640 1 -- 172.21.15.19:0/751701327 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead0004440 con 0x7fead80e7040 2024-08-31T12:16:05.389 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.351+0000 7feae5964640 1 -- 172.21.15.19:0/751701327 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fead816e130 con 0x7fead80e7040 2024-08-31T12:16:05.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.351+0000 7f1afaffd640 1 -- 172.21.15.19:0/490765926 <== osd.2 v2:172.21.15.19:6808/2332368824 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1b00009e50 con 0x7f1ae80800a0 2024-08-31T12:16:05.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.350+0000 7fc2ccff9640 1 --2- 172.21.15.19:0/1553236222 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc2b4077a50 0x7fc2b4079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.350+0000 7fc2ccff9640 1 -- 172.21.15.19:0/1553236222 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc2bc099830 con 0x7fc2c80dfa30 2024-08-31T12:16:05.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.351+0000 7fc2ccff9640 1 --2- 172.21.15.19:0/1553236222 >> [v2:172.21.15.43:6802/4271971521,v1:172.21.15.43:6803/4271971521] conn(0x7fc2b407fcc0 0x7fc2b4082110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.351+0000 7fc2ccff9640 1 -- 172.21.15.19:0/1553236222 --> [v2:172.21.15.43:6802/4271971521,v1:172.21.15.43:6803/4271971521] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc2b40827e0 con 0x7fc2b407fcc0 2024-08-31T12:16:05.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.351+0000 7fc2ccff9640 1 -- 172.21.15.19:0/1553236222 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fc2bc09e2a0 con 0x7fc2c80dfa30 2024-08-31T12:16:05.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.351+0000 7fc2cffff640 1 --2- 172.21.15.19:0/1553236222 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc2b4077a50 0x7fc2b4079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.351+0000 7feae5964640 1 -- 172.21.15.19:0/751701327 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fead816f390 con 0x7fead80e7040 2024-08-31T12:16:05.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.351+0000 7fc491d9b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc4940c5760 0x7fc494159150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.351+0000 7f88367fc640 1 -- 172.21.15.19:0/728413247 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88380039f0 con 0x7f8840151040 2024-08-31T12:16:05.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.351+0000 7f1b09b61640 1 --2- 172.21.15.19:0/490765926 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1ae8077e50 0x7f1ae807a360 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1afc034260 tx=0x7f1b0406c940 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:05.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.351+0000 7f88367fc640 1 -- 172.21.15.19:0/728413247 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f883803e070 con 0x7f8840151040 2024-08-31T12:16:05.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.351+0000 7f8847f1a640 1 -- 172.21.15.19:0/728413247 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88401f3ac0 con 0x7f8840151040 2024-08-31T12:16:05.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.351+0000 7fe20ca1b640 1 --2- 172.21.15.19:0/1984143439 >> [v2:172.21.15.19:6824/3785550598,v1:172.21.15.19:6825/3785550598] conn(0x7fe1f40809d0 0x7fe1f4082ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.351+0000 7fc498dad640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc494154a30 0x7fc494141aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.351+0000 7fc498dad640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fc494154e40 con 0x7fc4940c5760 2024-08-31T12:16:05.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.351+0000 7fc498dad640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc4940740e0 con 0x7fc4940c3e50 2024-08-31T12:16:05.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.351+0000 7fc498dad640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fc4940c6360 con 0x7fc494154a30 2024-08-31T12:16:05.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.351+0000 7fc491d9b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc4940c5760 0x7fc494159150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:56666/0 (socket says 172.21.15.19:56666) 2024-08-31T12:16:05.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.351+0000 7fc491d9b640 1 -- 172.21.15.19:0/2311917927 learned_addr learned my addr 172.21.15.19:0/2311917927 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:05.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.352+0000 7f8847f1a640 1 -- 172.21.15.19:0/728413247 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88401f0560 con 0x7f8840151040 2024-08-31T12:16:05.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.352+0000 7f0e27fff640 1 --2- 172.21.15.19:0/856725592 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0e14077a50 0x7f0e14079f60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0e2814a950 tx=0x7f0e18008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:05.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.352+0000 7fc2cffff640 1 --2- 172.21.15.19:0/1553236222 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc2b4077a50 0x7fc2b4079f60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc2c40046b0 tx=0x7fc2c4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:05.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.352+0000 7fc2d4ee7640 1 --2- 172.21.15.19:0/1553236222 >> [v2:172.21.15.43:6802/4271971521,v1:172.21.15.43:6803/4271971521] conn(0x7fc2b407fcc0 0x7fc2b4082110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.352+0000 7fe20ca1b640 1 --2- 172.21.15.19:0/1984143439 >> [v2:172.21.15.19:6824/3785550598,v1:172.21.15.19:6825/3785550598] conn(0x7fe1f40809d0 0x7fe1f4082ec0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:05.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.352+0000 7fc492d9d640 1 --2- 172.21.15.19:0/2311917927 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc494154a30 0x7fc494141aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.352+0000 7fc491d9b640 1 -- 172.21.15.19:0/2311917927 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc494154a30 msgr2=0x7fc494141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.352+0000 7fc491d9b640 1 --2- 172.21.15.19:0/2311917927 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc494154a30 0x7fc494141aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.352+0000 7fc491d9b640 1 -- 172.21.15.19:0/2311917927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc4940c3e50 msgr2=0x7fc494159f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.352+0000 7fc491d9b640 1 --2- 172.21.15.19:0/2311917927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc4940c3e50 0x7fc494159f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.352+0000 7fc491d9b640 1 -- 172.21.15.19:0/2311917927 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc49415a470 con 0x7fc4940c5760 2024-08-31T12:16:05.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.352+0000 7feae5964640 1 -- 172.21.15.19:0/751701327 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fead804e900 con 0x7fead80e7040 2024-08-31T12:16:05.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.353+0000 7fc491d9b640 1 --2- 172.21.15.19:0/2311917927 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc4940c5760 0x7fc494159150 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc488009850 tx=0x7fc4880042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:05.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.353+0000 7f0e2d08f640 1 --2- 172.21.15.19:0/856725592 >> [v2:172.21.15.19:6800/680447146,v1:172.21.15.19:6801/680447146] conn(0x7f0e1407fca0 0x7f0e140820f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:05.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.353+0000 7f882bfff640 1 -- 172.21.15.19:0/728413247 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f88400ca9a0 con 0x7f8840151040 2024-08-31T12:16:05.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.353+0000 7fd5f4ff9640 1 -- 172.21.15.19:0/2199610839 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fd5e401a070 con 0x7fd5f004d230 2024-08-31T12:16:05.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.353+0000 7f88367fc640 1 -- 172.21.15.19:0/728413247 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8838004040 con 0x7f8840151040 2024-08-31T12:16:05.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.353+0000 7fc2d4ee7640 1 --2- 172.21.15.19:0/1553236222 >> [v2:172.21.15.43:6802/4271971521,v1:172.21.15.43:6803/4271971521] conn(0x7fc2b407fcc0 0x7fc2b4082110 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:05.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.353+0000 7fe204ff9640 1 -- 172.21.15.19:0/1984143439 <== osd.3 v2:172.21.15.19:6824/3785550598 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe1f4083590 con 0x7fe1f40809d0 2024-08-31T12:16:05.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.354+0000 7f0e257fa640 1 -- 172.21.15.19:0/856725592 <== osd.0 v2:172.21.15.19:6800/680447146 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0e140827c0 con 0x7f0e1407fca0 2024-08-31T12:16:05.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.354+0000 7fd5f4ff9640 1 --2- 172.21.15.19:0/2199610839 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd5dc077a80 0x7fd5dc079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.354+0000 7fc2ccff9640 1 -- 172.21.15.19:0/1553236222 <== osd.5 v2:172.21.15.43:6802/4271971521 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc2b40827e0 con 0x7fc2b407fcc0 2024-08-31T12:16:05.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.354+0000 7fd5f4ff9640 1 -- 172.21.15.19:0/2199610839 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd5e4098bc0 con 0x7fd5f004d230 2024-08-31T12:16:05.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.355+0000 7fd5f4ff9640 1 --2- 172.21.15.19:0/2199610839 >> [v2:172.21.15.43:6818/1985519349,v1:172.21.15.43:6819/1985519349] conn(0x7fd5dc07fcf0 0x7fd5dc082140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.355+0000 7fd5f4ff9640 1 -- 172.21.15.19:0/2199610839 --> [v2:172.21.15.43:6818/1985519349,v1:172.21.15.43:6819/1985519349] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd5e4016ec0 con 0x7fd5dc07fcf0 2024-08-31T12:16:05.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.355+0000 7fd5f4ff9640 1 -- 172.21.15.19:0/2199610839 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd5e4098fb0 con 0x7fd5f004d230 2024-08-31T12:16:05.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.355+0000 7fd5f77fe640 1 --2- 172.21.15.19:0/2199610839 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd5dc077a80 0x7fd5dc079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.355+0000 7fd5fcaef640 1 --2- 172.21.15.19:0/2199610839 >> [v2:172.21.15.43:6818/1985519349,v1:172.21.15.43:6819/1985519349] conn(0x7fd5dc07fcf0 0x7fd5dc082140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.354+0000 7fc482ffd640 1 -- 172.21.15.19:0/2311917927 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc488035040 con 0x7fc4940c5760 2024-08-31T12:16:05.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.355+0000 7fc482ffd640 1 -- 172.21.15.19:0/2311917927 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc488003af0 con 0x7fc4940c5760 2024-08-31T12:16:05.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.355+0000 7fc482ffd640 1 -- 172.21.15.19:0/2311917927 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc488035040 con 0x7fc4940c5760 2024-08-31T12:16:05.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.355+0000 7fc498dad640 1 -- 172.21.15.19:0/2311917927 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc494155c60 con 0x7fc4940c5760 2024-08-31T12:16:05.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.355+0000 7fc498dad640 1 -- 172.21.15.19:0/2311917927 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc494156220 con 0x7fc4940c5760 2024-08-31T12:16:05.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.356+0000 7feadd7fa640 1 -- 172.21.15.19:0/751701327 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fead00035f0 con 0x7fead80e7040 2024-08-31T12:16:05.401 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.356+0000 7f88367fc640 1 -- 172.21.15.19:0/728413247 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f88380041e0 con 0x7f8840151040 2024-08-31T12:16:05.401 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.357+0000 7f88367fc640 1 --2- 172.21.15.19:0/728413247 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8824077b20 0x7f882407a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.401 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.357+0000 7f88367fc640 1 -- 172.21.15.19:0/728413247 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f88380baaa0 con 0x7f8840151040 2024-08-31T12:16:05.401 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.357+0000 7fd5f77fe640 1 --2- 172.21.15.19:0/2199610839 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd5dc077a80 0x7fd5dc079f90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd5f00dceb0 tx=0x7fd5ec004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:05.401 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.357+0000 7feadd7fa640 1 --2- 172.21.15.19:0/751701327 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7feacc077b20 0x7feacc07a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.357+0000 7fc498dad640 1 -- 172.21.15.19:0/2311917927 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc4940c5b70 con 0x7fc4940c5760 2024-08-31T12:16:05.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.357+0000 7feadd7fa640 1 -- 172.21.15.19:0/751701327 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fead009a440 con 0x7fead80e7040 2024-08-31T12:16:05.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.357+0000 7f88367fc640 1 --2- 172.21.15.19:0/728413247 >> [v2:172.21.15.19:6816/1522610034,v1:172.21.15.19:6817/1522610034] conn(0x7f882407fd70 0x7f88240821c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.357+0000 7f88367fc640 1 -- 172.21.15.19:0/728413247 --> [v2:172.21.15.19:6816/1522610034,v1:172.21.15.19:6817/1522610034] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f883802cec0 con 0x7f882407fd70 2024-08-31T12:16:05.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.357+0000 7f88367fc640 1 -- 172.21.15.19:0/728413247 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f88380bf2a0 con 0x7f8840151040 2024-08-31T12:16:05.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.357+0000 7feadd7fa640 1 --2- 172.21.15.19:0/751701327 >> [v2:172.21.15.43:6810/792572234,v1:172.21.15.43:6811/792572234] conn(0x7feacc07fdf0 0x7feacc082240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.357+0000 7feadd7fa640 1 -- 172.21.15.19:0/751701327 --> [v2:172.21.15.43:6810/792572234,v1:172.21.15.43:6811/792572234] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fead0022e50 con 0x7feacc07fdf0 2024-08-31T12:16:05.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.357+0000 7feadd7fa640 1 -- 172.21.15.19:0/751701327 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fead009a830 con 0x7fead80e7040 2024-08-31T12:16:05.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.358+0000 7feae4962640 1 --2- 172.21.15.19:0/751701327 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7feacc077b20 0x7feacc07a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.404 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.358+0000 7fd5fcaef640 1 --2- 172.21.15.19:0/2199610839 >> [v2:172.21.15.43:6818/1985519349,v1:172.21.15.43:6819/1985519349] conn(0x7fd5dc07fcf0 0x7fd5dc082140 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:05.404 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.358+0000 7feadffff640 1 --2- 172.21.15.19:0/751701327 >> [v2:172.21.15.43:6810/792572234,v1:172.21.15.43:6811/792572234] conn(0x7feacc07fdf0 0x7feacc082240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.358+0000 7f8845cb6640 1 --2- 172.21.15.19:0/728413247 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8824077b20 0x7f882407a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.358+0000 7f88454b5640 1 --2- 172.21.15.19:0/728413247 >> [v2:172.21.15.19:6816/1522610034,v1:172.21.15.19:6817/1522610034] conn(0x7f882407fd70 0x7f88240821c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.358+0000 7feae4962640 1 --2- 172.21.15.19:0/751701327 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7feacc077b20 0x7feacc07a030 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fead804ffd0 tx=0x7feae006c940 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:05.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.358+0000 7feadffff640 1 --2- 172.21.15.19:0/751701327 >> [v2:172.21.15.43:6810/792572234,v1:172.21.15.43:6811/792572234] conn(0x7feacc07fdf0 0x7feacc082240 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:05.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.359+0000 7f8845cb6640 1 --2- 172.21.15.19:0/728413247 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8824077b20 0x7f882407a030 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f883c00ac80 tx=0x7f883c010040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:05.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.359+0000 7fd5f4ff9640 1 -- 172.21.15.19:0/2199610839 <== osd.4 v2:172.21.15.43:6818/1985519349 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd5e4016ec0 con 0x7fd5dc07fcf0 2024-08-31T12:16:05.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.359+0000 7f88454b5640 1 --2- 172.21.15.19:0/728413247 >> [v2:172.21.15.19:6816/1522610034,v1:172.21.15.19:6817/1522610034] conn(0x7f882407fd70 0x7f88240821c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:05.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.359+0000 7fc482ffd640 1 -- 172.21.15.19:0/2311917927 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fc488050060 con 0x7fc4940c5760 2024-08-31T12:16:05.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.359+0000 7fc482ffd640 1 --2- 172.21.15.19:0/2311917927 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc474077e80 0x7fc47407a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.360+0000 7feadd7fa640 1 -- 172.21.15.19:0/751701327 <== osd.7 v2:172.21.15.43:6810/792572234 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fead0022e50 con 0x7feacc07fdf0 2024-08-31T12:16:05.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.360+0000 7fc482ffd640 1 -- 172.21.15.19:0/2311917927 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc488003800 con 0x7fc4940c5760 2024-08-31T12:16:05.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.360+0000 7fc482ffd640 1 --2- 172.21.15.19:0/2311917927 >> [v2:172.21.15.43:6826/2863384694,v1:172.21.15.43:6827/2863384694] conn(0x7fc4740800d0 0x7fc474082520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.360+0000 7fc482ffd640 1 -- 172.21.15.19:0/2311917927 --> [v2:172.21.15.43:6826/2863384694,v1:172.21.15.43:6827/2863384694] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc488050e80 con 0x7fc4740800d0 2024-08-31T12:16:05.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.360+0000 7fc482ffd640 1 -- 172.21.15.19:0/2311917927 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fc4880c62c0 con 0x7fc4940c5760 2024-08-31T12:16:05.408 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.360+0000 7fc49259c640 1 --2- 172.21.15.19:0/2311917927 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc474077e80 0x7fc47407a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.408 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.360+0000 7f88367fc640 1 -- 172.21.15.19:0/728413247 <== osd.1 v2:172.21.15.19:6816/1522610034 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f883802cec0 con 0x7f882407fd70 2024-08-31T12:16:05.408 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.360+0000 7fc492d9d640 1 --2- 172.21.15.19:0/2311917927 >> [v2:172.21.15.43:6826/2863384694,v1:172.21.15.43:6827/2863384694] conn(0x7fc4740800d0 0x7fc474082520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.360+0000 7fc49259c640 1 --2- 172.21.15.19:0/2311917927 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc474077e80 0x7fc47407a390 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc48c009f20 tx=0x7fc48c0094d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:05.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.361+0000 7fc492d9d640 1 --2- 172.21.15.19:0/2311917927 >> [v2:172.21.15.43:6826/2863384694,v1:172.21.15.43:6827/2863384694] conn(0x7fc4740800d0 0x7fc474082520 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:05.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.362+0000 7fc482ffd640 1 -- 172.21.15.19:0/2311917927 <== osd.6 v2:172.21.15.43:6826/2863384694 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc488050e80 con 0x7fc4740800d0 2024-08-31T12:16:05.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.380+0000 7f1b0b364640 1 -- 172.21.15.19:0/490765926 --> [v2:172.21.15.19:6808/2332368824,v1:172.21.15.19:6809/2332368824] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1afc16e6d0 con 0x7f1ae80800a0 2024-08-31T12:16:05.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.381+0000 7f1afaffd640 1 -- 172.21.15.19:0/490765926 <== osd.2 v2:172.21.15.19:6808/2332368824 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f1afc16e6d0 con 0x7f1ae80800a0 2024-08-31T12:16:05.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.381+0000 7f1af8ff9640 1 -- 172.21.15.19:0/490765926 >> [v2:172.21.15.19:6808/2332368824,v1:172.21.15.19:6809/2332368824] conn(0x7f1ae80800a0 msgr2=0x7f1ae80824f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.381+0000 7f1af8ff9640 1 --2- 172.21.15.19:0/490765926 >> [v2:172.21.15.19:6808/2332368824,v1:172.21.15.19:6809/2332368824] conn(0x7f1ae80800a0 0x7f1ae80824f0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.382+0000 7f1af8ff9640 1 -- 172.21.15.19:0/490765926 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1ae8077e50 msgr2=0x7f1ae807a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.382+0000 7f1af8ff9640 1 --2- 172.21.15.19:0/490765926 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1ae8077e50 0x7f1ae807a360 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1afc034260 tx=0x7f1b0406c940 comp rx=0 tx=0).stop 2024-08-31T12:16:05.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.382+0000 7f1af8ff9640 1 -- 172.21.15.19:0/490765926 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1afc004a70 msgr2=0x7f1afc16ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.382+0000 7f1af8ff9640 1 --2- 172.21.15.19:0/490765926 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1afc004a70 0x7f1afc16ea10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1b0000c3d0 tx=0x7f1b0000c890 comp rx=0 tx=0).stop 2024-08-31T12:16:05.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.382+0000 7f1b0ab63640 1 -- 172.21.15.19:0/490765926 reap_dead start 2024-08-31T12:16:05.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.382+0000 7f1af8ff9640 1 -- 172.21.15.19:0/490765926 shutdown_connections 2024-08-31T12:16:05.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.383+0000 7f1af8ff9640 1 -- 172.21.15.19:0/490765926 >> 172.21.15.19:0/490765926 conn(0x7f1afc050280 msgr2=0x7f1afc034fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:05.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.383+0000 7f0e2eaf2640 1 -- 172.21.15.19:0/856725592 --> [v2:172.21.15.19:6800/680447146,v1:172.21.15.19:6801/680447146] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0e280cc210 con 0x7f0e1407fca0 2024-08-31T12:16:05.412 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.383+0000 7f1af8ff9640 1 -- 172.21.15.19:0/490765926 shutdown_connections 2024-08-31T12:16:05.412 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.383+0000 7f1af8ff9640 1 -- 172.21.15.19:0/490765926 wait complete. 2024-08-31T12:16:05.412 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.384+0000 7f0e257fa640 1 -- 172.21.15.19:0/856725592 <== osd.0 v2:172.21.15.19:6800/680447146 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0e280cc210 con 0x7f0e1407fca0 2024-08-31T12:16:05.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.384+0000 7f0e2eaf2640 1 -- 172.21.15.19:0/856725592 >> [v2:172.21.15.19:6800/680447146,v1:172.21.15.19:6801/680447146] conn(0x7f0e1407fca0 msgr2=0x7f0e140820f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.384+0000 7f0e2eaf2640 1 --2- 172.21.15.19:0/856725592 >> [v2:172.21.15.19:6800/680447146,v1:172.21.15.19:6801/680447146] conn(0x7f0e1407fca0 0x7f0e140820f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.385+0000 7f0e2eaf2640 1 -- 172.21.15.19:0/856725592 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0e14077a50 msgr2=0x7f0e14079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.385+0000 7f0e2eaf2640 1 --2- 172.21.15.19:0/856725592 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0e14077a50 0x7f0e14079f60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0e2814a950 tx=0x7f0e18008040 comp rx=0 tx=0).stop 2024-08-31T12:16:05.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.385+0000 7f0e2eaf2640 1 -- 172.21.15.19:0/856725592 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e28070310 msgr2=0x7f0e28145f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.385+0000 7f0e2eaf2640 1 --2- 172.21.15.19:0/856725592 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e28070310 0x7f0e28145f30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c00c500 tx=0x7f0e1c00c9c0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.385+0000 7f0e2d08f640 1 -- 172.21.15.19:0/856725592 reap_dead start 2024-08-31T12:16:05.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.385+0000 7f0e2eaf2640 1 -- 172.21.15.19:0/856725592 shutdown_connections 2024-08-31T12:16:05.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.385+0000 7f0e2eaf2640 1 -- 172.21.15.19:0/856725592 >> 172.21.15.19:0/856725592 conn(0x7f0e2806cae0 msgr2=0x7f0e280c8100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:05.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.386+0000 7f0e2eaf2640 1 -- 172.21.15.19:0/856725592 shutdown_connections 2024-08-31T12:16:05.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.386+0000 7fe1ee7fc640 1 -- 172.21.15.19:0/1984143439 --> [v2:172.21.15.19:6824/3785550598,v1:172.21.15.19:6825/3785550598] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe20815f240 con 0x7fe1f40809d0 2024-08-31T12:16:05.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.386+0000 7f0e2eaf2640 1 -- 172.21.15.19:0/856725592 wait complete. 2024-08-31T12:16:05.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.386+0000 7fe204ff9640 1 -- 172.21.15.19:0/1984143439 <== osd.3 v2:172.21.15.19:6824/3785550598 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fe20815f240 con 0x7fe1f40809d0 2024-08-31T12:16:05.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.387+0000 7fc2d56e8640 1 -- 172.21.15.19:0/1553236222 --> [v2:172.21.15.43:6802/4271971521,v1:172.21.15.43:6803/4271971521] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc2c804fcb0 con 0x7fc2b407fcc0 2024-08-31T12:16:05.416 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.387+0000 7fe1ee7fc640 1 -- 172.21.15.19:0/1984143439 >> [v2:172.21.15.19:6824/3785550598,v1:172.21.15.19:6825/3785550598] conn(0x7fe1f40809d0 msgr2=0x7fe1f4082ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.416 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.387+0000 7fe1ee7fc640 1 --2- 172.21.15.19:0/1984143439 >> [v2:172.21.15.19:6824/3785550598,v1:172.21.15.19:6825/3785550598] conn(0x7fe1f40809d0 0x7fe1f4082ec0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.416 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.387+0000 7fe1ee7fc640 1 -- 172.21.15.19:0/1984143439 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe1f4077c50 msgr2=0x7fe1f407a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.416 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.387+0000 7fe1ee7fc640 1 --2- 172.21.15.19:0/1984143439 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe1f4077c50 0x7fe1f407a160 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc0095d0 tx=0x7fe1fc006b30 comp rx=0 tx=0).stop 2024-08-31T12:16:05.416 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.387+0000 7fc2ccff9640 1 -- 172.21.15.19:0/1553236222 <== osd.5 v2:172.21.15.43:6802/4271971521 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fc2c804fcb0 con 0x7fc2b407fcc0 2024-08-31T12:16:05.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.387+0000 7fe1ee7fc640 1 -- 172.21.15.19:0/1984143439 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe20806f5c0 msgr2=0x7fe2080c5eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.387+0000 7fe1ee7fc640 1 --2- 172.21.15.19:0/1984143439 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe20806f5c0 0x7fe2080c5eb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe1f8006010 tx=0x7fe1f8003b90 comp rx=0 tx=0).stop 2024-08-31T12:16:05.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.387+0000 7fe20ca1b640 1 -- 172.21.15.19:0/1984143439 reap_dead start 2024-08-31T12:16:05.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.387+0000 7fe1ee7fc640 1 -- 172.21.15.19:0/1984143439 shutdown_connections 2024-08-31T12:16:05.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.387+0000 7fe1ee7fc640 1 -- 172.21.15.19:0/1984143439 >> 172.21.15.19:0/1984143439 conn(0x7fe20806cae0 msgr2=0x7fe20814c8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:05.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.388+0000 7fe1ee7fc640 1 -- 172.21.15.19:0/1984143439 shutdown_connections 2024-08-31T12:16:05.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.388+0000 7fe1ee7fc640 1 -- 172.21.15.19:0/1984143439 wait complete. 2024-08-31T12:16:05.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.388+0000 7fc2d56e8640 1 -- 172.21.15.19:0/1553236222 >> [v2:172.21.15.43:6802/4271971521,v1:172.21.15.43:6803/4271971521] conn(0x7fc2b407fcc0 msgr2=0x7fc2b4082110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.388+0000 7fc2d56e8640 1 --2- 172.21.15.19:0/1553236222 >> [v2:172.21.15.43:6802/4271971521,v1:172.21.15.43:6803/4271971521] conn(0x7fc2b407fcc0 0x7fc2b4082110 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.388+0000 7fc2d56e8640 1 -- 172.21.15.19:0/1553236222 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc2b4077a50 msgr2=0x7fc2b4079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.388+0000 7fc2d56e8640 1 --2- 172.21.15.19:0/1553236222 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc2b4077a50 0x7fc2b4079f60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc2c40046b0 tx=0x7fc2c4009040 comp rx=0 tx=0).stop 2024-08-31T12:16:05.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.388+0000 7fc2d56e8640 1 -- 172.21.15.19:0/1553236222 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc2c80dfa30 msgr2=0x7fc2c80d7320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.388+0000 7fc2d56e8640 1 --2- 172.21.15.19:0/1553236222 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc2c80dfa30 0x7fc2c80d7320 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc00d700 tx=0x7fc2bc00dbc0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.389+0000 7fc2d4ee7640 1 -- 172.21.15.19:0/1553236222 reap_dead start 2024-08-31T12:16:05.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.389+0000 7fc2d56e8640 1 -- 172.21.15.19:0/1553236222 shutdown_connections 2024-08-31T12:16:05.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.389+0000 7fc2d56e8640 1 -- 172.21.15.19:0/1553236222 >> 172.21.15.19:0/1553236222 conn(0x7fc2c8050fa0 msgr2=0x7fc2c804bf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:05.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.389+0000 7fd5fd2f0640 1 -- 172.21.15.19:0/2199610839 --> [v2:172.21.15.43:6818/1985519349,v1:172.21.15.43:6819/1985519349] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd5d8002dd0 con 0x7fd5dc07fcf0 2024-08-31T12:16:05.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.389+0000 7fc2d56e8640 1 -- 172.21.15.19:0/1553236222 shutdown_connections 2024-08-31T12:16:05.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.389+0000 7fc2d56e8640 1 -- 172.21.15.19:0/1553236222 wait complete. 2024-08-31T12:16:05.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.389+0000 7fd5f4ff9640 1 -- 172.21.15.19:0/2199610839 <== osd.4 v2:172.21.15.43:6818/1985519349 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fd5d8002dd0 con 0x7fd5dc07fcf0 2024-08-31T12:16:05.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.391+0000 7fd5e27fc640 1 -- 172.21.15.19:0/2199610839 >> [v2:172.21.15.43:6818/1985519349,v1:172.21.15.43:6819/1985519349] conn(0x7fd5dc07fcf0 msgr2=0x7fd5dc082140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.391+0000 7fd5e27fc640 1 --2- 172.21.15.19:0/2199610839 >> [v2:172.21.15.43:6818/1985519349,v1:172.21.15.43:6819/1985519349] conn(0x7fd5dc07fcf0 0x7fd5dc082140 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.391+0000 7fd5e27fc640 1 -- 172.21.15.19:0/2199610839 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd5dc077a80 msgr2=0x7fd5dc079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.391+0000 7fd5e27fc640 1 --2- 172.21.15.19:0/2199610839 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd5dc077a80 0x7fd5dc079f90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd5f00dceb0 tx=0x7fd5ec004730 comp rx=0 tx=0).stop 2024-08-31T12:16:05.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.391+0000 7fd5e27fc640 1 -- 172.21.15.19:0/2199610839 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd5f004d230 msgr2=0x7fd5f0177600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.391+0000 7fd5e27fc640 1 --2- 172.21.15.19:0/2199610839 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd5f004d230 0x7fd5f0177600 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd5e400c3c0 tx=0x7fd5e400c880 comp rx=0 tx=0).stop 2024-08-31T12:16:05.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.391+0000 7fd5fcaef640 1 -- 172.21.15.19:0/2199610839 reap_dead start 2024-08-31T12:16:05.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.392+0000 7fd5e27fc640 1 -- 172.21.15.19:0/2199610839 shutdown_connections 2024-08-31T12:16:05.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.392+0000 7fd5e27fc640 1 -- 172.21.15.19:0/2199610839 >> 172.21.15.19:0/2199610839 conn(0x7fd5f0050f40 msgr2=0x7fd5f00dd4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:05.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.392+0000 7fd5e27fc640 1 -- 172.21.15.19:0/2199610839 shutdown_connections 2024-08-31T12:16:05.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.392+0000 7fd5e27fc640 1 -- 172.21.15.19:0/2199610839 wait complete. 2024-08-31T12:16:05.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.392+0000 7fc498dad640 1 -- 172.21.15.19:0/2311917927 --> [v2:172.21.15.43:6826/2863384694,v1:172.21.15.43:6827/2863384694] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc494147ac0 con 0x7fc4740800d0 2024-08-31T12:16:05.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.392+0000 7f882bfff640 1 -- 172.21.15.19:0/728413247 --> [v2:172.21.15.19:6816/1522610034,v1:172.21.15.19:6817/1522610034] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f88401458c0 con 0x7f882407fd70 2024-08-31T12:16:05.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.393+0000 7feae5964640 1 -- 172.21.15.19:0/751701327 --> [v2:172.21.15.43:6810/792572234,v1:172.21.15.43:6811/792572234] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fead804e290 con 0x7feacc07fdf0 2024-08-31T12:16:05.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.393+0000 7f88367fc640 1 -- 172.21.15.19:0/728413247 <== osd.1 v2:172.21.15.19:6816/1522610034 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f88401458c0 con 0x7f882407fd70 2024-08-31T12:16:05.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.393+0000 7fc482ffd640 1 -- 172.21.15.19:0/2311917927 <== osd.6 v2:172.21.15.43:6826/2863384694 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fc494147ac0 con 0x7fc4740800d0 2024-08-31T12:16:05.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.393+0000 7feadd7fa640 1 -- 172.21.15.19:0/751701327 <== osd.7 v2:172.21.15.43:6810/792572234 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fead804e290 con 0x7feacc07fdf0 2024-08-31T12:16:05.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.394+0000 7f882bfff640 1 -- 172.21.15.19:0/728413247 >> [v2:172.21.15.19:6816/1522610034,v1:172.21.15.19:6817/1522610034] conn(0x7f882407fd70 msgr2=0x7f88240821c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.394+0000 7f882bfff640 1 --2- 172.21.15.19:0/728413247 >> [v2:172.21.15.19:6816/1522610034,v1:172.21.15.19:6817/1522610034] conn(0x7f882407fd70 0x7f88240821c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.394+0000 7feae5964640 1 -- 172.21.15.19:0/751701327 >> [v2:172.21.15.43:6810/792572234,v1:172.21.15.43:6811/792572234] conn(0x7feacc07fdf0 msgr2=0x7feacc082240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.394+0000 7feae5964640 1 --2- 172.21.15.19:0/751701327 >> [v2:172.21.15.43:6810/792572234,v1:172.21.15.43:6811/792572234] conn(0x7feacc07fdf0 0x7feacc082240 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.394+0000 7feae5964640 1 -- 172.21.15.19:0/751701327 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7feacc077b20 msgr2=0x7feacc07a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.394+0000 7feae5964640 1 --2- 172.21.15.19:0/751701327 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7feacc077b20 0x7feacc07a030 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fead804ffd0 tx=0x7feae006c940 comp rx=0 tx=0).stop 2024-08-31T12:16:05.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.394+0000 7f882bfff640 1 -- 172.21.15.19:0/728413247 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8824077b20 msgr2=0x7f882407a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.394+0000 7f882bfff640 1 --2- 172.21.15.19:0/728413247 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8824077b20 0x7f882407a030 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f883c00ac80 tx=0x7f883c010040 comp rx=0 tx=0).stop 2024-08-31T12:16:05.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.394+0000 7f882bfff640 1 -- 172.21.15.19:0/728413247 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8840151040 msgr2=0x7f8840144ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.394+0000 7f882bfff640 1 --2- 172.21.15.19:0/728413247 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8840151040 0x7f8840144ce0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f884006ed70 tx=0x7f8838030650 comp rx=0 tx=0).stop 2024-08-31T12:16:05.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.394+0000 7feae5964640 1 -- 172.21.15.19:0/751701327 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fead80e7040 msgr2=0x7fead8185a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.394+0000 7feae5964640 1 --2- 172.21.15.19:0/751701327 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fead80e7040 0x7fead8185a90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fead000e970 tx=0x7fead000ee30 comp rx=0 tx=0).stop 2024-08-31T12:16:05.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.394+0000 7feae5163640 1 -- 172.21.15.19:0/751701327 reap_dead start 2024-08-31T12:16:05.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.395+0000 7f88464b7640 1 -- 172.21.15.19:0/728413247 reap_dead start 2024-08-31T12:16:05.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.395+0000 7f882bfff640 1 -- 172.21.15.19:0/728413247 shutdown_connections 2024-08-31T12:16:05.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.395+0000 7f882bfff640 1 -- 172.21.15.19:0/728413247 >> 172.21.15.19:0/728413247 conn(0x7f884006cae0 msgr2=0x7f88400c6860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:05.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.395+0000 7f882bfff640 1 -- 172.21.15.19:0/728413247 shutdown_connections 2024-08-31T12:16:05.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.395+0000 7f882bfff640 1 -- 172.21.15.19:0/728413247 wait complete. 2024-08-31T12:16:05.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.395+0000 7feae5964640 1 -- 172.21.15.19:0/751701327 shutdown_connections 2024-08-31T12:16:05.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.395+0000 7feae5964640 1 -- 172.21.15.19:0/751701327 >> 172.21.15.19:0/751701327 conn(0x7fead80543b0 msgr2=0x7fead804f150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:05.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.395+0000 7feae5964640 1 -- 172.21.15.19:0/751701327 shutdown_connections 2024-08-31T12:16:05.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.395+0000 7feae5964640 1 -- 172.21.15.19:0/751701327 wait complete. 2024-08-31T12:16:05.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.395+0000 7fc498dad640 1 -- 172.21.15.19:0/2311917927 >> [v2:172.21.15.43:6826/2863384694,v1:172.21.15.43:6827/2863384694] conn(0x7fc4740800d0 msgr2=0x7fc474082520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.395+0000 7fc498dad640 1 --2- 172.21.15.19:0/2311917927 >> [v2:172.21.15.43:6826/2863384694,v1:172.21.15.43:6827/2863384694] conn(0x7fc4740800d0 0x7fc474082520 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.395+0000 7fc498dad640 1 -- 172.21.15.19:0/2311917927 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc474077e80 msgr2=0x7fc47407a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.395+0000 7fc498dad640 1 --2- 172.21.15.19:0/2311917927 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc474077e80 0x7fc47407a390 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc48c009f20 tx=0x7fc48c0094d0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.395+0000 7fc498dad640 1 -- 172.21.15.19:0/2311917927 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc4940c5760 msgr2=0x7fc494159150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.395+0000 7fc498dad640 1 --2- 172.21.15.19:0/2311917927 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc4940c5760 0x7fc494159150 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc488009850 tx=0x7fc4880042f0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.395+0000 7fc492d9d640 1 -- 172.21.15.19:0/2311917927 reap_dead start 2024-08-31T12:16:05.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.396+0000 7fc498dad640 1 -- 172.21.15.19:0/2311917927 shutdown_connections 2024-08-31T12:16:05.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.396+0000 7fc498dad640 1 -- 172.21.15.19:0/2311917927 >> 172.21.15.19:0/2311917927 conn(0x7fc494071100 msgr2=0x7fc4940c7bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:05.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.396+0000 7fc498dad640 1 -- 172.21.15.19:0/2311917927 shutdown_connections 2024-08-31T12:16:05.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.396+0000 7fc498dad640 1 -- 172.21.15.19:0/2311917927 wait complete. 2024-08-31T12:16:05.430 INFO:teuthology.orchestra.run.smithi019.stdout:55834574855 2024-08-31T12:16:05.431 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-08-31T12:16:05.432 INFO:teuthology.orchestra.run.smithi019.stdout:64424509447 2024-08-31T12:16:05.432 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-08-31T12:16:05.433 INFO:teuthology.orchestra.run.smithi019.stdout:64424509446 2024-08-31T12:16:05.433 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-08-31T12:16:05.434 INFO:teuthology.orchestra.run.smithi019.stdout:60129542153 2024-08-31T12:16:05.434 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-08-31T12:16:05.435 INFO:teuthology.orchestra.run.smithi019.stdout:60129542151 2024-08-31T12:16:05.435 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-08-31T12:16:05.436 INFO:teuthology.orchestra.run.smithi019.stdout:64424509447 2024-08-31T12:16:05.436 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-08-31T12:16:05.437 INFO:teuthology.orchestra.run.smithi019.stdout:60129542151 2024-08-31T12:16:05.437 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-08-31T12:16:05.438 INFO:teuthology.orchestra.run.smithi019.stdout:64424509447 2024-08-31T12:16:05.438 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-08-31T12:16:05.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.683+0000 7f0591c54640 1 -- 172.21.15.19:0/3710316969 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f058c141e70 msgr2=0x7f058c0c5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.684+0000 7f0591c54640 1 --2- 172.21.15.19:0/3710316969 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f058c141e70 0x7f058c0c5880 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f058c06dfb0 tx=0x7f058002a4d0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.685 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.684+0000 7f0591c54640 1 -- 172.21.15.19:0/3710316969 shutdown_connections 2024-08-31T12:16:05.685 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.684+0000 7f0591c54640 1 --2- 172.21.15.19:0/3710316969 >> v2:172.21.15.19:3300/0 conn(0x7f058c0c7280 0x7f058c15c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.685 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.684+0000 7f0591c54640 1 --2- 172.21.15.19:0/3710316969 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f058c141e70 0x7f058c0c5880 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.684+0000 7f0591c54640 1 -- 172.21.15.19:0/3710316969 >> 172.21.15.19:0/3710316969 conn(0x7f058c06cae0 msgr2=0x7f058c06cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:05.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.684+0000 7f46b7247640 1 -- 172.21.15.19:0/1976203831 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f46a8045a10 msgr2=0x7f46a804c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.684+0000 7f46b7247640 1 --2- 172.21.15.19:0/1976203831 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f46a8045a10 0x7f46a804c470 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f46a804bda0 tx=0x7f46a002b9a0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.684+0000 7f0591c54640 1 -- 172.21.15.19:0/3710316969 shutdown_connections 2024-08-31T12:16:05.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.685+0000 7f46b7247640 1 -- 172.21.15.19:0/1976203831 shutdown_connections 2024-08-31T12:16:05.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.685+0000 7f46b7247640 1 --2- 172.21.15.19:0/1976203831 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f46a8045a10 0x7f46a804c470 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.685+0000 7f46b7247640 1 --2- 172.21.15.19:0/1976203831 >> v2:172.21.15.43:3300/0 conn(0x7f46a8046480 0x7f46a80d5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.685+0000 7f46b7247640 1 -- 172.21.15.19:0/1976203831 >> 172.21.15.19:0/1976203831 conn(0x7f46a80508a0 msgr2=0x7f46a8050cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:05.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.685+0000 7f0591c54640 1 -- 172.21.15.19:0/3710316969 wait complete. 2024-08-31T12:16:05.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.685+0000 7f46b7247640 1 -- 172.21.15.19:0/1976203831 shutdown_connections 2024-08-31T12:16:05.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.685+0000 7f46b7247640 1 -- 172.21.15.19:0/1976203831 wait complete. 2024-08-31T12:16:05.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.685+0000 7f540f59e640 1 -- 172.21.15.19:0/3510631248 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f541015ae00 msgr2=0x7f5410073ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.685+0000 7f540f59e640 1 --2- 172.21.15.19:0/3510631248 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f541015ae00 0x7f5410073ef0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5404009830 tx=0x7f540402a510 comp rx=0 tx=0).stop 2024-08-31T12:16:05.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.686+0000 7f540f59e640 1 -- 172.21.15.19:0/3510631248 shutdown_connections 2024-08-31T12:16:05.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.686+0000 7f540f59e640 1 --2- 172.21.15.19:0/3510631248 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f541015ae00 0x7f5410073ef0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.686+0000 7f540f59e640 1 --2- 172.21.15.19:0/3510631248 >> v2:172.21.15.19:3300/0 conn(0x7f5410071260 0x7f5410072e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.686+0000 7f540f59e640 1 -- 172.21.15.19:0/3510631248 >> 172.21.15.19:0/3510631248 conn(0x7f54100cc540 msgr2=0x7f54100cc950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:05.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.686+0000 7f540f59e640 1 -- 172.21.15.19:0/3510631248 shutdown_connections 2024-08-31T12:16:05.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.686+0000 7f540f59e640 1 -- 172.21.15.19:0/3510631248 wait complete. 2024-08-31T12:16:05.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.688+0000 7f0591c54640 1 Processor -- start 2024-08-31T12:16:05.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.688+0000 7f0591c54640 1 -- start start 2024-08-31T12:16:05.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.688+0000 7f46b7247640 1 Processor -- start 2024-08-31T12:16:05.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.688+0000 7f46b7247640 1 -- start start 2024-08-31T12:16:05.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.689+0000 7f3599a9e640 1 -- 172.21.15.19:0/4244229915 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f357c003a80 msgr2=0x7f357c003ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.689+0000 7f3599a9e640 1 --2- 172.21.15.19:0/4244229915 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f357c003a80 0x7f357c003ed0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f358c00c640 tx=0x7f357c005830 comp rx=0 tx=0).stop 2024-08-31T12:16:05.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.689+0000 7f359929d640 1 -- 172.21.15.19:0/4244229915 reap_dead start 2024-08-31T12:16:05.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.689+0000 7f3599a9e640 1 -- 172.21.15.19:0/4244229915 shutdown_connections 2024-08-31T12:16:05.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.689+0000 7f3599a9e640 1 -- 172.21.15.19:0/4244229915 >> 172.21.15.19:0/4244229915 conn(0x7f359406cae0 msgr2=0x7f359406cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:05.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.689+0000 7f3599a9e640 1 -- 172.21.15.19:0/4244229915 shutdown_connections 2024-08-31T12:16:05.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.690+0000 7f3599a9e640 1 -- 172.21.15.19:0/4244229915 wait complete. 2024-08-31T12:16:05.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.690+0000 7f46b7247640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f46a8045a10 0x7f46a80d5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.690+0000 7f46b7247640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46a8046480 0x7f46a81749c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.690+0000 7f46b7247640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f46a804c9b0 0x7f46a80d0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.690+0000 7f46b7247640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f46a804be70 con 0x7f46a8045a10 2024-08-31T12:16:05.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.690+0000 7f46b7247640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f46a80dbf70 con 0x7f46a8046480 2024-08-31T12:16:05.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.690+0000 7f46b7247640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f46a8045720 con 0x7f46a804c9b0 2024-08-31T12:16:05.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.691+0000 7f0591c54640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f058c0c5dc0 0x7f058c15a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.691+0000 7f540f59e640 1 Processor -- start 2024-08-31T12:16:05.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.691+0000 7f46b5a44640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46a8046480 0x7f46a81749c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.691+0000 7f46b6245640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f46a8045a10 0x7f46a80d5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.691+0000 7f46b5a44640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46a8046480 0x7f46a81749c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:47140/0 (socket says 172.21.15.19:47140) 2024-08-31T12:16:05.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.691+0000 7f46b5a44640 1 -- 172.21.15.19:0/3024971063 learned_addr learned my addr 172.21.15.19:0/3024971063 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:05.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.691+0000 7f46b84a9640 1 --2- 172.21.15.19:0/3024971063 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f46a804c9b0 0x7f46a80d0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.691+0000 7f540f59e640 1 -- start start 2024-08-31T12:16:05.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.691+0000 7ff918d17640 1 -- 172.21.15.19:0/2919716033 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff90c04b370 msgr2=0x7ff90c04b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.691+0000 7ff918d17640 1 --2- 172.21.15.19:0/2919716033 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff90c04b370 0x7ff90c04b780 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff914066480 tx=0x7ff9140a6df0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.691+0000 7ff918d17640 1 -- 172.21.15.19:0/2919716033 shutdown_connections 2024-08-31T12:16:05.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.691+0000 7ff918d17640 1 --2- 172.21.15.19:0/2919716033 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff90c04b370 0x7ff90c04b780 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.691+0000 7ff918d17640 1 --2- 172.21.15.19:0/2919716033 >> v2:172.21.15.43:3300/0 conn(0x7ff90c044d20 0x7ff90c0454b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.696 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.691+0000 7ff918d17640 1 -- 172.21.15.19:0/2919716033 >> 172.21.15.19:0/2919716033 conn(0x7ff90c04fe30 msgr2=0x7ff90c050240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:05.696 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.691+0000 7ff918d17640 1 -- 172.21.15.19:0/2919716033 shutdown_connections 2024-08-31T12:16:05.696 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.691+0000 7f46b6245640 1 -- 172.21.15.19:0/3024971063 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f46a804c9b0 msgr2=0x7f46a80d0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.696 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.691+0000 7f46b6245640 1 --2- 172.21.15.19:0/3024971063 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f46a804c9b0 0x7f46a80d0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.691+0000 7f46b6245640 1 -- 172.21.15.19:0/3024971063 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46a8046480 msgr2=0x7f46a81749c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.691+0000 7f46b6245640 1 --2- 172.21.15.19:0/3024971063 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46a8046480 0x7f46a81749c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.691+0000 7f46b6245640 1 -- 172.21.15.19:0/3024971063 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46a80d91c0 con 0x7f46a8045a10 2024-08-31T12:16:05.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.691+0000 7f46b6245640 1 --2- 172.21.15.19:0/3024971063 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f46a8045a10 0x7f46a80d5b60 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f46ac009af0 tx=0x7f46ac00d2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:05.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.692+0000 7f0591c54640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f058c0c7280 0x7f058c1e33a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.692+0000 7f0591c54640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f058c141e70 0x7f058c0cb7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.692+0000 7f0591c54640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f058c142280 con 0x7f058c0c5dc0 2024-08-31T12:16:05.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.692+0000 7f0591c54640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f058c0c87b0 con 0x7f058c0c7280 2024-08-31T12:16:05.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.692+0000 7f0591c54640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f058c15ab90 con 0x7f058c141e70 2024-08-31T12:16:05.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.692+0000 7ff918d17640 1 -- 172.21.15.19:0/2919716033 wait complete. 2024-08-31T12:16:05.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.692+0000 7f0590c52640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f058c0c5dc0 0x7f058c15a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.692+0000 7f0590c52640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f058c0c5dc0 0x7f058c15a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:56716/0 (socket says 172.21.15.19:56716) 2024-08-31T12:16:05.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.692+0000 7f0590c52640 1 -- 172.21.15.19:0/1063617660 learned_addr learned my addr 172.21.15.19:0/1063617660 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:05.699 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.693+0000 7f46a6ffd640 1 -- 172.21.15.19:0/3024971063 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46ac00dbf0 con 0x7f46a8045a10 2024-08-31T12:16:05.699 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.693+0000 7f46a6ffd640 1 -- 172.21.15.19:0/3024971063 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f46ac00dd90 con 0x7f46a8045a10 2024-08-31T12:16:05.699 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.693+0000 7f46a6ffd640 1 -- 172.21.15.19:0/3024971063 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46ac019660 con 0x7f46a8045a10 2024-08-31T12:16:05.699 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.693+0000 7f540f59e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5410071260 0x7f541014ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.699 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.693+0000 7f540f59e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5410074430 0x7f5410146170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.700 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.693+0000 7f540f59e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f541015ae00 0x7f541014f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.700 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.693+0000 7f540f59e640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f5410074840 con 0x7f541015ae00 2024-08-31T12:16:05.700 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.693+0000 7f540f59e640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f54100709d0 con 0x7f5410074430 2024-08-31T12:16:05.700 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.693+0000 7f540f59e640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f5410070b50 con 0x7f5410071260 2024-08-31T12:16:05.700 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.694+0000 7f540ed9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f541015ae00 0x7f541014f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.700 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.694+0000 7f540ed9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f541015ae00 0x7f541014f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:56724/0 (socket says 172.21.15.19:56724) 2024-08-31T12:16:05.700 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.694+0000 7f540ed9d640 1 -- 172.21.15.19:0/4233751590 learned_addr learned my addr 172.21.15.19:0/4233751590 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:05.700 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.694+0000 7f540ed9d640 1 -- 172.21.15.19:0/4233751590 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5410071260 msgr2=0x7f541014ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.700 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.694+0000 7f540ed9d640 1 --2- 172.21.15.19:0/4233751590 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5410071260 0x7f541014ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.700 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.694+0000 7f540ed9d640 1 -- 172.21.15.19:0/4233751590 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5410074430 msgr2=0x7f5410146170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.700 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.694+0000 7f540ed9d640 1 --2- 172.21.15.19:0/4233751590 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5410074430 0x7f5410146170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.694+0000 7f540ed9d640 1 -- 172.21.15.19:0/4233751590 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5410150ed0 con 0x7f541015ae00 2024-08-31T12:16:05.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.693+0000 7f0591453640 1 --2- 172.21.15.19:0/1063617660 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f058c141e70 0x7f058c0cb7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.693+0000 7f0590c52640 1 -- 172.21.15.19:0/1063617660 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f058c141e70 msgr2=0x7f058c0cb7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.693+0000 7f0590c52640 1 --2- 172.21.15.19:0/1063617660 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f058c141e70 0x7f058c0cb7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.693+0000 7f0590c52640 1 -- 172.21.15.19:0/1063617660 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f058c0c7280 msgr2=0x7f058c1e33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.693+0000 7f0590c52640 1 --2- 172.21.15.19:0/1063617660 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f058c0c7280 0x7f058c1e33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.693+0000 7f0590c52640 1 -- 172.21.15.19:0/1063617660 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f058c15a6c0 con 0x7f058c0c5dc0 2024-08-31T12:16:05.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.693+0000 7f0591453640 1 --2- 172.21.15.19:0/1063617660 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f058c141e70 0x7f058c0cb7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-31T12:16:05.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.694+0000 7f0590c52640 1 --2- 172.21.15.19:0/1063617660 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f058c0c5dc0 0x7f058c15a180 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0580009920 tx=0x7f05800045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:05.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.693+0000 7f3599a9e640 1 Processor -- start 2024-08-31T12:16:05.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.693+0000 7f3599a9e640 1 -- start start 2024-08-31T12:16:05.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.694+0000 7f9db759e640 1 -- 172.21.15.19:0/994932173 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9db00483a0 msgr2=0x7f9db0007230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.694+0000 7f9db759e640 1 --2- 172.21.15.19:0/994932173 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9db00483a0 0x7f9db0007230 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9da8009830 tx=0x7f9da802a5a0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.694+0000 7f05897fa640 1 -- 172.21.15.19:0/1063617660 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0580031600 con 0x7f058c0c5dc0 2024-08-31T12:16:05.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.694+0000 7f0591c54640 1 -- 172.21.15.19:0/1063617660 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f058c13ae90 con 0x7f058c0c5dc0 2024-08-31T12:16:05.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.694+0000 7f05897fa640 1 -- 172.21.15.19:0/1063617660 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0580031c90 con 0x7f058c0c5dc0 2024-08-31T12:16:05.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.694+0000 7f540ed9d640 1 --2- 172.21.15.19:0/4233751590 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f541015ae00 0x7f541014f740 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f540800cae0 tx=0x7f540800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:05.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.694+0000 7f0591c54640 1 -- 172.21.15.19:0/1063617660 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f058c1ef920 con 0x7f058c0c5dc0 2024-08-31T12:16:05.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.694+0000 7ff918d17640 1 Processor -- start 2024-08-31T12:16:05.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.694+0000 7f05897fa640 1 -- 172.21.15.19:0/1063617660 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0580031600 con 0x7f058c0c5dc0 2024-08-31T12:16:05.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.694+0000 7f46b7247640 1 -- 172.21.15.19:0/3024971063 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46a8174f00 con 0x7f46a8045a10 2024-08-31T12:16:05.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.694+0000 7f46b7247640 1 -- 172.21.15.19:0/3024971063 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46a80dade0 con 0x7f46a8045a10 2024-08-31T12:16:05.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.695+0000 7f9db759e640 1 -- 172.21.15.19:0/994932173 shutdown_connections 2024-08-31T12:16:05.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.695+0000 7f9db759e640 1 --2- 172.21.15.19:0/994932173 >> v2:172.21.15.43:3300/0 conn(0x7f9db00e2330 0x7f9db00e2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.695+0000 7f9db759e640 1 --2- 172.21.15.19:0/994932173 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9db00483a0 0x7f9db0007230 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.695+0000 7f9db759e640 1 -- 172.21.15.19:0/994932173 >> 172.21.15.19:0/994932173 conn(0x7f9db0050fa0 msgr2=0x7f9db00513b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:05.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.694+0000 7ff918d17640 1 -- start start 2024-08-31T12:16:05.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.695+0000 7f53feffd640 1 -- 172.21.15.19:0/4233751590 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5408016020 con 0x7f541015ae00 2024-08-31T12:16:05.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.695+0000 7f540f59e640 1 -- 172.21.15.19:0/4233751590 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54100c59c0 con 0x7f541015ae00 2024-08-31T12:16:05.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.695+0000 7f540f59e640 1 -- 172.21.15.19:0/4233751590 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54100c5c60 con 0x7f541015ae00 2024-08-31T12:16:05.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.695+0000 7f53feffd640 1 -- 172.21.15.19:0/4233751590 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f540800acd0 con 0x7f541015ae00 2024-08-31T12:16:05.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.696+0000 7f3599a9e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3594158960 0x7f35940cc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.696+0000 7f3599a9e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f35941e22c0 0x7f3594157d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.696+0000 7f3599a9e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f35941e26d0 0x7f3594158270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.696+0000 7f3599a9e640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f35940c87b0 con 0x7f3594158960 2024-08-31T12:16:05.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.695+0000 7f5ddffff640 1 -- 172.21.15.19:0/1357918330 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5de015b140 msgr2=0x7f5de00c8c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.695+0000 7f5ddffff640 1 --2- 172.21.15.19:0/1357918330 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5de015b140 0x7f5de00c8c00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5dd0009860 tx=0x7f5dd002a590 comp rx=0 tx=0).stop 2024-08-31T12:16:05.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.696+0000 7f3599a9e640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3594159880 con 0x7f35941e26d0 2024-08-31T12:16:05.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.696+0000 7f3599a9e640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f3594142280 con 0x7f35941e22c0 2024-08-31T12:16:05.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.696+0000 7f9db759e640 1 -- 172.21.15.19:0/994932173 shutdown_connections 2024-08-31T12:16:05.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.696+0000 7f53feffd640 1 -- 172.21.15.19:0/4233751590 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5408012680 con 0x7f541015ae00 2024-08-31T12:16:05.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.696+0000 7f359929d640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f35941e26d0 0x7f3594158270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.696+0000 7f359929d640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f35941e26d0 0x7f3594158270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:47188/0 (socket says 172.21.15.19:47188) 2024-08-31T12:16:05.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.696+0000 7f359929d640 1 -- 172.21.15.19:0/3490878220 learned_addr learned my addr 172.21.15.19:0/3490878220 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:05.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.696+0000 7f5ddffff640 1 -- 172.21.15.19:0/1357918330 shutdown_connections 2024-08-31T12:16:05.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.696+0000 7f5ddffff640 1 --2- 172.21.15.19:0/1357918330 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5de015b140 0x7f5de00c8c00 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.696+0000 7f5ddffff640 1 -- 172.21.15.19:0/1357918330 >> 172.21.15.19:0/1357918330 conn(0x7f5de006ee80 msgr2=0x7f5de0070360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:05.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.696+0000 7ff918d17640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff90c044d20 0x7ff90c0da500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.697+0000 7ff918d17640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff90c04b370 0x7ff90c0d1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.697+0000 7ff918d17640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff90c0df620 0x7ff90c0db080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.697+0000 7ff9137fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff90c044d20 0x7ff90c0da500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.697+0000 7f359929d640 1 -- 172.21.15.19:0/3490878220 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f35941e22c0 msgr2=0x7f3594157d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.697+0000 7f3593fff640 1 --2- 172.21.15.19:0/3490878220 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f35941e22c0 0x7f3594157d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.697+0000 7f540f59e640 1 -- 172.21.15.19:0/4233751590 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53f4005290 con 0x7f541015ae00 2024-08-31T12:16:05.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.697+0000 7f359929d640 1 --2- 172.21.15.19:0/3490878220 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f35941e22c0 0x7f3594157d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.697+0000 7f359929d640 1 -- 172.21.15.19:0/3490878220 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3594158960 msgr2=0x7f35940cc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.697+0000 7f359929d640 1 --2- 172.21.15.19:0/3490878220 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3594158960 0x7f35940cc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.697+0000 7f359929d640 1 -- 172.21.15.19:0/3490878220 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35941ef030 con 0x7f35941e26d0 2024-08-31T12:16:05.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.696+0000 7f9db759e640 1 -- 172.21.15.19:0/994932173 wait complete. 2024-08-31T12:16:05.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.697+0000 7f3593fff640 1 --2- 172.21.15.19:0/3490878220 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f35941e22c0 0x7f3594157d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-31T12:16:05.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.697+0000 7f46a4ff9640 1 -- 172.21.15.19:0/3024971063 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46a8033480 con 0x7f46a8045a10 2024-08-31T12:16:05.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.697+0000 7f5ddffff640 1 -- 172.21.15.19:0/1357918330 shutdown_connections 2024-08-31T12:16:05.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.697+0000 7f0591c54640 1 -- 172.21.15.19:0/1063617660 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f056c005290 con 0x7f058c0c5dc0 2024-08-31T12:16:05.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.697+0000 7ff9137fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff90c044d20 0x7ff90c0da500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:47190/0 (socket says 172.21.15.19:47190) 2024-08-31T12:16:05.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.697+0000 7ff912ffd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff90c04b370 0x7ff90c0d1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.697+0000 7ff912ffd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff90c04b370 0x7ff90c0d1250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.19:48972/0 (socket says 172.21.15.19:48972) 2024-08-31T12:16:05.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.698+0000 7ff9137fe640 1 -- 172.21.15.19:0/429619655 learned_addr learned my addr 172.21.15.19:0/429619655 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:05.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.698+0000 7ff918d17640 1 -- 172.21.15.19:0/429619655 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7ff90c005dd0 con 0x7ff90c0df620 2024-08-31T12:16:05.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.698+0000 7ff918d17640 1 -- 172.21.15.19:0/429619655 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff90c005c50 con 0x7ff90c044d20 2024-08-31T12:16:05.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.698+0000 7ff918d17640 1 -- 172.21.15.19:0/429619655 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7ff90c005f50 con 0x7ff90c04b370 2024-08-31T12:16:05.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.698+0000 7f359929d640 1 --2- 172.21.15.19:0/3490878220 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f35941e26d0 0x7f3594158270 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f358400c180 tx=0x7f358400c580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:05.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.698+0000 7f5ddffff640 1 -- 172.21.15.19:0/1357918330 wait complete. 2024-08-31T12:16:05.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.698+0000 7ff913fff640 1 --2- 172.21.15.19:0/429619655 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff90c0df620 0x7ff90c0db080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.699+0000 7f53feffd640 1 -- 172.21.15.19:0/4233751590 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f5408019020 con 0x7f541015ae00 2024-08-31T12:16:05.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.698+0000 7f05897fa640 1 -- 172.21.15.19:0/1063617660 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f0580005020 con 0x7f058c0c5dc0 2024-08-31T12:16:05.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.699+0000 7f05897fa640 1 --2- 172.21.15.19:0/1063617660 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0578077e50 0x7f057807a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.699+0000 7f05897fa640 1 -- 172.21.15.19:0/1063617660 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0580031400 con 0x7f058c0c5dc0 2024-08-31T12:16:05.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.699+0000 7f058bfff640 1 --2- 172.21.15.19:0/1063617660 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0578077e50 0x7f057807a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.699+0000 7f35917fa640 1 -- 172.21.15.19:0/3490878220 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3584005a80 con 0x7f35941e26d0 2024-08-31T12:16:05.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.699+0000 7f35917fa640 1 -- 172.21.15.19:0/3490878220 <== mon.1 v2:172.21.15.43:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3584005d30 con 0x7f35941e26d0 2024-08-31T12:16:05.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.698+0000 7ff913fff640 1 -- 172.21.15.19:0/429619655 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff90c04b370 msgr2=0x7ff90c0d1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.698+0000 7ff913fff640 1 --2- 172.21.15.19:0/429619655 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff90c04b370 0x7ff90c0d1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.698+0000 7ff913fff640 1 -- 172.21.15.19:0/429619655 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff90c044d20 msgr2=0x7ff90c0da500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.698+0000 7ff913fff640 1 --2- 172.21.15.19:0/429619655 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff90c044d20 0x7ff90c0da500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.698+0000 7ff913fff640 1 -- 172.21.15.19:0/429619655 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff90c17a340 con 0x7ff90c0df620 2024-08-31T12:16:05.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.699+0000 7ff913fff640 1 --2- 172.21.15.19:0/429619655 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff90c0df620 0x7ff90c0db080 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff914000b60 tx=0x7ff9140ae5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:05.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.699+0000 7ff9137fe640 1 --2- 172.21.15.19:0/429619655 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff90c044d20 0x7ff90c0da500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-31T12:16:05.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.699+0000 7f9db759e640 1 Processor -- start 2024-08-31T12:16:05.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.699+0000 7f9db759e640 1 -- start start 2024-08-31T12:16:05.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.700+0000 7f14bed39640 1 -- 172.21.15.19:0/1762829416 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f14a0003a50 msgr2=0x7f14a0003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.700+0000 7f14bed39640 1 --2- 172.21.15.19:0/1762829416 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f14a0003a50 0x7f14a0003ec0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f14b000c640 tx=0x7f14a0005830 comp rx=0 tx=0).stop 2024-08-31T12:16:05.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.700+0000 7f53feffd640 1 --2- 172.21.15.19:0/4233751590 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f53f0077b50 0x7f53f007a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.699+0000 7f46a6ffd640 1 -- 172.21.15.19:0/3024971063 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f46ac019800 con 0x7f46a8045a10 2024-08-31T12:16:05.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.700+0000 7f46a6ffd640 1 --2- 172.21.15.19:0/3024971063 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4694077b20 0x7f469407a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.700+0000 7f46a6ffd640 1 -- 172.21.15.19:0/3024971063 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f46ac099d30 con 0x7f46a8045a10 2024-08-31T12:16:05.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.700+0000 7f35917fa640 1 -- 172.21.15.19:0/3490878220 <== mon.1 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3584007290 con 0x7f35941e26d0 2024-08-31T12:16:05.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.700+0000 7f3599a9e640 1 -- 172.21.15.19:0/3490878220 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35941ef310 con 0x7f35941e26d0 2024-08-31T12:16:05.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.700+0000 7f53feffd640 1 -- 172.21.15.19:0/4233751590 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5408099f30 con 0x7f541015ae00 2024-08-31T12:16:05.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.700+0000 7f3599a9e640 1 -- 172.21.15.19:0/3490878220 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35941ef8d0 con 0x7f35941e26d0 2024-08-31T12:16:05.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.700+0000 7f058bfff640 1 --2- 172.21.15.19:0/1063617660 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0578077e50 0x7f057807a360 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f057c006fd0 tx=0x7f057c006bc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:05.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.700+0000 7f540e59c640 1 --2- 172.21.15.19:0/4233751590 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f53f0077b50 0x7f53f007a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.700+0000 7f14bd2d6640 1 -- 172.21.15.19:0/1762829416 reap_dead start 2024-08-31T12:16:05.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.700+0000 7f14bed39640 1 -- 172.21.15.19:0/1762829416 shutdown_connections 2024-08-31T12:16:05.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.700+0000 7f14bed39640 1 -- 172.21.15.19:0/1762829416 >> 172.21.15.19:0/1762829416 conn(0x7f14b80cc950 msgr2=0x7f14b80ccd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:05.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.701+0000 7f14bed39640 1 -- 172.21.15.19:0/1762829416 shutdown_connections 2024-08-31T12:16:05.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.701+0000 7f46b5a44640 1 --2- 172.21.15.19:0/3024971063 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4694077b20 0x7f469407a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.701+0000 7f14bed39640 1 -- 172.21.15.19:0/1762829416 wait complete. 2024-08-31T12:16:05.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.701+0000 7f5ddffff640 1 Processor -- start 2024-08-31T12:16:05.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.701+0000 7f5ddffff640 1 -- start start 2024-08-31T12:16:05.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.701+0000 7f540e59c640 1 --2- 172.21.15.19:0/4233751590 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f53f0077b50 0x7f53f007a060 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f54000058c0 tx=0x7f5400005850 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:05.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.701+0000 7f46b5a44640 1 --2- 172.21.15.19:0/3024971063 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4694077b20 0x7f469407a030 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f46a000a7d0 tx=0x7f46a0004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:05.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.702+0000 7f3599a9e640 1 -- 172.21.15.19:0/3490878220 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f359415d060 con 0x7f35941e26d0 2024-08-31T12:16:05.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.702+0000 7f9db759e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9db00483a0 0x7f9db0044a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.702+0000 7f9db759e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9db00dfa30 0x7f9db01694d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.702+0000 7f9db759e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9db00e2330 0x7f9db0175720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.702+0000 7f9db659c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9db00483a0 0x7f9db0044a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.702+0000 7f9db659c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9db00483a0 0x7f9db0044a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:56762/0 (socket says 172.21.15.19:56762) 2024-08-31T12:16:05.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.702+0000 7f9db5d9b640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9db00dfa30 0x7f9db01694d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.702+0000 7f9db5d9b640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9db00dfa30 0x7f9db01694d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:47194/0 (socket says 172.21.15.19:47194) 2024-08-31T12:16:05.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.702+0000 7f9db659c640 1 -- 172.21.15.19:0/1734715190 learned_addr learned my addr 172.21.15.19:0/1734715190 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:05.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.702+0000 7f9db6d9d640 1 --2- 172.21.15.19:0/1734715190 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9db00e2330 0x7f9db0175720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.702+0000 7f9db759e640 1 -- 172.21.15.19:0/1734715190 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f9db004a9f0 con 0x7f9db00483a0 2024-08-31T12:16:05.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.702+0000 7f9db759e640 1 -- 172.21.15.19:0/1734715190 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9db00df510 con 0x7f9db00dfa30 2024-08-31T12:16:05.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.702+0000 7f9db759e640 1 -- 172.21.15.19:0/1734715190 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f9db004ab70 con 0x7f9db00e2330 2024-08-31T12:16:05.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.703+0000 7f9db659c640 1 -- 172.21.15.19:0/1734715190 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9db00e2330 msgr2=0x7f9db0175720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.703+0000 7f9db659c640 1 --2- 172.21.15.19:0/1734715190 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9db00e2330 0x7f9db0175720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.703+0000 7f9db659c640 1 -- 172.21.15.19:0/1734715190 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9db00dfa30 msgr2=0x7f9db01694d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.703+0000 7f14bed39640 1 Processor -- start 2024-08-31T12:16:05.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.703+0000 7f9db659c640 1 --2- 172.21.15.19:0/1734715190 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9db00dfa30 0x7f9db01694d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.703+0000 7f9db659c640 1 -- 172.21.15.19:0/1734715190 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9db0178e50 con 0x7f9db00483a0 2024-08-31T12:16:05.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.703+0000 7f35917fa640 1 -- 172.21.15.19:0/3490878220 <== mon.1 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f3584007430 con 0x7f35941e26d0 2024-08-31T12:16:05.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.703+0000 7f9db659c640 1 --2- 172.21.15.19:0/1734715190 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9db00483a0 0x7f9db0044a40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9dac00c430 tx=0x7f9dac00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:05.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.703+0000 7f5ddffff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5de006cae0 0x7f5de0154800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.703+0000 7f5ddffff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5de00c6b00 0x7f5de00cc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.727 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.703+0000 7f14bed39640 1 -- start start 2024-08-31T12:16:05.727 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.703+0000 7f5ddffff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5de015b140 0x7f5de00cc610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.727 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.703+0000 7f5ddeffd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5de006cae0 0x7f5de0154800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.703+0000 7f35917fa640 1 --2- 172.21.15.19:0/3490878220 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3580077ad0 0x7f3580079fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.703+0000 7f5ddffff640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f5de00c92c0 con 0x7f5de006cae0 2024-08-31T12:16:05.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.703+0000 7f5ddeffd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5de006cae0 0x7f5de0154800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:56764/0 (socket says 172.21.15.19:56764) 2024-08-31T12:16:05.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.704+0000 7f5ddeffd640 1 -- 172.21.15.19:0/1451091092 learned_addr learned my addr 172.21.15.19:0/1451091092 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:05.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.704+0000 7f5dde7fc640 1 --2- 172.21.15.19:0/1451091092 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5de00c6b00 0x7f5de00cc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.704+0000 7f5ddffff640 1 -- 172.21.15.19:0/1451091092 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5de00c9140 con 0x7f5de015b140 2024-08-31T12:16:05.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.704+0000 7f5ddffff640 1 -- 172.21.15.19:0/1451091092 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f5de00c4770 con 0x7f5de00c6b00 2024-08-31T12:16:05.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.704+0000 7f35917fa640 1 -- 172.21.15.19:0/3490878220 <== mon.1 v2:172.21.15.43:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3584098740 con 0x7f35941e26d0 2024-08-31T12:16:05.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.704+0000 7f5ddf7fe640 1 --2- 172.21.15.19:0/1451091092 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5de015b140 0x7f5de00cc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.704+0000 7f3598a9c640 1 --2- 172.21.15.19:0/3490878220 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3580077ad0 0x7f3580079fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.704+0000 7f5dde7fc640 1 -- 172.21.15.19:0/1451091092 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5de015b140 msgr2=0x7f5de00cc610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.704+0000 7f5dde7fc640 1 --2- 172.21.15.19:0/1451091092 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5de015b140 0x7f5de00cc610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.704+0000 7f5dde7fc640 1 -- 172.21.15.19:0/1451091092 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5de006cae0 msgr2=0x7f5de0154800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.704+0000 7f5dde7fc640 1 --2- 172.21.15.19:0/1451091092 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5de006cae0 0x7f5de0154800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.704+0000 7f5dde7fc640 1 -- 172.21.15.19:0/1451091092 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5de0154d40 con 0x7f5de00c6b00 2024-08-31T12:16:05.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.704+0000 7f5ddeffd640 1 --2- 172.21.15.19:0/1451091092 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5de006cae0 0x7f5de0154800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:16:05.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.704+0000 7f5dde7fc640 1 --2- 172.21.15.19:0/1451091092 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5de00c6b00 0x7f5de00cc0d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5de0073f40 tx=0x7f5dd0033ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:05.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.704+0000 7f3598a9c640 1 --2- 172.21.15.19:0/3490878220 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3580077ad0 0x7f3580079fe0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f358c033000 tx=0x7f358c001d70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:05.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.704+0000 7f5dc77fe640 1 -- 172.21.15.19:0/1451091092 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd0031b50 con 0x7f5de00c6b00 2024-08-31T12:16:05.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.705+0000 7f5dc77fe640 1 -- 172.21.15.19:0/1451091092 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5dd0031de0 con 0x7f5de00c6b00 2024-08-31T12:16:05.732 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.705+0000 7f5dc77fe640 1 -- 172.21.15.19:0/1451091092 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd0005280 con 0x7f5de00c6b00 2024-08-31T12:16:05.732 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.705+0000 7f5ddffff640 1 -- 172.21.15.19:0/1451091092 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5de0153800 con 0x7f5de00c6b00 2024-08-31T12:16:05.732 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.705+0000 7f5ddffff640 1 -- 172.21.15.19:0/1451091092 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5de0157c70 con 0x7f5de00c6b00 2024-08-31T12:16:05.732 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.705+0000 7f14bed39640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f14b81521e0 0x7f14b8159b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.732 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.705+0000 7f14bed39640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14b814a210 0x7f14b814fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.732 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.705+0000 7f14bed39640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f14b814a620 0x7f14b8150400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.732 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.705+0000 7f14bed39640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f14b81515e0 con 0x7f14b81521e0 2024-08-31T12:16:05.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.705+0000 7f14bed39640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f14b8151460 con 0x7f14b814a210 2024-08-31T12:16:05.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.705+0000 7f14bed39640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f14b8073db0 con 0x7f14b814a620 2024-08-31T12:16:05.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.705+0000 7f14bcad5640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f14b81521e0 0x7f14b8159b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.705+0000 7f14bcad5640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f14b81521e0 0x7f14b8159b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:56766/0 (socket says 172.21.15.19:56766) 2024-08-31T12:16:05.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.705+0000 7f14bcad5640 1 -- 172.21.15.19:0/586451489 learned_addr learned my addr 172.21.15.19:0/586451489 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:05.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.705+0000 7f14b7fff640 1 --2- 172.21.15.19:0/586451489 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14b814a210 0x7f14b814fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.705+0000 7f14bd2d6640 1 --2- 172.21.15.19:0/586451489 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f14b814a620 0x7f14b8150400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.705+0000 7f14bcad5640 1 -- 172.21.15.19:0/586451489 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f14b814a620 msgr2=0x7f14b8150400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.705+0000 7f14bcad5640 1 --2- 172.21.15.19:0/586451489 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f14b814a620 0x7f14b8150400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.705+0000 7f14bcad5640 1 -- 172.21.15.19:0/586451489 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14b814a210 msgr2=0x7f14b814fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:05.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.705+0000 7f14bcad5640 1 --2- 172.21.15.19:0/586451489 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14b814a210 0x7f14b814fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:05.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.705+0000 7f14bcad5640 1 -- 172.21.15.19:0/586451489 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14b8152c30 con 0x7f14b81521e0 2024-08-31T12:16:05.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.705+0000 7f14b7fff640 1 --2- 172.21.15.19:0/586451489 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14b814a210 0x7f14b814fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:16:05.734 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.705+0000 7f14bcad5640 1 --2- 172.21.15.19:0/586451489 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f14b81521e0 0x7f14b8159b10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f14b000c760 tx=0x7f14b0004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:05.734 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.706+0000 7f5ddffff640 1 -- 172.21.15.19:0/1451091092 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5de00c3600 con 0x7f5de00c6b00 2024-08-31T12:16:05.734 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.707+0000 7f53feffd640 1 -- 172.21.15.19:0/4233751590 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f5408020030 con 0x7f541015ae00 2024-08-31T12:16:05.735 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.708+0000 7f5dc77fe640 1 -- 172.21.15.19:0/1451091092 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f5dd0065b80 con 0x7f5de00c6b00 2024-08-31T12:16:05.735 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.708+0000 7f5dc77fe640 1 --2- 172.21.15.19:0/1451091092 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5dcc077a50 0x7f5dcc079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.735 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.708+0000 7f5dc77fe640 1 -- 172.21.15.19:0/1451091092 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5dd00bab30 con 0x7f5de00c6b00 2024-08-31T12:16:05.735 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.708+0000 7f5ddeffd640 1 --2- 172.21.15.19:0/1451091092 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5dcc077a50 0x7f5dcc079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.735 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.709+0000 7f5ddeffd640 1 --2- 172.21.15.19:0/1451091092 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5dcc077a50 0x7f5dcc079f60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5dd8001d40 tx=0x7f5dd8001cd0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:05.735 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.709+0000 7f35917fa640 1 -- 172.21.15.19:0/3490878220 <== mon.1 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f359415d060 con 0x7f35941e26d0 2024-08-31T12:16:05.736 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.712+0000 7f5dc77fe640 1 -- 172.21.15.19:0/1451091092 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f5dd003f030 con 0x7f5de00c6b00 2024-08-31T12:16:05.736 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.713+0000 7f46a6ffd640 1 -- 172.21.15.19:0/3024971063 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f46ac015030 con 0x7f46a8045a10 2024-08-31T12:16:05.736 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.722+0000 7f05897fa640 1 -- 172.21.15.19:0/1063617660 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f058003f030 con 0x7f058c0c5dc0 2024-08-31T12:16:05.737 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.722+0000 7ff8fffff640 1 -- 172.21.15.19:0/429619655 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9140b4040 con 0x7ff90c0df620 2024-08-31T12:16:05.737 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.722+0000 7ff8fffff640 1 -- 172.21.15.19:0/429619655 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff9140bd070 con 0x7ff90c0df620 2024-08-31T12:16:05.737 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.722+0000 7ff8fffff640 1 -- 172.21.15.19:0/429619655 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9140ae960 con 0x7ff90c0df620 2024-08-31T12:16:05.737 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.722+0000 7ff918d17640 1 -- 172.21.15.19:0/429619655 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff90c17a5e0 con 0x7ff90c0df620 2024-08-31T12:16:05.737 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.722+0000 7ff918d17640 1 -- 172.21.15.19:0/429619655 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff90c0d9770 con 0x7ff90c0df620 2024-08-31T12:16:05.737 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.722+0000 7f9da6ffd640 1 -- 172.21.15.19:0/1734715190 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dac011790 con 0x7f9db00483a0 2024-08-31T12:16:05.737 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.722+0000 7f9db759e640 1 -- 172.21.15.19:0/1734715190 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9db0179130 con 0x7f9db00483a0 2024-08-31T12:16:05.737 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.722+0000 7f9da6ffd640 1 -- 172.21.15.19:0/1734715190 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9dac017070 con 0x7f9db00483a0 2024-08-31T12:16:05.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.722+0000 7f14b57fa640 1 -- 172.21.15.19:0/586451489 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b002bb70 con 0x7f14b81521e0 2024-08-31T12:16:05.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.722+0000 7f14b57fa640 1 -- 172.21.15.19:0/586451489 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f14b002bd10 con 0x7f14b81521e0 2024-08-31T12:16:05.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.722+0000 7f14bed39640 1 -- 172.21.15.19:0/586451489 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14b8152f10 con 0x7f14b81521e0 2024-08-31T12:16:05.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.723+0000 7f14b57fa640 1 -- 172.21.15.19:0/586451489 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b003c560 con 0x7f14b81521e0 2024-08-31T12:16:05.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.723+0000 7f14bed39640 1 -- 172.21.15.19:0/586451489 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14b8158630 con 0x7f14b81521e0 2024-08-31T12:16:05.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.723+0000 7ff8fdffb640 1 -- 172.21.15.19:0/429619655 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8ec005290 con 0x7ff90c0df620 2024-08-31T12:16:05.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.723+0000 7f9da6ffd640 1 -- 172.21.15.19:0/1734715190 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dac011bf0 con 0x7f9db00483a0 2024-08-31T12:16:05.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.723+0000 7f9db759e640 1 -- 172.21.15.19:0/1734715190 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9db0169a10 con 0x7f9db00483a0 2024-08-31T12:16:05.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.725+0000 7f149effd640 1 -- 172.21.15.19:0/586451489 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14b80cc080 con 0x7f14b81521e0 2024-08-31T12:16:05.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.727+0000 7ff8fffff640 1 -- 172.21.15.19:0/429619655 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7ff9140aeb00 con 0x7ff90c0df620 2024-08-31T12:16:05.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.725+0000 7f9db759e640 1 -- 172.21.15.19:0/1734715190 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9db00e2740 con 0x7f9db00483a0 2024-08-31T12:16:05.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.728+0000 7ff8fffff640 1 --2- 172.21.15.19:0/429619655 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff8f8077b50 0x7ff8f807a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.729+0000 7ff8fffff640 1 -- 172.21.15.19:0/429619655 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff9141369b0 con 0x7ff90c0df620 2024-08-31T12:16:05.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.729+0000 7f14b57fa640 1 -- 172.21.15.19:0/586451489 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f14b003c700 con 0x7f14b81521e0 2024-08-31T12:16:05.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.730+0000 7f14b57fa640 1 --2- 172.21.15.19:0/586451489 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f14a4077a80 0x7f14a4079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.730+0000 7f14b57fa640 1 -- 172.21.15.19:0/586451489 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f14b00b9c90 con 0x7f14b81521e0 2024-08-31T12:16:05.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.730+0000 7f14b7fff640 1 --2- 172.21.15.19:0/586451489 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f14a4077a80 0x7f14a4079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.731+0000 7ff9137fe640 1 --2- 172.21.15.19:0/429619655 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff8f8077b50 0x7ff8f807a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.730+0000 7f9da6ffd640 1 -- 172.21.15.19:0/1734715190 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f9dac012420 con 0x7f9db00483a0 2024-08-31T12:16:05.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.731+0000 7f9da6ffd640 1 --2- 172.21.15.19:0/1734715190 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9d98077a80 0x7f9d98079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:05.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.731+0000 7f14b7fff640 1 --2- 172.21.15.19:0/586451489 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f14a4077a80 0x7f14a4079f90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f14ac005eb0 tx=0x7f14ac00f040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:05.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.732+0000 7f9da6ffd640 1 -- 172.21.15.19:0/1734715190 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9dac099c70 con 0x7f9db00483a0 2024-08-31T12:16:05.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.732+0000 7ff9137fe640 1 --2- 172.21.15.19:0/429619655 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff8f8077b50 0x7ff8f807a060 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff9080058c0 tx=0x7ff9080092a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:05.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.732+0000 7f9db5d9b640 1 --2- 172.21.15.19:0/1734715190 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9d98077a80 0x7f9d98079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:05.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.733+0000 7f9db5d9b640 1 --2- 172.21.15.19:0/1734715190 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9d98077a80 0x7f9d98079f90 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9da8034000 tx=0x7f9da8004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:05.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.737+0000 7ff8fffff640 1 -- 172.21.15.19:0/429619655 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7ff9140ff790 con 0x7ff90c0df620 2024-08-31T12:16:05.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.746+0000 7f9da6ffd640 1 -- 172.21.15.19:0/1734715190 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f9dac062400 con 0x7f9db00483a0 2024-08-31T12:16:05.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:05.756+0000 7f14b57fa640 1 -- 172.21.15.19:0/586451489 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f14b80cc080 con 0x7f14b81521e0 2024-08-31T12:16:06.094 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.093+0000 7f540f59e640 1 -- 172.21.15.19:0/4233751590 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f53f4005850 con 0x7f541015ae00 2024-08-31T12:16:06.095 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.094+0000 7f53feffd640 1 -- 172.21.15.19:0/4233751590 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5408062d10 con 0x7f541015ae00 2024-08-31T12:16:06.095 INFO:teuthology.orchestra.run.smithi019.stdout:55834574854 2024-08-31T12:16:06.098 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.097+0000 7f53fcff9640 1 -- 172.21.15.19:0/4233751590 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f53f0077b50 msgr2=0x7f53f007a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:06.098 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.097+0000 7f53fcff9640 1 --2- 172.21.15.19:0/4233751590 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f53f0077b50 0x7f53f007a060 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f54000058c0 tx=0x7f5400005850 comp rx=0 tx=0).stop 2024-08-31T12:16:06.098 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.098+0000 7f53fcff9640 1 -- 172.21.15.19:0/4233751590 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f541015ae00 msgr2=0x7f541014f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:06.098 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.098+0000 7f53fcff9640 1 --2- 172.21.15.19:0/4233751590 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f541015ae00 0x7f541014f740 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f540800cae0 tx=0x7f540800a060 comp rx=0 tx=0).stop 2024-08-31T12:16:06.099 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.098+0000 7f53fcff9640 1 -- 172.21.15.19:0/4233751590 shutdown_connections 2024-08-31T12:16:06.100 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.098+0000 7f53fcff9640 1 --2- 172.21.15.19:0/4233751590 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f53f0077b50 0x7f53f007a060 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:06.100 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.098+0000 7f53fcff9640 1 --2- 172.21.15.19:0/4233751590 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f541015ae00 0x7f541014f740 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:06.100 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.098+0000 7f53fcff9640 1 --2- 172.21.15.19:0/4233751590 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5410074430 0x7f5410146170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:06.100 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.098+0000 7f53fcff9640 1 --2- 172.21.15.19:0/4233751590 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5410071260 0x7f541014ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:06.100 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.098+0000 7f53fcff9640 1 -- 172.21.15.19:0/4233751590 >> 172.21.15.19:0/4233751590 conn(0x7f54100cc540 msgr2=0x7f5410158a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:06.100 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.099+0000 7f53fcff9640 1 -- 172.21.15.19:0/4233751590 shutdown_connections 2024-08-31T12:16:06.100 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.099+0000 7f53fcff9640 1 -- 172.21.15.19:0/4233751590 wait complete. 2024-08-31T12:16:06.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.114+0000 7f3599a9e640 1 -- 172.21.15.19:0/3490878220 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f3594073fb0 con 0x7f35941e26d0 2024-08-31T12:16:06.115 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.114+0000 7f35917fa640 1 -- 172.21.15.19:0/3490878220 <== mon.1 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3594073fb0 con 0x7f35941e26d0 2024-08-31T12:16:06.117 INFO:teuthology.orchestra.run.smithi019.stdout:60129542151 2024-08-31T12:16:06.117 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.2 2024-08-31T12:16:06.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.118+0000 7f3599a9e640 1 -- 172.21.15.19:0/3490878220 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3580077ad0 msgr2=0x7f3580079fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:06.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.118+0000 7f3599a9e640 1 --2- 172.21.15.19:0/3490878220 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3580077ad0 0x7f3580079fe0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f358c033000 tx=0x7f358c001d70 comp rx=0 tx=0).stop 2024-08-31T12:16:06.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.118+0000 7f3599a9e640 1 -- 172.21.15.19:0/3490878220 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f35941e26d0 msgr2=0x7f3594158270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:06.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.118+0000 7f3599a9e640 1 --2- 172.21.15.19:0/3490878220 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f35941e26d0 0x7f3594158270 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f358400c180 tx=0x7f358400c580 comp rx=0 tx=0).stop 2024-08-31T12:16:06.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.118+0000 7f3599a9e640 1 -- 172.21.15.19:0/3490878220 shutdown_connections 2024-08-31T12:16:06.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.118+0000 7f3599a9e640 1 --2- 172.21.15.19:0/3490878220 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3580077ad0 0x7f3580079fe0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:06.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.118+0000 7f3599a9e640 1 --2- 172.21.15.19:0/3490878220 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f35941e26d0 0x7f3594158270 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:06.120 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.118+0000 7f3599a9e640 1 --2- 172.21.15.19:0/3490878220 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f35941e22c0 0x7f3594157d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:06.120 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.118+0000 7f3599a9e640 1 --2- 172.21.15.19:0/3490878220 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3594158960 0x7f35940cc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:06.120 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.118+0000 7f3599a9e640 1 -- 172.21.15.19:0/3490878220 >> 172.21.15.19:0/3490878220 conn(0x7f359406cae0 msgr2=0x7f35940ca100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:06.120 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.118+0000 7f3599a9e640 1 -- 172.21.15.19:0/3490878220 shutdown_connections 2024-08-31T12:16:06.120 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.118+0000 7f3599a9e640 1 -- 172.21.15.19:0/3490878220 wait complete. 2024-08-31T12:16:06.134 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.134+0000 7f5ddffff640 1 -- 172.21.15.19:0/1451091092 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f5de0151760 con 0x7f5de00c6b00 2024-08-31T12:16:06.135 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.134+0000 7f46a4ff9640 1 -- 172.21.15.19:0/3024971063 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f46a80d8ee0 con 0x7f46a8045a10 2024-08-31T12:16:06.135 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.134+0000 7f5dc77fe640 1 -- 172.21.15.19:0/1451091092 <== mon.2 v2:172.21.15.43:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5dd0031770 con 0x7f5de00c6b00 2024-08-31T12:16:06.136 INFO:teuthology.orchestra.run.smithi019.stdout:60129542149 2024-08-31T12:16:06.137 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542151 for osd.5 2024-08-31T12:16:06.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.135+0000 7f46a6ffd640 1 -- 172.21.15.19:0/3024971063 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f46ac062b10 con 0x7f46a8045a10 2024-08-31T12:16:06.137 INFO:teuthology.orchestra.run.smithi019.stdout:64424509446 2024-08-31T12:16:06.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.138+0000 7f5ddffff640 1 -- 172.21.15.19:0/1451091092 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5dcc077a50 msgr2=0x7f5dcc079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:06.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.138+0000 7f5ddffff640 1 --2- 172.21.15.19:0/1451091092 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5dcc077a50 0x7f5dcc079f60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5dd8001d40 tx=0x7f5dd8001cd0 comp rx=0 tx=0).stop 2024-08-31T12:16:06.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.138+0000 7f5ddffff640 1 -- 172.21.15.19:0/1451091092 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5de00c6b00 msgr2=0x7f5de00cc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:06.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.138+0000 7f5ddffff640 1 --2- 172.21.15.19:0/1451091092 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5de00c6b00 0x7f5de00cc0d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5de0073f40 tx=0x7f5dd0033ba0 comp rx=0 tx=0).stop 2024-08-31T12:16:06.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.138+0000 7f5ddffff640 1 -- 172.21.15.19:0/1451091092 shutdown_connections 2024-08-31T12:16:06.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.138+0000 7f5ddffff640 1 --2- 172.21.15.19:0/1451091092 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5dcc077a50 0x7f5dcc079f60 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:06.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.138+0000 7f5ddffff640 1 --2- 172.21.15.19:0/1451091092 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5de015b140 0x7f5de00cc610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:06.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.138+0000 7f5ddffff640 1 --2- 172.21.15.19:0/1451091092 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5de00c6b00 0x7f5de00cc0d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:06.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.138+0000 7f5ddffff640 1 --2- 172.21.15.19:0/1451091092 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5de006cae0 0x7f5de0154800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:06.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.138+0000 7f5ddffff640 1 -- 172.21.15.19:0/1451091092 >> 172.21.15.19:0/1451091092 conn(0x7f5de006ee80 msgr2=0x7f5de014cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:06.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.138+0000 7f46a4ff9640 1 -- 172.21.15.19:0/3024971063 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4694077b20 msgr2=0x7f469407a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:06.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.138+0000 7f46a4ff9640 1 --2- 172.21.15.19:0/3024971063 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4694077b20 0x7f469407a030 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f46a000a7d0 tx=0x7f46a0004730 comp rx=0 tx=0).stop 2024-08-31T12:16:06.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.138+0000 7f46a4ff9640 1 -- 172.21.15.19:0/3024971063 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f46a8045a10 msgr2=0x7f46a80d5b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:06.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.138+0000 7f46a4ff9640 1 --2- 172.21.15.19:0/3024971063 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f46a8045a10 0x7f46a80d5b60 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f46ac009af0 tx=0x7f46ac00d2e0 comp rx=0 tx=0).stop 2024-08-31T12:16:06.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.138+0000 7f46a4ff9640 1 -- 172.21.15.19:0/3024971063 shutdown_connections 2024-08-31T12:16:06.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.138+0000 7f46a4ff9640 1 --2- 172.21.15.19:0/3024971063 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4694077b20 0x7f469407a030 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:06.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.138+0000 7f46a4ff9640 1 --2- 172.21.15.19:0/3024971063 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f46a804c9b0 0x7f46a80d0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:06.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.138+0000 7f46a4ff9640 1 --2- 172.21.15.19:0/3024971063 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46a8046480 0x7f46a81749c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:06.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.138+0000 7f46a4ff9640 1 --2- 172.21.15.19:0/3024971063 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f46a8045a10 0x7f46a80d5b60 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:06.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.138+0000 7f46a4ff9640 1 -- 172.21.15.19:0/3024971063 >> 172.21.15.19:0/3024971063 conn(0x7f46a80508a0 msgr2=0x7f46a8050cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:06.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.138+0000 7f5ddffff640 1 -- 172.21.15.19:0/1451091092 shutdown_connections 2024-08-31T12:16:06.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.138+0000 7f5ddffff640 1 -- 172.21.15.19:0/1451091092 wait complete. 2024-08-31T12:16:06.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.138+0000 7f46a4ff9640 1 -- 172.21.15.19:0/3024971063 shutdown_connections 2024-08-31T12:16:06.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.138+0000 7f46a4ff9640 1 -- 172.21.15.19:0/3024971063 wait complete. 2024-08-31T12:16:06.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.146+0000 7ff8fdffb640 1 -- 172.21.15.19:0/429619655 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7ff8ec005850 con 0x7ff90c0df620 2024-08-31T12:16:06.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.146+0000 7ff8fffff640 1 -- 172.21.15.19:0/429619655 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff9140ff090 con 0x7ff90c0df620 2024-08-31T12:16:06.147 INFO:teuthology.orchestra.run.smithi019.stdout:64424509446 2024-08-31T12:16:06.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.146+0000 7f149effd640 1 -- 172.21.15.19:0/586451489 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f14b8072b80 con 0x7f14b81521e0 2024-08-31T12:16:06.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.147+0000 7f14b57fa640 1 -- 172.21.15.19:0/586451489 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f14b0041030 con 0x7f14b81521e0 2024-08-31T12:16:06.148 INFO:teuthology.orchestra.run.smithi019.stdout:64424509446 2024-08-31T12:16:06.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.149+0000 7ff8fdffb640 1 -- 172.21.15.19:0/429619655 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff8f8077b50 msgr2=0x7ff8f807a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:06.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.149+0000 7ff8fdffb640 1 --2- 172.21.15.19:0/429619655 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff8f8077b50 0x7ff8f807a060 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff9080058c0 tx=0x7ff9080092a0 comp rx=0 tx=0).stop 2024-08-31T12:16:06.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.149+0000 7ff8fdffb640 1 -- 172.21.15.19:0/429619655 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff90c0df620 msgr2=0x7ff90c0db080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:06.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.149+0000 7ff8fdffb640 1 --2- 172.21.15.19:0/429619655 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff90c0df620 0x7ff90c0db080 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff914000b60 tx=0x7ff9140ae5d0 comp rx=0 tx=0).stop 2024-08-31T12:16:06.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.149+0000 7ff8fdffb640 1 -- 172.21.15.19:0/429619655 shutdown_connections 2024-08-31T12:16:06.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.149+0000 7ff8fdffb640 1 --2- 172.21.15.19:0/429619655 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff8f8077b50 0x7ff8f807a060 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:06.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.149+0000 7ff8fdffb640 1 --2- 172.21.15.19:0/429619655 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff90c0df620 0x7ff90c0db080 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:06.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.149+0000 7ff8fdffb640 1 --2- 172.21.15.19:0/429619655 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff90c04b370 0x7ff90c0d1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:06.151 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.149+0000 7ff8fdffb640 1 --2- 172.21.15.19:0/429619655 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff90c044d20 0x7ff90c0da500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:06.151 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.149+0000 7ff8fdffb640 1 -- 172.21.15.19:0/429619655 >> 172.21.15.19:0/429619655 conn(0x7ff90c04fe30 msgr2=0x7ff90c0054b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:06.151 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.149+0000 7ff8fdffb640 1 -- 172.21.15.19:0/429619655 shutdown_connections 2024-08-31T12:16:06.151 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.149+0000 7ff8fdffb640 1 -- 172.21.15.19:0/429619655 wait complete. 2024-08-31T12:16:06.151 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.149+0000 7f149effd640 1 -- 172.21.15.19:0/586451489 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f14a4077a80 msgr2=0x7f14a4079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:06.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.149+0000 7f149effd640 1 --2- 172.21.15.19:0/586451489 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f14a4077a80 0x7f14a4079f90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f14ac005eb0 tx=0x7f14ac00f040 comp rx=0 tx=0).stop 2024-08-31T12:16:06.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.149+0000 7f149effd640 1 -- 172.21.15.19:0/586451489 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f14b81521e0 msgr2=0x7f14b8159b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:06.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.149+0000 7f149effd640 1 --2- 172.21.15.19:0/586451489 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f14b81521e0 0x7f14b8159b10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f14b000c760 tx=0x7f14b0004d10 comp rx=0 tx=0).stop 2024-08-31T12:16:06.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.150+0000 7f149effd640 1 -- 172.21.15.19:0/586451489 shutdown_connections 2024-08-31T12:16:06.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.150+0000 7f149effd640 1 --2- 172.21.15.19:0/586451489 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f14a4077a80 0x7f14a4079f90 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:06.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.150+0000 7f149effd640 1 --2- 172.21.15.19:0/586451489 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f14b814a620 0x7f14b8150400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:06.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.150+0000 7f149effd640 1 --2- 172.21.15.19:0/586451489 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14b814a210 0x7f14b814fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:06.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.150+0000 7f149effd640 1 --2- 172.21.15.19:0/586451489 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f14b81521e0 0x7f14b8159b10 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:06.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.150+0000 7f149effd640 1 -- 172.21.15.19:0/586451489 >> 172.21.15.19:0/586451489 conn(0x7f14b80cc950 msgr2=0x7f14b814c9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:06.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.150+0000 7f149effd640 1 -- 172.21.15.19:0/586451489 shutdown_connections 2024-08-31T12:16:06.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.150+0000 7f149effd640 1 -- 172.21.15.19:0/586451489 wait complete. 2024-08-31T12:16:06.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.151+0000 7f0591c54640 1 -- 172.21.15.19:0/1063617660 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f056c005850 con 0x7f058c0c5dc0 2024-08-31T12:16:06.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.152+0000 7f05897fa640 1 -- 172.21.15.19:0/1063617660 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f058003f030 con 0x7f058c0c5dc0 2024-08-31T12:16:06.153 INFO:teuthology.orchestra.run.smithi019.stdout:64424509445 2024-08-31T12:16:06.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.155+0000 7f0591c54640 1 -- 172.21.15.19:0/1063617660 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0578077e50 msgr2=0x7f057807a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:06.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.155+0000 7f0591c54640 1 --2- 172.21.15.19:0/1063617660 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0578077e50 0x7f057807a360 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f057c006fd0 tx=0x7f057c006bc0 comp rx=0 tx=0).stop 2024-08-31T12:16:06.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.155+0000 7f0591c54640 1 -- 172.21.15.19:0/1063617660 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f058c0c5dc0 msgr2=0x7f058c15a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:06.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.155+0000 7f0591c54640 1 --2- 172.21.15.19:0/1063617660 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f058c0c5dc0 0x7f058c15a180 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0580009920 tx=0x7f05800045e0 comp rx=0 tx=0).stop 2024-08-31T12:16:06.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.155+0000 7f0591c54640 1 -- 172.21.15.19:0/1063617660 shutdown_connections 2024-08-31T12:16:06.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.155+0000 7f0591c54640 1 --2- 172.21.15.19:0/1063617660 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0578077e50 0x7f057807a360 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:06.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.155+0000 7f0591c54640 1 --2- 172.21.15.19:0/1063617660 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f058c141e70 0x7f058c0cb7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:06.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.156+0000 7f0591c54640 1 --2- 172.21.15.19:0/1063617660 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f058c0c7280 0x7f058c1e33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:06.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.156+0000 7f0591c54640 1 --2- 172.21.15.19:0/1063617660 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f058c0c5dc0 0x7f058c15a180 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:06.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.156+0000 7f0591c54640 1 -- 172.21.15.19:0/1063617660 >> 172.21.15.19:0/1063617660 conn(0x7f058c06cae0 msgr2=0x7f058c13eae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:06.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.156+0000 7f0591c54640 1 -- 172.21.15.19:0/1063617660 shutdown_connections 2024-08-31T12:16:06.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.156+0000 7f0591c54640 1 -- 172.21.15.19:0/1063617660 wait complete. 2024-08-31T12:16:06.158 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509446 for osd.0 2024-08-31T12:16:06.159 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542149 for osd.4 2024-08-31T12:16:06.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.163+0000 7f9db759e640 1 -- 172.21.15.19:0/1734715190 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f9db0179300 con 0x7f9db00483a0 2024-08-31T12:16:06.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.164+0000 7f9da6ffd640 1 -- 172.21.15.19:0/1734715190 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f9dac0690e0 con 0x7f9db00483a0 2024-08-31T12:16:06.165 INFO:teuthology.orchestra.run.smithi019.stdout:60129542149 2024-08-31T12:16:06.168 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.166+0000 7f9db759e640 1 -- 172.21.15.19:0/1734715190 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9d98077a80 msgr2=0x7f9d98079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:06.168 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.166+0000 7f9db759e640 1 --2- 172.21.15.19:0/1734715190 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9d98077a80 0x7f9d98079f90 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9da8034000 tx=0x7f9da8004730 comp rx=0 tx=0).stop 2024-08-31T12:16:06.168 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.166+0000 7f9db759e640 1 -- 172.21.15.19:0/1734715190 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9db00483a0 msgr2=0x7f9db0044a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:06.168 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.166+0000 7f9db759e640 1 --2- 172.21.15.19:0/1734715190 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9db00483a0 0x7f9db0044a40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9dac00c430 tx=0x7f9dac00c8f0 comp rx=0 tx=0).stop 2024-08-31T12:16:06.168 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.166+0000 7f9db759e640 1 -- 172.21.15.19:0/1734715190 shutdown_connections 2024-08-31T12:16:06.168 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.166+0000 7f9db759e640 1 --2- 172.21.15.19:0/1734715190 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9d98077a80 0x7f9d98079f90 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:06.168 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.166+0000 7f9db759e640 1 --2- 172.21.15.19:0/1734715190 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9db00e2330 0x7f9db0175720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:06.168 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.166+0000 7f9db759e640 1 --2- 172.21.15.19:0/1734715190 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9db00dfa30 0x7f9db01694d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:06.168 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.166+0000 7f9db759e640 1 --2- 172.21.15.19:0/1734715190 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9db00483a0 0x7f9db0044a40 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:06.168 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.166+0000 7f9db759e640 1 -- 172.21.15.19:0/1734715190 >> 172.21.15.19:0/1734715190 conn(0x7f9db0050fa0 msgr2=0x7f9db004b5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:06.168 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.166+0000 7f9db759e640 1 -- 172.21.15.19:0/1734715190 shutdown_connections 2024-08-31T12:16:06.168 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:06.167+0000 7f9db759e640 1 -- 172.21.15.19:0/1734715190 wait complete. 2024-08-31T12:16:06.170 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509446 for osd.1 2024-08-31T12:16:06.170 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509445 for osd.3 2024-08-31T12:16:06.171 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509446 for osd.6 2024-08-31T12:16:06.179 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542149 for osd.7 2024-08-31T12:16:07.119 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-08-31T12:16:07.138 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-08-31T12:16:07.159 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-08-31T12:16:07.159 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-08-31T12:16:07.171 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-08-31T12:16:07.171 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-08-31T12:16:07.171 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-08-31T12:16:07.179 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-08-31T12:16:07.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.289+0000 7f1a76924640 1 -- 172.21.15.19:0/3087659089 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1a70070bb0 msgr2=0x7f1a7015bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:07.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.289+0000 7f1a76924640 1 --2- 172.21.15.19:0/3087659089 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1a70070bb0 0x7f1a7015bbc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1a68005fd0 tx=0x7f1a6802aac0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.289+0000 7f1a76924640 1 -- 172.21.15.19:0/3087659089 shutdown_connections 2024-08-31T12:16:07.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.289+0000 7f1a76924640 1 --2- 172.21.15.19:0/3087659089 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1a70070bb0 0x7f1a7015bbc0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.289+0000 7f1a76924640 1 -- 172.21.15.19:0/3087659089 >> 172.21.15.19:0/3087659089 conn(0x7f1a700cc950 msgr2=0x7f1a700ccd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:07.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.297+0000 7f1a76924640 1 -- 172.21.15.19:0/3087659089 shutdown_connections 2024-08-31T12:16:07.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.297+0000 7f1a76924640 1 -- 172.21.15.19:0/3087659089 wait complete. 2024-08-31T12:16:07.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.300+0000 7f1a76924640 1 Processor -- start 2024-08-31T12:16:07.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.300+0000 7f1a76924640 1 -- start start 2024-08-31T12:16:07.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.302+0000 7f1a76924640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1a70070bb0 0x7f1a701f6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:07.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.303+0000 7f1a76924640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1a700c7170 0x7f1a701564c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:07.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.303+0000 7f1a76924640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1a7014d480 0x7f1a70145f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:07.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.303+0000 7f1a6ffff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1a70070bb0 0x7f1a701f6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:07.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.303+0000 7f1a6ffff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1a70070bb0 0x7f1a701f6510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:47232/0 (socket says 172.21.15.19:47232) 2024-08-31T12:16:07.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.303+0000 7f1a6f7fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1a700c7170 0x7f1a701564c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:07.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.303+0000 7f1a6f7fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1a700c7170 0x7f1a701564c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.19:49018/0 (socket says 172.21.15.19:49018) 2024-08-31T12:16:07.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.303+0000 7f1a6ffff640 1 -- 172.21.15.19:0/512466581 learned_addr learned my addr 172.21.15.19:0/512466581 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:07.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.303+0000 7f1a76924640 1 -- 172.21.15.19:0/512466581 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f1a700703b0 con 0x7f1a7014d480 2024-08-31T12:16:07.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.304+0000 7f1a76924640 1 -- 172.21.15.19:0/512466581 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1a700700d0 con 0x7f1a70070bb0 2024-08-31T12:16:07.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.304+0000 7f1a76924640 1 -- 172.21.15.19:0/512466581 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f1a70070690 con 0x7f1a700c7170 2024-08-31T12:16:07.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.304+0000 7f1a74ec1640 1 --2- 172.21.15.19:0/512466581 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1a7014d480 0x7f1a70145f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:07.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.305+0000 7f1a74ec1640 1 -- 172.21.15.19:0/512466581 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1a700c7170 msgr2=0x7f1a701564c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:07.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.305+0000 7f1a74ec1640 1 --2- 172.21.15.19:0/512466581 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1a700c7170 0x7f1a701564c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.305+0000 7f1a74ec1640 1 -- 172.21.15.19:0/512466581 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1a70070bb0 msgr2=0x7f1a701f6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:07.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.305+0000 7f1a74ec1640 1 --2- 172.21.15.19:0/512466581 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1a70070bb0 0x7f1a701f6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.305+0000 7f1a74ec1640 1 -- 172.21.15.19:0/512466581 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a70148940 con 0x7f1a7014d480 2024-08-31T12:16:07.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.306+0000 7f1a74ec1640 1 --2- 172.21.15.19:0/512466581 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1a7014d480 0x7f1a70145f50 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f1a6400b550 tx=0x7f1a6400ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:07.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.306+0000 7f1a6cff9640 1 -- 172.21.15.19:0/512466581 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a64014070 con 0x7f1a7014d480 2024-08-31T12:16:07.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.306+0000 7f1a6ffff640 1 --2- 172.21.15.19:0/512466581 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1a70070bb0 0x7f1a701f6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-31T12:16:07.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.306+0000 7f1a6f7fe640 1 --2- 172.21.15.19:0/512466581 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1a700c7170 0x7f1a701564c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-31T12:16:07.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.307+0000 7f1a76924640 1 -- 172.21.15.19:0/512466581 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a700cbbb0 con 0x7f1a7014d480 2024-08-31T12:16:07.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.307+0000 7f1a76924640 1 -- 172.21.15.19:0/512466581 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a700cbe50 con 0x7f1a7014d480 2024-08-31T12:16:07.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.310+0000 7f1a76924640 1 -- 172.21.15.19:0/512466581 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a70073ac0 con 0x7f1a7014d480 2024-08-31T12:16:07.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.312+0000 7f1a6cff9640 1 -- 172.21.15.19:0/512466581 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1a64010790 con 0x7f1a7014d480 2024-08-31T12:16:07.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.313+0000 7f1a6cff9640 1 -- 172.21.15.19:0/512466581 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a64010be0 con 0x7f1a7014d480 2024-08-31T12:16:07.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.313+0000 7f1a6cff9640 1 -- 172.21.15.19:0/512466581 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f1a64017070 con 0x7f1a7014d480 2024-08-31T12:16:07.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.314+0000 7f1a6cff9640 1 --2- 172.21.15.19:0/512466581 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1a5c077b80 0x7f1a5c07a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:07.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.314+0000 7f1a6cff9640 1 -- 172.21.15.19:0/512466581 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f1a640994b0 con 0x7f1a7014d480 2024-08-31T12:16:07.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.318+0000 7f1a6ffff640 1 --2- 172.21.15.19:0/512466581 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1a5c077b80 0x7f1a5c07a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:07.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.320+0000 7f1a6cff9640 1 -- 172.21.15.19:0/512466581 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f1a700cbbb0 con 0x7f1a7014d480 2024-08-31T12:16:07.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.325+0000 7f1a6ffff640 1 --2- 172.21.15.19:0/512466581 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1a5c077b80 0x7f1a5c07a090 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1a60000f60 tx=0x7f1a60009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:07.356 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.355+0000 7fa3e1f67640 1 -- 172.21.15.19:0/4109093568 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3d4048970 msgr2=0x7fa3d40d5bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:07.356 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.355+0000 7fa3e1f67640 1 --2- 172.21.15.19:0/4109093568 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3d4048970 0x7fa3d40d5bc0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc075480 tx=0x7fa3dc0a6ec0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.355+0000 7f968147e640 1 -- 172.21.15.19:0/2872141569 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f967c150170 msgr2=0x7f967c150580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:07.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.355+0000 7f968147e640 1 --2- 172.21.15.19:0/2872141569 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f967c150170 0x7f967c150580 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9674009be0 tx=0x7f967402a970 comp rx=0 tx=0).stop 2024-08-31T12:16:07.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.355+0000 7f968147e640 1 -- 172.21.15.19:0/2872141569 shutdown_connections 2024-08-31T12:16:07.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.355+0000 7f968147e640 1 --2- 172.21.15.19:0/2872141569 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f967c150170 0x7f967c150580 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.355+0000 7f968147e640 1 -- 172.21.15.19:0/2872141569 >> 172.21.15.19:0/2872141569 conn(0x7f967c0cda10 msgr2=0x7f967c0c9e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:07.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.356+0000 7fee35cb6640 1 -- 172.21.15.19:0/1217882957 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fee300b3630 msgr2=0x7fee300ca6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:07.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.356+0000 7fee35cb6640 1 --2- 172.21.15.19:0/1217882957 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fee300b3630 0x7fee300ca6d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fee28009be0 tx=0x7fee2802a920 comp rx=0 tx=0).stop 2024-08-31T12:16:07.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.356+0000 7fa3e1f67640 1 -- 172.21.15.19:0/4109093568 shutdown_connections 2024-08-31T12:16:07.358 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.356+0000 7fa3e1f67640 1 --2- 172.21.15.19:0/4109093568 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3d4048970 0x7fa3d40d5bc0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.358 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.356+0000 7fa3e1f67640 1 -- 172.21.15.19:0/4109093568 >> 172.21.15.19:0/4109093568 conn(0x7fa3d4050fa0 msgr2=0x7fa3d40513b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:07.359 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.356+0000 7fa3e1f67640 1 -- 172.21.15.19:0/4109093568 shutdown_connections 2024-08-31T12:16:07.359 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.356+0000 7f968147e640 1 -- 172.21.15.19:0/2872141569 shutdown_connections 2024-08-31T12:16:07.359 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.356+0000 7f968147e640 1 -- 172.21.15.19:0/2872141569 wait complete. 2024-08-31T12:16:07.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.359+0000 7fee35cb6640 1 -- 172.21.15.19:0/1217882957 shutdown_connections 2024-08-31T12:16:07.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.359+0000 7fee35cb6640 1 --2- 172.21.15.19:0/1217882957 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fee300b3630 0x7fee300ca6d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.359+0000 7fee35cb6640 1 -- 172.21.15.19:0/1217882957 >> 172.21.15.19:0/1217882957 conn(0x7fee30069350 msgr2=0x7fee300d07a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:07.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.356+0000 7fa3e1f67640 1 -- 172.21.15.19:0/4109093568 wait complete. 2024-08-31T12:16:07.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.359+0000 7fa3e1f67640 1 Processor -- start 2024-08-31T12:16:07.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.359+0000 7fa3e1f67640 1 -- start start 2024-08-31T12:16:07.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.360+0000 7fee35cb6640 1 -- 172.21.15.19:0/1217882957 shutdown_connections 2024-08-31T12:16:07.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.359+0000 7f968147e640 1 Processor -- start 2024-08-31T12:16:07.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.359+0000 7f968147e640 1 -- start start 2024-08-31T12:16:07.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.361+0000 7f968147e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f967c150170 0x7f967c1f5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:07.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.361+0000 7f967affd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f967c150170 0x7f967c1f5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:07.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.361+0000 7f967affd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f967c150170 0x7f967c1f5970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:56806/0 (socket says 172.21.15.19:56806) 2024-08-31T12:16:07.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.361+0000 7f967affd640 1 -- 172.21.15.19:0/2314858609 learned_addr learned my addr 172.21.15.19:0/2314858609 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:07.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.361+0000 7f968147e640 1 --2- 172.21.15.19:0/2314858609 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f967c15b9b0 0x7f967c0cc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:07.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.361+0000 7f968147e640 1 --2- 172.21.15.19:0/2314858609 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f967c15a0d0 0x7f967c1487e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:07.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.361+0000 7f968147e640 1 -- 172.21.15.19:0/2314858609 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f967c0c8810 con 0x7f967c150170 2024-08-31T12:16:07.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.361+0000 7f968147e640 1 -- 172.21.15.19:0/2314858609 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f967c0c8530 con 0x7f967c15a0d0 2024-08-31T12:16:07.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.361+0000 7f968147e640 1 -- 172.21.15.19:0/2314858609 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f967c1555d0 con 0x7f967c15b9b0 2024-08-31T12:16:07.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.360+0000 7fee35cb6640 1 -- 172.21.15.19:0/1217882957 wait complete. 2024-08-31T12:16:07.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.363+0000 7fee35cb6640 1 Processor -- start 2024-08-31T12:16:07.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.361+0000 7fa3e1f67640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa3d4048970 0x7fa3d40d80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:07.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.363+0000 7fee35cb6640 1 -- start start 2024-08-31T12:16:07.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.365+0000 7fee35cb6640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee3006ebe0 0x7fee3014a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:07.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.365+0000 7fee35cb6640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fee300b3630 0x7fee30073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:07.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.366+0000 7fee35cb6640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fee300cb5b0 0x7fee3014b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:07.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.366+0000 7fee2f7fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee3006ebe0 0x7fee3014a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:07.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.366+0000 7fee2f7fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee3006ebe0 0x7fee3014a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:47268/0 (socket says 172.21.15.19:47268) 2024-08-31T12:16:07.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.366+0000 7fee2effd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fee300b3630 0x7fee30073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:07.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.366+0000 7fee2effd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fee300b3630 0x7fee30073680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.19:49056/0 (socket says 172.21.15.19:49056) 2024-08-31T12:16:07.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.366+0000 7fee2ffff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fee300cb5b0 0x7fee3014b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:07.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.366+0000 7fee2ffff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fee300cb5b0 0x7fee3014b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:56812/0 (socket says 172.21.15.19:56812) 2024-08-31T12:16:07.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.366+0000 7fee35cb6640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fee300cac10 con 0x7fee300cb5b0 2024-08-31T12:16:07.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.366+0000 7fee35cb6640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fee3015bc50 con 0x7fee3006ebe0 2024-08-31T12:16:07.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.366+0000 7fee35cb6640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fee3015bf30 con 0x7fee300b3630 2024-08-31T12:16:07.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.362+0000 7fa3e0f65640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa3d4048970 0x7fa3d40d80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:07.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.362+0000 7fa3e0f65640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa3d4048970 0x7fa3d40d80f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:56808/0 (socket says 172.21.15.19:56808) 2024-08-31T12:16:07.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.362+0000 7fa3e0f65640 1 -- 172.21.15.19:0/2863317481 learned_addr learned my addr 172.21.15.19:0/2863317481 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:07.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.362+0000 7fa3e1f67640 1 --2- 172.21.15.19:0/2863317481 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa3d4048ec0 0x7fa3d40cf4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:07.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.362+0000 7fa3e1f67640 1 --2- 172.21.15.19:0/2863317481 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3d404c4e0 0x7fa3d4007aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:07.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.362+0000 7fa3e1f67640 1 -- 172.21.15.19:0/2863317481 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d4049940 con 0x7fa3d4048970 2024-08-31T12:16:07.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.362+0000 7fa3e1f67640 1 -- 172.21.15.19:0/2863317481 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d4049410 con 0x7fa3d404c4e0 2024-08-31T12:16:07.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.362+0000 7fa3e1f67640 1 -- 172.21.15.19:0/2863317481 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fa3d404d760 con 0x7fa3d4048ec0 2024-08-31T12:16:07.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.362+0000 7fa3e1766640 1 --2- 172.21.15.19:0/2863317481 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3d404c4e0 0x7fa3d4007aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:07.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.363+0000 7fa3e0f65640 1 -- 172.21.15.19:0/2863317481 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa3d4048ec0 msgr2=0x7fa3d40cf4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:16:07.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.363+0000 7fa3e0f65640 1 --2- 172.21.15.19:0/2863317481 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa3d4048ec0 0x7fa3d40cf4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.363+0000 7fa3e0f65640 1 -- 172.21.15.19:0/2863317481 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3d404c4e0 msgr2=0x7fa3d4007aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:07.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.363+0000 7fa3e0f65640 1 --2- 172.21.15.19:0/2863317481 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3d404c4e0 0x7fa3d4007aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.363+0000 7fa3e0f65640 1 -- 172.21.15.19:0/2863317481 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3d4033b30 con 0x7fa3d4048970 2024-08-31T12:16:07.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.363+0000 7fa3e1766640 1 --2- 172.21.15.19:0/2863317481 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3d404c4e0 0x7fa3d4007aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-31T12:16:07.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.363+0000 7fa3e0f65640 1 --2- 172.21.15.19:0/2863317481 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa3d4048970 0x7fa3d40d80f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc0526b0 tx=0x7fa3dc0afcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:07.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.366+0000 7fa3d97fa640 1 -- 172.21.15.19:0/2863317481 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3dc085420 con 0x7fa3d4048970 2024-08-31T12:16:07.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.362+0000 7f9673fff640 1 --2- 172.21.15.19:0/2314858609 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f967c15b9b0 0x7f967c0cc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:07.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.362+0000 7f967affd640 1 -- 172.21.15.19:0/2314858609 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f967c15b9b0 msgr2=0x7f967c0cc3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:07.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.362+0000 7f967affd640 1 --2- 172.21.15.19:0/2314858609 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f967c15b9b0 0x7f967c0cc3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.362+0000 7f967affd640 1 -- 172.21.15.19:0/2314858609 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f967c15a0d0 msgr2=0x7f967c1487e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:16:07.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.362+0000 7f967affd640 1 --2- 172.21.15.19:0/2314858609 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f967c15a0d0 0x7f967c1487e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.362+0000 7f967affd640 1 -- 172.21.15.19:0/2314858609 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f967c1f5eb0 con 0x7f967c150170 2024-08-31T12:16:07.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.363+0000 7f967affd640 1 --2- 172.21.15.19:0/2314858609 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f967c150170 0x7f967c1f5970 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f966c01d770 tx=0x7f966c01dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:07.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.363+0000 7f96737fe640 1 -- 172.21.15.19:0/2314858609 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f966c027020 con 0x7f967c150170 2024-08-31T12:16:07.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.363+0000 7f96737fe640 1 -- 172.21.15.19:0/2314858609 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f966c01b650 con 0x7f967c150170 2024-08-31T12:16:07.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.363+0000 7f96737fe640 1 -- 172.21.15.19:0/2314858609 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f966c01ba60 con 0x7f967c150170 2024-08-31T12:16:07.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.363+0000 7f968147e640 1 -- 172.21.15.19:0/2314858609 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f967c1f8ef0 con 0x7f967c150170 2024-08-31T12:16:07.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.363+0000 7f968147e640 1 -- 172.21.15.19:0/2314858609 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f967c1f94b0 con 0x7f967c150170 2024-08-31T12:16:07.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.366+0000 7f96737fe640 1 -- 172.21.15.19:0/2314858609 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f966c02a020 con 0x7f967c150170 2024-08-31T12:16:07.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.366+0000 7fee2f7fe640 1 -- 172.21.15.19:0/3063856160 learned_addr learned my addr 172.21.15.19:0/3063856160 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:07.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.367+0000 7fee2ffff640 1 -- 172.21.15.19:0/3063856160 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fee300b3630 msgr2=0x7fee30073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:07.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.367+0000 7fee2ffff640 1 --2- 172.21.15.19:0/3063856160 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fee300b3630 0x7fee30073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.367+0000 7fee2ffff640 1 -- 172.21.15.19:0/3063856160 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee3006ebe0 msgr2=0x7fee3014a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:07.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.367+0000 7fee2ffff640 1 --2- 172.21.15.19:0/3063856160 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee3006ebe0 0x7fee3014a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.367+0000 7fee2ffff640 1 -- 172.21.15.19:0/3063856160 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee3014aef0 con 0x7fee300cb5b0 2024-08-31T12:16:07.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.367+0000 7fee2ffff640 1 --2- 172.21.15.19:0/3063856160 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fee300cb5b0 0x7fee3014b2c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fee2400c4d0 tx=0x7fee2400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:07.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.367+0000 7fee2f7fe640 1 --2- 172.21.15.19:0/3063856160 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee3006ebe0 0x7fee3014a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-08-31T12:16:07.373 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.367+0000 7f96737fe640 1 --2- 172.21.15.19:0/2314858609 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f965c077b50 0x7f965c07a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:07.373 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.368+0000 7f96737fe640 1 -- 172.21.15.19:0/2314858609 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f966c0aa4d0 con 0x7f967c150170 2024-08-31T12:16:07.373 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.368+0000 7f9673fff640 1 --2- 172.21.15.19:0/2314858609 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f965c077b50 0x7f965c07a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:07.373 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.368+0000 7f96717fa640 1 -- 172.21.15.19:0/2314858609 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9660005290 con 0x7f967c150170 2024-08-31T12:16:07.374 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.368+0000 7fee17fff640 1 -- 172.21.15.19:0/3063856160 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee240052d0 con 0x7fee300cb5b0 2024-08-31T12:16:07.374 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.368+0000 7fee17fff640 1 -- 172.21.15.19:0/3063856160 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fee24005470 con 0x7fee300cb5b0 2024-08-31T12:16:07.374 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.368+0000 7fee17fff640 1 -- 172.21.15.19:0/3063856160 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee24003830 con 0x7fee300cb5b0 2024-08-31T12:16:07.374 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.369+0000 7fee35cb6640 1 -- 172.21.15.19:0/3063856160 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee301ff910 con 0x7fee300cb5b0 2024-08-31T12:16:07.374 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.369+0000 7fee35cb6640 1 -- 172.21.15.19:0/3063856160 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee3014a1b0 con 0x7fee300cb5b0 2024-08-31T12:16:07.374 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.366+0000 7fa3e1f67640 1 -- 172.21.15.19:0/2863317481 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3d40cea00 con 0x7fa3d4048970 2024-08-31T12:16:07.374 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.366+0000 7fa3e1f67640 1 -- 172.21.15.19:0/2863317481 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3d40ceca0 con 0x7fa3d4048970 2024-08-31T12:16:07.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.370+0000 7fa3d97fa640 1 -- 172.21.15.19:0/2863317481 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa3dc0855c0 con 0x7fa3d4048970 2024-08-31T12:16:07.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.370+0000 7fa3d97fa640 1 -- 172.21.15.19:0/2863317481 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3dc0859f0 con 0x7fa3d4048970 2024-08-31T12:16:07.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.370+0000 7fee35cb6640 1 -- 172.21.15.19:0/3063856160 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee3015aad0 con 0x7fee300cb5b0 2024-08-31T12:16:07.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.372+0000 7fa3d97fa640 1 -- 172.21.15.19:0/2863317481 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fa3dc0843c0 con 0x7fa3d4048970 2024-08-31T12:16:07.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.373+0000 7fa3e1f67640 1 -- 172.21.15.19:0/2863317481 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3d40483a0 con 0x7fa3d4048970 2024-08-31T12:16:07.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.373+0000 7f9673fff640 1 --2- 172.21.15.19:0/2314858609 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f965c077b50 0x7f965c07a060 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f967c0c4bf0 tx=0x7f9664009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:07.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.373+0000 7fa3d97fa640 1 --2- 172.21.15.19:0/2863317481 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa3c0077b20 0x7fa3c007a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:07.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.374+0000 7fa3d97fa640 1 -- 172.21.15.19:0/2863317481 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa3dc0a7e00 con 0x7fa3d4048970 2024-08-31T12:16:07.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.374+0000 7fa3dbfff640 1 --2- 172.21.15.19:0/2863317481 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa3c0077b20 0x7fa3c007a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:07.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.375+0000 7fa3dbfff640 1 --2- 172.21.15.19:0/2863317481 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa3c0077b20 0x7fa3c007a030 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa3d40d7190 tx=0x7fa3d00079d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:07.379 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.379+0000 7f96737fe640 1 -- 172.21.15.19:0/2314858609 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f966c031020 con 0x7f967c150170 2024-08-31T12:16:07.381 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.381+0000 7fee17fff640 1 -- 172.21.15.19:0/3063856160 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fee240039d0 con 0x7fee300cb5b0 2024-08-31T12:16:07.382 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.381+0000 7fee17fff640 1 --2- 172.21.15.19:0/3063856160 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fee1c077ad0 0x7fee1c079fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:07.382 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.382+0000 7fee2f7fe640 1 --2- 172.21.15.19:0/3063856160 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fee1c077ad0 0x7fee1c079fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:07.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.383+0000 7fee2f7fe640 1 --2- 172.21.15.19:0/3063856160 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fee1c077ad0 0x7fee1c079fe0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fee28006010 tx=0x7fee28004ab0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:07.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.389+0000 7fa3d97fa640 1 -- 172.21.15.19:0/2863317481 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fa3dc0fed40 con 0x7fa3d4048970 2024-08-31T12:16:07.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.396+0000 7fee17fff640 1 -- 172.21.15.19:0/3063856160 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fee2405e350 con 0x7fee300cb5b0 2024-08-31T12:16:07.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.400+0000 7fee17fff640 1 -- 172.21.15.19:0/3063856160 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fee3015aad0 con 0x7fee300cb5b0 2024-08-31T12:16:07.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.414+0000 7fb9ca778640 1 -- 172.21.15.19:0/4028774951 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb9c415b950 msgr2=0x7fb9c40cc2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:07.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.414+0000 7fb9ca778640 1 --2- 172.21.15.19:0/4028774951 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb9c415b950 0x7fb9c40cc2e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb9b4009830 tx=0x7fb9b402a510 comp rx=0 tx=0).stop 2024-08-31T12:16:07.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.414+0000 7fb9ca778640 1 -- 172.21.15.19:0/4028774951 shutdown_connections 2024-08-31T12:16:07.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.414+0000 7fb9ca778640 1 --2- 172.21.15.19:0/4028774951 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb9c4150700 0x7fb9c4150b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.414+0000 7fb9ca778640 1 --2- 172.21.15.19:0/4028774951 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb9c415b950 0x7fb9c40cc2e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.414+0000 7fb9ca778640 1 -- 172.21.15.19:0/4028774951 >> 172.21.15.19:0/4028774951 conn(0x7fb9c406cae0 msgr2=0x7fb9c406cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:07.416 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.415+0000 7f315347e640 1 -- 172.21.15.19:0/2199556763 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f314c0c8070 msgr2=0x7f314c153b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:07.416 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.415+0000 7f315347e640 1 --2- 172.21.15.19:0/2199556763 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f314c0c8070 0x7f314c153b20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3148009860 tx=0x7f314802a620 comp rx=0 tx=0).stop 2024-08-31T12:16:07.416 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.414+0000 7fb9ca778640 1 -- 172.21.15.19:0/4028774951 shutdown_connections 2024-08-31T12:16:07.416 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.414+0000 7fb9ca778640 1 -- 172.21.15.19:0/4028774951 wait complete. 2024-08-31T12:16:07.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.416+0000 7f315347e640 1 -- 172.21.15.19:0/2199556763 shutdown_connections 2024-08-31T12:16:07.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.416+0000 7f315347e640 1 --2- 172.21.15.19:0/2199556763 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f314c0c8070 0x7f314c153b20 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.416+0000 7f315347e640 1 -- 172.21.15.19:0/2199556763 >> 172.21.15.19:0/2199556763 conn(0x7f314c0ce0b0 msgr2=0x7f314c0c9f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:07.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.417+0000 7f315347e640 1 -- 172.21.15.19:0/2199556763 shutdown_connections 2024-08-31T12:16:07.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.417+0000 7fb9ca778640 1 Processor -- start 2024-08-31T12:16:07.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.417+0000 7fe8ecadf640 1 -- 172.21.15.19:0/3751247024 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8d80039e0 msgr2=0x7fe8d8003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:07.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.417+0000 7fe8ecadf640 1 --2- 172.21.15.19:0/3751247024 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8d80039e0 0x7fe8d8003e30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc02e720 tx=0x7fe8d8005830 comp rx=0 tx=0).stop 2024-08-31T12:16:07.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.417+0000 7f315347e640 1 -- 172.21.15.19:0/2199556763 wait complete. 2024-08-31T12:16:07.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.417+0000 7fe8e7fff640 1 -- 172.21.15.19:0/3751247024 reap_dead start 2024-08-31T12:16:07.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.417+0000 7fe8ecadf640 1 -- 172.21.15.19:0/3751247024 shutdown_connections 2024-08-31T12:16:07.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.417+0000 7fe8ecadf640 1 -- 172.21.15.19:0/3751247024 >> 172.21.15.19:0/3751247024 conn(0x7fe8e0050fa0 msgr2=0x7fe8e00513b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:07.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.417+0000 7fb9ca778640 1 -- start start 2024-08-31T12:16:07.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.418+0000 7fe8ecadf640 1 -- 172.21.15.19:0/3751247024 shutdown_connections 2024-08-31T12:16:07.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.419+0000 7fb9ca778640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb9c4072940 0x7fb9c414b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:07.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.419+0000 7fb9ca778640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb9c4150700 0x7fb9c415dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:07.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.418+0000 7fe8ecadf640 1 -- 172.21.15.19:0/3751247024 wait complete. 2024-08-31T12:16:07.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.420+0000 7fb9c8f75640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb9c4150700 0x7fb9c415dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:07.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.420+0000 7fb9c8f75640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb9c4150700 0x7fb9c415dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:56836/0 (socket says 172.21.15.19:56836) 2024-08-31T12:16:07.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.420+0000 7fb9c8f75640 1 -- 172.21.15.19:0/1888095369 learned_addr learned my addr 172.21.15.19:0/1888095369 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:07.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.420+0000 7fb9ca778640 1 --2- 172.21.15.19:0/1888095369 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb9c415b950 0x7fb9c4146bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:07.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.420+0000 7fb9ca778640 1 -- 172.21.15.19:0/1888095369 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c415bd60 con 0x7fb9c4150700 2024-08-31T12:16:07.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.420+0000 7fb9ca778640 1 -- 172.21.15.19:0/1888095369 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c415e890 con 0x7fb9c415b950 2024-08-31T12:16:07.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.420+0000 7fb9ca778640 1 -- 172.21.15.19:0/1888095369 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fb9c40cc820 con 0x7fb9c4072940 2024-08-31T12:16:07.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.421+0000 7fb9c9776640 1 --2- 172.21.15.19:0/1888095369 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb9c4072940 0x7fb9c414b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:07.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.421+0000 7fb9c9f77640 1 --2- 172.21.15.19:0/1888095369 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb9c415b950 0x7fb9c4146bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:07.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.421+0000 7fe8ecadf640 1 Processor -- start 2024-08-31T12:16:07.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.421+0000 7fb9c8f75640 1 -- 172.21.15.19:0/1888095369 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb9c4072940 msgr2=0x7fb9c414b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:07.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.421+0000 7fb9c8f75640 1 --2- 172.21.15.19:0/1888095369 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb9c4072940 0x7fb9c414b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.421+0000 7fb9c8f75640 1 -- 172.21.15.19:0/1888095369 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb9c415b950 msgr2=0x7fb9c4146bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:07.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.421+0000 7fb9c8f75640 1 --2- 172.21.15.19:0/1888095369 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb9c415b950 0x7fb9c4146bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.421+0000 7fb9c8f75640 1 -- 172.21.15.19:0/1888095369 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9c41f5ca0 con 0x7fb9c4150700 2024-08-31T12:16:07.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.422+0000 7fb9c8f75640 1 --2- 172.21.15.19:0/1888095369 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb9c4150700 0x7fb9c415dc20 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb9b4000c00 tx=0x7fb9b40316c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:07.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.420+0000 7efcc4580640 1 -- 172.21.15.19:0/1022219272 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efcbc072b80 msgr2=0x7efcbc1570f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:07.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.420+0000 7efcc4580640 1 --2- 172.21.15.19:0/1022219272 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efcbc072b80 0x7efcbc1570f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7efcbc157ad0 tx=0x7efcb8032bb0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.420+0000 7efcc4580640 1 -- 172.21.15.19:0/1022219272 shutdown_connections 2024-08-31T12:16:07.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.420+0000 7efcc4580640 1 --2- 172.21.15.19:0/1022219272 >> v2:172.21.15.43:3300/0 conn(0x7efcbc070920 0x7efcbc070d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.420+0000 7efcc4580640 1 --2- 172.21.15.19:0/1022219272 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efcbc0c6e60 0x7efcbc150e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.420+0000 7efcc4580640 1 --2- 172.21.15.19:0/1022219272 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efcbc072b80 0x7efcbc1570f0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.420+0000 7efcc4580640 1 -- 172.21.15.19:0/1022219272 >> 172.21.15.19:0/1022219272 conn(0x7efcbc0cc950 msgr2=0x7efcbc0ccd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:07.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.420+0000 7efcc4580640 1 -- 172.21.15.19:0/1022219272 shutdown_connections 2024-08-31T12:16:07.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.420+0000 7efcc4580640 1 -- 172.21.15.19:0/1022219272 wait complete. 2024-08-31T12:16:07.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.420+0000 7f315347e640 1 Processor -- start 2024-08-31T12:16:07.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.420+0000 7f315347e640 1 -- start start 2024-08-31T12:16:07.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.422+0000 7f315347e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f314c070df0 0x7f314c1f5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:07.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.422+0000 7f315347e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f314c0c8070 0x7f314c14a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:07.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.422+0000 7f315347e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f314c15bdf0 0x7f314c15d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:07.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.422+0000 7f315347e640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f314c15b8d0 con 0x7f314c070df0 2024-08-31T12:16:07.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.422+0000 7f315347e640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f314c071960 con 0x7f314c15bdf0 2024-08-31T12:16:07.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.422+0000 7f315347e640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f314c0c8480 con 0x7f314c0c8070 2024-08-31T12:16:07.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.423+0000 7f315121a640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f314c070df0 0x7f314c1f5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:07.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.423+0000 7f315121a640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f314c070df0 0x7f314c1f5a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:56848/0 (socket says 172.21.15.19:56848) 2024-08-31T12:16:07.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.423+0000 7f315121a640 1 -- 172.21.15.19:0/127232850 learned_addr learned my addr 172.21.15.19:0/127232850 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:07.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.423+0000 7f3150a19640 1 --2- 172.21.15.19:0/127232850 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f314c0c8070 0x7f314c14a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:07.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.423+0000 7fb9b9ffb640 1 -- 172.21.15.19:0/1888095369 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b4031b50 con 0x7fb9c4150700 2024-08-31T12:16:07.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.423+0000 7fb9b9ffb640 1 -- 172.21.15.19:0/1888095369 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb9b4031de0 con 0x7fb9c4150700 2024-08-31T12:16:07.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.423+0000 7fb9b9ffb640 1 -- 172.21.15.19:0/1888095369 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b4033330 con 0x7fb9c4150700 2024-08-31T12:16:07.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.423+0000 7f3151a1b640 1 --2- 172.21.15.19:0/127232850 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f314c15bdf0 0x7f314c15d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:07.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.423+0000 7f315121a640 1 -- 172.21.15.19:0/127232850 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f314c0c8070 msgr2=0x7f314c14a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:07.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.423+0000 7f315121a640 1 --2- 172.21.15.19:0/127232850 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f314c0c8070 0x7f314c14a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.423+0000 7f315121a640 1 -- 172.21.15.19:0/127232850 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f314c15bdf0 msgr2=0x7f314c15d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:07.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.423+0000 7f315121a640 1 --2- 172.21.15.19:0/127232850 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f314c15bdf0 0x7f314c15d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.423+0000 7f315121a640 1 -- 172.21.15.19:0/127232850 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f314c147b40 con 0x7f314c070df0 2024-08-31T12:16:07.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.423+0000 7efcc4580640 1 Processor -- start 2024-08-31T12:16:07.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.423+0000 7f3150a19640 1 --2- 172.21.15.19:0/127232850 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f314c0c8070 0x7f314c14a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-31T12:16:07.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.421+0000 7fe8ecadf640 1 -- start start 2024-08-31T12:16:07.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.423+0000 7fb9ca778640 1 -- 172.21.15.19:0/1888095369 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9c41f5f60 con 0x7fb9c4150700 2024-08-31T12:16:07.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.423+0000 7fe8ecadf640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8d80039e0 0x7fe8e00d9e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:07.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.424+0000 7fe8ecadf640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe8e00daf20 0x7fe8e00de710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:07.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.424+0000 7fb9ca778640 1 -- 172.21.15.19:0/1888095369 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9c41f88a0 con 0x7fb9c4150700 2024-08-31T12:16:07.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.424+0000 7f315121a640 1 --2- 172.21.15.19:0/127232850 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f314c070df0 0x7f314c1f5a50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f314400cbf0 tx=0x7f314400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:07.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.424+0000 7f313dffb640 1 -- 172.21.15.19:0/127232850 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3144016020 con 0x7f314c070df0 2024-08-31T12:16:07.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.424+0000 7f315347e640 1 -- 172.21.15.19:0/127232850 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f314c1f5f90 con 0x7f314c070df0 2024-08-31T12:16:07.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.425+0000 7f313dffb640 1 -- 172.21.15.19:0/127232850 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f314400ade0 con 0x7f314c070df0 2024-08-31T12:16:07.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.425+0000 7f315347e640 1 -- 172.21.15.19:0/127232850 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f314c1f94d0 con 0x7f314c070df0 2024-08-31T12:16:07.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.425+0000 7f313dffb640 1 -- 172.21.15.19:0/127232850 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3144016020 con 0x7f314c070df0 2024-08-31T12:16:07.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.425+0000 7fe8e77fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8d80039e0 0x7fe8e00d9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:07.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.425+0000 7fe8e77fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8d80039e0 0x7fe8e00d9e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:47322/0 (socket says 172.21.15.19:47322) 2024-08-31T12:16:07.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.426+0000 7fe8e77fe640 1 -- 172.21.15.19:0/2357358490 learned_addr learned my addr 172.21.15.19:0/2357358490 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:07.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.426+0000 7fe8e6ffd640 1 --2- 172.21.15.19:0/2357358490 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe8e00daf20 0x7fe8e00de710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:07.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.426+0000 7fe8ecadf640 1 --2- 172.21.15.19:0/2357358490 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe8e00dec50 0x7fe8e00d8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:07.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.426+0000 7fe8ecadf640 1 -- 172.21.15.19:0/2357358490 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fe8e00092c0 con 0x7fe8e00daf20 2024-08-31T12:16:07.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.426+0000 7fe8ecadf640 1 -- 172.21.15.19:0/2357358490 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe8e0008da0 con 0x7fe8d80039e0 2024-08-31T12:16:07.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.426+0000 7fe8ecadf640 1 -- 172.21.15.19:0/2357358490 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fe8e00444e0 con 0x7fe8e00dec50 2024-08-31T12:16:07.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.426+0000 7fe8e7fff640 1 --2- 172.21.15.19:0/2357358490 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe8e00dec50 0x7fe8e00d8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:07.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.427+0000 7fb9ca778640 1 -- 172.21.15.19:0/1888095369 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9c41f6130 con 0x7fb9c4150700 2024-08-31T12:16:07.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.426+0000 7efcc4580640 1 -- start start 2024-08-31T12:16:07.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.427+0000 7fb9b9ffb640 1 -- 172.21.15.19:0/1888095369 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fb9b40334d0 con 0x7fb9c4150700 2024-08-31T12:16:07.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.427+0000 7fe8e6ffd640 1 -- 172.21.15.19:0/2357358490 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe8e00dec50 msgr2=0x7fe8e00d8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:07.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.427+0000 7fe8e6ffd640 1 --2- 172.21.15.19:0/2357358490 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe8e00dec50 0x7fe8e00d8aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.427+0000 7fe8e6ffd640 1 -- 172.21.15.19:0/2357358490 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8d80039e0 msgr2=0x7fe8e00d9e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:07.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.427+0000 7fe8e6ffd640 1 --2- 172.21.15.19:0/2357358490 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8d80039e0 0x7fe8e00d9e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.427+0000 7fe8e6ffd640 1 -- 172.21.15.19:0/2357358490 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8e00da3a0 con 0x7fe8e00daf20 2024-08-31T12:16:07.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.427+0000 7fe8e6ffd640 1 --2- 172.21.15.19:0/2357358490 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe8e00daf20 0x7fe8e00de710 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe8e00336a0 tx=0x7fe8d400fe70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:07.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.428+0000 7fb9b9ffb640 1 --2- 172.21.15.19:0/1888095369 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb9a8077b50 0x7fb9a807a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:07.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.428+0000 7fb9b9ffb640 1 -- 172.21.15.19:0/1888095369 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb9b40ba8f0 con 0x7fb9c4150700 2024-08-31T12:16:07.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.428+0000 7efcc4580640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efcbc070920 0x7efcbc13d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:07.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.428+0000 7efcc4580640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efcbc072b80 0x7efcbc155400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:07.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.428+0000 7efcc4580640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efcbc0c6e60 0x7efcbc155f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:07.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.428+0000 7efcc231c640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efcbc070920 0x7efcbc13d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:07.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.428+0000 7efcc231c640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efcbc070920 0x7efcbc13d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:47332/0 (socket says 172.21.15.19:47332) 2024-08-31T12:16:07.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.428+0000 7efcc1b1b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efcbc072b80 0x7efcbc155400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:07.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.428+0000 7efcc1b1b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efcbc072b80 0x7efcbc155400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.19:49150/0 (socket says 172.21.15.19:49150) 2024-08-31T12:16:07.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.428+0000 7efcc2b1d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efcbc0c6e60 0x7efcbc155f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:07.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.428+0000 7efcc2b1d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efcbc0c6e60 0x7efcbc155f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:56856/0 (socket says 172.21.15.19:56856) 2024-08-31T12:16:07.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.429+0000 7efcc4580640 1 -- 172.21.15.19:0/1707373632 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7efcbc073f30 con 0x7efcbc0c6e60 2024-08-31T12:16:07.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.429+0000 7efcc231c640 1 -- 172.21.15.19:0/1707373632 learned_addr learned my addr 172.21.15.19:0/1707373632 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:07.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.429+0000 7efcc4580640 1 -- 172.21.15.19:0/1707373632 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7efcbc073db0 con 0x7efcbc070920 2024-08-31T12:16:07.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.429+0000 7efcc4580640 1 -- 172.21.15.19:0/1707373632 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7efcbc1513b0 con 0x7efcbc072b80 2024-08-31T12:16:07.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.429+0000 7efcc231c640 1 -- 172.21.15.19:0/1707373632 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efcbc072b80 msgr2=0x7efcbc155400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:07.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.429+0000 7efcc231c640 1 --2- 172.21.15.19:0/1707373632 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efcbc072b80 0x7efcbc155400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.429+0000 7efcc231c640 1 -- 172.21.15.19:0/1707373632 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efcbc0c6e60 msgr2=0x7efcbc155f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:07.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.429+0000 7efcc231c640 1 --2- 172.21.15.19:0/1707373632 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efcbc0c6e60 0x7efcbc155f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.429+0000 7efcc231c640 1 -- 172.21.15.19:0/1707373632 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcbc13d880 con 0x7efcbc070920 2024-08-31T12:16:07.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.429+0000 7efcc2b1d640 1 --2- 172.21.15.19:0/1707373632 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efcbc0c6e60 0x7efcbc155f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:16:07.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.429+0000 7efcc231c640 1 --2- 172.21.15.19:0/1707373632 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efcbc070920 0x7efcbc13d340 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efcb8013470 tx=0x7efcb80045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:07.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.430+0000 7efcb2ffd640 1 -- 172.21.15.19:0/1707373632 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb800fe60 con 0x7efcbc070920 2024-08-31T12:16:07.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.430+0000 7fb9c9776640 1 --2- 172.21.15.19:0/1888095369 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb9a8077b50 0x7fb9a807a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:07.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.430+0000 7efcb2ffd640 1 -- 172.21.15.19:0/1707373632 <== mon.1 v2:172.21.15.43:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efcb8004a70 con 0x7efcbc070920 2024-08-31T12:16:07.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.430+0000 7efcb2ffd640 1 -- 172.21.15.19:0/1707373632 <== mon.1 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb80110d0 con 0x7efcbc070920 2024-08-31T12:16:07.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.430+0000 7efcc4580640 1 -- 172.21.15.19:0/1707373632 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcbc155940 con 0x7efcbc070920 2024-08-31T12:16:07.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.430+0000 7efcc4580640 1 -- 172.21.15.19:0/1707373632 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcbc13c860 con 0x7efcbc070920 2024-08-31T12:16:07.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.431+0000 7f315347e640 1 -- 172.21.15.19:0/127232850 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f312c005290 con 0x7f314c070df0 2024-08-31T12:16:07.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.432+0000 7f313dffb640 1 -- 172.21.15.19:0/127232850 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f3144019020 con 0x7f314c070df0 2024-08-31T12:16:07.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.432+0000 7efcb2ffd640 1 -- 172.21.15.19:0/1707373632 <== mon.1 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7efcb8011270 con 0x7efcbc070920 2024-08-31T12:16:07.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.432+0000 7f313dffb640 1 --2- 172.21.15.19:0/127232850 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3130077c80 0x7f313007a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:07.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.433+0000 7f313dffb640 1 -- 172.21.15.19:0/127232850 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3144099940 con 0x7f314c070df0 2024-08-31T12:16:07.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.433+0000 7fb9c9776640 1 --2- 172.21.15.19:0/1888095369 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb9a8077b50 0x7fb9a807a060 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb9c00013f0 tx=0x7fb9c0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:07.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.433+0000 7efcb0ff9640 1 -- 172.21.15.19:0/1707373632 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcbc0c5240 con 0x7efcbc070920 2024-08-31T12:16:07.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.433+0000 7efcb2ffd640 1 --2- 172.21.15.19:0/1707373632 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7efca0077a80 0x7efca0079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:07.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.433+0000 7efcb2ffd640 1 -- 172.21.15.19:0/1707373632 <== mon.1 v2:172.21.15.43:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7efcb80b9490 con 0x7efcbc070920 2024-08-31T12:16:07.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.433+0000 7efcc1b1b640 1 --2- 172.21.15.19:0/1707373632 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7efca0077a80 0x7efca0079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:07.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.434+0000 7f3150a19640 1 --2- 172.21.15.19:0/127232850 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3130077c80 0x7f313007a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:07.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.434+0000 7f3150a19640 1 --2- 172.21.15.19:0/127232850 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3130077c80 0x7f313007a190 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3140001670 tx=0x7f3140009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:07.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.434+0000 7efcc1b1b640 1 --2- 172.21.15.19:0/1707373632 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7efca0077a80 0x7efca0079f90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7efcbc1ef890 tx=0x7efcac005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:07.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.435+0000 7fe8d3fff640 1 -- 172.21.15.19:0/2357358490 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d401d070 con 0x7fe8e00daf20 2024-08-31T12:16:07.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.435+0000 7fe8ecadf640 1 -- 172.21.15.19:0/2357358490 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8e00d9980 con 0x7fe8e00daf20 2024-08-31T12:16:07.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.435+0000 7fb9b9ffb640 1 -- 172.21.15.19:0/1888095369 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fb9b403f030 con 0x7fb9c4150700 2024-08-31T12:16:07.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.435+0000 7fe8ecadf640 1 -- 172.21.15.19:0/2357358490 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8e00479c0 con 0x7fe8e00daf20 2024-08-31T12:16:07.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.437+0000 7fe8ecadf640 1 -- 172.21.15.19:0/2357358490 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8e00e2d00 con 0x7fe8e00daf20 2024-08-31T12:16:07.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.439+0000 7efcb2ffd640 1 -- 172.21.15.19:0/1707373632 <== mon.1 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7efcb804a060 con 0x7efcbc070920 2024-08-31T12:16:07.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.440+0000 7fe8d3fff640 1 -- 172.21.15.19:0/2357358490 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe8d400dcd0 con 0x7fe8e00daf20 2024-08-31T12:16:07.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.440+0000 7fe8d3fff640 1 -- 172.21.15.19:0/2357358490 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d400dea0 con 0x7fe8e00daf20 2024-08-31T12:16:07.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.441+0000 7f313dffb640 1 -- 172.21.15.19:0/127232850 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f3144020020 con 0x7f314c070df0 2024-08-31T12:16:07.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.444+0000 7fe8d3fff640 1 -- 172.21.15.19:0/2357358490 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fe8d4020070 con 0x7fe8e00daf20 2024-08-31T12:16:07.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.445+0000 7fe8d3fff640 1 --2- 172.21.15.19:0/2357358490 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe8cc077a80 0x7fe8cc079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:07.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.445+0000 7fe8d3fff640 1 -- 172.21.15.19:0/2357358490 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe8d4099c70 con 0x7fe8e00daf20 2024-08-31T12:16:07.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.450+0000 7fe8d3fff640 1 -- 172.21.15.19:0/2357358490 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fe8e00e2d00 con 0x7fe8e00daf20 2024-08-31T12:16:07.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.451+0000 7fe8e77fe640 1 --2- 172.21.15.19:0/2357358490 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe8cc077a80 0x7fe8cc079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:07.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.467+0000 7fe8e77fe640 1 --2- 172.21.15.19:0/2357358490 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe8cc077a80 0x7fe8cc079f90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc000c00 tx=0x7fe8dc0036f0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:07.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.742+0000 7f1a76924640 1 -- 172.21.15.19:0/512466581 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f1a700636d0 con 0x7f1a7014d480 2024-08-31T12:16:07.744 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.744+0000 7f1a6cff9640 1 -- 172.21.15.19:0/512466581 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1a64062200 con 0x7f1a7014d480 2024-08-31T12:16:07.744 INFO:teuthology.orchestra.run.smithi019.stdout:55834574855 2024-08-31T12:16:07.748 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.747+0000 7f1a567fc640 1 -- 172.21.15.19:0/512466581 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1a5c077b80 msgr2=0x7f1a5c07a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:07.748 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.747+0000 7f1a567fc640 1 --2- 172.21.15.19:0/512466581 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1a5c077b80 0x7f1a5c07a090 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1a60000f60 tx=0x7f1a60009040 comp rx=0 tx=0).stop 2024-08-31T12:16:07.748 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.748+0000 7f1a567fc640 1 -- 172.21.15.19:0/512466581 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1a7014d480 msgr2=0x7f1a70145f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:07.748 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.748+0000 7f1a567fc640 1 --2- 172.21.15.19:0/512466581 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1a7014d480 0x7f1a70145f50 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f1a6400b550 tx=0x7f1a6400ba10 comp rx=0 tx=0).stop 2024-08-31T12:16:07.749 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.748+0000 7f1a567fc640 1 -- 172.21.15.19:0/512466581 shutdown_connections 2024-08-31T12:16:07.749 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.748+0000 7f1a567fc640 1 --2- 172.21.15.19:0/512466581 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1a5c077b80 0x7f1a5c07a090 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.748+0000 7f1a567fc640 1 --2- 172.21.15.19:0/512466581 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1a7014d480 0x7f1a70145f50 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.748+0000 7f1a567fc640 1 --2- 172.21.15.19:0/512466581 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1a700c7170 0x7f1a701564c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.748+0000 7f1a567fc640 1 --2- 172.21.15.19:0/512466581 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1a70070bb0 0x7f1a701f6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.748+0000 7f1a567fc640 1 -- 172.21.15.19:0/512466581 >> 172.21.15.19:0/512466581 conn(0x7f1a700cc950 msgr2=0x7f1a70147170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:07.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.749+0000 7f1a567fc640 1 -- 172.21.15.19:0/512466581 shutdown_connections 2024-08-31T12:16:07.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.749+0000 7f1a567fc640 1 -- 172.21.15.19:0/512466581 wait complete. 2024-08-31T12:16:07.766 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.2 2024-08-31T12:16:07.767 DEBUG:teuthology.parallel:result is None 2024-08-31T12:16:07.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.780+0000 7f96717fa640 1 -- 172.21.15.19:0/2314858609 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f9660005580 con 0x7f967c150170 2024-08-31T12:16:07.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.781+0000 7f96737fe640 1 -- 172.21.15.19:0/2314858609 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f966c073220 con 0x7f967c150170 2024-08-31T12:16:07.782 INFO:teuthology.orchestra.run.smithi019.stdout:60129542151 2024-08-31T12:16:07.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.783+0000 7f96717fa640 1 -- 172.21.15.19:0/2314858609 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f965c077b50 msgr2=0x7f965c07a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:07.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.783+0000 7f96717fa640 1 --2- 172.21.15.19:0/2314858609 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f965c077b50 0x7f965c07a060 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f967c0c4bf0 tx=0x7f9664009040 comp rx=0 tx=0).stop 2024-08-31T12:16:07.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.783+0000 7f96717fa640 1 -- 172.21.15.19:0/2314858609 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f967c150170 msgr2=0x7f967c1f5970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:07.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.783+0000 7f96717fa640 1 --2- 172.21.15.19:0/2314858609 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f967c150170 0x7f967c1f5970 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f966c01d770 tx=0x7f966c01dc30 comp rx=0 tx=0).stop 2024-08-31T12:16:07.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.783+0000 7f96717fa640 1 -- 172.21.15.19:0/2314858609 shutdown_connections 2024-08-31T12:16:07.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.783+0000 7f96717fa640 1 --2- 172.21.15.19:0/2314858609 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f965c077b50 0x7f965c07a060 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.783+0000 7f96717fa640 1 --2- 172.21.15.19:0/2314858609 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f967c15a0d0 0x7f967c1487e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.783+0000 7f96717fa640 1 --2- 172.21.15.19:0/2314858609 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f967c15b9b0 0x7f967c0cc3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.783+0000 7f96717fa640 1 --2- 172.21.15.19:0/2314858609 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f967c150170 0x7f967c1f5970 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.783+0000 7f96717fa640 1 -- 172.21.15.19:0/2314858609 >> 172.21.15.19:0/2314858609 conn(0x7f967c0cda10 msgr2=0x7f967c071ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:07.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.783+0000 7f96717fa640 1 -- 172.21.15.19:0/2314858609 shutdown_connections 2024-08-31T12:16:07.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.783+0000 7f96717fa640 1 -- 172.21.15.19:0/2314858609 wait complete. 2024-08-31T12:16:07.795 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.4 2024-08-31T12:16:07.795 DEBUG:teuthology.parallel:result is None 2024-08-31T12:16:07.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.813+0000 7fa3e1f67640 1 -- 172.21.15.19:0/2863317481 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fa3d4033f60 con 0x7fa3d4048970 2024-08-31T12:16:07.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.814+0000 7fa3d97fa640 1 -- 172.21.15.19:0/2863317481 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa3dc1071f0 con 0x7fa3d4048970 2024-08-31T12:16:07.814 INFO:teuthology.orchestra.run.smithi019.stdout:64424509448 2024-08-31T12:16:07.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.817+0000 7fa3e1f67640 1 -- 172.21.15.19:0/2863317481 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa3c0077b20 msgr2=0x7fa3c007a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:07.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.817+0000 7fa3e1f67640 1 --2- 172.21.15.19:0/2863317481 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa3c0077b20 0x7fa3c007a030 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa3d40d7190 tx=0x7fa3d00079d0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.817+0000 7fa3e1f67640 1 -- 172.21.15.19:0/2863317481 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa3d4048970 msgr2=0x7fa3d40d80f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:07.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.817+0000 7fa3e1f67640 1 --2- 172.21.15.19:0/2863317481 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa3d4048970 0x7fa3d40d80f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc0526b0 tx=0x7fa3dc0afcf0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.817+0000 7fa3e1f67640 1 -- 172.21.15.19:0/2863317481 shutdown_connections 2024-08-31T12:16:07.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.817+0000 7fa3e1f67640 1 --2- 172.21.15.19:0/2863317481 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa3c0077b20 0x7fa3c007a030 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.817+0000 7fa3e1f67640 1 --2- 172.21.15.19:0/2863317481 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3d404c4e0 0x7fa3d4007aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.817+0000 7fa3e1f67640 1 --2- 172.21.15.19:0/2863317481 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa3d4048ec0 0x7fa3d40cf4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.817+0000 7fa3e1f67640 1 --2- 172.21.15.19:0/2863317481 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa3d4048970 0x7fa3d40d80f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.817+0000 7fa3e1f67640 1 -- 172.21.15.19:0/2863317481 >> 172.21.15.19:0/2863317481 conn(0x7fa3d4050fa0 msgr2=0x7fa3d40d0110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:07.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.817+0000 7fa3e1f67640 1 -- 172.21.15.19:0/2863317481 shutdown_connections 2024-08-31T12:16:07.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.818+0000 7fa3e1f67640 1 -- 172.21.15.19:0/2863317481 wait complete. 2024-08-31T12:16:07.822 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.821+0000 7fee35cb6640 1 -- 172.21.15.19:0/3063856160 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fee3014b0c0 con 0x7fee300cb5b0 2024-08-31T12:16:07.823 INFO:teuthology.orchestra.run.smithi019.stdout:60129542153 2024-08-31T12:16:07.823 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.822+0000 7fee17fff640 1 -- 172.21.15.19:0/3063856160 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fee3014b0c0 con 0x7fee300cb5b0 2024-08-31T12:16:07.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.825+0000 7fee35cb6640 1 -- 172.21.15.19:0/3063856160 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fee1c077ad0 msgr2=0x7fee1c079fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:07.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.825+0000 7fee35cb6640 1 --2- 172.21.15.19:0/3063856160 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fee1c077ad0 0x7fee1c079fe0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fee28006010 tx=0x7fee28004ab0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.825+0000 7fee35cb6640 1 -- 172.21.15.19:0/3063856160 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fee300cb5b0 msgr2=0x7fee3014b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:07.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.825+0000 7fee35cb6640 1 --2- 172.21.15.19:0/3063856160 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fee300cb5b0 0x7fee3014b2c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fee2400c4d0 tx=0x7fee2400c990 comp rx=0 tx=0).stop 2024-08-31T12:16:07.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.826+0000 7fee35cb6640 1 -- 172.21.15.19:0/3063856160 shutdown_connections 2024-08-31T12:16:07.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.826+0000 7fee35cb6640 1 --2- 172.21.15.19:0/3063856160 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fee1c077ad0 0x7fee1c079fe0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.826+0000 7fee35cb6640 1 --2- 172.21.15.19:0/3063856160 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fee300cb5b0 0x7fee3014b2c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.826+0000 7fee35cb6640 1 --2- 172.21.15.19:0/3063856160 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fee300b3630 0x7fee30073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.826+0000 7fee35cb6640 1 --2- 172.21.15.19:0/3063856160 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee3006ebe0 0x7fee3014a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.826+0000 7fee35cb6640 1 -- 172.21.15.19:0/3063856160 >> 172.21.15.19:0/3063856160 conn(0x7fee30069350 msgr2=0x7fee300d0690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:07.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.826+0000 7fee35cb6640 1 -- 172.21.15.19:0/3063856160 shutdown_connections 2024-08-31T12:16:07.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.826+0000 7fee35cb6640 1 -- 172.21.15.19:0/3063856160 wait complete. 2024-08-31T12:16:07.831 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509448 for osd.0 2024-08-31T12:16:07.831 DEBUG:teuthology.parallel:result is None 2024-08-31T12:16:07.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.840+0000 7efcb0ff9640 1 -- 172.21.15.19:0/1707373632 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7efcbc150800 con 0x7efcbc070920 2024-08-31T12:16:07.841 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.840+0000 7efcb2ffd640 1 -- 172.21.15.19:0/1707373632 <== mon.1 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7efcb8041070 con 0x7efcbc070920 2024-08-31T12:16:07.841 INFO:teuthology.orchestra.run.smithi019.stdout:60129542151 2024-08-31T12:16:07.842 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542153 for osd.5 2024-08-31T12:16:07.843 DEBUG:teuthology.parallel:result is None 2024-08-31T12:16:07.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.844+0000 7efcb0ff9640 1 -- 172.21.15.19:0/1707373632 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7efca0077a80 msgr2=0x7efca0079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:07.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.844+0000 7efcb0ff9640 1 --2- 172.21.15.19:0/1707373632 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7efca0077a80 0x7efca0079f90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7efcbc1ef890 tx=0x7efcac005f90 comp rx=0 tx=0).stop 2024-08-31T12:16:07.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.844+0000 7efcb0ff9640 1 -- 172.21.15.19:0/1707373632 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efcbc070920 msgr2=0x7efcbc13d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:07.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.844+0000 7efcb0ff9640 1 --2- 172.21.15.19:0/1707373632 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efcbc070920 0x7efcbc13d340 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efcb8013470 tx=0x7efcb80045e0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.844+0000 7efcb0ff9640 1 -- 172.21.15.19:0/1707373632 shutdown_connections 2024-08-31T12:16:07.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.844+0000 7efcb0ff9640 1 --2- 172.21.15.19:0/1707373632 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7efca0077a80 0x7efca0079f90 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.844+0000 7efcb0ff9640 1 --2- 172.21.15.19:0/1707373632 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efcbc0c6e60 0x7efcbc155f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.844+0000 7efcb0ff9640 1 --2- 172.21.15.19:0/1707373632 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efcbc072b80 0x7efcbc155400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.844+0000 7efcb0ff9640 1 --2- 172.21.15.19:0/1707373632 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efcbc070920 0x7efcbc13d340 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.844+0000 7efcb0ff9640 1 -- 172.21.15.19:0/1707373632 >> 172.21.15.19:0/1707373632 conn(0x7efcbc0cc950 msgr2=0x7efcbc150c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:07.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.844+0000 7efcb0ff9640 1 -- 172.21.15.19:0/1707373632 shutdown_connections 2024-08-31T12:16:07.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.844+0000 7efcb0ff9640 1 -- 172.21.15.19:0/1707373632 wait complete. 2024-08-31T12:16:07.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.856+0000 7fb9ca778640 1 -- 172.21.15.19:0/1888095369 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fb9c414cca0 con 0x7fb9c4150700 2024-08-31T12:16:07.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.857+0000 7fb9b9ffb640 1 -- 172.21.15.19:0/1888095369 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb9b40836d0 con 0x7fb9c4150700 2024-08-31T12:16:07.858 INFO:teuthology.orchestra.run.smithi019.stdout:64424509447 2024-08-31T12:16:07.859 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.7 2024-08-31T12:16:07.859 DEBUG:teuthology.parallel:result is None 2024-08-31T12:16:07.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.859+0000 7fb9ca778640 1 -- 172.21.15.19:0/1888095369 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb9a8077b50 msgr2=0x7fb9a807a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:07.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.859+0000 7fb9ca778640 1 --2- 172.21.15.19:0/1888095369 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb9a8077b50 0x7fb9a807a060 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb9c00013f0 tx=0x7fb9c0009040 comp rx=0 tx=0).stop 2024-08-31T12:16:07.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.859+0000 7fb9ca778640 1 -- 172.21.15.19:0/1888095369 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb9c4150700 msgr2=0x7fb9c415dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:07.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.859+0000 7fb9ca778640 1 --2- 172.21.15.19:0/1888095369 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb9c4150700 0x7fb9c415dc20 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb9b4000c00 tx=0x7fb9b40316c0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.859+0000 7fb9ca778640 1 -- 172.21.15.19:0/1888095369 shutdown_connections 2024-08-31T12:16:07.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.859+0000 7fb9ca778640 1 --2- 172.21.15.19:0/1888095369 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb9a8077b50 0x7fb9a807a060 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.859+0000 7fb9ca778640 1 --2- 172.21.15.19:0/1888095369 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb9c415b950 0x7fb9c4146bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.859+0000 7fb9ca778640 1 --2- 172.21.15.19:0/1888095369 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb9c4150700 0x7fb9c415dc20 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.859+0000 7fb9ca778640 1 --2- 172.21.15.19:0/1888095369 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb9c4072940 0x7fb9c414b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.859+0000 7fb9ca778640 1 -- 172.21.15.19:0/1888095369 >> 172.21.15.19:0/1888095369 conn(0x7fb9c406cae0 msgr2=0x7fb9c406cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:07.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.859+0000 7fb9ca778640 1 -- 172.21.15.19:0/1888095369 shutdown_connections 2024-08-31T12:16:07.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.859+0000 7fb9ca778640 1 -- 172.21.15.19:0/1888095369 wait complete. 2024-08-31T12:16:07.871 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509447 for osd.3 2024-08-31T12:16:07.871 DEBUG:teuthology.parallel:result is None 2024-08-31T12:16:07.877 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.877+0000 7f315347e640 1 -- 172.21.15.19:0/127232850 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f312c005580 con 0x7f314c070df0 2024-08-31T12:16:07.878 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.877+0000 7f313dffb640 1 -- 172.21.15.19:0/127232850 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3144062500 con 0x7f314c070df0 2024-08-31T12:16:07.878 INFO:teuthology.orchestra.run.smithi019.stdout:64424509448 2024-08-31T12:16:07.880 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.879+0000 7f315347e640 1 -- 172.21.15.19:0/127232850 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3130077c80 msgr2=0x7f313007a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:07.880 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.879+0000 7f315347e640 1 --2- 172.21.15.19:0/127232850 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3130077c80 0x7f313007a190 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3140001670 tx=0x7f3140009040 comp rx=0 tx=0).stop 2024-08-31T12:16:07.880 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.879+0000 7f315347e640 1 -- 172.21.15.19:0/127232850 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f314c070df0 msgr2=0x7f314c1f5a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:07.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.879+0000 7f315347e640 1 --2- 172.21.15.19:0/127232850 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f314c070df0 0x7f314c1f5a50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f314400cbf0 tx=0x7f314400a430 comp rx=0 tx=0).stop 2024-08-31T12:16:07.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.879+0000 7f315347e640 1 -- 172.21.15.19:0/127232850 shutdown_connections 2024-08-31T12:16:07.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.879+0000 7f315347e640 1 --2- 172.21.15.19:0/127232850 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3130077c80 0x7f313007a190 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.879+0000 7f315347e640 1 --2- 172.21.15.19:0/127232850 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f314c15bdf0 0x7f314c15d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.879+0000 7f315347e640 1 --2- 172.21.15.19:0/127232850 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f314c0c8070 0x7f314c14a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.879+0000 7f315347e640 1 --2- 172.21.15.19:0/127232850 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f314c070df0 0x7f314c1f5a50 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.879+0000 7f315347e640 1 -- 172.21.15.19:0/127232850 >> 172.21.15.19:0/127232850 conn(0x7f314c0ce0b0 msgr2=0x7f314c0c6070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:07.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.879+0000 7f315347e640 1 -- 172.21.15.19:0/127232850 shutdown_connections 2024-08-31T12:16:07.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.879+0000 7f315347e640 1 -- 172.21.15.19:0/127232850 wait complete. 2024-08-31T12:16:07.882 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.881+0000 7fe8ecadf640 1 -- 172.21.15.19:0/2357358490 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fe8e00518e0 con 0x7fe8e00daf20 2024-08-31T12:16:07.882 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.881+0000 7fe8d3fff640 1 -- 172.21.15.19:0/2357358490 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe8e00518e0 con 0x7fe8e00daf20 2024-08-31T12:16:07.882 INFO:teuthology.orchestra.run.smithi019.stdout:64424509448 2024-08-31T12:16:07.885 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.883+0000 7fe8ecadf640 1 -- 172.21.15.19:0/2357358490 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe8cc077a80 msgr2=0x7fe8cc079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:07.885 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.883+0000 7fe8ecadf640 1 --2- 172.21.15.19:0/2357358490 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe8cc077a80 0x7fe8cc079f90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc000c00 tx=0x7fe8dc0036f0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.885 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.883+0000 7fe8ecadf640 1 -- 172.21.15.19:0/2357358490 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe8e00daf20 msgr2=0x7fe8e00de710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:07.885 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.883+0000 7fe8ecadf640 1 --2- 172.21.15.19:0/2357358490 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe8e00daf20 0x7fe8e00de710 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe8e00336a0 tx=0x7fe8d400fe70 comp rx=0 tx=0).stop 2024-08-31T12:16:07.885 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.883+0000 7fe8ecadf640 1 -- 172.21.15.19:0/2357358490 shutdown_connections 2024-08-31T12:16:07.885 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.883+0000 7fe8ecadf640 1 --2- 172.21.15.19:0/2357358490 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe8cc077a80 0x7fe8cc079f90 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.885 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.883+0000 7fe8ecadf640 1 --2- 172.21.15.19:0/2357358490 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe8e00dec50 0x7fe8e00d8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.885 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.883+0000 7fe8ecadf640 1 --2- 172.21.15.19:0/2357358490 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe8e00daf20 0x7fe8e00de710 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.885 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.883+0000 7fe8ecadf640 1 --2- 172.21.15.19:0/2357358490 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8d80039e0 0x7fe8e00d9e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:07.886 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.883+0000 7fe8ecadf640 1 -- 172.21.15.19:0/2357358490 >> 172.21.15.19:0/2357358490 conn(0x7fe8e0050fa0 msgr2=0x7fe8e00e64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:07.886 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.884+0000 7fe8ecadf640 1 -- 172.21.15.19:0/2357358490 shutdown_connections 2024-08-31T12:16:07.886 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:07.884+0000 7fe8ecadf640 1 -- 172.21.15.19:0/2357358490 wait complete. 2024-08-31T12:16:07.892 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509448 for osd.1 2024-08-31T12:16:07.892 DEBUG:teuthology.parallel:result is None 2024-08-31T12:16:07.896 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509448 for osd.6 2024-08-31T12:16:07.896 DEBUG:teuthology.parallel:result is None 2024-08-31T12:16:07.896 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-08-31T12:16:07.896 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-31T12:16:08.050 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.048+0000 7f1d7899e640 1 -- 172.21.15.19:0/706190082 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d74155300 msgr2=0x7f1d7406d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:08.050 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.048+0000 7f1d7899e640 1 --2- 172.21.15.19:0/706190082 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d74155300 0x7f1d7406d000 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1d64005fd0 tx=0x7f1d6402ab20 comp rx=0 tx=0).stop 2024-08-31T12:16:08.050 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.049+0000 7f1d7899e640 1 -- 172.21.15.19:0/706190082 shutdown_connections 2024-08-31T12:16:08.050 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.049+0000 7f1d7899e640 1 --2- 172.21.15.19:0/706190082 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d74155300 0x7f1d7406d000 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:08.050 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.049+0000 7f1d7899e640 1 --2- 172.21.15.19:0/706190082 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d7406e0e0 0x7f1d740c9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:08.050 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.049+0000 7f1d7899e640 1 -- 172.21.15.19:0/706190082 >> 172.21.15.19:0/706190082 conn(0x7f1d740690d0 msgr2=0x7f1d740d0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:08.050 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.049+0000 7f1d7899e640 1 -- 172.21.15.19:0/706190082 shutdown_connections 2024-08-31T12:16:08.050 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.049+0000 7f1d7899e640 1 -- 172.21.15.19:0/706190082 wait complete. 2024-08-31T12:16:08.051 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.050+0000 7f1d7899e640 1 Processor -- start 2024-08-31T12:16:08.051 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.050+0000 7f1d7899e640 1 -- start start 2024-08-31T12:16:08.054 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.052+0000 7f1d7899e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d7406e0e0 0x7f1d740c67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:08.054 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.052+0000 7f1d7899e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d740c8a40 0x7f1d741f5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:08.054 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.052+0000 7f1d7899e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d74155300 0x7f1d740c4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:08.054 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.052+0000 7f1d7899e640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f1d740c8330 con 0x7f1d740c8a40 2024-08-31T12:16:08.054 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.052+0000 7f1d7899e640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1d7415b250 con 0x7f1d7406e0e0 2024-08-31T12:16:08.054 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.052+0000 7f1d7899e640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f1d740c8610 con 0x7f1d74155300 2024-08-31T12:16:08.054 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.052+0000 7f1d71d9b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d740c8a40 0x7f1d741f5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:08.054 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.052+0000 7f1d71d9b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d740c8a40 0x7f1d741f5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:56882/0 (socket says 172.21.15.19:56882) 2024-08-31T12:16:08.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.052+0000 7f1d71d9b640 1 -- 172.21.15.19:0/3473936520 learned_addr learned my addr 172.21.15.19:0/3473936520 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:08.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.052+0000 7f1d7259c640 1 --2- 172.21.15.19:0/3473936520 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d7406e0e0 0x7f1d740c67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:08.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.052+0000 7f1d72d9d640 1 --2- 172.21.15.19:0/3473936520 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d74155300 0x7f1d740c4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:08.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.052+0000 7f1d71d9b640 1 -- 172.21.15.19:0/3473936520 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d74155300 msgr2=0x7f1d740c4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:08.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.052+0000 7f1d71d9b640 1 --2- 172.21.15.19:0/3473936520 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d74155300 0x7f1d740c4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:08.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.052+0000 7f1d71d9b640 1 -- 172.21.15.19:0/3473936520 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d7406e0e0 msgr2=0x7f1d740c67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:08.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.052+0000 7f1d71d9b640 1 --2- 172.21.15.19:0/3473936520 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d7406e0e0 0x7f1d740c67b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:08.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.052+0000 7f1d71d9b640 1 -- 172.21.15.19:0/3473936520 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d740c6cf0 con 0x7f1d740c8a40 2024-08-31T12:16:08.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.052+0000 7f1d72d9d640 1 --2- 172.21.15.19:0/3473936520 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d74155300 0x7f1d740c4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-31T12:16:08.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.052+0000 7f1d71d9b640 1 --2- 172.21.15.19:0/3473936520 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d740c8a40 0x7f1d741f5040 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c00d650 tx=0x7f1d5c00db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:08.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.052+0000 7f1d6affd640 1 -- 172.21.15.19:0/3473936520 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d5c017070 con 0x7f1d740c8a40 2024-08-31T12:16:08.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.052+0000 7f1d6affd640 1 -- 172.21.15.19:0/3473936520 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1d5c005310 con 0x7f1d740c8a40 2024-08-31T12:16:08.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.053+0000 7f1d7899e640 1 -- 172.21.15.19:0/3473936520 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d741f5d00 con 0x7f1d740c8a40 2024-08-31T12:16:08.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.053+0000 7f1d6affd640 1 -- 172.21.15.19:0/3473936520 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d5c009820 con 0x7f1d740c8a40 2024-08-31T12:16:08.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.053+0000 7f1d7899e640 1 -- 172.21.15.19:0/3473936520 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d741f5a20 con 0x7f1d740c8a40 2024-08-31T12:16:08.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.054+0000 7f1d7899e640 1 -- 172.21.15.19:0/3473936520 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d740c9850 con 0x7f1d740c8a40 2024-08-31T12:16:08.057 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.054+0000 7f1d6affd640 1 -- 172.21.15.19:0/3473936520 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f1d5c0099c0 con 0x7f1d740c8a40 2024-08-31T12:16:08.057 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.055+0000 7f1d6affd640 1 --2- 172.21.15.19:0/3473936520 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1d54077a80 0x7f1d54079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:08.057 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.055+0000 7f1d6affd640 1 -- 172.21.15.19:0/3473936520 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f1d5c0997b0 con 0x7f1d740c8a40 2024-08-31T12:16:08.057 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.055+0000 7f1d7259c640 1 --2- 172.21.15.19:0/3473936520 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1d54077a80 0x7f1d54079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:08.057 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.055+0000 7f1d7259c640 1 --2- 172.21.15.19:0/3473936520 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1d54077a80 0x7f1d54079f90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1d740cb2b0 tx=0x7f1d6c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:08.060 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.060+0000 7f1d6affd640 1 -- 172.21.15.19:0/3473936520 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f1d5c062520 con 0x7f1d740c8a40 2024-08-31T12:16:08.242 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.241+0000 7f1d7899e640 1 -- 172.21.15.19:0/3473936520 --> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1d740c2770 con 0x7f1d54077a80 2024-08-31T12:16:08.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.246+0000 7f1d6affd640 1 -- 172.21.15.19:0/3473936520 <== mgr.4100 v2:172.21.15.43:6800/2824336655 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251824 (secure 0 0 0) 0x7f1d740c2770 con 0x7f1d54077a80 2024-08-31T12:16:08.250 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:16:08.251 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-08-31T12:16:08.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.249+0000 7f1d7899e640 1 -- 172.21.15.19:0/3473936520 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1d54077a80 msgr2=0x7f1d54079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:08.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.249+0000 7f1d7899e640 1 --2- 172.21.15.19:0/3473936520 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1d54077a80 0x7f1d54079f90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1d740cb2b0 tx=0x7f1d6c009040 comp rx=0 tx=0).stop 2024-08-31T12:16:08.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.249+0000 7f1d7899e640 1 -- 172.21.15.19:0/3473936520 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d740c8a40 msgr2=0x7f1d741f5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:08.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.249+0000 7f1d7899e640 1 --2- 172.21.15.19:0/3473936520 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d740c8a40 0x7f1d741f5040 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c00d650 tx=0x7f1d5c00db10 comp rx=0 tx=0).stop 2024-08-31T12:16:08.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.249+0000 7f1d7899e640 1 -- 172.21.15.19:0/3473936520 shutdown_connections 2024-08-31T12:16:08.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.249+0000 7f1d7899e640 1 --2- 172.21.15.19:0/3473936520 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1d54077a80 0x7f1d54079f90 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:08.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.249+0000 7f1d7899e640 1 --2- 172.21.15.19:0/3473936520 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d74155300 0x7f1d740c4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:08.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.249+0000 7f1d7899e640 1 --2- 172.21.15.19:0/3473936520 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d740c8a40 0x7f1d741f5040 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:08.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.249+0000 7f1d7899e640 1 --2- 172.21.15.19:0/3473936520 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d7406e0e0 0x7f1d740c67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:08.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.249+0000 7f1d7899e640 1 -- 172.21.15.19:0/3473936520 >> 172.21.15.19:0/3473936520 conn(0x7f1d740690d0 msgr2=0x7f1d74150f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:08.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.249+0000 7f1d7899e640 1 -- 172.21.15.19:0/3473936520 shutdown_connections 2024-08-31T12:16:08.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.250+0000 7f1d7899e640 1 -- 172.21.15.19:0/3473936520 wait complete. 2024-08-31T12:16:08.261 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-08-31T12:16:06.493278+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":145,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":252328,"kb_used_data":2856,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754722392,"statfs":{"total":773094113280,"available":772835729408,"internally_reserved":0,"allocated":2924544,"data_stored":1770364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.208385"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306730+0000","last_change":"2024-08-31T12:15:57.306560+0000","last_active":"2024-08-31T12:15:58.306730+0000","last_peered":"2024-08-31T12:15:58.306730+0000","last_clean":"2024-08-31T12:15:58.306730+0000","last_became_active":"2024-08-31T12:15:57.306148+0000","last_became_peered":"2024-08-31T12:15:57.306148+0000","last_unstale":"2024-08-31T12:15:58.306730+0000","last_undegraded":"2024-08-31T12:15:58.306730+0000","last_fullsized":"2024-08-31T12:15:58.306730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:35:53.831742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300935+0000","last_change":"2024-08-31T12:15:57.307314+0000","last_active":"2024-08-31T12:15:58.300935+0000","last_peered":"2024-08-31T12:15:58.300935+0000","last_clean":"2024-08-31T12:15:58.300935+0000","last_became_active":"2024-08-31T12:15:57.306940+0000","last_became_peered":"2024-08-31T12:15:57.306940+0000","last_unstale":"2024-08-31T12:15:58.300935+0000","last_undegraded":"2024-08-31T12:15:58.300935+0000","last_fullsized":"2024-08-31T12:15:58.300935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:38:30.321214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937267+0000","last_change":"2024-08-31T12:15:57.309372+0000","last_active":"2024-08-31T12:15:58.937267+0000","last_peered":"2024-08-31T12:15:58.937267+0000","last_clean":"2024-08-31T12:15:58.937267+0000","last_became_active":"2024-08-31T12:15:57.305795+0000","last_became_peered":"2024-08-31T12:15:57.305795+0000","last_unstale":"2024-08-31T12:15:58.937267+0000","last_undegraded":"2024-08-31T12:15:58.937267+0000","last_fullsized":"2024-08-31T12:15:58.937267+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:37:45.891394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302363+0000","last_change":"2024-08-31T12:15:57.307339+0000","last_active":"2024-08-31T12:15:58.302363+0000","last_peered":"2024-08-31T12:15:58.302363+0000","last_clean":"2024-08-31T12:15:58.302363+0000","last_became_active":"2024-08-31T12:15:57.306957+0000","last_became_peered":"2024-08-31T12:15:57.306957+0000","last_unstale":"2024-08-31T12:15:58.302363+0000","last_undegraded":"2024-08-31T12:15:58.302363+0000","last_fullsized":"2024-08-31T12:15:58.302363+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:43:33.705099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306757+0000","last_change":"2024-08-31T12:15:57.317492+0000","last_active":"2024-08-31T12:15:58.306757+0000","last_peered":"2024-08-31T12:15:58.306757+0000","last_clean":"2024-08-31T12:15:58.306757+0000","last_became_active":"2024-08-31T12:15:57.317079+0000","last_became_peered":"2024-08-31T12:15:57.317079+0000","last_unstale":"2024-08-31T12:15:58.306757+0000","last_undegraded":"2024-08-31T12:15:58.306757+0000","last_fullsized":"2024-08-31T12:15:58.306757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:44:40.712699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302342+0000","last_change":"2024-08-31T12:15:57.307505+0000","last_active":"2024-08-31T12:15:58.302342+0000","last_peered":"2024-08-31T12:15:58.302342+0000","last_clean":"2024-08-31T12:15:58.302342+0000","last_became_active":"2024-08-31T12:15:57.307010+0000","last_became_peered":"2024-08-31T12:15:57.307010+0000","last_unstale":"2024-08-31T12:15:58.302342+0000","last_undegraded":"2024-08-31T12:15:58.302342+0000","last_fullsized":"2024-08-31T12:15:58.302342+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:39:31.100565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306247+0000","last_change":"2024-08-31T12:15:57.316892+0000","last_active":"2024-08-31T12:15:58.306247+0000","last_peered":"2024-08-31T12:15:58.306247+0000","last_clean":"2024-08-31T12:15:58.306247+0000","last_became_active":"2024-08-31T12:15:57.316504+0000","last_became_peered":"2024-08-31T12:15:57.316504+0000","last_unstale":"2024-08-31T12:15:58.306247+0000","last_undegraded":"2024-08-31T12:15:58.306247+0000","last_fullsized":"2024-08-31T12:15:58.306247+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:05:48.614918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937976+0000","last_change":"2024-08-31T12:15:57.309224+0000","last_active":"2024-08-31T12:15:58.937976+0000","last_peered":"2024-08-31T12:15:58.937976+0000","last_clean":"2024-08-31T12:15:58.937976+0000","last_became_active":"2024-08-31T12:15:57.308914+0000","last_became_peered":"2024-08-31T12:15:57.308914+0000","last_unstale":"2024-08-31T12:15:58.937976+0000","last_undegraded":"2024-08-31T12:15:58.937976+0000","last_fullsized":"2024-08-31T12:15:58.937976+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:37:02.435678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306762+0000","last_change":"2024-08-31T12:15:57.316723+0000","last_active":"2024-08-31T12:15:58.306762+0000","last_peered":"2024-08-31T12:15:58.306762+0000","last_clean":"2024-08-31T12:15:58.306762+0000","last_became_active":"2024-08-31T12:15:57.316354+0000","last_became_peered":"2024-08-31T12:15:57.316354+0000","last_unstale":"2024-08-31T12:15:58.306762+0000","last_undegraded":"2024-08-31T12:15:58.306762+0000","last_fullsized":"2024-08-31T12:15:58.306762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:31:19.860950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302888+0000","last_change":"2024-08-31T12:15:57.311161+0000","last_active":"2024-08-31T12:15:58.302888+0000","last_peered":"2024-08-31T12:15:58.302888+0000","last_clean":"2024-08-31T12:15:58.302888+0000","last_became_active":"2024-08-31T12:15:57.310807+0000","last_became_peered":"2024-08-31T12:15:57.310807+0000","last_unstale":"2024-08-31T12:15:58.302888+0000","last_undegraded":"2024-08-31T12:15:58.302888+0000","last_fullsized":"2024-08-31T12:15:58.302888+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:56:03.280339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302118+0000","last_change":"2024-08-31T12:15:57.300930+0000","last_active":"2024-08-31T12:15:58.302118+0000","last_peered":"2024-08-31T12:15:58.302118+0000","last_clean":"2024-08-31T12:15:58.302118+0000","last_became_active":"2024-08-31T12:15:57.300610+0000","last_became_peered":"2024-08-31T12:15:57.300610+0000","last_unstale":"2024-08-31T12:15:58.302118+0000","last_undegraded":"2024-08-31T12:15:58.302118+0000","last_fullsized":"2024-08-31T12:15:58.302118+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:42:39.731742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936435+0000","last_change":"2024-08-31T12:15:57.303151+0000","last_active":"2024-08-31T12:15:58.936435+0000","last_peered":"2024-08-31T12:15:58.936435+0000","last_clean":"2024-08-31T12:15:58.936435+0000","last_became_active":"2024-08-31T12:15:57.302622+0000","last_became_peered":"2024-08-31T12:15:57.302622+0000","last_unstale":"2024-08-31T12:15:58.936435+0000","last_undegraded":"2024-08-31T12:15:58.936435+0000","last_fullsized":"2024-08-31T12:15:58.936435+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:49:18.493629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937309+0000","last_change":"2024-08-31T12:15:57.305676+0000","last_active":"2024-08-31T12:15:58.937309+0000","last_peered":"2024-08-31T12:15:58.937309+0000","last_clean":"2024-08-31T12:15:58.937309+0000","last_became_active":"2024-08-31T12:15:57.305321+0000","last_became_peered":"2024-08-31T12:15:57.305321+0000","last_unstale":"2024-08-31T12:15:58.937309+0000","last_undegraded":"2024-08-31T12:15:58.937309+0000","last_fullsized":"2024-08-31T12:15:58.937309+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:42:39.731742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936688+0000","last_change":"2024-08-31T12:15:57.309671+0000","last_active":"2024-08-31T12:15:58.936688+0000","last_peered":"2024-08-31T12:15:58.936688+0000","last_clean":"2024-08-31T12:15:58.936688+0000","last_became_active":"2024-08-31T12:15:57.307013+0000","last_became_peered":"2024-08-31T12:15:57.307013+0000","last_unstale":"2024-08-31T12:15:58.936688+0000","last_undegraded":"2024-08-31T12:15:58.936688+0000","last_fullsized":"2024-08-31T12:15:58.936688+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:18:55.451164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300929+0000","last_change":"2024-08-31T12:15:57.298150+0000","last_active":"2024-08-31T12:15:58.300929+0000","last_peered":"2024-08-31T12:15:58.300929+0000","last_clean":"2024-08-31T12:15:58.300929+0000","last_became_active":"2024-08-31T12:15:57.297182+0000","last_became_peered":"2024-08-31T12:15:57.297182+0000","last_unstale":"2024-08-31T12:15:58.300929+0000","last_undegraded":"2024-08-31T12:15:58.300929+0000","last_fullsized":"2024-08-31T12:15:58.300929+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:16:42.625396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302762+0000","last_change":"2024-08-31T12:15:57.314224+0000","last_active":"2024-08-31T12:15:58.302762+0000","last_peered":"2024-08-31T12:15:58.302762+0000","last_clean":"2024-08-31T12:15:58.302762+0000","last_became_active":"2024-08-31T12:15:57.313887+0000","last_became_peered":"2024-08-31T12:15:57.313887+0000","last_unstale":"2024-08-31T12:15:58.302762+0000","last_undegraded":"2024-08-31T12:15:58.302762+0000","last_fullsized":"2024-08-31T12:15:58.302762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:54:45.720621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303640+0000","last_change":"2024-08-31T12:15:57.308899+0000","last_active":"2024-08-31T12:15:58.303640+0000","last_peered":"2024-08-31T12:15:58.303640+0000","last_clean":"2024-08-31T12:15:58.303640+0000","last_became_active":"2024-08-31T12:15:57.308388+0000","last_became_peered":"2024-08-31T12:15:57.308388+0000","last_unstale":"2024-08-31T12:15:58.303640+0000","last_undegraded":"2024-08-31T12:15:58.303640+0000","last_fullsized":"2024-08-31T12:15:58.303640+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:54:03.871840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936410+0000","last_change":"2024-08-31T12:15:57.300201+0000","last_active":"2024-08-31T12:15:58.936410+0000","last_peered":"2024-08-31T12:15:58.936410+0000","last_clean":"2024-08-31T12:15:58.936410+0000","last_became_active":"2024-08-31T12:15:57.299826+0000","last_became_peered":"2024-08-31T12:15:57.299826+0000","last_unstale":"2024-08-31T12:15:58.936410+0000","last_undegraded":"2024-08-31T12:15:58.936410+0000","last_fullsized":"2024-08-31T12:15:58.936410+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:14:34.488102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.307053+0000","last_change":"2024-08-31T12:15:57.317606+0000","last_active":"2024-08-31T12:15:58.307053+0000","last_peered":"2024-08-31T12:15:58.307053+0000","last_clean":"2024-08-31T12:15:58.307053+0000","last_became_active":"2024-08-31T12:15:57.317185+0000","last_became_peered":"2024-08-31T12:15:57.317185+0000","last_unstale":"2024-08-31T12:15:58.307053+0000","last_undegraded":"2024-08-31T12:15:58.307053+0000","last_fullsized":"2024-08-31T12:15:58.307053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:25:46.346490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302171+0000","last_change":"2024-08-31T12:15:57.298778+0000","last_active":"2024-08-31T12:15:58.302171+0000","last_peered":"2024-08-31T12:15:58.302171+0000","last_clean":"2024-08-31T12:15:58.302171+0000","last_became_active":"2024-08-31T12:15:57.298430+0000","last_became_peered":"2024-08-31T12:15:57.298430+0000","last_unstale":"2024-08-31T12:15:58.302171+0000","last_undegraded":"2024-08-31T12:15:58.302171+0000","last_fullsized":"2024-08-31T12:15:58.302171+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:22:13.435300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937190+0000","last_change":"2024-08-31T12:15:57.317176+0000","last_active":"2024-08-31T12:15:58.937190+0000","last_peered":"2024-08-31T12:15:58.937190+0000","last_clean":"2024-08-31T12:15:58.937190+0000","last_became_active":"2024-08-31T12:15:57.316870+0000","last_became_peered":"2024-08-31T12:15:57.316870+0000","last_unstale":"2024-08-31T12:15:58.937190+0000","last_undegraded":"2024-08-31T12:15:58.937190+0000","last_fullsized":"2024-08-31T12:15:58.937190+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:43:12.703971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302636+0000","last_change":"2024-08-31T12:15:57.308907+0000","last_active":"2024-08-31T12:15:58.302636+0000","last_peered":"2024-08-31T12:15:58.302636+0000","last_clean":"2024-08-31T12:15:58.302636+0000","last_became_active":"2024-08-31T12:15:57.308459+0000","last_became_peered":"2024-08-31T12:15:57.308459+0000","last_unstale":"2024-08-31T12:15:58.302636+0000","last_undegraded":"2024-08-31T12:15:58.302636+0000","last_fullsized":"2024-08-31T12:15:58.302636+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:16:46.648912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937760+0000","last_change":"2024-08-31T12:15:57.304032+0000","last_active":"2024-08-31T12:15:58.937760+0000","last_peered":"2024-08-31T12:15:58.937760+0000","last_clean":"2024-08-31T12:15:58.937760+0000","last_became_active":"2024-08-31T12:15:57.303401+0000","last_became_peered":"2024-08-31T12:15:57.303401+0000","last_unstale":"2024-08-31T12:15:58.937760+0000","last_undegraded":"2024-08-31T12:15:58.937760+0000","last_fullsized":"2024-08-31T12:15:58.937760+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:56:03.280339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303013+0000","last_change":"2024-08-31T12:15:57.305412+0000","last_active":"2024-08-31T12:15:58.303013+0000","last_peered":"2024-08-31T12:15:58.303013+0000","last_clean":"2024-08-31T12:15:58.303013+0000","last_became_active":"2024-08-31T12:15:57.305075+0000","last_became_peered":"2024-08-31T12:15:57.305075+0000","last_unstale":"2024-08-31T12:15:58.303013+0000","last_undegraded":"2024-08-31T12:15:58.303013+0000","last_fullsized":"2024-08-31T12:15:58.303013+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:30:13.715505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937828+0000","last_change":"2024-08-31T12:15:57.300179+0000","last_active":"2024-08-31T12:15:58.937828+0000","last_peered":"2024-08-31T12:15:58.937828+0000","last_clean":"2024-08-31T12:15:58.937828+0000","last_became_active":"2024-08-31T12:15:57.299861+0000","last_became_peered":"2024-08-31T12:15:57.299861+0000","last_unstale":"2024-08-31T12:15:58.937828+0000","last_undegraded":"2024-08-31T12:15:58.937828+0000","last_fullsized":"2024-08-31T12:15:58.937828+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:47:51.267409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303001+0000","last_change":"2024-08-31T12:15:57.308519+0000","last_active":"2024-08-31T12:15:58.303001+0000","last_peered":"2024-08-31T12:15:58.303001+0000","last_clean":"2024-08-31T12:15:58.303001+0000","last_became_active":"2024-08-31T12:15:57.308189+0000","last_became_peered":"2024-08-31T12:15:57.308189+0000","last_unstale":"2024-08-31T12:15:58.303001+0000","last_undegraded":"2024-08-31T12:15:58.303001+0000","last_fullsized":"2024-08-31T12:15:58.303001+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:37:02.435678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300666+0000","last_change":"2024-08-31T12:15:57.308321+0000","last_active":"2024-08-31T12:15:58.300666+0000","last_peered":"2024-08-31T12:15:58.300666+0000","last_clean":"2024-08-31T12:15:58.300666+0000","last_became_active":"2024-08-31T12:15:57.304514+0000","last_became_peered":"2024-08-31T12:15:57.304514+0000","last_unstale":"2024-08-31T12:15:58.300666+0000","last_undegraded":"2024-08-31T12:15:58.300666+0000","last_fullsized":"2024-08-31T12:15:58.300666+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:09:49.617826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936830+0000","last_change":"2024-08-31T12:15:57.299268+0000","last_active":"2024-08-31T12:15:58.936830+0000","last_peered":"2024-08-31T12:15:58.936830+0000","last_clean":"2024-08-31T12:15:58.936830+0000","last_became_active":"2024-08-31T12:15:57.298766+0000","last_became_peered":"2024-08-31T12:15:57.298766+0000","last_unstale":"2024-08-31T12:15:58.936830+0000","last_undegraded":"2024-08-31T12:15:58.936830+0000","last_fullsized":"2024-08-31T12:15:58.936830+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:37:45.891394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937525+0000","last_change":"2024-08-31T12:15:57.310121+0000","last_active":"2024-08-31T12:15:58.937525+0000","last_peered":"2024-08-31T12:15:58.937525+0000","last_clean":"2024-08-31T12:15:58.937525+0000","last_became_active":"2024-08-31T12:15:57.309440+0000","last_became_peered":"2024-08-31T12:15:57.309440+0000","last_unstale":"2024-08-31T12:15:58.937525+0000","last_undegraded":"2024-08-31T12:15:58.937525+0000","last_fullsized":"2024-08-31T12:15:58.937525+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:53:19.523811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302889+0000","last_change":"2024-08-31T12:15:57.306214+0000","last_active":"2024-08-31T12:15:58.302889+0000","last_peered":"2024-08-31T12:15:58.302889+0000","last_clean":"2024-08-31T12:15:58.302889+0000","last_became_active":"2024-08-31T12:15:57.305878+0000","last_became_peered":"2024-08-31T12:15:57.305878+0000","last_unstale":"2024-08-31T12:15:58.302889+0000","last_undegraded":"2024-08-31T12:15:58.302889+0000","last_fullsized":"2024-08-31T12:15:58.302889+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:05:48.614918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.938282+0000","last_change":"2024-08-31T12:15:57.307376+0000","last_active":"2024-08-31T12:15:58.938282+0000","last_peered":"2024-08-31T12:15:58.938282+0000","last_clean":"2024-08-31T12:15:58.938282+0000","last_became_active":"2024-08-31T12:15:57.307056+0000","last_became_peered":"2024-08-31T12:15:57.307056+0000","last_unstale":"2024-08-31T12:15:58.938282+0000","last_undegraded":"2024-08-31T12:15:58.938282+0000","last_fullsized":"2024-08-31T12:15:58.938282+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:52:18.004608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302319+0000","last_change":"2024-08-31T12:15:57.313798+0000","last_active":"2024-08-31T12:15:58.302319+0000","last_peered":"2024-08-31T12:15:58.302319+0000","last_clean":"2024-08-31T12:15:58.302319+0000","last_became_active":"2024-08-31T12:15:57.313463+0000","last_became_peered":"2024-08-31T12:15:57.313463+0000","last_unstale":"2024-08-31T12:15:58.302319+0000","last_undegraded":"2024-08-31T12:15:58.302319+0000","last_fullsized":"2024-08-31T12:15:58.302319+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:47:51.267409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936240+0000","last_change":"2024-08-31T12:15:57.293350+0000","last_active":"2024-08-31T12:15:58.936240+0000","last_peered":"2024-08-31T12:15:58.936240+0000","last_clean":"2024-08-31T12:15:58.936240+0000","last_became_active":"2024-08-31T12:15:57.292981+0000","last_became_peered":"2024-08-31T12:15:57.292981+0000","last_unstale":"2024-08-31T12:15:58.936240+0000","last_undegraded":"2024-08-31T12:15:58.936240+0000","last_fullsized":"2024-08-31T12:15:58.936240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:52:00.312013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937453+0000","last_change":"2024-08-31T12:15:56.293497+0000","last_active":"2024-08-31T12:15:58.937453+0000","last_peered":"2024-08-31T12:15:58.937453+0000","last_clean":"2024-08-31T12:15:58.937453+0000","last_became_active":"2024-08-31T12:15:56.293164+0000","last_became_peered":"2024-08-31T12:15:56.293164+0000","last_unstale":"2024-08-31T12:15:58.937453+0000","last_undegraded":"2024-08-31T12:15:58.937453+0000","last_fullsized":"2024-08-31T12:15:58.937453+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:43:17.443433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937814+0000","last_change":"2024-08-31T12:15:57.312738+0000","last_active":"2024-08-31T12:15:58.937814+0000","last_peered":"2024-08-31T12:15:58.937814+0000","last_clean":"2024-08-31T12:15:58.937814+0000","last_became_active":"2024-08-31T12:15:57.312360+0000","last_became_peered":"2024-08-31T12:15:57.312360+0000","last_unstale":"2024-08-31T12:15:58.937814+0000","last_undegraded":"2024-08-31T12:15:58.937814+0000","last_fullsized":"2024-08-31T12:15:58.937814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:22:13.435300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.301828+0000","last_change":"2024-08-31T12:15:56.283526+0000","last_active":"2024-08-31T12:15:58.301828+0000","last_peered":"2024-08-31T12:15:58.301828+0000","last_clean":"2024-08-31T12:15:58.301828+0000","last_became_active":"2024-08-31T12:15:56.283147+0000","last_became_peered":"2024-08-31T12:15:56.283147+0000","last_unstale":"2024-08-31T12:15:58.301828+0000","last_undegraded":"2024-08-31T12:15:58.301828+0000","last_fullsized":"2024-08-31T12:15:58.301828+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:51:59.306912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937727+0000","last_change":"2024-08-31T12:15:56.291170+0000","last_active":"2024-08-31T12:15:58.937727+0000","last_peered":"2024-08-31T12:15:58.937727+0000","last_clean":"2024-08-31T12:15:58.937727+0000","last_became_active":"2024-08-31T12:15:56.290835+0000","last_became_peered":"2024-08-31T12:15:56.290835+0000","last_unstale":"2024-08-31T12:15:58.937727+0000","last_undegraded":"2024-08-31T12:15:58.937727+0000","last_fullsized":"2024-08-31T12:15:58.937727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:15:51.561831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.938053+0000","last_change":"2024-08-31T12:15:57.308183+0000","last_active":"2024-08-31T12:15:58.938053+0000","last_peered":"2024-08-31T12:15:58.938053+0000","last_clean":"2024-08-31T12:15:58.938053+0000","last_became_active":"2024-08-31T12:15:57.306962+0000","last_became_peered":"2024-08-31T12:15:57.306962+0000","last_unstale":"2024-08-31T12:15:58.938053+0000","last_undegraded":"2024-08-31T12:15:58.938053+0000","last_fullsized":"2024-08-31T12:15:58.938053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:01:18.188264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303216+0000","last_change":"2024-08-31T12:15:56.288126+0000","last_active":"2024-08-31T12:15:58.303216+0000","last_peered":"2024-08-31T12:15:58.303216+0000","last_clean":"2024-08-31T12:15:58.303216+0000","last_became_active":"2024-08-31T12:15:56.287552+0000","last_became_peered":"2024-08-31T12:15:56.287552+0000","last_unstale":"2024-08-31T12:15:58.303216+0000","last_undegraded":"2024-08-31T12:15:58.303216+0000","last_fullsized":"2024-08-31T12:15:58.303216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:54:29.988079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300810+0000","last_change":"2024-08-31T12:15:57.305029+0000","last_active":"2024-08-31T12:15:58.300810+0000","last_peered":"2024-08-31T12:15:58.300810+0000","last_clean":"2024-08-31T12:15:58.300810+0000","last_became_active":"2024-08-31T12:15:57.304700+0000","last_became_peered":"2024-08-31T12:15:57.304700+0000","last_unstale":"2024-08-31T12:15:58.300810+0000","last_undegraded":"2024-08-31T12:15:58.300810+0000","last_fullsized":"2024-08-31T12:15:58.300810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:58:17.482204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302528+0000","last_change":"2024-08-31T12:15:56.284907+0000","last_active":"2024-08-31T12:15:58.302528+0000","last_peered":"2024-08-31T12:15:58.302528+0000","last_clean":"2024-08-31T12:15:58.302528+0000","last_became_active":"2024-08-31T12:15:56.284558+0000","last_became_peered":"2024-08-31T12:15:56.284558+0000","last_unstale":"2024-08-31T12:15:58.302528+0000","last_undegraded":"2024-08-31T12:15:58.302528+0000","last_fullsized":"2024-08-31T12:15:58.302528+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:29:53.117724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303538+0000","last_change":"2024-08-31T12:15:57.308679+0000","last_active":"2024-08-31T12:15:58.303538+0000","last_peered":"2024-08-31T12:15:58.303538+0000","last_clean":"2024-08-31T12:15:58.303538+0000","last_became_active":"2024-08-31T12:15:57.308260+0000","last_became_peered":"2024-08-31T12:15:57.308260+0000","last_unstale":"2024-08-31T12:15:58.303538+0000","last_undegraded":"2024-08-31T12:15:58.303538+0000","last_fullsized":"2024-08-31T12:15:58.303538+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:31:19.860950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303564+0000","last_change":"2024-08-31T12:15:56.290763+0000","last_active":"2024-08-31T12:15:58.303564+0000","last_peered":"2024-08-31T12:15:58.303564+0000","last_clean":"2024-08-31T12:15:58.303564+0000","last_became_active":"2024-08-31T12:15:56.290400+0000","last_became_peered":"2024-08-31T12:15:56.290400+0000","last_unstale":"2024-08-31T12:15:58.303564+0000","last_undegraded":"2024-08-31T12:15:58.303564+0000","last_fullsized":"2024-08-31T12:15:58.303564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:53:08.929860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.938122+0000","last_change":"2024-08-31T12:15:57.311657+0000","last_active":"2024-08-31T12:15:58.938122+0000","last_peered":"2024-08-31T12:15:58.938122+0000","last_clean":"2024-08-31T12:15:58.938122+0000","last_became_active":"2024-08-31T12:15:57.310564+0000","last_became_peered":"2024-08-31T12:15:57.310564+0000","last_unstale":"2024-08-31T12:15:58.938122+0000","last_undegraded":"2024-08-31T12:15:58.938122+0000","last_fullsized":"2024-08-31T12:15:58.938122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:37:02.435678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:02.303958+0000","last_change":"2024-08-31T12:15:56.286792+0000","last_active":"2024-08-31T12:16:02.303958+0000","last_peered":"2024-08-31T12:16:02.303958+0000","last_clean":"2024-08-31T12:16:02.303958+0000","last_became_active":"2024-08-31T12:15:56.286458+0000","last_became_peered":"2024-08-31T12:15:56.286458+0000","last_unstale":"2024-08-31T12:16:02.303958+0000","last_undegraded":"2024-08-31T12:16:02.303958+0000","last_fullsized":"2024-08-31T12:16:02.303958+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:49:17.488528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936603+0000","last_change":"2024-08-31T12:15:57.309485+0000","last_active":"2024-08-31T12:15:58.936603+0000","last_peered":"2024-08-31T12:15:58.936603+0000","last_clean":"2024-08-31T12:15:58.936603+0000","last_became_active":"2024-08-31T12:15:57.309102+0000","last_became_peered":"2024-08-31T12:15:57.309102+0000","last_unstale":"2024-08-31T12:15:58.936603+0000","last_undegraded":"2024-08-31T12:15:58.936603+0000","last_fullsized":"2024-08-31T12:15:58.936603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:17:34.692296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:02.306321+0000","last_change":"2024-08-31T12:15:56.289460+0000","last_active":"2024-08-31T12:16:02.306321+0000","last_peered":"2024-08-31T12:16:02.306321+0000","last_clean":"2024-08-31T12:16:02.306321+0000","last_became_active":"2024-08-31T12:15:56.289096+0000","last_became_peered":"2024-08-31T12:15:56.289096+0000","last_unstale":"2024-08-31T12:16:02.306321+0000","last_undegraded":"2024-08-31T12:16:02.306321+0000","last_fullsized":"2024-08-31T12:16:02.306321+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:06:34.313772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306865+0000","last_change":"2024-08-31T12:15:57.317968+0000","last_active":"2024-08-31T12:15:58.306865+0000","last_peered":"2024-08-31T12:15:58.306865+0000","last_clean":"2024-08-31T12:15:58.306865+0000","last_became_active":"2024-08-31T12:15:57.317632+0000","last_became_peered":"2024-08-31T12:15:57.317632+0000","last_unstale":"2024-08-31T12:15:58.306865+0000","last_undegraded":"2024-08-31T12:15:58.306865+0000","last_fullsized":"2024-08-31T12:15:58.306865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:22:02.569106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:02.304711+0000","last_change":"2024-08-31T12:15:56.284014+0000","last_active":"2024-08-31T12:16:02.304711+0000","last_peered":"2024-08-31T12:16:02.304711+0000","last_clean":"2024-08-31T12:16:02.304711+0000","last_became_active":"2024-08-31T12:15:56.283675+0000","last_became_peered":"2024-08-31T12:15:56.283675+0000","last_unstale":"2024-08-31T12:16:02.304711+0000","last_undegraded":"2024-08-31T12:16:02.304711+0000","last_fullsized":"2024-08-31T12:16:02.304711+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:51:59.306912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937018+0000","last_change":"2024-08-31T12:15:57.312440+0000","last_active":"2024-08-31T12:15:58.937018+0000","last_peered":"2024-08-31T12:15:58.937018+0000","last_clean":"2024-08-31T12:15:58.937018+0000","last_became_active":"2024-08-31T12:15:57.312007+0000","last_became_peered":"2024-08-31T12:15:57.312007+0000","last_unstale":"2024-08-31T12:15:58.937018+0000","last_undegraded":"2024-08-31T12:15:58.937018+0000","last_fullsized":"2024-08-31T12:15:58.937018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:23:43.553625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303030+0000","last_change":"2024-08-31T12:15:56.290968+0000","last_active":"2024-08-31T12:15:58.303030+0000","last_peered":"2024-08-31T12:15:58.303030+0000","last_clean":"2024-08-31T12:15:58.303030+0000","last_became_active":"2024-08-31T12:15:56.290603+0000","last_became_peered":"2024-08-31T12:15:56.290603+0000","last_unstale":"2024-08-31T12:15:58.303030+0000","last_undegraded":"2024-08-31T12:15:58.303030+0000","last_fullsized":"2024-08-31T12:15:58.303030+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:50:23.215239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.307024+0000","last_change":"2024-08-31T12:15:57.312790+0000","last_active":"2024-08-31T12:15:58.307024+0000","last_peered":"2024-08-31T12:15:58.307024+0000","last_clean":"2024-08-31T12:15:58.307024+0000","last_became_active":"2024-08-31T12:15:57.312383+0000","last_became_peered":"2024-08-31T12:15:57.312383+0000","last_unstale":"2024-08-31T12:15:58.307024+0000","last_undegraded":"2024-08-31T12:15:58.307024+0000","last_fullsized":"2024-08-31T12:15:58.307024+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:20:41.042375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.301697+0000","last_change":"2024-08-31T12:15:56.289795+0000","last_active":"2024-08-31T12:15:58.301697+0000","last_peered":"2024-08-31T12:15:58.301697+0000","last_clean":"2024-08-31T12:15:58.301697+0000","last_became_active":"2024-08-31T12:15:56.289477+0000","last_became_peered":"2024-08-31T12:15:56.289477+0000","last_unstale":"2024-08-31T12:15:58.301697+0000","last_undegraded":"2024-08-31T12:15:58.301697+0000","last_fullsized":"2024-08-31T12:15:58.301697+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:15:51.561831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302374+0000","last_change":"2024-08-31T12:15:57.312865+0000","last_active":"2024-08-31T12:15:58.302374+0000","last_peered":"2024-08-31T12:15:58.302374+0000","last_clean":"2024-08-31T12:15:58.302374+0000","last_became_active":"2024-08-31T12:15:57.312462+0000","last_became_peered":"2024-08-31T12:15:57.312462+0000","last_unstale":"2024-08-31T12:15:58.302374+0000","last_undegraded":"2024-08-31T12:15:58.302374+0000","last_fullsized":"2024-08-31T12:15:58.302374+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:16:42.625396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:02.306724+0000","last_change":"2024-08-31T12:15:56.285412+0000","last_active":"2024-08-31T12:16:02.306724+0000","last_peered":"2024-08-31T12:16:02.306724+0000","last_clean":"2024-08-31T12:16:02.306724+0000","last_became_active":"2024-08-31T12:15:56.285108+0000","last_became_peered":"2024-08-31T12:15:56.285108+0000","last_unstale":"2024-08-31T12:16:02.306724+0000","last_undegraded":"2024-08-31T12:16:02.306724+0000","last_fullsized":"2024-08-31T12:16:02.306724+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:29:45.580957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306407+0000","last_change":"2024-08-31T12:15:57.311701+0000","last_active":"2024-08-31T12:15:58.306407+0000","last_peered":"2024-08-31T12:15:58.306407+0000","last_clean":"2024-08-31T12:15:58.306407+0000","last_became_active":"2024-08-31T12:15:57.311368+0000","last_became_peered":"2024-08-31T12:15:57.311368+0000","last_unstale":"2024-08-31T12:15:58.306407+0000","last_undegraded":"2024-08-31T12:15:58.306407+0000","last_fullsized":"2024-08-31T12:15:58.306407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:30:13.715505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:00.126954+0000","last_change":"2024-08-31T12:15:56.289979+0000","last_active":"2024-08-31T12:16:00.126954+0000","last_peered":"2024-08-31T12:16:00.126954+0000","last_clean":"2024-08-31T12:16:00.126954+0000","last_became_active":"2024-08-31T12:15:56.288289+0000","last_became_peered":"2024-08-31T12:15:56.288289+0000","last_unstale":"2024-08-31T12:16:00.126954+0000","last_undegraded":"2024-08-31T12:16:00.126954+0000","last_fullsized":"2024-08-31T12:16:00.126954+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:46:55.057087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306825+0000","last_change":"2024-08-31T12:15:57.311580+0000","last_active":"2024-08-31T12:15:58.306825+0000","last_peered":"2024-08-31T12:15:58.306825+0000","last_clean":"2024-08-31T12:15:58.306825+0000","last_became_active":"2024-08-31T12:15:57.311323+0000","last_became_peered":"2024-08-31T12:15:57.311323+0000","last_unstale":"2024-08-31T12:15:58.306825+0000","last_undegraded":"2024-08-31T12:15:58.306825+0000","last_fullsized":"2024-08-31T12:15:58.306825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:54:03.871840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:02.304157+0000","last_change":"2024-08-31T12:15:56.290496+0000","last_active":"2024-08-31T12:16:02.304157+0000","last_peered":"2024-08-31T12:16:02.304157+0000","last_clean":"2024-08-31T12:16:02.304157+0000","last_became_active":"2024-08-31T12:15:56.290196+0000","last_became_peered":"2024-08-31T12:15:56.290196+0000","last_unstale":"2024-08-31T12:16:02.304157+0000","last_undegraded":"2024-08-31T12:16:02.304157+0000","last_fullsized":"2024-08-31T12:16:02.304157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:52:20.055757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306986+0000","last_change":"2024-08-31T12:15:57.313317+0000","last_active":"2024-08-31T12:15:58.306986+0000","last_peered":"2024-08-31T12:15:58.306986+0000","last_clean":"2024-08-31T12:15:58.306986+0000","last_became_active":"2024-08-31T12:15:57.312145+0000","last_became_peered":"2024-08-31T12:15:57.312145+0000","last_unstale":"2024-08-31T12:15:58.306986+0000","last_undegraded":"2024-08-31T12:15:58.306986+0000","last_fullsized":"2024-08-31T12:15:58.306986+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:09:31.904512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300816+0000","last_change":"2024-08-31T12:15:56.281809+0000","last_active":"2024-08-31T12:15:58.300816+0000","last_peered":"2024-08-31T12:15:58.300816+0000","last_clean":"2024-08-31T12:15:58.300816+0000","last_became_active":"2024-08-31T12:15:56.281368+0000","last_became_peered":"2024-08-31T12:15:56.281368+0000","last_unstale":"2024-08-31T12:15:58.300816+0000","last_undegraded":"2024-08-31T12:15:58.300816+0000","last_fullsized":"2024-08-31T12:15:58.300816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:17:33.687196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303702+0000","last_change":"2024-08-31T12:15:57.305874+0000","last_active":"2024-08-31T12:15:58.303702+0000","last_peered":"2024-08-31T12:15:58.303702+0000","last_clean":"2024-08-31T12:15:58.303702+0000","last_became_active":"2024-08-31T12:15:57.305507+0000","last_became_peered":"2024-08-31T12:15:57.305507+0000","last_unstale":"2024-08-31T12:15:58.303702+0000","last_undegraded":"2024-08-31T12:15:58.303702+0000","last_fullsized":"2024-08-31T12:15:58.303702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:22:02.569106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937377+0000","last_change":"2024-08-31T12:15:56.291851+0000","last_active":"2024-08-31T12:15:58.937377+0000","last_peered":"2024-08-31T12:15:58.937377+0000","last_clean":"2024-08-31T12:15:58.937377+0000","last_became_active":"2024-08-31T12:15:56.291446+0000","last_became_peered":"2024-08-31T12:15:56.291446+0000","last_unstale":"2024-08-31T12:15:58.937377+0000","last_undegraded":"2024-08-31T12:15:58.937377+0000","last_fullsized":"2024-08-31T12:15:58.937377+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:54:29.988079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306538+0000","last_change":"2024-08-31T12:15:57.316154+0000","last_active":"2024-08-31T12:15:58.306538+0000","last_peered":"2024-08-31T12:15:58.306538+0000","last_clean":"2024-08-31T12:15:58.306538+0000","last_became_active":"2024-08-31T12:15:57.315830+0000","last_became_peered":"2024-08-31T12:15:57.315830+0000","last_unstale":"2024-08-31T12:15:58.306538+0000","last_undegraded":"2024-08-31T12:15:58.306538+0000","last_fullsized":"2024-08-31T12:15:58.306538+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:05:53.039520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":111,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306756+0000","last_change":"2024-08-31T12:15:47.084912+0000","last_active":"2024-08-31T12:15:58.306756+0000","last_peered":"2024-08-31T12:15:58.306756+0000","last_clean":"2024-08-31T12:15:58.306756+0000","last_became_active":"2024-08-31T12:15:47.083771+0000","last_became_peered":"2024-08-31T12:15:47.083771+0000","last_unstale":"2024-08-31T12:15:58.306756+0000","last_undegraded":"2024-08-31T12:15:58.306756+0000","last_fullsized":"2024-08-31T12:15:58.306756+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:45.213688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:45.213688+0000","last_clean_scrub_stamp":"2024-08-31T12:15:45.213688+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:25:24.827530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.301865+0000","last_change":"2024-08-31T12:15:56.285355+0000","last_active":"2024-08-31T12:15:58.301865+0000","last_peered":"2024-08-31T12:15:58.301865+0000","last_clean":"2024-08-31T12:15:58.301865+0000","last_became_active":"2024-08-31T12:15:56.284470+0000","last_became_peered":"2024-08-31T12:15:56.284470+0000","last_unstale":"2024-08-31T12:15:58.301865+0000","last_undegraded":"2024-08-31T12:15:58.301865+0000","last_fullsized":"2024-08-31T12:15:58.301865+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:52:20.055757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936697+0000","last_change":"2024-08-31T12:15:57.309039+0000","last_active":"2024-08-31T12:15:58.936697+0000","last_peered":"2024-08-31T12:15:58.936697+0000","last_clean":"2024-08-31T12:15:58.936697+0000","last_became_active":"2024-08-31T12:15:57.308721+0000","last_became_peered":"2024-08-31T12:15:57.308721+0000","last_unstale":"2024-08-31T12:15:58.936697+0000","last_undegraded":"2024-08-31T12:15:58.936697+0000","last_fullsized":"2024-08-31T12:15:58.936697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:43:12.703971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306092+0000","last_change":"2024-08-31T12:15:56.293221+0000","last_active":"2024-08-31T12:15:58.306092+0000","last_peered":"2024-08-31T12:15:58.306092+0000","last_clean":"2024-08-31T12:15:58.306092+0000","last_became_active":"2024-08-31T12:15:56.292910+0000","last_became_peered":"2024-08-31T12:15:56.292910+0000","last_unstale":"2024-08-31T12:15:58.306092+0000","last_undegraded":"2024-08-31T12:15:58.306092+0000","last_fullsized":"2024-08-31T12:15:58.306092+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:14:33.483002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937638+0000","last_change":"2024-08-31T12:15:57.307266+0000","last_active":"2024-08-31T12:15:58.937638+0000","last_peered":"2024-08-31T12:15:58.937638+0000","last_clean":"2024-08-31T12:15:58.937638+0000","last_became_active":"2024-08-31T12:15:57.306529+0000","last_became_peered":"2024-08-31T12:15:57.306529+0000","last_unstale":"2024-08-31T12:15:58.937638+0000","last_undegraded":"2024-08-31T12:15:58.937638+0000","last_fullsized":"2024-08-31T12:15:58.937638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:42:39.731742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.938249+0000","last_change":"2024-08-31T12:15:56.291822+0000","last_active":"2024-08-31T12:15:58.938249+0000","last_peered":"2024-08-31T12:15:58.938249+0000","last_clean":"2024-08-31T12:15:58.938249+0000","last_became_active":"2024-08-31T12:15:56.291064+0000","last_became_peered":"2024-08-31T12:15:56.291064+0000","last_unstale":"2024-08-31T12:15:58.938249+0000","last_undegraded":"2024-08-31T12:15:58.938249+0000","last_fullsized":"2024-08-31T12:15:58.938249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:53:08.929860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300606+0000","last_change":"2024-08-31T12:15:57.299212+0000","last_active":"2024-08-31T12:15:58.300606+0000","last_peered":"2024-08-31T12:15:58.300606+0000","last_clean":"2024-08-31T12:15:58.300606+0000","last_became_active":"2024-08-31T12:15:57.298892+0000","last_became_peered":"2024-08-31T12:15:57.298892+0000","last_unstale":"2024-08-31T12:15:58.300606+0000","last_undegraded":"2024-08-31T12:15:58.300606+0000","last_fullsized":"2024-08-31T12:15:58.300606+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:53:19.523811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303164+0000","last_change":"2024-08-31T12:15:56.286720+0000","last_active":"2024-08-31T12:15:58.303164+0000","last_peered":"2024-08-31T12:15:58.303164+0000","last_clean":"2024-08-31T12:15:58.303164+0000","last_became_active":"2024-08-31T12:15:56.285648+0000","last_became_peered":"2024-08-31T12:15:56.285648+0000","last_unstale":"2024-08-31T12:15:58.303164+0000","last_undegraded":"2024-08-31T12:15:58.303164+0000","last_fullsized":"2024-08-31T12:15:58.303164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:39:31.946792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936342+0000","last_change":"2024-08-31T12:15:57.311785+0000","last_active":"2024-08-31T12:15:58.936342+0000","last_peered":"2024-08-31T12:15:58.936342+0000","last_clean":"2024-08-31T12:15:58.936342+0000","last_became_active":"2024-08-31T12:15:57.311469+0000","last_became_peered":"2024-08-31T12:15:57.311469+0000","last_unstale":"2024-08-31T12:15:58.936342+0000","last_undegraded":"2024-08-31T12:15:58.936342+0000","last_fullsized":"2024-08-31T12:15:58.936342+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:29:54.122825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300691+0000","last_change":"2024-08-31T12:15:56.284302+0000","last_active":"2024-08-31T12:15:58.300691+0000","last_peered":"2024-08-31T12:15:58.300691+0000","last_clean":"2024-08-31T12:15:58.300691+0000","last_became_active":"2024-08-31T12:15:56.283979+0000","last_became_peered":"2024-08-31T12:15:56.283979+0000","last_unstale":"2024-08-31T12:15:58.300691+0000","last_undegraded":"2024-08-31T12:15:58.300691+0000","last_fullsized":"2024-08-31T12:15:58.300691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:51:59.306912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303594+0000","last_change":"2024-08-31T12:15:57.305383+0000","last_active":"2024-08-31T12:15:58.303594+0000","last_peered":"2024-08-31T12:15:58.303594+0000","last_clean":"2024-08-31T12:15:58.303594+0000","last_became_active":"2024-08-31T12:15:57.305048+0000","last_became_peered":"2024-08-31T12:15:57.305048+0000","last_unstale":"2024-08-31T12:15:58.303594+0000","last_undegraded":"2024-08-31T12:15:58.303594+0000","last_fullsized":"2024-08-31T12:15:58.303594+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:34:56.946695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306963+0000","last_change":"2024-08-31T12:15:56.297648+0000","last_active":"2024-08-31T12:15:58.306963+0000","last_peered":"2024-08-31T12:15:58.306963+0000","last_clean":"2024-08-31T12:15:58.306963+0000","last_became_active":"2024-08-31T12:15:56.297306+0000","last_became_peered":"2024-08-31T12:15:56.297306+0000","last_unstale":"2024-08-31T12:15:58.306963+0000","last_undegraded":"2024-08-31T12:15:58.306963+0000","last_fullsized":"2024-08-31T12:15:58.306963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:29:54.693819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300681+0000","last_change":"2024-08-31T12:15:57.297230+0000","last_active":"2024-08-31T12:15:58.300681+0000","last_peered":"2024-08-31T12:15:58.300681+0000","last_clean":"2024-08-31T12:15:58.300681+0000","last_became_active":"2024-08-31T12:15:57.296892+0000","last_became_peered":"2024-08-31T12:15:57.296892+0000","last_unstale":"2024-08-31T12:15:58.300681+0000","last_undegraded":"2024-08-31T12:15:58.300681+0000","last_fullsized":"2024-08-31T12:15:58.300681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:42:36.033586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300841+0000","last_change":"2024-08-31T12:15:56.286504+0000","last_active":"2024-08-31T12:15:58.300841+0000","last_peered":"2024-08-31T12:15:58.300841+0000","last_clean":"2024-08-31T12:15:58.300841+0000","last_became_active":"2024-08-31T12:15:56.286173+0000","last_became_peered":"2024-08-31T12:15:56.286173+0000","last_unstale":"2024-08-31T12:15:58.300841+0000","last_undegraded":"2024-08-31T12:15:58.300841+0000","last_fullsized":"2024-08-31T12:15:58.300841+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:06:34.313772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.938135+0000","last_change":"2024-08-31T12:15:57.308013+0000","last_active":"2024-08-31T12:15:58.938135+0000","last_peered":"2024-08-31T12:15:58.938135+0000","last_clean":"2024-08-31T12:15:58.938135+0000","last_became_active":"2024-08-31T12:15:57.307701+0000","last_became_peered":"2024-08-31T12:15:57.307701+0000","last_unstale":"2024-08-31T12:15:58.938135+0000","last_undegraded":"2024-08-31T12:15:58.938135+0000","last_fullsized":"2024-08-31T12:15:58.938135+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:06:15.161589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:02.304975+0000","last_change":"2024-08-31T12:15:56.294852+0000","last_active":"2024-08-31T12:16:02.304975+0000","last_peered":"2024-08-31T12:16:02.304975+0000","last_clean":"2024-08-31T12:16:02.304975+0000","last_became_active":"2024-08-31T12:15:56.294496+0000","last_became_peered":"2024-08-31T12:15:56.294496+0000","last_unstale":"2024-08-31T12:16:02.304975+0000","last_undegraded":"2024-08-31T12:16:02.304975+0000","last_fullsized":"2024-08-31T12:16:02.304975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:49:17.488528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937973+0000","last_change":"2024-08-31T12:15:57.305202+0000","last_active":"2024-08-31T12:15:58.937973+0000","last_peered":"2024-08-31T12:15:58.937973+0000","last_clean":"2024-08-31T12:15:58.937973+0000","last_became_active":"2024-08-31T12:15:57.304602+0000","last_became_peered":"2024-08-31T12:15:57.304602+0000","last_unstale":"2024-08-31T12:15:58.937973+0000","last_undegraded":"2024-08-31T12:15:58.937973+0000","last_fullsized":"2024-08-31T12:15:58.937973+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:56:03.280339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306090+0000","last_change":"2024-08-31T12:15:56.297542+0000","last_active":"2024-08-31T12:15:58.306090+0000","last_peered":"2024-08-31T12:15:58.306090+0000","last_clean":"2024-08-31T12:15:58.306090+0000","last_became_active":"2024-08-31T12:15:56.297168+0000","last_became_peered":"2024-08-31T12:15:56.297168+0000","last_unstale":"2024-08-31T12:15:58.306090+0000","last_undegraded":"2024-08-31T12:15:58.306090+0000","last_fullsized":"2024-08-31T12:15:58.306090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:56:41.924772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936311+0000","last_change":"2024-08-31T12:15:57.298845+0000","last_active":"2024-08-31T12:15:58.936311+0000","last_peered":"2024-08-31T12:15:58.936311+0000","last_clean":"2024-08-31T12:15:58.936311+0000","last_became_active":"2024-08-31T12:15:57.298520+0000","last_became_peered":"2024-08-31T12:15:57.298520+0000","last_unstale":"2024-08-31T12:15:58.936311+0000","last_undegraded":"2024-08-31T12:15:58.936311+0000","last_fullsized":"2024-08-31T12:15:58.936311+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:06:35.318873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300879+0000","last_change":"2024-08-31T12:15:56.290475+0000","last_active":"2024-08-31T12:15:58.300879+0000","last_peered":"2024-08-31T12:15:58.300879+0000","last_clean":"2024-08-31T12:15:58.300879+0000","last_became_active":"2024-08-31T12:15:56.289960+0000","last_became_peered":"2024-08-31T12:15:56.289960+0000","last_unstale":"2024-08-31T12:15:58.300879+0000","last_undegraded":"2024-08-31T12:15:58.300879+0000","last_fullsized":"2024-08-31T12:15:58.300879+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:29:53.117724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.938261+0000","last_change":"2024-08-31T12:15:57.301460+0000","last_active":"2024-08-31T12:15:58.938261+0000","last_peered":"2024-08-31T12:15:58.938261+0000","last_clean":"2024-08-31T12:15:58.938261+0000","last_became_active":"2024-08-31T12:15:57.301157+0000","last_became_peered":"2024-08-31T12:15:57.301157+0000","last_unstale":"2024-08-31T12:15:58.938261+0000","last_undegraded":"2024-08-31T12:15:58.938261+0000","last_fullsized":"2024-08-31T12:15:58.938261+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:22:52.173578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.301007+0000","last_change":"2024-08-31T12:15:56.289720+0000","last_active":"2024-08-31T12:15:58.301007+0000","last_peered":"2024-08-31T12:15:58.301007+0000","last_clean":"2024-08-31T12:15:58.301007+0000","last_became_active":"2024-08-31T12:15:56.289332+0000","last_became_peered":"2024-08-31T12:15:56.289332+0000","last_unstale":"2024-08-31T12:15:58.301007+0000","last_undegraded":"2024-08-31T12:15:58.301007+0000","last_fullsized":"2024-08-31T12:15:58.301007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:15:51.561831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937371+0000","last_change":"2024-08-31T12:15:57.313802+0000","last_active":"2024-08-31T12:15:58.937371+0000","last_peered":"2024-08-31T12:15:58.937371+0000","last_clean":"2024-08-31T12:15:58.937371+0000","last_became_active":"2024-08-31T12:15:57.313428+0000","last_became_peered":"2024-08-31T12:15:57.313428+0000","last_unstale":"2024-08-31T12:15:58.937371+0000","last_undegraded":"2024-08-31T12:15:58.937371+0000","last_fullsized":"2024-08-31T12:15:58.937371+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:22:13.435300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:02.305854+0000","last_change":"2024-08-31T12:15:56.296105+0000","last_active":"2024-08-31T12:16:02.305854+0000","last_peered":"2024-08-31T12:16:02.305854+0000","last_clean":"2024-08-31T12:16:02.305854+0000","last_became_active":"2024-08-31T12:15:56.295524+0000","last_became_peered":"2024-08-31T12:15:56.295524+0000","last_unstale":"2024-08-31T12:16:02.305854+0000","last_undegraded":"2024-08-31T12:16:02.305854+0000","last_fullsized":"2024-08-31T12:16:02.305854+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:41:55.482044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.301683+0000","last_change":"2024-08-31T12:15:57.300155+0000","last_active":"2024-08-31T12:15:58.301683+0000","last_peered":"2024-08-31T12:15:58.301683+0000","last_clean":"2024-08-31T12:15:58.301683+0000","last_became_active":"2024-08-31T12:15:57.299436+0000","last_became_peered":"2024-08-31T12:15:57.299436+0000","last_unstale":"2024-08-31T12:15:58.301683+0000","last_undegraded":"2024-08-31T12:15:58.301683+0000","last_fullsized":"2024-08-31T12:15:58.301683+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:37:45.891394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303455+0000","last_change":"2024-08-31T12:15:57.300529+0000","last_active":"2024-08-31T12:15:58.303455+0000","last_peered":"2024-08-31T12:15:58.303455+0000","last_clean":"2024-08-31T12:15:58.303455+0000","last_became_active":"2024-08-31T12:15:57.300159+0000","last_became_peered":"2024-08-31T12:15:57.300159+0000","last_unstale":"2024-08-31T12:15:58.303455+0000","last_undegraded":"2024-08-31T12:15:58.303455+0000","last_fullsized":"2024-08-31T12:15:58.303455+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:35:53.831742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:02.304978+0000","last_change":"2024-08-31T12:15:56.297791+0000","last_active":"2024-08-31T12:16:02.304978+0000","last_peered":"2024-08-31T12:16:02.304978+0000","last_clean":"2024-08-31T12:16:02.304978+0000","last_became_active":"2024-08-31T12:15:56.296976+0000","last_became_peered":"2024-08-31T12:15:56.296976+0000","last_unstale":"2024-08-31T12:16:02.304978+0000","last_undegraded":"2024-08-31T12:16:02.304978+0000","last_fullsized":"2024-08-31T12:16:02.304978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:07:54.963050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303405+0000","last_change":"2024-08-31T12:15:57.301167+0000","last_active":"2024-08-31T12:15:58.303405+0000","last_peered":"2024-08-31T12:15:58.303405+0000","last_clean":"2024-08-31T12:15:58.303405+0000","last_became_active":"2024-08-31T12:15:57.300646+0000","last_became_peered":"2024-08-31T12:15:57.300646+0000","last_unstale":"2024-08-31T12:15:58.303405+0000","last_undegraded":"2024-08-31T12:15:58.303405+0000","last_fullsized":"2024-08-31T12:15:58.303405+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:05:53.039520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:02.303751+0000","last_change":"2024-08-31T12:15:56.295139+0000","last_active":"2024-08-31T12:16:02.303751+0000","last_peered":"2024-08-31T12:16:02.303751+0000","last_clean":"2024-08-31T12:16:02.303751+0000","last_became_active":"2024-08-31T12:15:56.294560+0000","last_became_peered":"2024-08-31T12:15:56.294560+0000","last_unstale":"2024-08-31T12:16:02.303751+0000","last_undegraded":"2024-08-31T12:16:02.303751+0000","last_fullsized":"2024-08-31T12:16:02.303751+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:52:16.999507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937898+0000","last_change":"2024-08-31T12:15:57.311074+0000","last_active":"2024-08-31T12:15:58.937898+0000","last_peered":"2024-08-31T12:15:58.937898+0000","last_clean":"2024-08-31T12:15:58.937898+0000","last_became_active":"2024-08-31T12:15:57.310262+0000","last_became_peered":"2024-08-31T12:15:57.310262+0000","last_unstale":"2024-08-31T12:15:58.937898+0000","last_undegraded":"2024-08-31T12:15:58.937898+0000","last_fullsized":"2024-08-31T12:15:58.937898+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:47:51.267409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937350+0000","last_change":"2024-08-31T12:15:56.285428+0000","last_active":"2024-08-31T12:15:58.937350+0000","last_peered":"2024-08-31T12:15:58.937350+0000","last_clean":"2024-08-31T12:15:58.937350+0000","last_became_active":"2024-08-31T12:15:56.284884+0000","last_became_peered":"2024-08-31T12:15:56.284884+0000","last_unstale":"2024-08-31T12:15:58.937350+0000","last_undegraded":"2024-08-31T12:15:58.937350+0000","last_fullsized":"2024-08-31T12:15:58.937350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:49:25.181432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937174+0000","last_change":"2024-08-31T12:15:57.305772+0000","last_active":"2024-08-31T12:15:58.937174+0000","last_peered":"2024-08-31T12:15:58.937174+0000","last_clean":"2024-08-31T12:15:58.937174+0000","last_became_active":"2024-08-31T12:15:57.305409+0000","last_became_peered":"2024-08-31T12:15:57.305409+0000","last_unstale":"2024-08-31T12:15:58.937174+0000","last_undegraded":"2024-08-31T12:15:58.937174+0000","last_fullsized":"2024-08-31T12:15:58.937174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:53:19.523811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:02.306147+0000","last_change":"2024-08-31T12:15:56.294197+0000","last_active":"2024-08-31T12:16:02.306147+0000","last_peered":"2024-08-31T12:16:02.306147+0000","last_clean":"2024-08-31T12:16:02.306147+0000","last_became_active":"2024-08-31T12:15:56.293864+0000","last_became_peered":"2024-08-31T12:15:56.293864+0000","last_unstale":"2024-08-31T12:16:02.306147+0000","last_undegraded":"2024-08-31T12:16:02.306147+0000","last_fullsized":"2024-08-31T12:16:02.306147+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:43:17.443433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509448,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35560,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336280,"statfs":{"total":96636764160,"available":96600350720,"internally_reserved":0,"allocated":221184,"data_stored":74713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509447,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":74713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509448,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":73261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509448,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":815104,"data_stored":664107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542151,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36132,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335708,"statfs":{"total":96636764160,"available":96599764992,"internally_reserved":0,"allocated":806912,"data_stored":664051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542153,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35552,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336288,"statfs":{"total":96636764160,"available":96600358912,"internally_reserved":0,"allocated":212992,"data_stored":73191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542151,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35560,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336280,"statfs":{"total":96636764160,"available":96600350720,"internally_reserved":0,"allocated":221184,"data_stored":73227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":73101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-31T12:16:08.265 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-31T12:16:08.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.416+0000 7fc9009f8640 1 -- 172.21.15.19:0/475582106 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc8fc155300 msgr2=0x7fc8fc06d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:08.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.416+0000 7fc9009f8640 1 --2- 172.21.15.19:0/475582106 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc8fc155300 0x7fc8fc06d000 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc15b0f0 tx=0x7fc8f402a670 comp rx=0 tx=0).stop 2024-08-31T12:16:08.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.416+0000 7fc9009f8640 1 -- 172.21.15.19:0/475582106 shutdown_connections 2024-08-31T12:16:08.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.416+0000 7fc9009f8640 1 --2- 172.21.15.19:0/475582106 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc8fc155300 0x7fc8fc06d000 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:08.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.416+0000 7fc9009f8640 1 --2- 172.21.15.19:0/475582106 >> v2:172.21.15.43:3300/0 conn(0x7fc8fc06e0e0 0x7fc8fc0c9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:08.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.416+0000 7fc9009f8640 1 -- 172.21.15.19:0/475582106 >> 172.21.15.19:0/475582106 conn(0x7fc8fc0690d0 msgr2=0x7fc8fc0d0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:08.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.416+0000 7fc9009f8640 1 -- 172.21.15.19:0/475582106 shutdown_connections 2024-08-31T12:16:08.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.416+0000 7fc9009f8640 1 -- 172.21.15.19:0/475582106 wait complete. 2024-08-31T12:16:08.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.418+0000 7fc9009f8640 1 Processor -- start 2024-08-31T12:16:08.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.418+0000 7fc9009f8640 1 -- start start 2024-08-31T12:16:08.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.419+0000 7fc9009f8640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc8fc06e0e0 0x7fc8fc0c61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:08.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.419+0000 7fc9009f8640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc8fc0c8a40 0x7fc8fc1e4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:08.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.419+0000 7fc9009f8640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8fc155300 0x7fc8fc0cd830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:08.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.419+0000 7fc9009f8640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fc8fc0c8330 con 0x7fc8fc06e0e0 2024-08-31T12:16:08.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.419+0000 7fc9009f8640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc8fc15b250 con 0x7fc8fc155300 2024-08-31T12:16:08.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.419+0000 7fc9009f8640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fc8fc0c8610 con 0x7fc8fc0c8a40 2024-08-31T12:16:08.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.419+0000 7fc8fa59c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc8fc06e0e0 0x7fc8fc0c61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:08.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.419+0000 7fc8fa59c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc8fc06e0e0 0x7fc8fc0c61b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:44824/0 (socket says 172.21.15.19:44824) 2024-08-31T12:16:08.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.419+0000 7fc8fa59c640 1 -- 172.21.15.19:0/3673801113 learned_addr learned my addr 172.21.15.19:0/3673801113 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:08.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.419+0000 7fc8fad9d640 1 --2- 172.21.15.19:0/3673801113 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8fc155300 0x7fc8fc0cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:08.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.419+0000 7fc8f9d9b640 1 --2- 172.21.15.19:0/3673801113 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc8fc0c8a40 0x7fc8fc1e4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:08.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.419+0000 7fc8fa59c640 1 -- 172.21.15.19:0/3673801113 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc8fc0c8a40 msgr2=0x7fc8fc1e4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:08.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.419+0000 7fc8fa59c640 1 --2- 172.21.15.19:0/3673801113 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc8fc0c8a40 0x7fc8fc1e4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:08.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.419+0000 7fc8fa59c640 1 -- 172.21.15.19:0/3673801113 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8fc155300 msgr2=0x7fc8fc0cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:08.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.419+0000 7fc8fa59c640 1 --2- 172.21.15.19:0/3673801113 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8fc155300 0x7fc8fc0cd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:08.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.419+0000 7fc8fa59c640 1 -- 172.21.15.19:0/3673801113 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8fc1e5130 con 0x7fc8fc06e0e0 2024-08-31T12:16:08.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.419+0000 7fc8fad9d640 1 --2- 172.21.15.19:0/3673801113 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8fc155300 0x7fc8fc0cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-31T12:16:08.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.419+0000 7fc8f9d9b640 1 --2- 172.21.15.19:0/3673801113 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc8fc0c8a40 0x7fc8fc1e4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-31T12:16:08.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.420+0000 7fc8fa59c640 1 --2- 172.21.15.19:0/3673801113 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc8fc06e0e0 0x7fc8fc0c61b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec00c760 tx=0x7fc8ec00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:08.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.420+0000 7fc8f2ffd640 1 -- 172.21.15.19:0/3673801113 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec003b70 con 0x7fc8fc06e0e0 2024-08-31T12:16:08.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.420+0000 7fc8f2ffd640 1 -- 172.21.15.19:0/3673801113 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc8ec003d30 con 0x7fc8fc06e0e0 2024-08-31T12:16:08.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.420+0000 7fc9009f8640 1 -- 172.21.15.19:0/3673801113 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8fc1e6530 con 0x7fc8fc06e0e0 2024-08-31T12:16:08.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.420+0000 7fc8f2ffd640 1 -- 172.21.15.19:0/3673801113 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec00aab0 con 0x7fc8fc06e0e0 2024-08-31T12:16:08.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.420+0000 7fc9009f8640 1 -- 172.21.15.19:0/3673801113 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8fc1e6af0 con 0x7fc8fc06e0e0 2024-08-31T12:16:08.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.421+0000 7fc9009f8640 1 -- 172.21.15.19:0/3673801113 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8fc0c9850 con 0x7fc8fc06e0e0 2024-08-31T12:16:08.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.422+0000 7fc8f2ffd640 1 -- 172.21.15.19:0/3673801113 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fc8ec00ac50 con 0x7fc8fc06e0e0 2024-08-31T12:16:08.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.422+0000 7fc8f2ffd640 1 --2- 172.21.15.19:0/3673801113 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc8dc077a80 0x7fc8dc079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:08.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.422+0000 7fc8f2ffd640 1 -- 172.21.15.19:0/3673801113 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc8ec099360 con 0x7fc8fc06e0e0 2024-08-31T12:16:08.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.422+0000 7fc8f9d9b640 1 --2- 172.21.15.19:0/3673801113 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc8dc077a80 0x7fc8dc079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:08.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.423+0000 7fc8f9d9b640 1 --2- 172.21.15.19:0/3673801113 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc8dc077a80 0x7fc8dc079f90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc0c5590 tx=0x7fc8e4005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:08.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.427+0000 7fc8f2ffd640 1 -- 172.21.15.19:0/3673801113 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fc8ec062160 con 0x7fc8fc06e0e0 2024-08-31T12:16:08.608 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.607+0000 7fc9009f8640 1 -- 172.21.15.19:0/3673801113 --> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc8fc1e61c0 con 0x7fc8dc077a80 2024-08-31T12:16:08.612 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.612+0000 7fc8f2ffd640 1 -- 172.21.15.19:0/3673801113 <== mgr.4100 v2:172.21.15.43:6800/2824336655 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251854 (secure 0 0 0) 0x7fc8fc1e61c0 con 0x7fc8dc077a80 2024-08-31T12:16:08.615 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:16:08.616 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-08-31T12:16:08.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.615+0000 7fc9009f8640 1 -- 172.21.15.19:0/3673801113 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc8dc077a80 msgr2=0x7fc8dc079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:08.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.615+0000 7fc9009f8640 1 --2- 172.21.15.19:0/3673801113 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc8dc077a80 0x7fc8dc079f90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc0c5590 tx=0x7fc8e4005f90 comp rx=0 tx=0).stop 2024-08-31T12:16:08.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.615+0000 7fc9009f8640 1 -- 172.21.15.19:0/3673801113 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc8fc06e0e0 msgr2=0x7fc8fc0c61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:08.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.615+0000 7fc9009f8640 1 --2- 172.21.15.19:0/3673801113 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc8fc06e0e0 0x7fc8fc0c61b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec00c760 tx=0x7fc8ec00cc20 comp rx=0 tx=0).stop 2024-08-31T12:16:08.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.615+0000 7fc9009f8640 1 -- 172.21.15.19:0/3673801113 shutdown_connections 2024-08-31T12:16:08.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.615+0000 7fc9009f8640 1 --2- 172.21.15.19:0/3673801113 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc8dc077a80 0x7fc8dc079f90 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:08.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.615+0000 7fc9009f8640 1 --2- 172.21.15.19:0/3673801113 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8fc155300 0x7fc8fc0cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:08.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.615+0000 7fc9009f8640 1 --2- 172.21.15.19:0/3673801113 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc8fc0c8a40 0x7fc8fc1e4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:08.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.615+0000 7fc9009f8640 1 --2- 172.21.15.19:0/3673801113 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc8fc06e0e0 0x7fc8fc0c61b0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:08.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.615+0000 7fc9009f8640 1 -- 172.21.15.19:0/3673801113 >> 172.21.15.19:0/3673801113 conn(0x7fc8fc0690d0 msgr2=0x7fc8fc0c3170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:08.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.615+0000 7fc9009f8640 1 -- 172.21.15.19:0/3673801113 shutdown_connections 2024-08-31T12:16:08.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.615+0000 7fc9009f8640 1 -- 172.21.15.19:0/3673801113 wait complete. 2024-08-31T12:16:08.627 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-08-31T12:16:08.493911+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":148,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113,"ondisk_log_size":113,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":252328,"kb_used_data":2856,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754722392,"statfs":{"total":773094113280,"available":772835729408,"internally_reserved":0,"allocated":2924544,"data_stored":1770364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.194257"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306730+0000","last_change":"2024-08-31T12:15:57.306560+0000","last_active":"2024-08-31T12:15:58.306730+0000","last_peered":"2024-08-31T12:15:58.306730+0000","last_clean":"2024-08-31T12:15:58.306730+0000","last_became_active":"2024-08-31T12:15:57.306148+0000","last_became_peered":"2024-08-31T12:15:57.306148+0000","last_unstale":"2024-08-31T12:15:58.306730+0000","last_undegraded":"2024-08-31T12:15:58.306730+0000","last_fullsized":"2024-08-31T12:15:58.306730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:35:53.831742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300935+0000","last_change":"2024-08-31T12:15:57.307314+0000","last_active":"2024-08-31T12:15:58.300935+0000","last_peered":"2024-08-31T12:15:58.300935+0000","last_clean":"2024-08-31T12:15:58.300935+0000","last_became_active":"2024-08-31T12:15:57.306940+0000","last_became_peered":"2024-08-31T12:15:57.306940+0000","last_unstale":"2024-08-31T12:15:58.300935+0000","last_undegraded":"2024-08-31T12:15:58.300935+0000","last_fullsized":"2024-08-31T12:15:58.300935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:38:30.321214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937267+0000","last_change":"2024-08-31T12:15:57.309372+0000","last_active":"2024-08-31T12:15:58.937267+0000","last_peered":"2024-08-31T12:15:58.937267+0000","last_clean":"2024-08-31T12:15:58.937267+0000","last_became_active":"2024-08-31T12:15:57.305795+0000","last_became_peered":"2024-08-31T12:15:57.305795+0000","last_unstale":"2024-08-31T12:15:58.937267+0000","last_undegraded":"2024-08-31T12:15:58.937267+0000","last_fullsized":"2024-08-31T12:15:58.937267+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:37:45.891394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302363+0000","last_change":"2024-08-31T12:15:57.307339+0000","last_active":"2024-08-31T12:15:58.302363+0000","last_peered":"2024-08-31T12:15:58.302363+0000","last_clean":"2024-08-31T12:15:58.302363+0000","last_became_active":"2024-08-31T12:15:57.306957+0000","last_became_peered":"2024-08-31T12:15:57.306957+0000","last_unstale":"2024-08-31T12:15:58.302363+0000","last_undegraded":"2024-08-31T12:15:58.302363+0000","last_fullsized":"2024-08-31T12:15:58.302363+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:43:33.705099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306757+0000","last_change":"2024-08-31T12:15:57.317492+0000","last_active":"2024-08-31T12:15:58.306757+0000","last_peered":"2024-08-31T12:15:58.306757+0000","last_clean":"2024-08-31T12:15:58.306757+0000","last_became_active":"2024-08-31T12:15:57.317079+0000","last_became_peered":"2024-08-31T12:15:57.317079+0000","last_unstale":"2024-08-31T12:15:58.306757+0000","last_undegraded":"2024-08-31T12:15:58.306757+0000","last_fullsized":"2024-08-31T12:15:58.306757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:44:40.712699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302342+0000","last_change":"2024-08-31T12:15:57.307505+0000","last_active":"2024-08-31T12:15:58.302342+0000","last_peered":"2024-08-31T12:15:58.302342+0000","last_clean":"2024-08-31T12:15:58.302342+0000","last_became_active":"2024-08-31T12:15:57.307010+0000","last_became_peered":"2024-08-31T12:15:57.307010+0000","last_unstale":"2024-08-31T12:15:58.302342+0000","last_undegraded":"2024-08-31T12:15:58.302342+0000","last_fullsized":"2024-08-31T12:15:58.302342+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:39:31.100565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306247+0000","last_change":"2024-08-31T12:15:57.316892+0000","last_active":"2024-08-31T12:15:58.306247+0000","last_peered":"2024-08-31T12:15:58.306247+0000","last_clean":"2024-08-31T12:15:58.306247+0000","last_became_active":"2024-08-31T12:15:57.316504+0000","last_became_peered":"2024-08-31T12:15:57.316504+0000","last_unstale":"2024-08-31T12:15:58.306247+0000","last_undegraded":"2024-08-31T12:15:58.306247+0000","last_fullsized":"2024-08-31T12:15:58.306247+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:05:48.614918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937976+0000","last_change":"2024-08-31T12:15:57.309224+0000","last_active":"2024-08-31T12:15:58.937976+0000","last_peered":"2024-08-31T12:15:58.937976+0000","last_clean":"2024-08-31T12:15:58.937976+0000","last_became_active":"2024-08-31T12:15:57.308914+0000","last_became_peered":"2024-08-31T12:15:57.308914+0000","last_unstale":"2024-08-31T12:15:58.937976+0000","last_undegraded":"2024-08-31T12:15:58.937976+0000","last_fullsized":"2024-08-31T12:15:58.937976+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:37:02.435678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306762+0000","last_change":"2024-08-31T12:15:57.316723+0000","last_active":"2024-08-31T12:15:58.306762+0000","last_peered":"2024-08-31T12:15:58.306762+0000","last_clean":"2024-08-31T12:15:58.306762+0000","last_became_active":"2024-08-31T12:15:57.316354+0000","last_became_peered":"2024-08-31T12:15:57.316354+0000","last_unstale":"2024-08-31T12:15:58.306762+0000","last_undegraded":"2024-08-31T12:15:58.306762+0000","last_fullsized":"2024-08-31T12:15:58.306762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:31:19.860950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302888+0000","last_change":"2024-08-31T12:15:57.311161+0000","last_active":"2024-08-31T12:15:58.302888+0000","last_peered":"2024-08-31T12:15:58.302888+0000","last_clean":"2024-08-31T12:15:58.302888+0000","last_became_active":"2024-08-31T12:15:57.310807+0000","last_became_peered":"2024-08-31T12:15:57.310807+0000","last_unstale":"2024-08-31T12:15:58.302888+0000","last_undegraded":"2024-08-31T12:15:58.302888+0000","last_fullsized":"2024-08-31T12:15:58.302888+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:56:03.280339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302118+0000","last_change":"2024-08-31T12:15:57.300930+0000","last_active":"2024-08-31T12:15:58.302118+0000","last_peered":"2024-08-31T12:15:58.302118+0000","last_clean":"2024-08-31T12:15:58.302118+0000","last_became_active":"2024-08-31T12:15:57.300610+0000","last_became_peered":"2024-08-31T12:15:57.300610+0000","last_unstale":"2024-08-31T12:15:58.302118+0000","last_undegraded":"2024-08-31T12:15:58.302118+0000","last_fullsized":"2024-08-31T12:15:58.302118+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:42:39.731742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936435+0000","last_change":"2024-08-31T12:15:57.303151+0000","last_active":"2024-08-31T12:15:58.936435+0000","last_peered":"2024-08-31T12:15:58.936435+0000","last_clean":"2024-08-31T12:15:58.936435+0000","last_became_active":"2024-08-31T12:15:57.302622+0000","last_became_peered":"2024-08-31T12:15:57.302622+0000","last_unstale":"2024-08-31T12:15:58.936435+0000","last_undegraded":"2024-08-31T12:15:58.936435+0000","last_fullsized":"2024-08-31T12:15:58.936435+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:49:18.493629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937309+0000","last_change":"2024-08-31T12:15:57.305676+0000","last_active":"2024-08-31T12:15:58.937309+0000","last_peered":"2024-08-31T12:15:58.937309+0000","last_clean":"2024-08-31T12:15:58.937309+0000","last_became_active":"2024-08-31T12:15:57.305321+0000","last_became_peered":"2024-08-31T12:15:57.305321+0000","last_unstale":"2024-08-31T12:15:58.937309+0000","last_undegraded":"2024-08-31T12:15:58.937309+0000","last_fullsized":"2024-08-31T12:15:58.937309+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:42:39.731742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936688+0000","last_change":"2024-08-31T12:15:57.309671+0000","last_active":"2024-08-31T12:15:58.936688+0000","last_peered":"2024-08-31T12:15:58.936688+0000","last_clean":"2024-08-31T12:15:58.936688+0000","last_became_active":"2024-08-31T12:15:57.307013+0000","last_became_peered":"2024-08-31T12:15:57.307013+0000","last_unstale":"2024-08-31T12:15:58.936688+0000","last_undegraded":"2024-08-31T12:15:58.936688+0000","last_fullsized":"2024-08-31T12:15:58.936688+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:18:55.451164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300929+0000","last_change":"2024-08-31T12:15:57.298150+0000","last_active":"2024-08-31T12:15:58.300929+0000","last_peered":"2024-08-31T12:15:58.300929+0000","last_clean":"2024-08-31T12:15:58.300929+0000","last_became_active":"2024-08-31T12:15:57.297182+0000","last_became_peered":"2024-08-31T12:15:57.297182+0000","last_unstale":"2024-08-31T12:15:58.300929+0000","last_undegraded":"2024-08-31T12:15:58.300929+0000","last_fullsized":"2024-08-31T12:15:58.300929+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:16:42.625396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302762+0000","last_change":"2024-08-31T12:15:57.314224+0000","last_active":"2024-08-31T12:15:58.302762+0000","last_peered":"2024-08-31T12:15:58.302762+0000","last_clean":"2024-08-31T12:15:58.302762+0000","last_became_active":"2024-08-31T12:15:57.313887+0000","last_became_peered":"2024-08-31T12:15:57.313887+0000","last_unstale":"2024-08-31T12:15:58.302762+0000","last_undegraded":"2024-08-31T12:15:58.302762+0000","last_fullsized":"2024-08-31T12:15:58.302762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:54:45.720621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303640+0000","last_change":"2024-08-31T12:15:57.308899+0000","last_active":"2024-08-31T12:15:58.303640+0000","last_peered":"2024-08-31T12:15:58.303640+0000","last_clean":"2024-08-31T12:15:58.303640+0000","last_became_active":"2024-08-31T12:15:57.308388+0000","last_became_peered":"2024-08-31T12:15:57.308388+0000","last_unstale":"2024-08-31T12:15:58.303640+0000","last_undegraded":"2024-08-31T12:15:58.303640+0000","last_fullsized":"2024-08-31T12:15:58.303640+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:54:03.871840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936410+0000","last_change":"2024-08-31T12:15:57.300201+0000","last_active":"2024-08-31T12:15:58.936410+0000","last_peered":"2024-08-31T12:15:58.936410+0000","last_clean":"2024-08-31T12:15:58.936410+0000","last_became_active":"2024-08-31T12:15:57.299826+0000","last_became_peered":"2024-08-31T12:15:57.299826+0000","last_unstale":"2024-08-31T12:15:58.936410+0000","last_undegraded":"2024-08-31T12:15:58.936410+0000","last_fullsized":"2024-08-31T12:15:58.936410+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:14:34.488102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.307053+0000","last_change":"2024-08-31T12:15:57.317606+0000","last_active":"2024-08-31T12:15:58.307053+0000","last_peered":"2024-08-31T12:15:58.307053+0000","last_clean":"2024-08-31T12:15:58.307053+0000","last_became_active":"2024-08-31T12:15:57.317185+0000","last_became_peered":"2024-08-31T12:15:57.317185+0000","last_unstale":"2024-08-31T12:15:58.307053+0000","last_undegraded":"2024-08-31T12:15:58.307053+0000","last_fullsized":"2024-08-31T12:15:58.307053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:25:46.346490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302171+0000","last_change":"2024-08-31T12:15:57.298778+0000","last_active":"2024-08-31T12:15:58.302171+0000","last_peered":"2024-08-31T12:15:58.302171+0000","last_clean":"2024-08-31T12:15:58.302171+0000","last_became_active":"2024-08-31T12:15:57.298430+0000","last_became_peered":"2024-08-31T12:15:57.298430+0000","last_unstale":"2024-08-31T12:15:58.302171+0000","last_undegraded":"2024-08-31T12:15:58.302171+0000","last_fullsized":"2024-08-31T12:15:58.302171+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:22:13.435300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937190+0000","last_change":"2024-08-31T12:15:57.317176+0000","last_active":"2024-08-31T12:15:58.937190+0000","last_peered":"2024-08-31T12:15:58.937190+0000","last_clean":"2024-08-31T12:15:58.937190+0000","last_became_active":"2024-08-31T12:15:57.316870+0000","last_became_peered":"2024-08-31T12:15:57.316870+0000","last_unstale":"2024-08-31T12:15:58.937190+0000","last_undegraded":"2024-08-31T12:15:58.937190+0000","last_fullsized":"2024-08-31T12:15:58.937190+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:43:12.703971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302636+0000","last_change":"2024-08-31T12:15:57.308907+0000","last_active":"2024-08-31T12:15:58.302636+0000","last_peered":"2024-08-31T12:15:58.302636+0000","last_clean":"2024-08-31T12:15:58.302636+0000","last_became_active":"2024-08-31T12:15:57.308459+0000","last_became_peered":"2024-08-31T12:15:57.308459+0000","last_unstale":"2024-08-31T12:15:58.302636+0000","last_undegraded":"2024-08-31T12:15:58.302636+0000","last_fullsized":"2024-08-31T12:15:58.302636+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:16:46.648912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937760+0000","last_change":"2024-08-31T12:15:57.304032+0000","last_active":"2024-08-31T12:15:58.937760+0000","last_peered":"2024-08-31T12:15:58.937760+0000","last_clean":"2024-08-31T12:15:58.937760+0000","last_became_active":"2024-08-31T12:15:57.303401+0000","last_became_peered":"2024-08-31T12:15:57.303401+0000","last_unstale":"2024-08-31T12:15:58.937760+0000","last_undegraded":"2024-08-31T12:15:58.937760+0000","last_fullsized":"2024-08-31T12:15:58.937760+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:56:03.280339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303013+0000","last_change":"2024-08-31T12:15:57.305412+0000","last_active":"2024-08-31T12:15:58.303013+0000","last_peered":"2024-08-31T12:15:58.303013+0000","last_clean":"2024-08-31T12:15:58.303013+0000","last_became_active":"2024-08-31T12:15:57.305075+0000","last_became_peered":"2024-08-31T12:15:57.305075+0000","last_unstale":"2024-08-31T12:15:58.303013+0000","last_undegraded":"2024-08-31T12:15:58.303013+0000","last_fullsized":"2024-08-31T12:15:58.303013+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:30:13.715505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937828+0000","last_change":"2024-08-31T12:15:57.300179+0000","last_active":"2024-08-31T12:15:58.937828+0000","last_peered":"2024-08-31T12:15:58.937828+0000","last_clean":"2024-08-31T12:15:58.937828+0000","last_became_active":"2024-08-31T12:15:57.299861+0000","last_became_peered":"2024-08-31T12:15:57.299861+0000","last_unstale":"2024-08-31T12:15:58.937828+0000","last_undegraded":"2024-08-31T12:15:58.937828+0000","last_fullsized":"2024-08-31T12:15:58.937828+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:47:51.267409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303001+0000","last_change":"2024-08-31T12:15:57.308519+0000","last_active":"2024-08-31T12:15:58.303001+0000","last_peered":"2024-08-31T12:15:58.303001+0000","last_clean":"2024-08-31T12:15:58.303001+0000","last_became_active":"2024-08-31T12:15:57.308189+0000","last_became_peered":"2024-08-31T12:15:57.308189+0000","last_unstale":"2024-08-31T12:15:58.303001+0000","last_undegraded":"2024-08-31T12:15:58.303001+0000","last_fullsized":"2024-08-31T12:15:58.303001+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:37:02.435678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300666+0000","last_change":"2024-08-31T12:15:57.308321+0000","last_active":"2024-08-31T12:15:58.300666+0000","last_peered":"2024-08-31T12:15:58.300666+0000","last_clean":"2024-08-31T12:15:58.300666+0000","last_became_active":"2024-08-31T12:15:57.304514+0000","last_became_peered":"2024-08-31T12:15:57.304514+0000","last_unstale":"2024-08-31T12:15:58.300666+0000","last_undegraded":"2024-08-31T12:15:58.300666+0000","last_fullsized":"2024-08-31T12:15:58.300666+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:09:49.617826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936830+0000","last_change":"2024-08-31T12:15:57.299268+0000","last_active":"2024-08-31T12:15:58.936830+0000","last_peered":"2024-08-31T12:15:58.936830+0000","last_clean":"2024-08-31T12:15:58.936830+0000","last_became_active":"2024-08-31T12:15:57.298766+0000","last_became_peered":"2024-08-31T12:15:57.298766+0000","last_unstale":"2024-08-31T12:15:58.936830+0000","last_undegraded":"2024-08-31T12:15:58.936830+0000","last_fullsized":"2024-08-31T12:15:58.936830+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:37:45.891394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937525+0000","last_change":"2024-08-31T12:15:57.310121+0000","last_active":"2024-08-31T12:15:58.937525+0000","last_peered":"2024-08-31T12:15:58.937525+0000","last_clean":"2024-08-31T12:15:58.937525+0000","last_became_active":"2024-08-31T12:15:57.309440+0000","last_became_peered":"2024-08-31T12:15:57.309440+0000","last_unstale":"2024-08-31T12:15:58.937525+0000","last_undegraded":"2024-08-31T12:15:58.937525+0000","last_fullsized":"2024-08-31T12:15:58.937525+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:53:19.523811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302889+0000","last_change":"2024-08-31T12:15:57.306214+0000","last_active":"2024-08-31T12:15:58.302889+0000","last_peered":"2024-08-31T12:15:58.302889+0000","last_clean":"2024-08-31T12:15:58.302889+0000","last_became_active":"2024-08-31T12:15:57.305878+0000","last_became_peered":"2024-08-31T12:15:57.305878+0000","last_unstale":"2024-08-31T12:15:58.302889+0000","last_undegraded":"2024-08-31T12:15:58.302889+0000","last_fullsized":"2024-08-31T12:15:58.302889+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:05:48.614918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.938282+0000","last_change":"2024-08-31T12:15:57.307376+0000","last_active":"2024-08-31T12:15:58.938282+0000","last_peered":"2024-08-31T12:15:58.938282+0000","last_clean":"2024-08-31T12:15:58.938282+0000","last_became_active":"2024-08-31T12:15:57.307056+0000","last_became_peered":"2024-08-31T12:15:57.307056+0000","last_unstale":"2024-08-31T12:15:58.938282+0000","last_undegraded":"2024-08-31T12:15:58.938282+0000","last_fullsized":"2024-08-31T12:15:58.938282+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:52:18.004608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302319+0000","last_change":"2024-08-31T12:15:57.313798+0000","last_active":"2024-08-31T12:15:58.302319+0000","last_peered":"2024-08-31T12:15:58.302319+0000","last_clean":"2024-08-31T12:15:58.302319+0000","last_became_active":"2024-08-31T12:15:57.313463+0000","last_became_peered":"2024-08-31T12:15:57.313463+0000","last_unstale":"2024-08-31T12:15:58.302319+0000","last_undegraded":"2024-08-31T12:15:58.302319+0000","last_fullsized":"2024-08-31T12:15:58.302319+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:47:51.267409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936240+0000","last_change":"2024-08-31T12:15:57.293350+0000","last_active":"2024-08-31T12:15:58.936240+0000","last_peered":"2024-08-31T12:15:58.936240+0000","last_clean":"2024-08-31T12:15:58.936240+0000","last_became_active":"2024-08-31T12:15:57.292981+0000","last_became_peered":"2024-08-31T12:15:57.292981+0000","last_unstale":"2024-08-31T12:15:58.936240+0000","last_undegraded":"2024-08-31T12:15:58.936240+0000","last_fullsized":"2024-08-31T12:15:58.936240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:52:00.312013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937453+0000","last_change":"2024-08-31T12:15:56.293497+0000","last_active":"2024-08-31T12:15:58.937453+0000","last_peered":"2024-08-31T12:15:58.937453+0000","last_clean":"2024-08-31T12:15:58.937453+0000","last_became_active":"2024-08-31T12:15:56.293164+0000","last_became_peered":"2024-08-31T12:15:56.293164+0000","last_unstale":"2024-08-31T12:15:58.937453+0000","last_undegraded":"2024-08-31T12:15:58.937453+0000","last_fullsized":"2024-08-31T12:15:58.937453+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:43:17.443433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937814+0000","last_change":"2024-08-31T12:15:57.312738+0000","last_active":"2024-08-31T12:15:58.937814+0000","last_peered":"2024-08-31T12:15:58.937814+0000","last_clean":"2024-08-31T12:15:58.937814+0000","last_became_active":"2024-08-31T12:15:57.312360+0000","last_became_peered":"2024-08-31T12:15:57.312360+0000","last_unstale":"2024-08-31T12:15:58.937814+0000","last_undegraded":"2024-08-31T12:15:58.937814+0000","last_fullsized":"2024-08-31T12:15:58.937814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:22:13.435300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.301828+0000","last_change":"2024-08-31T12:15:56.283526+0000","last_active":"2024-08-31T12:15:58.301828+0000","last_peered":"2024-08-31T12:15:58.301828+0000","last_clean":"2024-08-31T12:15:58.301828+0000","last_became_active":"2024-08-31T12:15:56.283147+0000","last_became_peered":"2024-08-31T12:15:56.283147+0000","last_unstale":"2024-08-31T12:15:58.301828+0000","last_undegraded":"2024-08-31T12:15:58.301828+0000","last_fullsized":"2024-08-31T12:15:58.301828+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:51:59.306912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937727+0000","last_change":"2024-08-31T12:15:56.291170+0000","last_active":"2024-08-31T12:15:58.937727+0000","last_peered":"2024-08-31T12:15:58.937727+0000","last_clean":"2024-08-31T12:15:58.937727+0000","last_became_active":"2024-08-31T12:15:56.290835+0000","last_became_peered":"2024-08-31T12:15:56.290835+0000","last_unstale":"2024-08-31T12:15:58.937727+0000","last_undegraded":"2024-08-31T12:15:58.937727+0000","last_fullsized":"2024-08-31T12:15:58.937727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:15:51.561831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.938053+0000","last_change":"2024-08-31T12:15:57.308183+0000","last_active":"2024-08-31T12:15:58.938053+0000","last_peered":"2024-08-31T12:15:58.938053+0000","last_clean":"2024-08-31T12:15:58.938053+0000","last_became_active":"2024-08-31T12:15:57.306962+0000","last_became_peered":"2024-08-31T12:15:57.306962+0000","last_unstale":"2024-08-31T12:15:58.938053+0000","last_undegraded":"2024-08-31T12:15:58.938053+0000","last_fullsized":"2024-08-31T12:15:58.938053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:01:18.188264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303216+0000","last_change":"2024-08-31T12:15:56.288126+0000","last_active":"2024-08-31T12:15:58.303216+0000","last_peered":"2024-08-31T12:15:58.303216+0000","last_clean":"2024-08-31T12:15:58.303216+0000","last_became_active":"2024-08-31T12:15:56.287552+0000","last_became_peered":"2024-08-31T12:15:56.287552+0000","last_unstale":"2024-08-31T12:15:58.303216+0000","last_undegraded":"2024-08-31T12:15:58.303216+0000","last_fullsized":"2024-08-31T12:15:58.303216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:54:29.988079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300810+0000","last_change":"2024-08-31T12:15:57.305029+0000","last_active":"2024-08-31T12:15:58.300810+0000","last_peered":"2024-08-31T12:15:58.300810+0000","last_clean":"2024-08-31T12:15:58.300810+0000","last_became_active":"2024-08-31T12:15:57.304700+0000","last_became_peered":"2024-08-31T12:15:57.304700+0000","last_unstale":"2024-08-31T12:15:58.300810+0000","last_undegraded":"2024-08-31T12:15:58.300810+0000","last_fullsized":"2024-08-31T12:15:58.300810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:58:17.482204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302528+0000","last_change":"2024-08-31T12:15:56.284907+0000","last_active":"2024-08-31T12:15:58.302528+0000","last_peered":"2024-08-31T12:15:58.302528+0000","last_clean":"2024-08-31T12:15:58.302528+0000","last_became_active":"2024-08-31T12:15:56.284558+0000","last_became_peered":"2024-08-31T12:15:56.284558+0000","last_unstale":"2024-08-31T12:15:58.302528+0000","last_undegraded":"2024-08-31T12:15:58.302528+0000","last_fullsized":"2024-08-31T12:15:58.302528+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:29:53.117724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303538+0000","last_change":"2024-08-31T12:15:57.308679+0000","last_active":"2024-08-31T12:15:58.303538+0000","last_peered":"2024-08-31T12:15:58.303538+0000","last_clean":"2024-08-31T12:15:58.303538+0000","last_became_active":"2024-08-31T12:15:57.308260+0000","last_became_peered":"2024-08-31T12:15:57.308260+0000","last_unstale":"2024-08-31T12:15:58.303538+0000","last_undegraded":"2024-08-31T12:15:58.303538+0000","last_fullsized":"2024-08-31T12:15:58.303538+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:31:19.860950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303564+0000","last_change":"2024-08-31T12:15:56.290763+0000","last_active":"2024-08-31T12:15:58.303564+0000","last_peered":"2024-08-31T12:15:58.303564+0000","last_clean":"2024-08-31T12:15:58.303564+0000","last_became_active":"2024-08-31T12:15:56.290400+0000","last_became_peered":"2024-08-31T12:15:56.290400+0000","last_unstale":"2024-08-31T12:15:58.303564+0000","last_undegraded":"2024-08-31T12:15:58.303564+0000","last_fullsized":"2024-08-31T12:15:58.303564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:53:08.929860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.938122+0000","last_change":"2024-08-31T12:15:57.311657+0000","last_active":"2024-08-31T12:15:58.938122+0000","last_peered":"2024-08-31T12:15:58.938122+0000","last_clean":"2024-08-31T12:15:58.938122+0000","last_became_active":"2024-08-31T12:15:57.310564+0000","last_became_peered":"2024-08-31T12:15:57.310564+0000","last_unstale":"2024-08-31T12:15:58.938122+0000","last_undegraded":"2024-08-31T12:15:58.938122+0000","last_fullsized":"2024-08-31T12:15:58.938122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:37:02.435678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:02.303958+0000","last_change":"2024-08-31T12:15:56.286792+0000","last_active":"2024-08-31T12:16:02.303958+0000","last_peered":"2024-08-31T12:16:02.303958+0000","last_clean":"2024-08-31T12:16:02.303958+0000","last_became_active":"2024-08-31T12:15:56.286458+0000","last_became_peered":"2024-08-31T12:15:56.286458+0000","last_unstale":"2024-08-31T12:16:02.303958+0000","last_undegraded":"2024-08-31T12:16:02.303958+0000","last_fullsized":"2024-08-31T12:16:02.303958+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:49:17.488528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936603+0000","last_change":"2024-08-31T12:15:57.309485+0000","last_active":"2024-08-31T12:15:58.936603+0000","last_peered":"2024-08-31T12:15:58.936603+0000","last_clean":"2024-08-31T12:15:58.936603+0000","last_became_active":"2024-08-31T12:15:57.309102+0000","last_became_peered":"2024-08-31T12:15:57.309102+0000","last_unstale":"2024-08-31T12:15:58.936603+0000","last_undegraded":"2024-08-31T12:15:58.936603+0000","last_fullsized":"2024-08-31T12:15:58.936603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:17:34.692296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:02.306321+0000","last_change":"2024-08-31T12:15:56.289460+0000","last_active":"2024-08-31T12:16:02.306321+0000","last_peered":"2024-08-31T12:16:02.306321+0000","last_clean":"2024-08-31T12:16:02.306321+0000","last_became_active":"2024-08-31T12:15:56.289096+0000","last_became_peered":"2024-08-31T12:15:56.289096+0000","last_unstale":"2024-08-31T12:16:02.306321+0000","last_undegraded":"2024-08-31T12:16:02.306321+0000","last_fullsized":"2024-08-31T12:16:02.306321+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:06:34.313772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306865+0000","last_change":"2024-08-31T12:15:57.317968+0000","last_active":"2024-08-31T12:15:58.306865+0000","last_peered":"2024-08-31T12:15:58.306865+0000","last_clean":"2024-08-31T12:15:58.306865+0000","last_became_active":"2024-08-31T12:15:57.317632+0000","last_became_peered":"2024-08-31T12:15:57.317632+0000","last_unstale":"2024-08-31T12:15:58.306865+0000","last_undegraded":"2024-08-31T12:15:58.306865+0000","last_fullsized":"2024-08-31T12:15:58.306865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:22:02.569106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:02.304711+0000","last_change":"2024-08-31T12:15:56.284014+0000","last_active":"2024-08-31T12:16:02.304711+0000","last_peered":"2024-08-31T12:16:02.304711+0000","last_clean":"2024-08-31T12:16:02.304711+0000","last_became_active":"2024-08-31T12:15:56.283675+0000","last_became_peered":"2024-08-31T12:15:56.283675+0000","last_unstale":"2024-08-31T12:16:02.304711+0000","last_undegraded":"2024-08-31T12:16:02.304711+0000","last_fullsized":"2024-08-31T12:16:02.304711+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:51:59.306912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937018+0000","last_change":"2024-08-31T12:15:57.312440+0000","last_active":"2024-08-31T12:15:58.937018+0000","last_peered":"2024-08-31T12:15:58.937018+0000","last_clean":"2024-08-31T12:15:58.937018+0000","last_became_active":"2024-08-31T12:15:57.312007+0000","last_became_peered":"2024-08-31T12:15:57.312007+0000","last_unstale":"2024-08-31T12:15:58.937018+0000","last_undegraded":"2024-08-31T12:15:58.937018+0000","last_fullsized":"2024-08-31T12:15:58.937018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:23:43.553625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303030+0000","last_change":"2024-08-31T12:15:56.290968+0000","last_active":"2024-08-31T12:15:58.303030+0000","last_peered":"2024-08-31T12:15:58.303030+0000","last_clean":"2024-08-31T12:15:58.303030+0000","last_became_active":"2024-08-31T12:15:56.290603+0000","last_became_peered":"2024-08-31T12:15:56.290603+0000","last_unstale":"2024-08-31T12:15:58.303030+0000","last_undegraded":"2024-08-31T12:15:58.303030+0000","last_fullsized":"2024-08-31T12:15:58.303030+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:50:23.215239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.307024+0000","last_change":"2024-08-31T12:15:57.312790+0000","last_active":"2024-08-31T12:15:58.307024+0000","last_peered":"2024-08-31T12:15:58.307024+0000","last_clean":"2024-08-31T12:15:58.307024+0000","last_became_active":"2024-08-31T12:15:57.312383+0000","last_became_peered":"2024-08-31T12:15:57.312383+0000","last_unstale":"2024-08-31T12:15:58.307024+0000","last_undegraded":"2024-08-31T12:15:58.307024+0000","last_fullsized":"2024-08-31T12:15:58.307024+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:20:41.042375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.301697+0000","last_change":"2024-08-31T12:15:56.289795+0000","last_active":"2024-08-31T12:15:58.301697+0000","last_peered":"2024-08-31T12:15:58.301697+0000","last_clean":"2024-08-31T12:15:58.301697+0000","last_became_active":"2024-08-31T12:15:56.289477+0000","last_became_peered":"2024-08-31T12:15:56.289477+0000","last_unstale":"2024-08-31T12:15:58.301697+0000","last_undegraded":"2024-08-31T12:15:58.301697+0000","last_fullsized":"2024-08-31T12:15:58.301697+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:15:51.561831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302374+0000","last_change":"2024-08-31T12:15:57.312865+0000","last_active":"2024-08-31T12:15:58.302374+0000","last_peered":"2024-08-31T12:15:58.302374+0000","last_clean":"2024-08-31T12:15:58.302374+0000","last_became_active":"2024-08-31T12:15:57.312462+0000","last_became_peered":"2024-08-31T12:15:57.312462+0000","last_unstale":"2024-08-31T12:15:58.302374+0000","last_undegraded":"2024-08-31T12:15:58.302374+0000","last_fullsized":"2024-08-31T12:15:58.302374+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:16:42.625396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'10","reported_seq":25,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:07.337317+0000","last_change":"2024-08-31T12:15:56.285412+0000","last_active":"2024-08-31T12:16:07.337317+0000","last_peered":"2024-08-31T12:16:07.337317+0000","last_clean":"2024-08-31T12:16:07.337317+0000","last_became_active":"2024-08-31T12:15:56.285108+0000","last_became_peered":"2024-08-31T12:15:56.285108+0000","last_unstale":"2024-08-31T12:16:07.337317+0000","last_undegraded":"2024-08-31T12:16:07.337317+0000","last_fullsized":"2024-08-31T12:16:07.337317+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:29:45.580957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306407+0000","last_change":"2024-08-31T12:15:57.311701+0000","last_active":"2024-08-31T12:15:58.306407+0000","last_peered":"2024-08-31T12:15:58.306407+0000","last_clean":"2024-08-31T12:15:58.306407+0000","last_became_active":"2024-08-31T12:15:57.311368+0000","last_became_peered":"2024-08-31T12:15:57.311368+0000","last_unstale":"2024-08-31T12:15:58.306407+0000","last_undegraded":"2024-08-31T12:15:58.306407+0000","last_fullsized":"2024-08-31T12:15:58.306407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:30:13.715505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:00.126954+0000","last_change":"2024-08-31T12:15:56.289979+0000","last_active":"2024-08-31T12:16:00.126954+0000","last_peered":"2024-08-31T12:16:00.126954+0000","last_clean":"2024-08-31T12:16:00.126954+0000","last_became_active":"2024-08-31T12:15:56.288289+0000","last_became_peered":"2024-08-31T12:15:56.288289+0000","last_unstale":"2024-08-31T12:16:00.126954+0000","last_undegraded":"2024-08-31T12:16:00.126954+0000","last_fullsized":"2024-08-31T12:16:00.126954+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:46:55.057087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306825+0000","last_change":"2024-08-31T12:15:57.311580+0000","last_active":"2024-08-31T12:15:58.306825+0000","last_peered":"2024-08-31T12:15:58.306825+0000","last_clean":"2024-08-31T12:15:58.306825+0000","last_became_active":"2024-08-31T12:15:57.311323+0000","last_became_peered":"2024-08-31T12:15:57.311323+0000","last_unstale":"2024-08-31T12:15:58.306825+0000","last_undegraded":"2024-08-31T12:15:58.306825+0000","last_fullsized":"2024-08-31T12:15:58.306825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:54:03.871840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:02.304157+0000","last_change":"2024-08-31T12:15:56.290496+0000","last_active":"2024-08-31T12:16:02.304157+0000","last_peered":"2024-08-31T12:16:02.304157+0000","last_clean":"2024-08-31T12:16:02.304157+0000","last_became_active":"2024-08-31T12:15:56.290196+0000","last_became_peered":"2024-08-31T12:15:56.290196+0000","last_unstale":"2024-08-31T12:16:02.304157+0000","last_undegraded":"2024-08-31T12:16:02.304157+0000","last_fullsized":"2024-08-31T12:16:02.304157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:52:20.055757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306986+0000","last_change":"2024-08-31T12:15:57.313317+0000","last_active":"2024-08-31T12:15:58.306986+0000","last_peered":"2024-08-31T12:15:58.306986+0000","last_clean":"2024-08-31T12:15:58.306986+0000","last_became_active":"2024-08-31T12:15:57.312145+0000","last_became_peered":"2024-08-31T12:15:57.312145+0000","last_unstale":"2024-08-31T12:15:58.306986+0000","last_undegraded":"2024-08-31T12:15:58.306986+0000","last_fullsized":"2024-08-31T12:15:58.306986+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:09:31.904512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300816+0000","last_change":"2024-08-31T12:15:56.281809+0000","last_active":"2024-08-31T12:15:58.300816+0000","last_peered":"2024-08-31T12:15:58.300816+0000","last_clean":"2024-08-31T12:15:58.300816+0000","last_became_active":"2024-08-31T12:15:56.281368+0000","last_became_peered":"2024-08-31T12:15:56.281368+0000","last_unstale":"2024-08-31T12:15:58.300816+0000","last_undegraded":"2024-08-31T12:15:58.300816+0000","last_fullsized":"2024-08-31T12:15:58.300816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:17:33.687196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303702+0000","last_change":"2024-08-31T12:15:57.305874+0000","last_active":"2024-08-31T12:15:58.303702+0000","last_peered":"2024-08-31T12:15:58.303702+0000","last_clean":"2024-08-31T12:15:58.303702+0000","last_became_active":"2024-08-31T12:15:57.305507+0000","last_became_peered":"2024-08-31T12:15:57.305507+0000","last_unstale":"2024-08-31T12:15:58.303702+0000","last_undegraded":"2024-08-31T12:15:58.303702+0000","last_fullsized":"2024-08-31T12:15:58.303702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:22:02.569106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937377+0000","last_change":"2024-08-31T12:15:56.291851+0000","last_active":"2024-08-31T12:15:58.937377+0000","last_peered":"2024-08-31T12:15:58.937377+0000","last_clean":"2024-08-31T12:15:58.937377+0000","last_became_active":"2024-08-31T12:15:56.291446+0000","last_became_peered":"2024-08-31T12:15:56.291446+0000","last_unstale":"2024-08-31T12:15:58.937377+0000","last_undegraded":"2024-08-31T12:15:58.937377+0000","last_fullsized":"2024-08-31T12:15:58.937377+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:54:29.988079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306538+0000","last_change":"2024-08-31T12:15:57.316154+0000","last_active":"2024-08-31T12:15:58.306538+0000","last_peered":"2024-08-31T12:15:58.306538+0000","last_clean":"2024-08-31T12:15:58.306538+0000","last_became_active":"2024-08-31T12:15:57.315830+0000","last_became_peered":"2024-08-31T12:15:57.315830+0000","last_unstale":"2024-08-31T12:15:58.306538+0000","last_undegraded":"2024-08-31T12:15:58.306538+0000","last_fullsized":"2024-08-31T12:15:58.306538+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:05:53.039520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":111,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306756+0000","last_change":"2024-08-31T12:15:47.084912+0000","last_active":"2024-08-31T12:15:58.306756+0000","last_peered":"2024-08-31T12:15:58.306756+0000","last_clean":"2024-08-31T12:15:58.306756+0000","last_became_active":"2024-08-31T12:15:47.083771+0000","last_became_peered":"2024-08-31T12:15:47.083771+0000","last_unstale":"2024-08-31T12:15:58.306756+0000","last_undegraded":"2024-08-31T12:15:58.306756+0000","last_fullsized":"2024-08-31T12:15:58.306756+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:45.213688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:45.213688+0000","last_clean_scrub_stamp":"2024-08-31T12:15:45.213688+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:25:24.827530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.301865+0000","last_change":"2024-08-31T12:15:56.285355+0000","last_active":"2024-08-31T12:15:58.301865+0000","last_peered":"2024-08-31T12:15:58.301865+0000","last_clean":"2024-08-31T12:15:58.301865+0000","last_became_active":"2024-08-31T12:15:56.284470+0000","last_became_peered":"2024-08-31T12:15:56.284470+0000","last_unstale":"2024-08-31T12:15:58.301865+0000","last_undegraded":"2024-08-31T12:15:58.301865+0000","last_fullsized":"2024-08-31T12:15:58.301865+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:52:20.055757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936697+0000","last_change":"2024-08-31T12:15:57.309039+0000","last_active":"2024-08-31T12:15:58.936697+0000","last_peered":"2024-08-31T12:15:58.936697+0000","last_clean":"2024-08-31T12:15:58.936697+0000","last_became_active":"2024-08-31T12:15:57.308721+0000","last_became_peered":"2024-08-31T12:15:57.308721+0000","last_unstale":"2024-08-31T12:15:58.936697+0000","last_undegraded":"2024-08-31T12:15:58.936697+0000","last_fullsized":"2024-08-31T12:15:58.936697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:43:12.703971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306092+0000","last_change":"2024-08-31T12:15:56.293221+0000","last_active":"2024-08-31T12:15:58.306092+0000","last_peered":"2024-08-31T12:15:58.306092+0000","last_clean":"2024-08-31T12:15:58.306092+0000","last_became_active":"2024-08-31T12:15:56.292910+0000","last_became_peered":"2024-08-31T12:15:56.292910+0000","last_unstale":"2024-08-31T12:15:58.306092+0000","last_undegraded":"2024-08-31T12:15:58.306092+0000","last_fullsized":"2024-08-31T12:15:58.306092+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:14:33.483002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937638+0000","last_change":"2024-08-31T12:15:57.307266+0000","last_active":"2024-08-31T12:15:58.937638+0000","last_peered":"2024-08-31T12:15:58.937638+0000","last_clean":"2024-08-31T12:15:58.937638+0000","last_became_active":"2024-08-31T12:15:57.306529+0000","last_became_peered":"2024-08-31T12:15:57.306529+0000","last_unstale":"2024-08-31T12:15:58.937638+0000","last_undegraded":"2024-08-31T12:15:58.937638+0000","last_fullsized":"2024-08-31T12:15:58.937638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:42:39.731742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"21'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.938287+0000","last_change":"2024-08-31T12:15:56.291822+0000","last_active":"2024-08-31T12:15:58.938287+0000","last_peered":"2024-08-31T12:15:58.938287+0000","last_clean":"2024-08-31T12:15:58.938287+0000","last_became_active":"2024-08-31T12:15:56.291064+0000","last_became_peered":"2024-08-31T12:15:56.291064+0000","last_unstale":"2024-08-31T12:15:58.938287+0000","last_undegraded":"2024-08-31T12:15:58.938287+0000","last_fullsized":"2024-08-31T12:15:58.938287+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:53:08.929860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300606+0000","last_change":"2024-08-31T12:15:57.299212+0000","last_active":"2024-08-31T12:15:58.300606+0000","last_peered":"2024-08-31T12:15:58.300606+0000","last_clean":"2024-08-31T12:15:58.300606+0000","last_became_active":"2024-08-31T12:15:57.298892+0000","last_became_peered":"2024-08-31T12:15:57.298892+0000","last_unstale":"2024-08-31T12:15:58.300606+0000","last_undegraded":"2024-08-31T12:15:58.300606+0000","last_fullsized":"2024-08-31T12:15:58.300606+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:53:19.523811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303164+0000","last_change":"2024-08-31T12:15:56.286720+0000","last_active":"2024-08-31T12:15:58.303164+0000","last_peered":"2024-08-31T12:15:58.303164+0000","last_clean":"2024-08-31T12:15:58.303164+0000","last_became_active":"2024-08-31T12:15:56.285648+0000","last_became_peered":"2024-08-31T12:15:56.285648+0000","last_unstale":"2024-08-31T12:15:58.303164+0000","last_undegraded":"2024-08-31T12:15:58.303164+0000","last_fullsized":"2024-08-31T12:15:58.303164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:39:31.946792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936342+0000","last_change":"2024-08-31T12:15:57.311785+0000","last_active":"2024-08-31T12:15:58.936342+0000","last_peered":"2024-08-31T12:15:58.936342+0000","last_clean":"2024-08-31T12:15:58.936342+0000","last_became_active":"2024-08-31T12:15:57.311469+0000","last_became_peered":"2024-08-31T12:15:57.311469+0000","last_unstale":"2024-08-31T12:15:58.936342+0000","last_undegraded":"2024-08-31T12:15:58.936342+0000","last_fullsized":"2024-08-31T12:15:58.936342+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:29:54.122825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300691+0000","last_change":"2024-08-31T12:15:56.284302+0000","last_active":"2024-08-31T12:15:58.300691+0000","last_peered":"2024-08-31T12:15:58.300691+0000","last_clean":"2024-08-31T12:15:58.300691+0000","last_became_active":"2024-08-31T12:15:56.283979+0000","last_became_peered":"2024-08-31T12:15:56.283979+0000","last_unstale":"2024-08-31T12:15:58.300691+0000","last_undegraded":"2024-08-31T12:15:58.300691+0000","last_fullsized":"2024-08-31T12:15:58.300691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:51:59.306912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303594+0000","last_change":"2024-08-31T12:15:57.305383+0000","last_active":"2024-08-31T12:15:58.303594+0000","last_peered":"2024-08-31T12:15:58.303594+0000","last_clean":"2024-08-31T12:15:58.303594+0000","last_became_active":"2024-08-31T12:15:57.305048+0000","last_became_peered":"2024-08-31T12:15:57.305048+0000","last_unstale":"2024-08-31T12:15:58.303594+0000","last_undegraded":"2024-08-31T12:15:58.303594+0000","last_fullsized":"2024-08-31T12:15:58.303594+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:34:56.946695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306963+0000","last_change":"2024-08-31T12:15:56.297648+0000","last_active":"2024-08-31T12:15:58.306963+0000","last_peered":"2024-08-31T12:15:58.306963+0000","last_clean":"2024-08-31T12:15:58.306963+0000","last_became_active":"2024-08-31T12:15:56.297306+0000","last_became_peered":"2024-08-31T12:15:56.297306+0000","last_unstale":"2024-08-31T12:15:58.306963+0000","last_undegraded":"2024-08-31T12:15:58.306963+0000","last_fullsized":"2024-08-31T12:15:58.306963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:29:54.693819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300681+0000","last_change":"2024-08-31T12:15:57.297230+0000","last_active":"2024-08-31T12:15:58.300681+0000","last_peered":"2024-08-31T12:15:58.300681+0000","last_clean":"2024-08-31T12:15:58.300681+0000","last_became_active":"2024-08-31T12:15:57.296892+0000","last_became_peered":"2024-08-31T12:15:57.296892+0000","last_unstale":"2024-08-31T12:15:58.300681+0000","last_undegraded":"2024-08-31T12:15:58.300681+0000","last_fullsized":"2024-08-31T12:15:58.300681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:42:36.033586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300841+0000","last_change":"2024-08-31T12:15:56.286504+0000","last_active":"2024-08-31T12:15:58.300841+0000","last_peered":"2024-08-31T12:15:58.300841+0000","last_clean":"2024-08-31T12:15:58.300841+0000","last_became_active":"2024-08-31T12:15:56.286173+0000","last_became_peered":"2024-08-31T12:15:56.286173+0000","last_unstale":"2024-08-31T12:15:58.300841+0000","last_undegraded":"2024-08-31T12:15:58.300841+0000","last_fullsized":"2024-08-31T12:15:58.300841+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:06:34.313772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.938135+0000","last_change":"2024-08-31T12:15:57.308013+0000","last_active":"2024-08-31T12:15:58.938135+0000","last_peered":"2024-08-31T12:15:58.938135+0000","last_clean":"2024-08-31T12:15:58.938135+0000","last_became_active":"2024-08-31T12:15:57.307701+0000","last_became_peered":"2024-08-31T12:15:57.307701+0000","last_unstale":"2024-08-31T12:15:58.938135+0000","last_undegraded":"2024-08-31T12:15:58.938135+0000","last_fullsized":"2024-08-31T12:15:58.938135+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:06:15.161589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:02.304975+0000","last_change":"2024-08-31T12:15:56.294852+0000","last_active":"2024-08-31T12:16:02.304975+0000","last_peered":"2024-08-31T12:16:02.304975+0000","last_clean":"2024-08-31T12:16:02.304975+0000","last_became_active":"2024-08-31T12:15:56.294496+0000","last_became_peered":"2024-08-31T12:15:56.294496+0000","last_unstale":"2024-08-31T12:16:02.304975+0000","last_undegraded":"2024-08-31T12:16:02.304975+0000","last_fullsized":"2024-08-31T12:16:02.304975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:49:17.488528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937973+0000","last_change":"2024-08-31T12:15:57.305202+0000","last_active":"2024-08-31T12:15:58.937973+0000","last_peered":"2024-08-31T12:15:58.937973+0000","last_clean":"2024-08-31T12:15:58.937973+0000","last_became_active":"2024-08-31T12:15:57.304602+0000","last_became_peered":"2024-08-31T12:15:57.304602+0000","last_unstale":"2024-08-31T12:15:58.937973+0000","last_undegraded":"2024-08-31T12:15:58.937973+0000","last_fullsized":"2024-08-31T12:15:58.937973+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:56:03.280339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306090+0000","last_change":"2024-08-31T12:15:56.297542+0000","last_active":"2024-08-31T12:15:58.306090+0000","last_peered":"2024-08-31T12:15:58.306090+0000","last_clean":"2024-08-31T12:15:58.306090+0000","last_became_active":"2024-08-31T12:15:56.297168+0000","last_became_peered":"2024-08-31T12:15:56.297168+0000","last_unstale":"2024-08-31T12:15:58.306090+0000","last_undegraded":"2024-08-31T12:15:58.306090+0000","last_fullsized":"2024-08-31T12:15:58.306090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:56:41.924772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936311+0000","last_change":"2024-08-31T12:15:57.298845+0000","last_active":"2024-08-31T12:15:58.936311+0000","last_peered":"2024-08-31T12:15:58.936311+0000","last_clean":"2024-08-31T12:15:58.936311+0000","last_became_active":"2024-08-31T12:15:57.298520+0000","last_became_peered":"2024-08-31T12:15:57.298520+0000","last_unstale":"2024-08-31T12:15:58.936311+0000","last_undegraded":"2024-08-31T12:15:58.936311+0000","last_fullsized":"2024-08-31T12:15:58.936311+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:06:35.318873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300879+0000","last_change":"2024-08-31T12:15:56.290475+0000","last_active":"2024-08-31T12:15:58.300879+0000","last_peered":"2024-08-31T12:15:58.300879+0000","last_clean":"2024-08-31T12:15:58.300879+0000","last_became_active":"2024-08-31T12:15:56.289960+0000","last_became_peered":"2024-08-31T12:15:56.289960+0000","last_unstale":"2024-08-31T12:15:58.300879+0000","last_undegraded":"2024-08-31T12:15:58.300879+0000","last_fullsized":"2024-08-31T12:15:58.300879+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:29:53.117724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.938261+0000","last_change":"2024-08-31T12:15:57.301460+0000","last_active":"2024-08-31T12:15:58.938261+0000","last_peered":"2024-08-31T12:15:58.938261+0000","last_clean":"2024-08-31T12:15:58.938261+0000","last_became_active":"2024-08-31T12:15:57.301157+0000","last_became_peered":"2024-08-31T12:15:57.301157+0000","last_unstale":"2024-08-31T12:15:58.938261+0000","last_undegraded":"2024-08-31T12:15:58.938261+0000","last_fullsized":"2024-08-31T12:15:58.938261+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:22:52.173578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.301007+0000","last_change":"2024-08-31T12:15:56.289720+0000","last_active":"2024-08-31T12:15:58.301007+0000","last_peered":"2024-08-31T12:15:58.301007+0000","last_clean":"2024-08-31T12:15:58.301007+0000","last_became_active":"2024-08-31T12:15:56.289332+0000","last_became_peered":"2024-08-31T12:15:56.289332+0000","last_unstale":"2024-08-31T12:15:58.301007+0000","last_undegraded":"2024-08-31T12:15:58.301007+0000","last_fullsized":"2024-08-31T12:15:58.301007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:15:51.561831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937371+0000","last_change":"2024-08-31T12:15:57.313802+0000","last_active":"2024-08-31T12:15:58.937371+0000","last_peered":"2024-08-31T12:15:58.937371+0000","last_clean":"2024-08-31T12:15:58.937371+0000","last_became_active":"2024-08-31T12:15:57.313428+0000","last_became_peered":"2024-08-31T12:15:57.313428+0000","last_unstale":"2024-08-31T12:15:58.937371+0000","last_undegraded":"2024-08-31T12:15:58.937371+0000","last_fullsized":"2024-08-31T12:15:58.937371+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:22:13.435300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:02.305854+0000","last_change":"2024-08-31T12:15:56.296105+0000","last_active":"2024-08-31T12:16:02.305854+0000","last_peered":"2024-08-31T12:16:02.305854+0000","last_clean":"2024-08-31T12:16:02.305854+0000","last_became_active":"2024-08-31T12:15:56.295524+0000","last_became_peered":"2024-08-31T12:15:56.295524+0000","last_unstale":"2024-08-31T12:16:02.305854+0000","last_undegraded":"2024-08-31T12:16:02.305854+0000","last_fullsized":"2024-08-31T12:16:02.305854+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:41:55.482044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.301683+0000","last_change":"2024-08-31T12:15:57.300155+0000","last_active":"2024-08-31T12:15:58.301683+0000","last_peered":"2024-08-31T12:15:58.301683+0000","last_clean":"2024-08-31T12:15:58.301683+0000","last_became_active":"2024-08-31T12:15:57.299436+0000","last_became_peered":"2024-08-31T12:15:57.299436+0000","last_unstale":"2024-08-31T12:15:58.301683+0000","last_undegraded":"2024-08-31T12:15:58.301683+0000","last_fullsized":"2024-08-31T12:15:58.301683+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:37:45.891394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303455+0000","last_change":"2024-08-31T12:15:57.300529+0000","last_active":"2024-08-31T12:15:58.303455+0000","last_peered":"2024-08-31T12:15:58.303455+0000","last_clean":"2024-08-31T12:15:58.303455+0000","last_became_active":"2024-08-31T12:15:57.300159+0000","last_became_peered":"2024-08-31T12:15:57.300159+0000","last_unstale":"2024-08-31T12:15:58.303455+0000","last_undegraded":"2024-08-31T12:15:58.303455+0000","last_fullsized":"2024-08-31T12:15:58.303455+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:35:53.831742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:02.304978+0000","last_change":"2024-08-31T12:15:56.297791+0000","last_active":"2024-08-31T12:16:02.304978+0000","last_peered":"2024-08-31T12:16:02.304978+0000","last_clean":"2024-08-31T12:16:02.304978+0000","last_became_active":"2024-08-31T12:15:56.296976+0000","last_became_peered":"2024-08-31T12:15:56.296976+0000","last_unstale":"2024-08-31T12:16:02.304978+0000","last_undegraded":"2024-08-31T12:16:02.304978+0000","last_fullsized":"2024-08-31T12:16:02.304978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:07:54.963050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303405+0000","last_change":"2024-08-31T12:15:57.301167+0000","last_active":"2024-08-31T12:15:58.303405+0000","last_peered":"2024-08-31T12:15:58.303405+0000","last_clean":"2024-08-31T12:15:58.303405+0000","last_became_active":"2024-08-31T12:15:57.300646+0000","last_became_peered":"2024-08-31T12:15:57.300646+0000","last_unstale":"2024-08-31T12:15:58.303405+0000","last_undegraded":"2024-08-31T12:15:58.303405+0000","last_fullsized":"2024-08-31T12:15:58.303405+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:05:53.039520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:02.303751+0000","last_change":"2024-08-31T12:15:56.295139+0000","last_active":"2024-08-31T12:16:02.303751+0000","last_peered":"2024-08-31T12:16:02.303751+0000","last_clean":"2024-08-31T12:16:02.303751+0000","last_became_active":"2024-08-31T12:15:56.294560+0000","last_became_peered":"2024-08-31T12:15:56.294560+0000","last_unstale":"2024-08-31T12:16:02.303751+0000","last_undegraded":"2024-08-31T12:16:02.303751+0000","last_fullsized":"2024-08-31T12:16:02.303751+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:52:16.999507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937898+0000","last_change":"2024-08-31T12:15:57.311074+0000","last_active":"2024-08-31T12:15:58.937898+0000","last_peered":"2024-08-31T12:15:58.937898+0000","last_clean":"2024-08-31T12:15:58.937898+0000","last_became_active":"2024-08-31T12:15:57.310262+0000","last_became_peered":"2024-08-31T12:15:57.310262+0000","last_unstale":"2024-08-31T12:15:58.937898+0000","last_undegraded":"2024-08-31T12:15:58.937898+0000","last_fullsized":"2024-08-31T12:15:58.937898+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:47:51.267409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937350+0000","last_change":"2024-08-31T12:15:56.285428+0000","last_active":"2024-08-31T12:15:58.937350+0000","last_peered":"2024-08-31T12:15:58.937350+0000","last_clean":"2024-08-31T12:15:58.937350+0000","last_became_active":"2024-08-31T12:15:56.284884+0000","last_became_peered":"2024-08-31T12:15:56.284884+0000","last_unstale":"2024-08-31T12:15:58.937350+0000","last_undegraded":"2024-08-31T12:15:58.937350+0000","last_fullsized":"2024-08-31T12:15:58.937350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:49:25.181432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937174+0000","last_change":"2024-08-31T12:15:57.305772+0000","last_active":"2024-08-31T12:15:58.937174+0000","last_peered":"2024-08-31T12:15:58.937174+0000","last_clean":"2024-08-31T12:15:58.937174+0000","last_became_active":"2024-08-31T12:15:57.305409+0000","last_became_peered":"2024-08-31T12:15:57.305409+0000","last_unstale":"2024-08-31T12:15:58.937174+0000","last_undegraded":"2024-08-31T12:15:58.937174+0000","last_fullsized":"2024-08-31T12:15:58.937174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:53:19.523811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:02.306147+0000","last_change":"2024-08-31T12:15:56.294197+0000","last_active":"2024-08-31T12:16:02.306147+0000","last_peered":"2024-08-31T12:16:02.306147+0000","last_clean":"2024-08-31T12:16:02.306147+0000","last_became_active":"2024-08-31T12:15:56.293864+0000","last_became_peered":"2024-08-31T12:15:56.293864+0000","last_unstale":"2024-08-31T12:16:02.306147+0000","last_undegraded":"2024-08-31T12:16:02.306147+0000","last_fullsized":"2024-08-31T12:16:02.306147+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:43:17.443433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509448,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35560,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336280,"statfs":{"total":96636764160,"available":96600350720,"internally_reserved":0,"allocated":221184,"data_stored":74713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509447,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":74713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509448,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":73261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509448,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":815104,"data_stored":664107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542151,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36132,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335708,"statfs":{"total":96636764160,"available":96599764992,"internally_reserved":0,"allocated":806912,"data_stored":664051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542153,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35552,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336288,"statfs":{"total":96636764160,"available":96600358912,"internally_reserved":0,"allocated":212992,"data_stored":73191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542151,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35560,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336280,"statfs":{"total":96636764160,"available":96600350720,"internally_reserved":0,"allocated":221184,"data_stored":73227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":73101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-31T12:16:08.631 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-08-31T12:16:08.631 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-08-31T12:16:08.631 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-08-31T12:16:08.631 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-08-31T12:16:08.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.780+0000 7f30aa7e6640 1 -- 172.21.15.19:0/2629006113 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f30a40cb280 msgr2=0x7f30a415b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:08.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.780+0000 7f30aa7e6640 1 --2- 172.21.15.19:0/2629006113 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f30a40cb280 0x7f30a415b120 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3098005fd0 tx=0x7f309802a670 comp rx=0 tx=0).stop 2024-08-31T12:16:08.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.780+0000 7f30aa7e6640 1 -- 172.21.15.19:0/2629006113 shutdown_connections 2024-08-31T12:16:08.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.780+0000 7f30aa7e6640 1 --2- 172.21.15.19:0/2629006113 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f30a415b660 0x7f30a4165aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:08.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.780+0000 7f30aa7e6640 1 --2- 172.21.15.19:0/2629006113 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f30a40cb280 0x7f30a415b120 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:08.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.780+0000 7f30aa7e6640 1 -- 172.21.15.19:0/2629006113 >> 172.21.15.19:0/2629006113 conn(0x7f30a4074810 msgr2=0x7f30a4068aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:08.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.780+0000 7f30aa7e6640 1 -- 172.21.15.19:0/2629006113 shutdown_connections 2024-08-31T12:16:08.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.780+0000 7f30aa7e6640 1 -- 172.21.15.19:0/2629006113 wait complete. 2024-08-31T12:16:08.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.782+0000 7f30aa7e6640 1 Processor -- start 2024-08-31T12:16:08.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.782+0000 7f30aa7e6640 1 -- start start 2024-08-31T12:16:08.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.783+0000 7f30aa7e6640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f30a40cb280 0x7f30a40cfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:08.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.783+0000 7f30aa7e6640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f30a40ccbb0 0x7f30a40d0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:08.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.783+0000 7f30aa7e6640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f30a415b660 0x7f30a4149450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:08.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.783+0000 7f30aa7e6640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f30a415ba70 con 0x7f30a415b660 2024-08-31T12:16:08.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.783+0000 7f30aa7e6640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f30a415f050 con 0x7f30a40cb280 2024-08-31T12:16:08.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.783+0000 7f30aa7e6640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f30a40cc780 con 0x7f30a40ccbb0 2024-08-31T12:16:08.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.783+0000 7f30a8d83640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f30a415b660 0x7f30a4149450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:08.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.783+0000 7f30a8d83640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f30a415b660 0x7f30a4149450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:44842/0 (socket says 172.21.15.19:44842) 2024-08-31T12:16:08.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.783+0000 7f30a8d83640 1 -- 172.21.15.19:0/3767584811 learned_addr learned my addr 172.21.15.19:0/3767584811 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:08.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.783+0000 7f30a3fff640 1 --2- 172.21.15.19:0/3767584811 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f30a40cb280 0x7f30a40cfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:08.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.783+0000 7f30a37fe640 1 --2- 172.21.15.19:0/3767584811 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f30a40ccbb0 0x7f30a40d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:08.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.784+0000 7f30a8d83640 1 -- 172.21.15.19:0/3767584811 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f30a40ccbb0 msgr2=0x7f30a40d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:08.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.784+0000 7f30a8d83640 1 --2- 172.21.15.19:0/3767584811 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f30a40ccbb0 0x7f30a40d0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:08.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.784+0000 7f30a8d83640 1 -- 172.21.15.19:0/3767584811 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f30a40cb280 msgr2=0x7f30a40cfeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:08.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.784+0000 7f30a8d83640 1 --2- 172.21.15.19:0/3767584811 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f30a40cb280 0x7f30a40cfeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:08.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.784+0000 7f30a8d83640 1 -- 172.21.15.19:0/3767584811 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30a40ce780 con 0x7f30a415b660 2024-08-31T12:16:08.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.784+0000 7f30a8d83640 1 --2- 172.21.15.19:0/3767584811 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f30a415b660 0x7f30a4149450 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f309c00ea40 tx=0x7f309c00ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:08.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.784+0000 7f30a0ff9640 1 -- 172.21.15.19:0/3767584811 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f309c00ce40 con 0x7f30a415b660 2024-08-31T12:16:08.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.784+0000 7f30a0ff9640 1 -- 172.21.15.19:0/3767584811 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f309c004c10 con 0x7f30a415b660 2024-08-31T12:16:08.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.784+0000 7f30aa7e6640 1 -- 172.21.15.19:0/3767584811 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30a40b5270 con 0x7f30a415b660 2024-08-31T12:16:08.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.784+0000 7f30a0ff9640 1 -- 172.21.15.19:0/3767584811 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f309c005020 con 0x7f30a415b660 2024-08-31T12:16:08.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.784+0000 7f30aa7e6640 1 -- 172.21.15.19:0/3767584811 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30a41528b0 con 0x7f30a415b660 2024-08-31T12:16:08.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.785+0000 7f30aa7e6640 1 -- 172.21.15.19:0/3767584811 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30a40cfae0 con 0x7f30a415b660 2024-08-31T12:16:08.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.786+0000 7f30a0ff9640 1 -- 172.21.15.19:0/3767584811 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f309c0051c0 con 0x7f30a415b660 2024-08-31T12:16:08.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.786+0000 7f30a0ff9640 1 --2- 172.21.15.19:0/3767584811 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3090077a50 0x7f3090079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:08.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.786+0000 7f30a0ff9640 1 -- 172.21.15.19:0/3767584811 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f309c09a360 con 0x7f30a415b660 2024-08-31T12:16:08.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.787+0000 7f30a3fff640 1 --2- 172.21.15.19:0/3767584811 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3090077a50 0x7f3090079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:08.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.787+0000 7f30a3fff640 1 --2- 172.21.15.19:0/3767584811 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3090077a50 0x7f3090079f60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3094001720 tx=0x7f3094009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:08.792 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:08.791+0000 7f30a0ff9640 1 -- 172.21.15.19:0/3767584811 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f309c062a70 con 0x7f30a415b660 2024-08-31T12:16:09.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.014+0000 7f30aa7e6640 1 -- 172.21.15.19:0/3767584811 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f30a4155660 con 0x7f30a415b660 2024-08-31T12:16:09.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.014+0000 7f30a0ff9640 1 -- 172.21.15.19:0/3767584811 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f309c067230 con 0x7f30a415b660 2024-08-31T12:16:09.015 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:16:09.015 INFO:teuthology.orchestra.run.smithi019.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-08-31T12:16:09.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.016+0000 7f30aa7e6640 1 -- 172.21.15.19:0/3767584811 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3090077a50 msgr2=0x7f3090079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:09.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.016+0000 7f30aa7e6640 1 --2- 172.21.15.19:0/3767584811 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3090077a50 0x7f3090079f60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3094001720 tx=0x7f3094009040 comp rx=0 tx=0).stop 2024-08-31T12:16:09.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.016+0000 7f30aa7e6640 1 -- 172.21.15.19:0/3767584811 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f30a415b660 msgr2=0x7f30a4149450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:09.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.016+0000 7f30aa7e6640 1 --2- 172.21.15.19:0/3767584811 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f30a415b660 0x7f30a4149450 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f309c00ea40 tx=0x7f309c00ef00 comp rx=0 tx=0).stop 2024-08-31T12:16:09.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.016+0000 7f30aa7e6640 1 -- 172.21.15.19:0/3767584811 shutdown_connections 2024-08-31T12:16:09.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.016+0000 7f30aa7e6640 1 --2- 172.21.15.19:0/3767584811 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3090077a50 0x7f3090079f60 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:09.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.017+0000 7f30aa7e6640 1 --2- 172.21.15.19:0/3767584811 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f30a415b660 0x7f30a4149450 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:09.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.017+0000 7f30aa7e6640 1 --2- 172.21.15.19:0/3767584811 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f30a40ccbb0 0x7f30a40d0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:09.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.017+0000 7f30aa7e6640 1 --2- 172.21.15.19:0/3767584811 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f30a40cb280 0x7f30a40cfeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:09.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.017+0000 7f30aa7e6640 1 -- 172.21.15.19:0/3767584811 >> 172.21.15.19:0/3767584811 conn(0x7f30a4074810 msgr2=0x7f30a4068aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:09.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.017+0000 7f30aa7e6640 1 -- 172.21.15.19:0/3767584811 shutdown_connections 2024-08-31T12:16:09.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.017+0000 7f30aa7e6640 1 -- 172.21.15.19:0/3767584811 wait complete. 2024-08-31T12:16:09.028 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-08-31T12:16:09.029 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-31T12:16:09.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.142+0000 7f2178357640 1 -- 172.21.15.19:0/1530272117 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2158003a50 msgr2=0x7f2158003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:09.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.142+0000 7f2178357640 1 --2- 172.21.15.19:0/1530272117 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2158003a50 0x7f2158003ec0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f216800c260 tx=0x7f2158005830 comp rx=0 tx=0).stop 2024-08-31T12:16:09.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.142+0000 7f2178357640 1 -- 172.21.15.19:0/1530272117 shutdown_connections 2024-08-31T12:16:09.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.142+0000 7f2178357640 1 --2- 172.21.15.19:0/1530272117 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2158003a50 0x7f2158003ec0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:09.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.142+0000 7f2178357640 1 -- 172.21.15.19:0/1530272117 >> 172.21.15.19:0/1530272117 conn(0x7f21700ca670 msgr2=0x7f21700cd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:09.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.142+0000 7f2178357640 1 -- 172.21.15.19:0/1530272117 shutdown_connections 2024-08-31T12:16:09.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.142+0000 7f2178357640 1 -- 172.21.15.19:0/1530272117 wait complete. 2024-08-31T12:16:09.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.143+0000 7f2178357640 1 Processor -- start 2024-08-31T12:16:09.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.143+0000 7f2178357640 1 -- start start 2024-08-31T12:16:09.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.145+0000 7f2178357640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2158003a50 0x7f21700c4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:09.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.145+0000 7f2178357640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f21700c9580 0x7f2170145d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:09.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.145+0000 7f2178357640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f21700c9990 0x7f2170146280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:09.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.145+0000 7f2178357640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f21700cd130 con 0x7f21700c9990 2024-08-31T12:16:09.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.145+0000 7f2178357640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2170073d70 con 0x7f2158003a50 2024-08-31T12:16:09.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.145+0000 7f2178357640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f217006eb40 con 0x7f21700c9580 2024-08-31T12:16:09.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.145+0000 7f21768f4640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f21700c9990 0x7f2170146280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:09.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.145+0000 7f21768f4640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f21700c9990 0x7f2170146280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:44864/0 (socket says 172.21.15.19:44864) 2024-08-31T12:16:09.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.145+0000 7f21768f4640 1 -- 172.21.15.19:0/930689515 learned_addr learned my addr 172.21.15.19:0/930689515 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:09.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.145+0000 7f21760f3640 1 --2- 172.21.15.19:0/930689515 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2158003a50 0x7f21700c4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:09.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.145+0000 7f21758f2640 1 --2- 172.21.15.19:0/930689515 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f21700c9580 0x7f2170145d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:09.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.145+0000 7f21768f4640 1 -- 172.21.15.19:0/930689515 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f21700c9580 msgr2=0x7f2170145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:09.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.145+0000 7f21768f4640 1 --2- 172.21.15.19:0/930689515 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f21700c9580 0x7f2170145d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:09.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.145+0000 7f21768f4640 1 -- 172.21.15.19:0/930689515 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2158003a50 msgr2=0x7f21700c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:09.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.145+0000 7f21768f4640 1 --2- 172.21.15.19:0/930689515 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2158003a50 0x7f21700c4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:09.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.145+0000 7f21768f4640 1 -- 172.21.15.19:0/930689515 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21700c5390 con 0x7f21700c9990 2024-08-31T12:16:09.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.145+0000 7f21768f4640 1 --2- 172.21.15.19:0/930689515 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f21700c9990 0x7f2170146280 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f217006e4d0 tx=0x7f21600019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:09.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.145+0000 7f216effd640 1 -- 172.21.15.19:0/930689515 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2160015020 con 0x7f21700c9990 2024-08-31T12:16:09.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.145+0000 7f216effd640 1 -- 172.21.15.19:0/930689515 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2160011960 con 0x7f21700c9990 2024-08-31T12:16:09.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.145+0000 7f2178357640 1 -- 172.21.15.19:0/930689515 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f217014ba90 con 0x7f21700c9990 2024-08-31T12:16:09.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.145+0000 7f216effd640 1 -- 172.21.15.19:0/930689515 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2160011db0 con 0x7f21700c9990 2024-08-31T12:16:09.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.146+0000 7f2178357640 1 -- 172.21.15.19:0/930689515 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21700c6350 con 0x7f21700c9990 2024-08-31T12:16:09.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.147+0000 7f2178357640 1 -- 172.21.15.19:0/930689515 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2170158f70 con 0x7f21700c9990 2024-08-31T12:16:09.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.147+0000 7f216effd640 1 -- 172.21.15.19:0/930689515 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f2160018020 con 0x7f21700c9990 2024-08-31T12:16:09.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.148+0000 7f216effd640 1 --2- 172.21.15.19:0/930689515 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f215c077a50 0x7f215c079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:09.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.148+0000 7f216effd640 1 -- 172.21.15.19:0/930689515 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f216009a0a0 con 0x7f21700c9990 2024-08-31T12:16:09.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.148+0000 7f21760f3640 1 --2- 172.21.15.19:0/930689515 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f215c077a50 0x7f215c079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:09.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.148+0000 7f21760f3640 1 --2- 172.21.15.19:0/930689515 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f215c077a50 0x7f215c079f60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2168033000 tx=0x7f217006e250 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:09.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.152+0000 7f216effd640 1 -- 172.21.15.19:0/930689515 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f216001f020 con 0x7f21700c9990 2024-08-31T12:16:09.374 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.373+0000 7f2178357640 1 -- 172.21.15.19:0/930689515 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f21700c7960 con 0x7f21700c9990 2024-08-31T12:16:09.374 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.373+0000 7f216effd640 1 -- 172.21.15.19:0/930689515 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+6898 (secure 0 0 0) 0x7f2160062ea0 con 0x7f21700c9990 2024-08-31T12:16:09.374 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:16:09.374 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":11,"btime":"2024-08-31T12:16:08:358605+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4371,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6833/4215834946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":4215834946},{"type":"v1","addr":"172.21.15.19:6833","nonce":4215834946}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/543344732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":543344732},{"type":"v1","addr":"172.21.15.19:6835","nonce":543344732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4380,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6837/699077015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6836","nonce":699077015},{"type":"v1","addr":"172.21.15.19:6837","nonce":699077015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":11,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-31T12:15:56.652936+0000","modified":"2024-08-31T12:16:08.358591+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4429,"mds_1":4426,"mds_2":4398},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4398":{"gid":4398,"name":"f","rank":2,"incarnation":11,"state":"up:creating","state_seq":1,"addr":"172.21.15.43:6837/2103490885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6835","nonce":2103490885},{"type":"v1","addr":"172.21.15.43:6837","nonce":2103490885}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4426":{"gid":4426,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.43:6839/1196858061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1196858061},{"type":"v1","addr":"172.21.15.43:6839","nonce":1196858061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4429":{"gid":4429,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.43:6838/3981040655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":3981040655},{"type":"v1","addr":"172.21.15.43:6838","nonce":3981040655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4429,"qdb_cluster":[4429,4426]},"id":1}]} 2024-08-31T12:16:09.374 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 11 2024-08-31T12:16:09.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.375+0000 7f2178357640 1 -- 172.21.15.19:0/930689515 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f215c077a50 msgr2=0x7f215c079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:09.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.375+0000 7f2178357640 1 --2- 172.21.15.19:0/930689515 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f215c077a50 0x7f215c079f60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2168033000 tx=0x7f217006e250 comp rx=0 tx=0).stop 2024-08-31T12:16:09.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.375+0000 7f2178357640 1 -- 172.21.15.19:0/930689515 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f21700c9990 msgr2=0x7f2170146280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:09.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.375+0000 7f2178357640 1 --2- 172.21.15.19:0/930689515 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f21700c9990 0x7f2170146280 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f217006e4d0 tx=0x7f21600019b0 comp rx=0 tx=0).stop 2024-08-31T12:16:09.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.375+0000 7f2178357640 1 -- 172.21.15.19:0/930689515 shutdown_connections 2024-08-31T12:16:09.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.375+0000 7f2178357640 1 --2- 172.21.15.19:0/930689515 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f215c077a50 0x7f215c079f60 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:09.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.375+0000 7f2178357640 1 --2- 172.21.15.19:0/930689515 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f21700c9990 0x7f2170146280 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:09.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.375+0000 7f2178357640 1 --2- 172.21.15.19:0/930689515 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f21700c9580 0x7f2170145d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:09.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.375+0000 7f2178357640 1 --2- 172.21.15.19:0/930689515 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2158003a50 0x7f21700c4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:09.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.375+0000 7f2178357640 1 -- 172.21.15.19:0/930689515 >> 172.21.15.19:0/930689515 conn(0x7f21700ca670 msgr2=0x7f21700635a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:09.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.375+0000 7f2178357640 1 -- 172.21.15.19:0/930689515 shutdown_connections 2024-08-31T12:16:09.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.375+0000 7f2178357640 1 -- 172.21.15.19:0/930689515 wait complete. 2024-08-31T12:16:09.387 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-31T12:16:09.501 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.499+0000 7f8a87f00640 1 -- 172.21.15.19:0/4229907008 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8a800cc820 msgr2=0x7f8a800b4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:09.501 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.499+0000 7f8a87f00640 1 --2- 172.21.15.19:0/4229907008 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8a800cc820 0x7f8a800b4f60 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8a8006f120 tx=0x7f8a7c02a5f0 comp rx=0 tx=0).stop 2024-08-31T12:16:09.501 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.499+0000 7f8a87f00640 1 -- 172.21.15.19:0/4229907008 shutdown_connections 2024-08-31T12:16:09.501 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.499+0000 7f8a87f00640 1 --2- 172.21.15.19:0/4229907008 >> v2:172.21.15.43:3300/0 conn(0x7f8a8006eb00 0x7f8a800b54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:09.501 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.500+0000 7f8a87f00640 1 --2- 172.21.15.19:0/4229907008 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8a800cc820 0x7f8a800b4f60 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:09.501 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.500+0000 7f8a87f00640 1 -- 172.21.15.19:0/4229907008 >> 172.21.15.19:0/4229907008 conn(0x7f8a80069260 msgr2=0x7f8a800d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:09.501 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.500+0000 7f8a87f00640 1 -- 172.21.15.19:0/4229907008 shutdown_connections 2024-08-31T12:16:09.501 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.500+0000 7f8a87f00640 1 -- 172.21.15.19:0/4229907008 wait complete. 2024-08-31T12:16:09.502 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.501+0000 7f8a87f00640 1 Processor -- start 2024-08-31T12:16:09.502 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.501+0000 7f8a87f00640 1 -- start start 2024-08-31T12:16:09.505 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.502+0000 7f8a87f00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8a8006eb00 0x7f8a801f1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:09.505 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.503+0000 7f8a87f00640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8a800cc820 0x7f8a8015e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:09.505 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.503+0000 7f8a87f00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8a8015b0a0 0x7f8a80152820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:09.505 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.503+0000 7f8a87f00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f8a800ccf90 con 0x7f8a8006eb00 2024-08-31T12:16:09.505 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.503+0000 7f8a87f00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8a800b4a40 con 0x7f8a8015b0a0 2024-08-31T12:16:09.505 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.503+0000 7f8a87f00640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f8a8006dff0 con 0x7f8a800cc820 2024-08-31T12:16:09.505 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.503+0000 7f8a85c9c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8a8006eb00 0x7f8a801f1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:09.505 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.503+0000 7f8a85c9c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8a8006eb00 0x7f8a801f1020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:44876/0 (socket says 172.21.15.19:44876) 2024-08-31T12:16:09.505 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.503+0000 7f8a85c9c640 1 -- 172.21.15.19:0/1170773374 learned_addr learned my addr 172.21.15.19:0/1170773374 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:09.505 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.503+0000 7f8a8649d640 1 --2- 172.21.15.19:0/1170773374 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8a8015b0a0 0x7f8a80152820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:09.506 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.503+0000 7f8a85c9c640 1 -- 172.21.15.19:0/1170773374 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8a800cc820 msgr2=0x7f8a8015e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:09.506 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.503+0000 7f8a8549b640 1 --2- 172.21.15.19:0/1170773374 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8a800cc820 0x7f8a8015e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:09.506 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.503+0000 7f8a85c9c640 1 --2- 172.21.15.19:0/1170773374 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8a800cc820 0x7f8a8015e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:09.506 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.503+0000 7f8a85c9c640 1 -- 172.21.15.19:0/1170773374 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8a8015b0a0 msgr2=0x7f8a80152820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:09.506 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.503+0000 7f8a85c9c640 1 --2- 172.21.15.19:0/1170773374 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8a8015b0a0 0x7f8a80152820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:09.506 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.503+0000 7f8a85c9c640 1 -- 172.21.15.19:0/1170773374 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a80156c80 con 0x7f8a8006eb00 2024-08-31T12:16:09.506 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.503+0000 7f8a8649d640 1 --2- 172.21.15.19:0/1170773374 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8a8015b0a0 0x7f8a80152820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-31T12:16:09.506 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.503+0000 7f8a8549b640 1 --2- 172.21.15.19:0/1170773374 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8a800cc820 0x7f8a8015e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-31T12:16:09.506 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.503+0000 7f8a85c9c640 1 --2- 172.21.15.19:0/1170773374 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8a8006eb00 0x7f8a801f1020 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c0099c0 tx=0x7f8a7c0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:09.506 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.503+0000 7f8a727fc640 1 -- 172.21.15.19:0/1170773374 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a7c03e030 con 0x7f8a8006eb00 2024-08-31T12:16:09.506 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.503+0000 7f8a727fc640 1 -- 172.21.15.19:0/1170773374 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8a7c0047a0 con 0x7f8a8006eb00 2024-08-31T12:16:09.506 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.503+0000 7f8a87f00640 1 -- 172.21.15.19:0/1170773374 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a8015c440 con 0x7f8a8006eb00 2024-08-31T12:16:09.506 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.503+0000 7f8a727fc640 1 -- 172.21.15.19:0/1170773374 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a7c035040 con 0x7f8a8006eb00 2024-08-31T12:16:09.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.503+0000 7f8a87f00640 1 -- 172.21.15.19:0/1170773374 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a801f1560 con 0x7f8a8006eb00 2024-08-31T12:16:09.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.504+0000 7f8a87f00640 1 -- 172.21.15.19:0/1170773374 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a800ca860 con 0x7f8a8006eb00 2024-08-31T12:16:09.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.505+0000 7f8a727fc640 1 -- 172.21.15.19:0/1170773374 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f8a7c041030 con 0x7f8a8006eb00 2024-08-31T12:16:09.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.505+0000 7f8a727fc640 1 --2- 172.21.15.19:0/1170773374 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8a64077a50 0x7f8a64079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:09.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.505+0000 7f8a727fc640 1 -- 172.21.15.19:0/1170773374 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f8a7c0b9e10 con 0x7f8a8006eb00 2024-08-31T12:16:09.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.506+0000 7f8a8549b640 1 --2- 172.21.15.19:0/1170773374 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8a64077a50 0x7f8a64079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:09.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.506+0000 7f8a8549b640 1 --2- 172.21.15.19:0/1170773374 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8a64077a50 0x7f8a64079f60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8a801eec50 tx=0x7f8a74009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:09.510 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.510+0000 7f8a727fc640 1 -- 172.21.15.19:0/1170773374 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f8a7c082c10 con 0x7f8a8006eb00 2024-08-31T12:16:09.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.729+0000 7f8a87f00640 1 -- 172.21.15.19:0/1170773374 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8a8006db30 con 0x7f8a8006eb00 2024-08-31T12:16:09.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.729+0000 7f8a727fc640 1 -- 172.21.15.19:0/1170773374 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+6898 (secure 0 0 0) 0x7f8a7c082510 con 0x7f8a8006eb00 2024-08-31T12:16:09.730 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:16:09.730 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":11,"btime":"2024-08-31T12:16:08:358605+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4371,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6833/4215834946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":4215834946},{"type":"v1","addr":"172.21.15.19:6833","nonce":4215834946}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/543344732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":543344732},{"type":"v1","addr":"172.21.15.19:6835","nonce":543344732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4380,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6837/699077015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6836","nonce":699077015},{"type":"v1","addr":"172.21.15.19:6837","nonce":699077015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":11,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-31T12:15:56.652936+0000","modified":"2024-08-31T12:16:08.358591+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4429,"mds_1":4426,"mds_2":4398},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4398":{"gid":4398,"name":"f","rank":2,"incarnation":11,"state":"up:creating","state_seq":1,"addr":"172.21.15.43:6837/2103490885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6835","nonce":2103490885},{"type":"v1","addr":"172.21.15.43:6837","nonce":2103490885}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4426":{"gid":4426,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.43:6839/1196858061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1196858061},{"type":"v1","addr":"172.21.15.43:6839","nonce":1196858061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4429":{"gid":4429,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.43:6838/3981040655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":3981040655},{"type":"v1","addr":"172.21.15.43:6838","nonce":3981040655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4429,"qdb_cluster":[4429,4426]},"id":1}]} 2024-08-31T12:16:09.730 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 11 2024-08-31T12:16:09.732 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.731+0000 7f8a87f00640 1 -- 172.21.15.19:0/1170773374 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8a64077a50 msgr2=0x7f8a64079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:09.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.731+0000 7f8a87f00640 1 --2- 172.21.15.19:0/1170773374 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8a64077a50 0x7f8a64079f60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8a801eec50 tx=0x7f8a74009040 comp rx=0 tx=0).stop 2024-08-31T12:16:09.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.731+0000 7f8a87f00640 1 -- 172.21.15.19:0/1170773374 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8a8006eb00 msgr2=0x7f8a801f1020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:09.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.731+0000 7f8a87f00640 1 --2- 172.21.15.19:0/1170773374 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8a8006eb00 0x7f8a801f1020 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c0099c0 tx=0x7f8a7c0039e0 comp rx=0 tx=0).stop 2024-08-31T12:16:09.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.731+0000 7f8a87f00640 1 -- 172.21.15.19:0/1170773374 shutdown_connections 2024-08-31T12:16:09.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.731+0000 7f8a87f00640 1 --2- 172.21.15.19:0/1170773374 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8a64077a50 0x7f8a64079f60 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:09.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.731+0000 7f8a87f00640 1 --2- 172.21.15.19:0/1170773374 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8a8015b0a0 0x7f8a80152820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:09.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.731+0000 7f8a87f00640 1 --2- 172.21.15.19:0/1170773374 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8a800cc820 0x7f8a8015e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:09.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.731+0000 7f8a87f00640 1 --2- 172.21.15.19:0/1170773374 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8a8006eb00 0x7f8a801f1020 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:09.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.731+0000 7f8a87f00640 1 -- 172.21.15.19:0/1170773374 >> 172.21.15.19:0/1170773374 conn(0x7f8a80069260 msgr2=0x7f8a80154c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:09.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.731+0000 7f8a87f00640 1 -- 172.21.15.19:0/1170773374 shutdown_connections 2024-08-31T12:16:09.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.731+0000 7f8a87f00640 1 -- 172.21.15.19:0/1170773374 wait complete. 2024-08-31T12:16:09.744 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-31T12:16:09.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.856+0000 7f3536f70640 1 -- 172.21.15.19:0/653581915 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f353006e0e0 msgr2=0x7f35300c9850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:09.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.856+0000 7f3536f70640 1 --2- 172.21.15.19:0/653581915 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f353006e0e0 0x7f35300c9850 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3518005fd0 tx=0x7f351802a670 comp rx=0 tx=0).stop 2024-08-31T12:16:09.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.856+0000 7f3536f70640 1 -- 172.21.15.19:0/653581915 shutdown_connections 2024-08-31T12:16:09.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.856+0000 7f3536f70640 1 --2- 172.21.15.19:0/653581915 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f353006e0e0 0x7f35300c9850 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:09.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.856+0000 7f3536f70640 1 -- 172.21.15.19:0/653581915 >> 172.21.15.19:0/653581915 conn(0x7f35300690d0 msgr2=0x7f35300d0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:09.858 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.856+0000 7f3536f70640 1 -- 172.21.15.19:0/653581915 shutdown_connections 2024-08-31T12:16:09.858 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.856+0000 7f3536f70640 1 -- 172.21.15.19:0/653581915 wait complete. 2024-08-31T12:16:09.858 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.857+0000 7f3536f70640 1 Processor -- start 2024-08-31T12:16:09.858 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.857+0000 7f3536f70640 1 -- start start 2024-08-31T12:16:09.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.859+0000 7f3536f70640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f353006e0e0 0x7f35301f2520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:09.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.859+0000 7f3536f70640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f35300c8a40 0x7f35300cf0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:09.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.859+0000 7f3536f70640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3530155300 0x7f35300c5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:09.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.859+0000 7f3536f70640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f353015b250 con 0x7f353006e0e0 2024-08-31T12:16:09.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.859+0000 7f3536f70640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f353015af70 con 0x7f3530155300 2024-08-31T12:16:09.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.859+0000 7f3536f70640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f35300c8610 con 0x7f35300c8a40 2024-08-31T12:16:09.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.859+0000 7f3534d0c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f353006e0e0 0x7f35301f2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:09.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.859+0000 7f3534d0c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f353006e0e0 0x7f35301f2520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:44894/0 (socket says 172.21.15.19:44894) 2024-08-31T12:16:09.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.859+0000 7f3534d0c640 1 -- 172.21.15.19:0/2901531982 learned_addr learned my addr 172.21.15.19:0/2901531982 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:09.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.859+0000 7f353550d640 1 --2- 172.21.15.19:0/2901531982 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3530155300 0x7f35300c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:09.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.859+0000 7f352ffff640 1 --2- 172.21.15.19:0/2901531982 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f35300c8a40 0x7f35300cf0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:09.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.859+0000 7f3534d0c640 1 -- 172.21.15.19:0/2901531982 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f35300c8a40 msgr2=0x7f35300cf0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:09.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.859+0000 7f3534d0c640 1 --2- 172.21.15.19:0/2901531982 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f35300c8a40 0x7f35300cf0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:09.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.859+0000 7f3534d0c640 1 -- 172.21.15.19:0/2901531982 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3530155300 msgr2=0x7f35300c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:09.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.859+0000 7f3534d0c640 1 --2- 172.21.15.19:0/2901531982 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3530155300 0x7f35300c5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:09.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.859+0000 7f3534d0c640 1 -- 172.21.15.19:0/2901531982 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35300d0250 con 0x7f353006e0e0 2024-08-31T12:16:09.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.859+0000 7f3534d0c640 1 --2- 172.21.15.19:0/2901531982 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f353006e0e0 0x7f35301f2520 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f352000ba50 tx=0x7f352000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:09.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.860+0000 7f352d7fa640 1 -- 172.21.15.19:0/2901531982 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3520019070 con 0x7f353006e0e0 2024-08-31T12:16:09.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.860+0000 7f352d7fa640 1 -- 172.21.15.19:0/2901531982 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f352000cd10 con 0x7f353006e0e0 2024-08-31T12:16:09.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.860+0000 7f3536f70640 1 -- 172.21.15.19:0/2901531982 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35300d0530 con 0x7f353006e0e0 2024-08-31T12:16:09.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.860+0000 7f352d7fa640 1 -- 172.21.15.19:0/2901531982 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35200146a0 con 0x7f353006e0e0 2024-08-31T12:16:09.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.860+0000 7f3536f70640 1 -- 172.21.15.19:0/2901531982 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35300ce8f0 con 0x7f353006e0e0 2024-08-31T12:16:09.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.861+0000 7f3536f70640 1 -- 172.21.15.19:0/2901531982 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35300cbe10 con 0x7f353006e0e0 2024-08-31T12:16:09.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.861+0000 7f352d7fa640 1 -- 172.21.15.19:0/2901531982 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f352001c070 con 0x7f353006e0e0 2024-08-31T12:16:09.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.862+0000 7f352d7fa640 1 --2- 172.21.15.19:0/2901531982 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3510077a50 0x7f3510079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:09.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.862+0000 7f352d7fa640 1 -- 172.21.15.19:0/2901531982 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3520099320 con 0x7f353006e0e0 2024-08-31T12:16:09.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.862+0000 7f352ffff640 1 --2- 172.21.15.19:0/2901531982 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3510077a50 0x7f3510079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:09.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.863+0000 7f352ffff640 1 --2- 172.21.15.19:0/2901531982 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3510077a50 0x7f3510079f60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3518033000 tx=0x7f3518004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:09.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:09.867+0000 7f352d7fa640 1 -- 172.21.15.19:0/2901531982 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f35300cbe10 con 0x7f353006e0e0 2024-08-31T12:16:10.054 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.053+0000 7f3536f70640 1 -- 172.21.15.19:0/2901531982 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f35300c9850 con 0x7f353006e0e0 2024-08-31T12:16:10.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.055+0000 7f352d7fa640 1 -- 172.21.15.19:0/2901531982 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16617 (secure 0 0 0) 0x7f3520061a40 con 0x7f353006e0e0 2024-08-31T12:16:10.056 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:16:10.056 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":21,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","created":"2024-08-31T12:15:32.084309+0000","modified":"2024-08-31T12:15:58.292413+0000","last_up_change":"2024-08-31T12:15:45.213688+0000","last_in_change":"2024-08-31T12:15:35.545202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-31T12:15:44.508197+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-31T12:15:54.512584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-31T12:15:55.618251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b80d6a3f-7e4e-4d4d-b1b4-0df71c9176ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6800","nonce":680447146},{"type":"v1","addr":"172.21.15.19:6801","nonce":680447146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":680447146},{"type":"v1","addr":"172.21.15.19:6803","nonce":680447146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":680447146},{"type":"v1","addr":"172.21.15.19:6807","nonce":680447146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":680447146},{"type":"v1","addr":"172.21.15.19:6805","nonce":680447146}]},"public_addr":"172.21.15.19:6801/680447146","cluster_addr":"172.21.15.19:6803/680447146","heartbeat_back_addr":"172.21.15.19:6807/680447146","heartbeat_front_addr":"172.21.15.19:6805/680447146","state":["exists","up"]},{"osd":1,"uuid":"6f986366-82a6-4133-9f2e-43f84c7b03bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":1522610034},{"type":"v1","addr":"172.21.15.19:6817","nonce":1522610034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":1522610034},{"type":"v1","addr":"172.21.15.19:6819","nonce":1522610034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":1522610034},{"type":"v1","addr":"172.21.15.19:6823","nonce":1522610034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":1522610034},{"type":"v1","addr":"172.21.15.19:6821","nonce":1522610034}]},"public_addr":"172.21.15.19:6817/1522610034","cluster_addr":"172.21.15.19:6819/1522610034","heartbeat_back_addr":"172.21.15.19:6823/1522610034","heartbeat_front_addr":"172.21.15.19:6821/1522610034","state":["exists","up"]},{"osd":2,"uuid":"ce8c2068-845c-4cbb-a4d1-077375d605ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6809","nonce":2332368824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6811","nonce":2332368824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6815","nonce":2332368824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6813","nonce":2332368824}]},"public_addr":"172.21.15.19:6809/2332368824","cluster_addr":"172.21.15.19:6811/2332368824","heartbeat_back_addr":"172.21.15.19:6815/2332368824","heartbeat_front_addr":"172.21.15.19:6813/2332368824","state":["exists","up"]},{"osd":3,"uuid":"b1cb0ef4-aa9b-421b-a94e-82de0c46e5cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":3785550598},{"type":"v1","addr":"172.21.15.19:6825","nonce":3785550598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":3785550598},{"type":"v1","addr":"172.21.15.19:6827","nonce":3785550598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":3785550598},{"type":"v1","addr":"172.21.15.19:6831","nonce":3785550598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":3785550598},{"type":"v1","addr":"172.21.15.19:6829","nonce":3785550598}]},"public_addr":"172.21.15.19:6825/3785550598","cluster_addr":"172.21.15.19:6827/3785550598","heartbeat_back_addr":"172.21.15.19:6831/3785550598","heartbeat_front_addr":"172.21.15.19:6829/3785550598","state":["exists","up"]},{"osd":4,"uuid":"825b58ec-a6da-4515-b6a5-81debe327011","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":1985519349},{"type":"v1","addr":"172.21.15.43:6819","nonce":1985519349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":1985519349},{"type":"v1","addr":"172.21.15.43:6821","nonce":1985519349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":1985519349},{"type":"v1","addr":"172.21.15.43:6825","nonce":1985519349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":1985519349},{"type":"v1","addr":"172.21.15.43:6823","nonce":1985519349}]},"public_addr":"172.21.15.43:6819/1985519349","cluster_addr":"172.21.15.43:6821/1985519349","heartbeat_back_addr":"172.21.15.43:6825/1985519349","heartbeat_front_addr":"172.21.15.43:6823/1985519349","state":["exists","up"]},{"osd":5,"uuid":"7f34e537-5997-40e4-92e7-2570507160c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":4271971521},{"type":"v1","addr":"172.21.15.43:6803","nonce":4271971521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":4271971521},{"type":"v1","addr":"172.21.15.43:6805","nonce":4271971521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":4271971521},{"type":"v1","addr":"172.21.15.43:6809","nonce":4271971521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":4271971521},{"type":"v1","addr":"172.21.15.43:6807","nonce":4271971521}]},"public_addr":"172.21.15.43:6803/4271971521","cluster_addr":"172.21.15.43:6805/4271971521","heartbeat_back_addr":"172.21.15.43:6809/4271971521","heartbeat_front_addr":"172.21.15.43:6807/4271971521","state":["exists","up"]},{"osd":6,"uuid":"195dd8ad-640a-4fa3-9ee4-06e4172e0464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2863384694},{"type":"v1","addr":"172.21.15.43:6827","nonce":2863384694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2863384694},{"type":"v1","addr":"172.21.15.43:6829","nonce":2863384694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":2863384694},{"type":"v1","addr":"172.21.15.43:6833","nonce":2863384694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2863384694},{"type":"v1","addr":"172.21.15.43:6831","nonce":2863384694}]},"public_addr":"172.21.15.43:6827/2863384694","cluster_addr":"172.21.15.43:6829/2863384694","heartbeat_back_addr":"172.21.15.43:6833/2863384694","heartbeat_front_addr":"172.21.15.43:6831/2863384694","state":["exists","up"]},{"osd":7,"uuid":"4645661f-f430-4f05-bdd1-69138dcd1bbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":792572234},{"type":"v1","addr":"172.21.15.43:6811","nonce":792572234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":792572234},{"type":"v1","addr":"172.21.15.43:6813","nonce":792572234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":792572234},{"type":"v1","addr":"172.21.15.43:6817","nonce":792572234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":792572234},{"type":"v1","addr":"172.21.15.43:6815","nonce":792572234}]},"public_addr":"172.21.15.43:6811/792572234","cluster_addr":"172.21.15.43:6813/792572234","heartbeat_back_addr":"172.21.15.43:6817/792572234","heartbeat_front_addr":"172.21.15.43:6815/792572234","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.156485+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.578126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.075440+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.639867+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.543602+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:41.919592+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.615557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:41.813337+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-31T12:16:10.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.057+0000 7f3536f70640 1 -- 172.21.15.19:0/2901531982 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3510077a50 msgr2=0x7f3510079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:10.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.057+0000 7f3536f70640 1 --2- 172.21.15.19:0/2901531982 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3510077a50 0x7f3510079f60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3518033000 tx=0x7f3518004730 comp rx=0 tx=0).stop 2024-08-31T12:16:10.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.057+0000 7f3536f70640 1 -- 172.21.15.19:0/2901531982 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f353006e0e0 msgr2=0x7f35301f2520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:10.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.057+0000 7f3536f70640 1 --2- 172.21.15.19:0/2901531982 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f353006e0e0 0x7f35301f2520 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f352000ba50 tx=0x7f352000bf10 comp rx=0 tx=0).stop 2024-08-31T12:16:10.059 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.057+0000 7f3536f70640 1 -- 172.21.15.19:0/2901531982 shutdown_connections 2024-08-31T12:16:10.059 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.057+0000 7f3536f70640 1 --2- 172.21.15.19:0/2901531982 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3510077a50 0x7f3510079f60 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:10.059 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.057+0000 7f3536f70640 1 --2- 172.21.15.19:0/2901531982 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3530155300 0x7f35300c5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:10.059 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.057+0000 7f3536f70640 1 --2- 172.21.15.19:0/2901531982 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f35300c8a40 0x7f35300cf0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:10.059 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.057+0000 7f3536f70640 1 --2- 172.21.15.19:0/2901531982 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f353006e0e0 0x7f35301f2520 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:10.059 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.057+0000 7f3536f70640 1 -- 172.21.15.19:0/2901531982 >> 172.21.15.19:0/2901531982 conn(0x7f35300690d0 msgr2=0x7f3530158de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:10.059 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.057+0000 7f3536f70640 1 -- 172.21.15.19:0/2901531982 shutdown_connections 2024-08-31T12:16:10.059 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.057+0000 7f3536f70640 1 -- 172.21.15.19:0/2901531982 wait complete. 2024-08-31T12:16:10.070 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-31T12:16:10.184 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.182+0000 7f838a8a9640 1 -- 172.21.15.19:0/1918416042 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83840c9850 msgr2=0x7f8384155300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:10.184 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.182+0000 7f838a8a9640 1 --2- 172.21.15.19:0/1918416042 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83840c9850 0x7f8384155300 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f837c005fd0 tx=0x7f837c02abc0 comp rx=0 tx=0).stop 2024-08-31T12:16:10.184 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.182+0000 7f838a8a9640 1 -- 172.21.15.19:0/1918416042 shutdown_connections 2024-08-31T12:16:10.184 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.182+0000 7f838a8a9640 1 --2- 172.21.15.19:0/1918416042 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83840c9850 0x7f8384155300 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:10.184 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.182+0000 7f838a8a9640 1 -- 172.21.15.19:0/1918416042 >> 172.21.15.19:0/1918416042 conn(0x7f83840690d0 msgr2=0x7f83840d0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:10.184 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.183+0000 7f838a8a9640 1 -- 172.21.15.19:0/1918416042 shutdown_connections 2024-08-31T12:16:10.184 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.183+0000 7f838a8a9640 1 -- 172.21.15.19:0/1918416042 wait complete. 2024-08-31T12:16:10.185 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.184+0000 7f838a8a9640 1 Processor -- start 2024-08-31T12:16:10.185 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.185+0000 7f838a8a9640 1 -- start start 2024-08-31T12:16:10.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.186+0000 7f838a8a9640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f838406e0e0 0x7f8384156d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:10.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.186+0000 7f838a8a9640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83840c8a40 0x7f83841f1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:10.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.186+0000 7f838a8a9640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f83840c9850 0x7f8384155a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:10.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.186+0000 7f838a8a9640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f838415b250 con 0x7f838406e0e0 2024-08-31T12:16:10.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.186+0000 7f838a8a9640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f838415af70 con 0x7f83840c8a40 2024-08-31T12:16:10.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.186+0000 7f838a8a9640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f83840c8610 con 0x7f83840c9850 2024-08-31T12:16:10.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.186+0000 7f8383fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f838406e0e0 0x7f8384156d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:10.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.186+0000 7f8383fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f838406e0e0 0x7f8384156d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:44916/0 (socket says 172.21.15.19:44916) 2024-08-31T12:16:10.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.186+0000 7f8383fff640 1 -- 172.21.15.19:0/398252060 learned_addr learned my addr 172.21.15.19:0/398252060 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:10.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.186+0000 7f83837fe640 1 --2- 172.21.15.19:0/398252060 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83840c8a40 0x7f83841f1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:10.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.186+0000 7f8388e46640 1 --2- 172.21.15.19:0/398252060 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f83840c9850 0x7f8384155a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:10.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.186+0000 7f8383fff640 1 -- 172.21.15.19:0/398252060 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f83840c9850 msgr2=0x7f8384155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:10.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.186+0000 7f8383fff640 1 --2- 172.21.15.19:0/398252060 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f83840c9850 0x7f8384155a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:10.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.186+0000 7f8383fff640 1 -- 172.21.15.19:0/398252060 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83840c8a40 msgr2=0x7f83841f1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:10.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.186+0000 7f8383fff640 1 --2- 172.21.15.19:0/398252060 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83840c8a40 0x7f83841f1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:10.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.186+0000 7f8383fff640 1 -- 172.21.15.19:0/398252060 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83841f22c0 con 0x7f838406e0e0 2024-08-31T12:16:10.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.186+0000 7f8383fff640 1 --2- 172.21.15.19:0/398252060 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f838406e0e0 0x7f8384156d30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f837400c430 tx=0x7f837400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:10.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.187+0000 7f8380ff9640 1 -- 172.21.15.19:0/398252060 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8374017070 con 0x7f838406e0e0 2024-08-31T12:16:10.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.187+0000 7f8380ff9640 1 -- 172.21.15.19:0/398252060 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f83740053c0 con 0x7f838406e0e0 2024-08-31T12:16:10.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.187+0000 7f838a8a9640 1 -- 172.21.15.19:0/398252060 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8384157940 con 0x7f838406e0e0 2024-08-31T12:16:10.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.187+0000 7f8380ff9640 1 -- 172.21.15.19:0/398252060 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8374003850 con 0x7f838406e0e0 2024-08-31T12:16:10.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.187+0000 7f838a8a9640 1 -- 172.21.15.19:0/398252060 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8384157f00 con 0x7f838406e0e0 2024-08-31T12:16:10.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.188+0000 7f838a8a9640 1 -- 172.21.15.19:0/398252060 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f838415ac20 con 0x7f838406e0e0 2024-08-31T12:16:10.191 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.188+0000 7f8380ff9640 1 -- 172.21.15.19:0/398252060 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f837401a070 con 0x7f838406e0e0 2024-08-31T12:16:10.191 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.189+0000 7f8380ff9640 1 --2- 172.21.15.19:0/398252060 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8370077a50 0x7f8370079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:10.191 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.189+0000 7f8380ff9640 1 -- 172.21.15.19:0/398252060 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f8374098940 con 0x7f838406e0e0 2024-08-31T12:16:10.191 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.189+0000 7f83837fe640 1 --2- 172.21.15.19:0/398252060 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8370077a50 0x7f8370079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:10.191 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.190+0000 7f83837fe640 1 --2- 172.21.15.19:0/398252060 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8370077a50 0x7f8370079f60 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f83840c37b0 tx=0x7f8378009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:10.194 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.193+0000 7f8380ff9640 1 -- 172.21.15.19:0/398252060 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f838415ac20 con 0x7f838406e0e0 2024-08-31T12:16:10.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.421+0000 7f838a8a9640 1 -- 172.21.15.19:0/398252060 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f838406de30 con 0x7f838406e0e0 2024-08-31T12:16:10.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.422+0000 7f8380ff9640 1 -- 172.21.15.19:0/398252060 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+6898 (secure 0 0 0) 0x7f8374061060 con 0x7f838406e0e0 2024-08-31T12:16:10.423 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:16:10.423 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":11,"btime":"2024-08-31T12:16:08:358605+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4371,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6833/4215834946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":4215834946},{"type":"v1","addr":"172.21.15.19:6833","nonce":4215834946}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/543344732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":543344732},{"type":"v1","addr":"172.21.15.19:6835","nonce":543344732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4380,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6837/699077015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6836","nonce":699077015},{"type":"v1","addr":"172.21.15.19:6837","nonce":699077015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":11,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-31T12:15:56.652936+0000","modified":"2024-08-31T12:16:08.358591+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4429,"mds_1":4426,"mds_2":4398},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4398":{"gid":4398,"name":"f","rank":2,"incarnation":11,"state":"up:creating","state_seq":1,"addr":"172.21.15.43:6837/2103490885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6835","nonce":2103490885},{"type":"v1","addr":"172.21.15.43:6837","nonce":2103490885}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4426":{"gid":4426,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.43:6839/1196858061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1196858061},{"type":"v1","addr":"172.21.15.43:6839","nonce":1196858061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4429":{"gid":4429,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.43:6838/3981040655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":3981040655},{"type":"v1","addr":"172.21.15.43:6838","nonce":3981040655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4429,"qdb_cluster":[4429,4426]},"id":1}]} 2024-08-31T12:16:10.423 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 11 2024-08-31T12:16:10.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.424+0000 7f838a8a9640 1 -- 172.21.15.19:0/398252060 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8370077a50 msgr2=0x7f8370079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:10.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.424+0000 7f838a8a9640 1 --2- 172.21.15.19:0/398252060 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8370077a50 0x7f8370079f60 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f83840c37b0 tx=0x7f8378009040 comp rx=0 tx=0).stop 2024-08-31T12:16:10.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.424+0000 7f838a8a9640 1 -- 172.21.15.19:0/398252060 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f838406e0e0 msgr2=0x7f8384156d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:10.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.424+0000 7f838a8a9640 1 --2- 172.21.15.19:0/398252060 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f838406e0e0 0x7f8384156d30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f837400c430 tx=0x7f837400c8f0 comp rx=0 tx=0).stop 2024-08-31T12:16:10.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.424+0000 7f838a8a9640 1 -- 172.21.15.19:0/398252060 shutdown_connections 2024-08-31T12:16:10.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.424+0000 7f838a8a9640 1 --2- 172.21.15.19:0/398252060 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8370077a50 0x7f8370079f60 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:10.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.424+0000 7f838a8a9640 1 --2- 172.21.15.19:0/398252060 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f83840c9850 0x7f8384155a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:10.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.424+0000 7f838a8a9640 1 --2- 172.21.15.19:0/398252060 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83840c8a40 0x7f83841f1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:10.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.424+0000 7f838a8a9640 1 --2- 172.21.15.19:0/398252060 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f838406e0e0 0x7f8384156d30 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:10.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.424+0000 7f838a8a9640 1 -- 172.21.15.19:0/398252060 >> 172.21.15.19:0/398252060 conn(0x7f83840690d0 msgr2=0x7f83840cba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:10.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.424+0000 7f838a8a9640 1 -- 172.21.15.19:0/398252060 shutdown_connections 2024-08-31T12:16:10.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:10.424+0000 7f838a8a9640 1 -- 172.21.15.19:0/398252060 wait complete. 2024-08-31T12:16:11.041 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 11, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-31T12:15:56.652936+0000', 'modified': '2024-08-31T12:16:08.358591+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 4429, 'mds_1': 4426, 'mds_2': 4398}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4398': {'gid': 4398, 'name': 'f', 'rank': 2, 'incarnation': 11, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.43:6837/2103490885', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6835', 'nonce': 2103490885}, {'type': 'v1', 'addr': '172.21.15.43:6837', 'nonce': 2103490885}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4426': {'gid': 4426, 'name': 'b', 'rank': 1, 'incarnation': 9, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.43:6839/1196858061', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6836', 'nonce': 1196858061}, {'type': 'v1', 'addr': '172.21.15.43:6839', 'nonce': 1196858061}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4429': {'gid': 4429, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.43:6838/3981040655', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6834', 'nonce': 3981040655}, {'type': 'v1', 'addr': '172.21.15.43:6838', 'nonce': 3981040655}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4429, 'qdb_cluster': [4429, 4426]} 2024-08-31T12:16:11.041 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4398:up:creating 2024-08-31T12:16:12.042 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-31T12:16:12.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.155+0000 7fd2fed62640 1 -- 172.21.15.19:0/189420665 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd2f80ce700 msgr2=0x7fd2f80ceb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:12.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.155+0000 7fd2fed62640 1 --2- 172.21.15.19:0/189420665 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd2f80ce700 0x7fd2f80ceb10 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd2e0005fd0 tx=0x7fd2e002a540 comp rx=0 tx=0).stop 2024-08-31T12:16:12.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.155+0000 7fd2fed62640 1 -- 172.21.15.19:0/189420665 shutdown_connections 2024-08-31T12:16:12.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.155+0000 7fd2fed62640 1 --2- 172.21.15.19:0/189420665 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd2f80ce700 0x7fd2f80ceb10 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:12.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.155+0000 7fd2fed62640 1 --2- 172.21.15.19:0/189420665 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd2f80cbce0 0x7fd2f8159960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:12.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.155+0000 7fd2fed62640 1 -- 172.21.15.19:0/189420665 >> 172.21.15.19:0/189420665 conn(0x7fd2f8068aa0 msgr2=0x7fd2f8068eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:12.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.155+0000 7fd2fed62640 1 -- 172.21.15.19:0/189420665 shutdown_connections 2024-08-31T12:16:12.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.155+0000 7fd2fed62640 1 -- 172.21.15.19:0/189420665 wait complete. 2024-08-31T12:16:12.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.157+0000 7fd2fed62640 1 Processor -- start 2024-08-31T12:16:12.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.157+0000 7fd2fed62640 1 -- start start 2024-08-31T12:16:12.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.158+0000 7fd2fed62640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd2f80cbce0 0x7fd2f815bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:12.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.158+0000 7fd2fed62640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd2f80ce700 0x7fd2f81f2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:12.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.158+0000 7fd2fed62640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd2f8152fd0 0x7fd2f815a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:12.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.158+0000 7fd2fed62640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f815cce0 con 0x7fd2f80ce700 2024-08-31T12:16:12.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.158+0000 7fd2fed62640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f80cf050 con 0x7fd2f8152fd0 2024-08-31T12:16:12.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.158+0000 7fd2fed62640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fd2f80c5790 con 0x7fd2f80cbce0 2024-08-31T12:16:12.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.158+0000 7fd2f7fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd2f80ce700 0x7fd2f81f2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:12.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.158+0000 7fd2f7fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd2f80ce700 0x7fd2f81f2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:44930/0 (socket says 172.21.15.19:44930) 2024-08-31T12:16:12.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.158+0000 7fd2f7fff640 1 -- 172.21.15.19:0/2993438693 learned_addr learned my addr 172.21.15.19:0/2993438693 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:12.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.158+0000 7fd2fd2ff640 1 --2- 172.21.15.19:0/2993438693 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd2f8152fd0 0x7fd2f815a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:12.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.158+0000 7fd2fcafe640 1 --2- 172.21.15.19:0/2993438693 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd2f80cbce0 0x7fd2f815bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:12.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.159+0000 7fd2f7fff640 1 -- 172.21.15.19:0/2993438693 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd2f80cbce0 msgr2=0x7fd2f815bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:12.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.159+0000 7fd2f7fff640 1 --2- 172.21.15.19:0/2993438693 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd2f80cbce0 0x7fd2f815bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:12.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.159+0000 7fd2f7fff640 1 -- 172.21.15.19:0/2993438693 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd2f8152fd0 msgr2=0x7fd2f815a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:12.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.159+0000 7fd2f7fff640 1 --2- 172.21.15.19:0/2993438693 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd2f8152fd0 0x7fd2f815a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:12.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.159+0000 7fd2f7fff640 1 -- 172.21.15.19:0/2993438693 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2f814eb90 con 0x7fd2f80ce700 2024-08-31T12:16:12.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.159+0000 7fd2fd2ff640 1 --2- 172.21.15.19:0/2993438693 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd2f8152fd0 0x7fd2f815a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:16:12.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.159+0000 7fd2fcafe640 1 --2- 172.21.15.19:0/2993438693 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd2f80cbce0 0x7fd2f815bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:16:12.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.159+0000 7fd2f7fff640 1 --2- 172.21.15.19:0/2993438693 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd2f80ce700 0x7fd2f81f2d40 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd2e800c730 tx=0x7fd2e800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:12.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.159+0000 7fd2f57fa640 1 -- 172.21.15.19:0/2993438693 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e8003b70 con 0x7fd2f80ce700 2024-08-31T12:16:12.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.159+0000 7fd2f57fa640 1 -- 172.21.15.19:0/2993438693 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd2e8003d30 con 0x7fd2f80ce700 2024-08-31T12:16:12.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.159+0000 7fd2fed62640 1 -- 172.21.15.19:0/2993438693 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2f814e0b0 con 0x7fd2f80ce700 2024-08-31T12:16:12.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.159+0000 7fd2f57fa640 1 -- 172.21.15.19:0/2993438693 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e800aab0 con 0x7fd2f80ce700 2024-08-31T12:16:12.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.159+0000 7fd2fed62640 1 -- 172.21.15.19:0/2993438693 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2f81e22a0 con 0x7fd2f80ce700 2024-08-31T12:16:12.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.160+0000 7fd2fed62640 1 -- 172.21.15.19:0/2993438693 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2f80c9290 con 0x7fd2f80ce700 2024-08-31T12:16:12.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.161+0000 7fd2f57fa640 1 -- 172.21.15.19:0/2993438693 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fd2e800ac50 con 0x7fd2f80ce700 2024-08-31T12:16:12.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.161+0000 7fd2f57fa640 1 --2- 172.21.15.19:0/2993438693 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd2d8077a80 0x7fd2d8079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:12.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.161+0000 7fd2f57fa640 1 -- 172.21.15.19:0/2993438693 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd2e80994b0 con 0x7fd2f80ce700 2024-08-31T12:16:12.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.162+0000 7fd2fcafe640 1 --2- 172.21.15.19:0/2993438693 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd2d8077a80 0x7fd2d8079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:12.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.162+0000 7fd2fcafe640 1 --2- 172.21.15.19:0/2993438693 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd2d8077a80 0x7fd2d8079f90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0005fd0 tx=0x7fd2f000f360 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:12.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.166+0000 7fd2f57fa640 1 -- 172.21.15.19:0/2993438693 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fd2e8062190 con 0x7fd2f80ce700 2024-08-31T12:16:12.212 INFO:tasks.ceph.mds.d.smithi043.stderr:2024-08-31T12:16:12.211+0000 7ff036f5a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-08-31T12:16:12.212 INFO:tasks.ceph.mds.d.smithi043.stderr:2024-08-31T12:16:12.211+0000 7ff036f5a640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-08-31T12:16:12.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.386+0000 7fd2fed62640 1 -- 172.21.15.19:0/2993438693 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd2f80c9c10 con 0x7fd2f80ce700 2024-08-31T12:16:12.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.387+0000 7fd2f57fa640 1 -- 172.21.15.19:0/2993438693 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+6898 (secure 0 0 0) 0x7fd2e8061a90 con 0x7fd2f80ce700 2024-08-31T12:16:12.388 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:16:12.388 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":11,"btime":"2024-08-31T12:16:08:358605+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4371,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6833/4215834946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":4215834946},{"type":"v1","addr":"172.21.15.19:6833","nonce":4215834946}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/543344732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":543344732},{"type":"v1","addr":"172.21.15.19:6835","nonce":543344732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4380,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6837/699077015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6836","nonce":699077015},{"type":"v1","addr":"172.21.15.19:6837","nonce":699077015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":11,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-31T12:15:56.652936+0000","modified":"2024-08-31T12:16:08.358591+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4429,"mds_1":4426,"mds_2":4398},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4398":{"gid":4398,"name":"f","rank":2,"incarnation":11,"state":"up:creating","state_seq":1,"addr":"172.21.15.43:6837/2103490885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6835","nonce":2103490885},{"type":"v1","addr":"172.21.15.43:6837","nonce":2103490885}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4426":{"gid":4426,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.43:6839/1196858061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1196858061},{"type":"v1","addr":"172.21.15.43:6839","nonce":1196858061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4429":{"gid":4429,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.43:6838/3981040655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":3981040655},{"type":"v1","addr":"172.21.15.43:6838","nonce":3981040655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4429,"qdb_cluster":[4429,4426]},"id":1}]} 2024-08-31T12:16:12.388 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 11 2024-08-31T12:16:12.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.389+0000 7fd2fed62640 1 -- 172.21.15.19:0/2993438693 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd2d8077a80 msgr2=0x7fd2d8079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:12.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.389+0000 7fd2fed62640 1 --2- 172.21.15.19:0/2993438693 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd2d8077a80 0x7fd2d8079f90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0005fd0 tx=0x7fd2f000f360 comp rx=0 tx=0).stop 2024-08-31T12:16:12.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.389+0000 7fd2fed62640 1 -- 172.21.15.19:0/2993438693 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd2f80ce700 msgr2=0x7fd2f81f2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:12.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.389+0000 7fd2fed62640 1 --2- 172.21.15.19:0/2993438693 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd2f80ce700 0x7fd2f81f2d40 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd2e800c730 tx=0x7fd2e800cbf0 comp rx=0 tx=0).stop 2024-08-31T12:16:12.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.389+0000 7fd2fed62640 1 -- 172.21.15.19:0/2993438693 shutdown_connections 2024-08-31T12:16:12.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.389+0000 7fd2fed62640 1 --2- 172.21.15.19:0/2993438693 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd2d8077a80 0x7fd2d8079f90 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:12.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.389+0000 7fd2fed62640 1 --2- 172.21.15.19:0/2993438693 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd2f8152fd0 0x7fd2f815a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:12.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.389+0000 7fd2fed62640 1 --2- 172.21.15.19:0/2993438693 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd2f80ce700 0x7fd2f81f2d40 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:12.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.389+0000 7fd2fed62640 1 --2- 172.21.15.19:0/2993438693 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd2f80cbce0 0x7fd2f815bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:12.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.389+0000 7fd2fed62640 1 -- 172.21.15.19:0/2993438693 >> 172.21.15.19:0/2993438693 conn(0x7fd2f8068aa0 msgr2=0x7fd2f8149330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:12.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.389+0000 7fd2fed62640 1 -- 172.21.15.19:0/2993438693 shutdown_connections 2024-08-31T12:16:12.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:12.389+0000 7fd2fed62640 1 -- 172.21.15.19:0/2993438693 wait complete. 2024-08-31T12:16:13.006 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 11, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-31T12:15:56.652936+0000', 'modified': '2024-08-31T12:16:08.358591+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 4429, 'mds_1': 4426, 'mds_2': 4398}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4398': {'gid': 4398, 'name': 'f', 'rank': 2, 'incarnation': 11, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.43:6837/2103490885', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6835', 'nonce': 2103490885}, {'type': 'v1', 'addr': '172.21.15.43:6837', 'nonce': 2103490885}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4426': {'gid': 4426, 'name': 'b', 'rank': 1, 'incarnation': 9, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.43:6839/1196858061', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6836', 'nonce': 1196858061}, {'type': 'v1', 'addr': '172.21.15.43:6839', 'nonce': 1196858061}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4429': {'gid': 4429, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.43:6838/3981040655', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6834', 'nonce': 3981040655}, {'type': 'v1', 'addr': '172.21.15.43:6838', 'nonce': 3981040655}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4429, 'qdb_cluster': [4429, 4426]} 2024-08-31T12:16:13.006 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4398:up:creating 2024-08-31T12:16:14.007 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-31T12:16:14.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.135+0000 7f7d8eade640 1 -- 172.21.15.19:0/3282048091 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7d88155300 msgr2=0x7f7d8806d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:14.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.135+0000 7f7d8eade640 1 --2- 172.21.15.19:0/3282048091 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7d88155300 0x7f7d8806d000 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7d8815b0f0 tx=0x7f7d8002ab40 comp rx=0 tx=0).stop 2024-08-31T12:16:14.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.135+0000 7f7d8eade640 1 -- 172.21.15.19:0/3282048091 shutdown_connections 2024-08-31T12:16:14.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.135+0000 7f7d8eade640 1 --2- 172.21.15.19:0/3282048091 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7d88155300 0x7f7d8806d000 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:14.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.135+0000 7f7d8eade640 1 --2- 172.21.15.19:0/3282048091 >> v2:172.21.15.43:3300/0 conn(0x7f7d880c8a40 0x7f7d880c9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:14.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.135+0000 7f7d8eade640 1 -- 172.21.15.19:0/3282048091 >> 172.21.15.19:0/3282048091 conn(0x7f7d880690d0 msgr2=0x7f7d880d0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:14.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.135+0000 7f7d8eade640 1 -- 172.21.15.19:0/3282048091 shutdown_connections 2024-08-31T12:16:14.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.135+0000 7f7d8eade640 1 -- 172.21.15.19:0/3282048091 wait complete. 2024-08-31T12:16:14.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.136+0000 7f7d8eade640 1 Processor -- start 2024-08-31T12:16:14.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.136+0000 7f7d8eade640 1 -- start start 2024-08-31T12:16:14.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.138+0000 7f7d8eade640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7d8806e0e0 0x7f7d881f15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:14.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.138+0000 7f7d8eade640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7d880c8a40 0x7f7d880cf960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:14.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.138+0000 7f7d8eade640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d88155300 0x7f7d8814c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:14.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.138+0000 7f7d8eade640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f7d880c8330 con 0x7f7d8806e0e0 2024-08-31T12:16:14.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.138+0000 7f7d8eade640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7d8815b250 con 0x7f7d88155300 2024-08-31T12:16:14.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.138+0000 7f7d8eade640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f7d880c8610 con 0x7f7d880c8a40 2024-08-31T12:16:14.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.138+0000 7f7d8c87a640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7d8806e0e0 0x7f7d881f15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:14.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.138+0000 7f7d8c87a640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7d8806e0e0 0x7f7d881f15e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:44944/0 (socket says 172.21.15.19:44944) 2024-08-31T12:16:14.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.138+0000 7f7d8c87a640 1 -- 172.21.15.19:0/4279575788 learned_addr learned my addr 172.21.15.19:0/4279575788 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:14.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.138+0000 7f7d87fff640 1 --2- 172.21.15.19:0/4279575788 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7d880c8a40 0x7f7d880cf960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:14.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.138+0000 7f7d8d07b640 1 --2- 172.21.15.19:0/4279575788 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d88155300 0x7f7d8814c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:14.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.138+0000 7f7d8c87a640 1 -- 172.21.15.19:0/4279575788 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7d880c8a40 msgr2=0x7f7d880cf960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:14.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.138+0000 7f7d8c87a640 1 --2- 172.21.15.19:0/4279575788 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7d880c8a40 0x7f7d880cf960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:14.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.138+0000 7f7d8c87a640 1 -- 172.21.15.19:0/4279575788 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d88155300 msgr2=0x7f7d8814c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:14.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.138+0000 7f7d8c87a640 1 --2- 172.21.15.19:0/4279575788 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d88155300 0x7f7d8814c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:14.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.138+0000 7f7d8c87a640 1 -- 172.21.15.19:0/4279575788 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d880d0950 con 0x7f7d8806e0e0 2024-08-31T12:16:14.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.139+0000 7f7d8c87a640 1 --2- 172.21.15.19:0/4279575788 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7d8806e0e0 0x7f7d881f15e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7d7800d740 tx=0x7f7d7800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:14.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.139+0000 7f7d857fa640 1 -- 172.21.15.19:0/4279575788 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d78017070 con 0x7f7d8806e0e0 2024-08-31T12:16:14.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.139+0000 7f7d857fa640 1 -- 172.21.15.19:0/4279575788 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7d78009d20 con 0x7f7d8806e0e0 2024-08-31T12:16:14.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.139+0000 7f7d8eade640 1 -- 172.21.15.19:0/4279575788 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d880c7160 con 0x7f7d8806e0e0 2024-08-31T12:16:14.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.139+0000 7f7d857fa640 1 -- 172.21.15.19:0/4279575788 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d780106c0 con 0x7f7d8806e0e0 2024-08-31T12:16:14.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.139+0000 7f7d8eade640 1 -- 172.21.15.19:0/4279575788 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d8814abb0 con 0x7f7d8806e0e0 2024-08-31T12:16:14.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.140+0000 7f7d8eade640 1 -- 172.21.15.19:0/4279575788 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d8815a6f0 con 0x7f7d8806e0e0 2024-08-31T12:16:14.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.141+0000 7f7d857fa640 1 -- 172.21.15.19:0/4279575788 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f7d7801a070 con 0x7f7d8806e0e0 2024-08-31T12:16:14.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.141+0000 7f7d857fa640 1 --2- 172.21.15.19:0/4279575788 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7d74077a50 0x7f7d74079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:14.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.141+0000 7f7d857fa640 1 -- 172.21.15.19:0/4279575788 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f7d780997d0 con 0x7f7d8806e0e0 2024-08-31T12:16:14.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.141+0000 7f7d87fff640 1 --2- 172.21.15.19:0/4279575788 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7d74077a50 0x7f7d74079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:14.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.142+0000 7f7d87fff640 1 --2- 172.21.15.19:0/4279575788 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7d74077a50 0x7f7d74079f60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7d88154490 tx=0x7f7d7c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:14.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.146+0000 7f7d857fa640 1 -- 172.21.15.19:0/4279575788 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f7d8815a6f0 con 0x7f7d8806e0e0 2024-08-31T12:16:14.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.365+0000 7f7d8eade640 1 -- 172.21.15.19:0/4279575788 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7d880ccf20 con 0x7f7d8806e0e0 2024-08-31T12:16:14.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.366+0000 7f7d857fa640 1 -- 172.21.15.19:0/4279575788 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+6901 (secure 0 0 0) 0x7f7d78061e60 con 0x7f7d8806e0e0 2024-08-31T12:16:14.367 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:16:14.367 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":12,"btime":"2024-08-31T12:16:13:439874+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4371,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6833/4215834946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":4215834946},{"type":"v1","addr":"172.21.15.19:6833","nonce":4215834946}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/543344732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":543344732},{"type":"v1","addr":"172.21.15.19:6835","nonce":543344732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4380,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6837/699077015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6836","nonce":699077015},{"type":"v1","addr":"172.21.15.19:6837","nonce":699077015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-31T12:15:56.652936+0000","modified":"2024-08-31T12:16:13.439859+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4429,"mds_1":4426,"mds_2":4398},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4398":{"gid":4398,"name":"f","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.43:6837/2103490885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6835","nonce":2103490885},{"type":"v1","addr":"172.21.15.43:6837","nonce":2103490885}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4426":{"gid":4426,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.43:6839/1196858061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1196858061},{"type":"v1","addr":"172.21.15.43:6839","nonce":1196858061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4429":{"gid":4429,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.43:6838/3981040655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":3981040655},{"type":"v1","addr":"172.21.15.43:6838","nonce":3981040655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4429,"qdb_cluster":[4398,4426,4429]},"id":1}]} 2024-08-31T12:16:14.367 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 12 2024-08-31T12:16:14.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.368+0000 7f7d8eade640 1 -- 172.21.15.19:0/4279575788 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7d74077a50 msgr2=0x7f7d74079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:14.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.368+0000 7f7d8eade640 1 --2- 172.21.15.19:0/4279575788 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7d74077a50 0x7f7d74079f60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7d88154490 tx=0x7f7d7c009040 comp rx=0 tx=0).stop 2024-08-31T12:16:14.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.368+0000 7f7d8eade640 1 -- 172.21.15.19:0/4279575788 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7d8806e0e0 msgr2=0x7f7d881f15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:14.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.368+0000 7f7d8eade640 1 --2- 172.21.15.19:0/4279575788 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7d8806e0e0 0x7f7d881f15e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7d7800d740 tx=0x7f7d7800dc00 comp rx=0 tx=0).stop 2024-08-31T12:16:14.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.368+0000 7f7d8eade640 1 -- 172.21.15.19:0/4279575788 shutdown_connections 2024-08-31T12:16:14.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.368+0000 7f7d8eade640 1 --2- 172.21.15.19:0/4279575788 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7d74077a50 0x7f7d74079f60 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:14.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.368+0000 7f7d8eade640 1 --2- 172.21.15.19:0/4279575788 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d88155300 0x7f7d8814c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:14.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.368+0000 7f7d8eade640 1 --2- 172.21.15.19:0/4279575788 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7d880c8a40 0x7f7d880cf960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:14.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.368+0000 7f7d8eade640 1 --2- 172.21.15.19:0/4279575788 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7d8806e0e0 0x7f7d881f15e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:14.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.368+0000 7f7d8eade640 1 -- 172.21.15.19:0/4279575788 >> 172.21.15.19:0/4279575788 conn(0x7f7d880690d0 msgr2=0x7f7d881544c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:14.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.368+0000 7f7d8eade640 1 -- 172.21.15.19:0/4279575788 shutdown_connections 2024-08-31T12:16:14.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:14.368+0000 7f7d8eade640 1 -- 172.21.15.19:0/4279575788 wait complete. 2024-08-31T12:16:14.984 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 12, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-08-31T12:15:56.652936+0000', 'modified': '2024-08-31T12:16:13.439859+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 4429, 'mds_1': 4426, 'mds_2': 4398}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4398': {'gid': 4398, 'name': 'f', 'rank': 2, 'incarnation': 11, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.43:6837/2103490885', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6835', 'nonce': 2103490885}, {'type': 'v1', 'addr': '172.21.15.43:6837', 'nonce': 2103490885}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4426': {'gid': 4426, 'name': 'b', 'rank': 1, 'incarnation': 9, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.43:6839/1196858061', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6836', 'nonce': 1196858061}, {'type': 'v1', 'addr': '172.21.15.43:6839', 'nonce': 1196858061}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4429': {'gid': 4429, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.43:6838/3981040655', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6834', 'nonce': 3981040655}, {'type': 'v1', 'addr': '172.21.15.43:6838', 'nonce': 3981040655}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4429, 'qdb_cluster': [4398, 4426, 4429]} 2024-08-31T12:16:14.984 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-08-31T12:16:14.985 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-08-31T12:16:15.108 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.107+0000 7fd0afb94640 1 -- 172.21.15.19:0/3558993385 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd0a806e0e0 msgr2=0x7fd0a80c9850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:15.109 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.107+0000 7fd0afb94640 1 --2- 172.21.15.19:0/3558993385 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd0a806e0e0 0x7fd0a80c9850 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd098009830 tx=0x7fd09802a640 comp rx=0 tx=0).stop 2024-08-31T12:16:15.109 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.107+0000 7fd0afb94640 1 -- 172.21.15.19:0/3558993385 shutdown_connections 2024-08-31T12:16:15.109 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.107+0000 7fd0afb94640 1 --2- 172.21.15.19:0/3558993385 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd0a806e0e0 0x7fd0a80c9850 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:15.109 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.107+0000 7fd0afb94640 1 -- 172.21.15.19:0/3558993385 >> 172.21.15.19:0/3558993385 conn(0x7fd0a80690d0 msgr2=0x7fd0a80d0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:15.109 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.107+0000 7fd0afb94640 1 -- 172.21.15.19:0/3558993385 shutdown_connections 2024-08-31T12:16:15.109 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.107+0000 7fd0afb94640 1 -- 172.21.15.19:0/3558993385 wait complete. 2024-08-31T12:16:15.110 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.109+0000 7fd0afb94640 1 Processor -- start 2024-08-31T12:16:15.110 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.109+0000 7fd0afb94640 1 -- start start 2024-08-31T12:16:15.112 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.110+0000 7fd0afb94640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd0a806e0e0 0x7fd0a81f5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:15.113 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.110+0000 7fd0afb94640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd0a80c8a40 0x7fd0a80cd320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:15.113 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.110+0000 7fd0afb94640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd0a8155300 0x7fd0a80c6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:15.113 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.110+0000 7fd0afb94640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a815b250 con 0x7fd0a80c8a40 2024-08-31T12:16:15.113 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.110+0000 7fd0afb94640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a815af70 con 0x7fd0a806e0e0 2024-08-31T12:16:15.113 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.110+0000 7fd0afb94640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fd0a80c8610 con 0x7fd0a8155300 2024-08-31T12:16:15.113 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.110+0000 7fd0ad12f640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd0a80c8a40 0x7fd0a80cd320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:15.113 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.111+0000 7fd0ad12f640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd0a80c8a40 0x7fd0a80cd320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:44964/0 (socket says 172.21.15.19:44964) 2024-08-31T12:16:15.113 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.111+0000 7fd0ad12f640 1 -- 172.21.15.19:0/206375269 learned_addr learned my addr 172.21.15.19:0/206375269 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:15.113 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.111+0000 7fd0ad930640 1 --2- 172.21.15.19:0/206375269 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd0a806e0e0 0x7fd0a81f5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:15.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.111+0000 7fd0ae131640 1 --2- 172.21.15.19:0/206375269 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd0a8155300 0x7fd0a80c6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:15.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.111+0000 7fd0ad12f640 1 -- 172.21.15.19:0/206375269 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd0a8155300 msgr2=0x7fd0a80c6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:15.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.111+0000 7fd0ad12f640 1 --2- 172.21.15.19:0/206375269 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd0a8155300 0x7fd0a80c6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:15.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.111+0000 7fd0ad12f640 1 -- 172.21.15.19:0/206375269 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd0a806e0e0 msgr2=0x7fd0a81f5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:15.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.111+0000 7fd0ad12f640 1 --2- 172.21.15.19:0/206375269 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd0a806e0e0 0x7fd0a81f5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:15.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.111+0000 7fd0ad12f640 1 -- 172.21.15.19:0/206375269 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0a81f57a0 con 0x7fd0a80c8a40 2024-08-31T12:16:15.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.111+0000 7fd0ad12f640 1 --2- 172.21.15.19:0/206375269 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd0a80c8a40 0x7fd0a80cd320 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd09802ac30 tx=0x7fd098033df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:15.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.111+0000 7fd09e7fc640 1 -- 172.21.15.19:0/206375269 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0980332e0 con 0x7fd0a80c8a40 2024-08-31T12:16:15.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.111+0000 7fd09e7fc640 1 -- 172.21.15.19:0/206375269 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd098033480 con 0x7fd0a80c8a40 2024-08-31T12:16:15.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.111+0000 7fd09e7fc640 1 -- 172.21.15.19:0/206375269 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd098005280 con 0x7fd0a80c8a40 2024-08-31T12:16:15.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.111+0000 7fd0afb94640 1 -- 172.21.15.19:0/206375269 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0a80c7460 con 0x7fd0a80c8a40 2024-08-31T12:16:15.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.111+0000 7fd0afb94640 1 -- 172.21.15.19:0/206375269 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0a81f59b0 con 0x7fd0a80c8a40 2024-08-31T12:16:15.115 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.113+0000 7fd09e7fc640 1 -- 172.21.15.19:0/206375269 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fd098005420 con 0x7fd0a80c8a40 2024-08-31T12:16:15.115 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.114+0000 7fd09e7fc640 1 --2- 172.21.15.19:0/206375269 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd08c077a50 0x7fd08c079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:15.115 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.114+0000 7fd09e7fc640 1 -- 172.21.15.19:0/206375269 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd0980ba6f0 con 0x7fd0a80c8a40 2024-08-31T12:16:15.116 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.114+0000 7fd0ad930640 1 --2- 172.21.15.19:0/206375269 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd08c077a50 0x7fd08c079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:15.116 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.115+0000 7fd0ad930640 1 --2- 172.21.15.19:0/206375269 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd08c077a50 0x7fd08c079f60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd0a4001720 tx=0x7fd0a4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:15.116 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.115+0000 7fd0b6bda740 1 Processor -- start 2024-08-31T12:16:15.116 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.115+0000 7fd0b6bda740 1 -- start start 2024-08-31T12:16:15.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.116+0000 7fd0b6bda740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x56280c168270 0x56280c552700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:15.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.116+0000 7fd0b6bda740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56280c13f3b0 0x56280c508e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:15.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.116+0000 7fd0b6bda740 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x56280c147590 0x56280c50e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:15.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.116+0000 7fd0b6bda740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x56280c481000 con 0x56280c168270 2024-08-31T12:16:15.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.116+0000 7fd0b6bda740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56280c470590 con 0x56280c13f3b0 2024-08-31T12:16:15.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.116+0000 7fd0b6bda740 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x56280c474540 con 0x56280c147590 2024-08-31T12:16:15.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.116+0000 7fd0ad930640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x56280c168270 0x56280c552700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:15.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.116+0000 7fd0ad930640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x56280c168270 0x56280c552700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:44980/0 (socket says 172.21.15.19:44980) 2024-08-31T12:16:15.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.116+0000 7fd0ad930640 1 -- 172.21.15.19:0/1107386911 learned_addr learned my addr 172.21.15.19:0/1107386911 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:15.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.116+0000 7fd0ae131640 1 --2- 172.21.15.19:0/1107386911 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x56280c147590 0x56280c50e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:15.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.116+0000 7fd0ad12f640 1 --2- 172.21.15.19:0/1107386911 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56280c13f3b0 0x56280c508e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:15.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.116+0000 7fd0ad930640 1 -- 172.21.15.19:0/1107386911 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x56280c147590 msgr2=0x56280c50e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-31T12:16:15.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.117+0000 7fd0ad930640 1 --2- 172.21.15.19:0/1107386911 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x56280c147590 0x56280c50e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:15.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.117+0000 7fd0ad930640 1 -- 172.21.15.19:0/1107386911 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56280c13f3b0 msgr2=0x56280c508e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-31T12:16:15.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.117+0000 7fd0ad930640 1 --2- 172.21.15.19:0/1107386911 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56280c13f3b0 0x56280c508e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:15.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.117+0000 7fd0ad930640 1 -- 172.21.15.19:0/1107386911 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56280c1d3850 con 0x56280c168270 2024-08-31T12:16:15.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.117+0000 7fd0ad930640 1 --2- 172.21.15.19:0/1107386911 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x56280c168270 0x56280c552700 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd0a400f970 tx=0x7fd0a4012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7f2feb4c497ad52 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:15.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.117+0000 7fd08b7fe640 1 -- 172.21.15.19:0/1107386911 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a4025070 con 0x56280c168270 2024-08-31T12:16:15.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.117+0000 7fd08b7fe640 1 -- 172.21.15.19:0/1107386911 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd0a401b050 con 0x56280c168270 2024-08-31T12:16:15.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.117+0000 7fd08b7fe640 1 -- 172.21.15.19:0/1107386911 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a4014de0 con 0x56280c168270 2024-08-31T12:16:15.120 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.118+0000 7fd0b6bda740 1 -- 172.21.15.19:0/1107386911 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x56280c168270 msgr2=0x56280c552700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:15.120 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.118+0000 7fd0b6bda740 1 --2- 172.21.15.19:0/1107386911 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x56280c168270 0x56280c552700 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd0a400f970 tx=0x7fd0a4012000 comp rx=0 tx=0).stop 2024-08-31T12:16:15.120 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.118+0000 7fd0b6bda740 1 -- 172.21.15.19:0/1107386911 shutdown_connections 2024-08-31T12:16:15.120 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.118+0000 7fd0b6bda740 1 --2- 172.21.15.19:0/1107386911 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x56280c147590 0x56280c50e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:15.120 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.118+0000 7fd0b6bda740 1 --2- 172.21.15.19:0/1107386911 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56280c13f3b0 0x56280c508e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:15.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.118+0000 7fd0b6bda740 1 --2- 172.21.15.19:0/1107386911 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x56280c168270 0x56280c552700 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:15.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.118+0000 7fd0b6bda740 1 -- 172.21.15.19:0/1107386911 >> 172.21.15.19:0/1107386911 conn(0x56280c15c3c0 msgr2=0x56280c551e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:15.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.118+0000 7fd0b6bda740 1 -- 172.21.15.19:0/1107386911 shutdown_connections 2024-08-31T12:16:15.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.118+0000 7fd0b6bda740 1 -- 172.21.15.19:0/1107386911 wait complete. 2024-08-31T12:16:15.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.119+0000 7fd0b6bda740 1 -- start start 2024-08-31T12:16:15.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.119+0000 7fd0b6bda740 1 Processor -- start 2024-08-31T12:16:15.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.120+0000 7fd0b6bda740 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x56280c13f3b0 0x56280c53e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:15.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.120+0000 7fd0b6bda740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x56280c147590 0x56280c5442f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:15.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.120+0000 7fd0b6bda740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56280c168270 0x56280c544830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:15.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.120+0000 7fd0b6bda740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x56280c1e2620 con 0x56280c147590 2024-08-31T12:16:15.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.120+0000 7fd0b6bda740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56280c3f7620 con 0x56280c168270 2024-08-31T12:16:15.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.120+0000 7fd0b6bda740 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x56280c561b40 con 0x56280c13f3b0 2024-08-31T12:16:15.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.120+0000 7fd0b6bda740 10 client.0 _ll_register_callbacks cb 0x56280c41b0a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-31T12:16:15.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.120+0000 7fd0ad12f640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x56280c147590 0x56280c5442f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:15.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.121+0000 7fd0ad12f640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x56280c147590 0x56280c5442f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:44986/0 (socket says 172.21.15.19:44986) 2024-08-31T12:16:15.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.121+0000 7fd0ad12f640 1 -- 172.21.15.19:0/315864046 learned_addr learned my addr 172.21.15.19:0/315864046 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:15.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.121+0000 7fd0ad930640 1 --2- 172.21.15.19:0/315864046 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x56280c13f3b0 0x56280c53e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:15.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.121+0000 7fd0ae131640 1 --2- 172.21.15.19:0/315864046 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56280c168270 0x56280c544830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:15.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.121+0000 7fd0ad12f640 1 -- 172.21.15.19:0/315864046 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x56280c13f3b0 msgr2=0x56280c53e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-31T12:16:15.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.121+0000 7fd0ad12f640 1 --2- 172.21.15.19:0/315864046 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x56280c13f3b0 0x56280c53e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:15.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.121+0000 7fd0ad12f640 1 -- 172.21.15.19:0/315864046 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56280c168270 msgr2=0x56280c544830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-31T12:16:15.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.121+0000 7fd0ad12f640 1 --2- 172.21.15.19:0/315864046 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56280c168270 0x56280c544830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:15.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.121+0000 7fd0ad12f640 1 -- 172.21.15.19:0/315864046 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56280c24a6b0 con 0x56280c147590 2024-08-31T12:16:15.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.121+0000 7fd0ae131640 1 --2- 172.21.15.19:0/315864046 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56280c168270 0x56280c544830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:16:15.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.121+0000 7fd0ad12f640 1 --2- 172.21.15.19:0/315864046 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x56280c147590 0x56280c5442f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd098009830 tx=0x7fd0980095b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84b1210bff1f051f server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:15.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.121+0000 7fd088ff9640 10 client.0 ms_handle_connect on v2:172.21.15.19:3300/0 2024-08-31T12:16:15.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.121+0000 7fd088ff9640 1 -- 172.21.15.19:0/315864046 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd098048070 con 0x56280c147590 2024-08-31T12:16:15.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.121+0000 7fd088ff9640 1 -- 172.21.15.19:0/315864046 <== mon.0 v2:172.21.15.19:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd098083060 con 0x56280c147590 2024-08-31T12:16:15.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.121+0000 7fd0b6bda740 1 -- 172.21.15.19:0/315864046 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0a401b050 con 0x56280c147590 2024-08-31T12:16:15.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.121+0000 7fd088ff9640 1 -- 172.21.15.19:0/315864046 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) ==== 24+0+0 (secure 0 0 0) 0x7fd098082910 con 0x56280c147590 2024-08-31T12:16:15.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.121+0000 7fd0b6bda740 10 client.4737 fetch_fsmap learned FSMap version 12 2024-08-31T12:16:15.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.121+0000 7fd0b6bda740 1 -- 172.21.15.19:0/315864046 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) -- 0x56280c53eea0 con 0x56280c147590 2024-08-31T12:16:15.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.121+0000 7fd088ff9640 1 -- 172.21.15.19:0/315864046 <== mon.0 v2:172.21.15.19:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd0a401b050 con 0x56280c147590 2024-08-31T12:16:15.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.122+0000 7fd088ff9640 1 -- 172.21.15.19:0/315864046 <== mon.0 v2:172.21.15.19:3300/0 5 ==== fsmap(e 12) ==== 4752+0+0 (secure 0 0 0) 0x7fd09808bb50 con 0x56280c147590 2024-08-31T12:16:15.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.122+0000 7fd0b6bda740 10 client.4737 fetch_fsmap finished waiting for FSMap version 12 2024-08-31T12:16:15.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.122+0000 7fd0b6bda740 10 client.4737 resolve_mds: resolved name 'f' to daemon mds.f 2024-08-31T12:16:15.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.122+0000 7fd0b6bda740 20 client.4737 populate_metadata read hostname 'smithi019' 2024-08-31T12:16:15.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.122+0000 7fd0b6bda740 1 --2- 172.21.15.19:0/315864046 >> [v2:172.21.15.43:6835/2103490885,v1:172.21.15.43:6837/2103490885] conn(0x56280c126e40 0x56280c56cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:15.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.122+0000 7fd0b6bda740 4 client.4737 mds_command: new command op to 4398 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-31T12:16:15.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.122+0000 7fd0b6bda740 1 -- 172.21.15.19:0/315864046 --> [v2:172.21.15.43:6835/2103490885,v1:172.21.15.43:6837/2103490885] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56280c482f50 con 0x56280c126e40 2024-08-31T12:16:15.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.122+0000 7fd0ad930640 1 --2- 172.21.15.19:0/315864046 >> [v2:172.21.15.43:6835/2103490885,v1:172.21.15.43:6837/2103490885] conn(0x56280c126e40 0x56280c56cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:15.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.122+0000 7fd0ad930640 1 --2- 172.21.15.19:0/315864046 >> [v2:172.21.15.43:6835/2103490885,v1:172.21.15.43:6837/2103490885] conn(0x56280c126e40 0x56280c56cc20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b908f3441114e8a5 server_cookie=20c1f286372b51a2 in_seq=0 out_seq=0 2024-08-31T12:16:15.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:15.122+0000 7fd088ff9640 10 client.4737 ms_handle_connect on v2:172.21.15.43:6835/2103490885 2024-08-31T12:16:17.212 INFO:tasks.ceph.mds.d.smithi043.stderr:2024-08-31T12:16:17.211+0000 7ff036f5a640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-08-31T12:16:18.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.446+0000 7fd088ff9640 1 -- 172.21.15.19:0/315864046 <== mds.2 v2:172.21.15.43:6835/2103490885 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56280c482f50 con 0x56280c126e40 2024-08-31T12:16:18.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.446+0000 7fd088ff9640 10 client.4737 handle_command_reply: tid=0 2024-08-31T12:16:18.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.446+0000 7fd0b6bda740 1 client.4737 shutdown 2024-08-31T12:16:18.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.446+0000 7fd0b6bda740 1 -- 172.21.15.19:0/315864046 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x56280c147590 msgr2=0x56280c5442f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:18.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.446+0000 7fd0b6bda740 1 --2- 172.21.15.19:0/315864046 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x56280c147590 0x56280c5442f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd098009830 tx=0x7fd0980095b0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.446+0000 7fd0b6bda740 1 -- 172.21.15.19:0/315864046 shutdown_connections 2024-08-31T12:16:18.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.447+0000 7fd0b6bda740 1 --2- 172.21.15.19:0/315864046 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56280c168270 0x56280c544830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.447+0000 7fd0b6bda740 1 --2- 172.21.15.19:0/315864046 >> [v2:172.21.15.43:6835/2103490885,v1:172.21.15.43:6837/2103490885] conn(0x56280c126e40 0x56280c56cc20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.447+0000 7fd0b6bda740 1 --2- 172.21.15.19:0/315864046 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x56280c147590 0x56280c5442f0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.449 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.447+0000 7fd0b6bda740 1 --2- 172.21.15.19:0/315864046 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x56280c13f3b0 0x56280c53e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.449 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.447+0000 7fd088ff9640 0 client.4737 ms_handle_reset on v2:172.21.15.43:6835/2103490885 2024-08-31T12:16:18.449 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.447+0000 7fd0b6bda740 1 -- 172.21.15.19:0/315864046 >> 172.21.15.19:0/315864046 conn(0x56280c15c3c0 msgr2=0x56280c50f6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:18.449 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.447+0000 7fd0b6bda740 1 -- 172.21.15.19:0/315864046 shutdown_connections 2024-08-31T12:16:18.449 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.447+0000 7fd0b6bda740 1 -- 172.21.15.19:0/315864046 wait complete. 2024-08-31T12:16:18.449 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.447+0000 7fd0b6bda740 20 client.4737 trim_cache size 0 max 16384 2024-08-31T12:16:18.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.467+0000 7fd0b6bda740 1 Processor -- start 2024-08-31T12:16:18.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.467+0000 7fd0b6bda740 1 -- start start 2024-08-31T12:16:18.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.468+0000 7fd0b6bda740 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x56280c13f3b0 0x56280c54f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:18.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.469+0000 7fd0b6bda740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x56280c147590 0x56280c50e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:18.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.469+0000 7fd0b6bda740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56280c168270 0x56280c50eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:18.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.469+0000 7fd0b6bda740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a8155f60 con 0x56280c147590 2024-08-31T12:16:18.474 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.469+0000 7fd0b6bda740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56280c561880 con 0x56280c168270 2024-08-31T12:16:18.474 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.469+0000 7fd0b6bda740 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x56280c53f690 con 0x56280c13f3b0 2024-08-31T12:16:18.474 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.469+0000 7fd0ad12f640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x56280c147590 0x56280c50e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:18.474 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.469+0000 7fd0ad12f640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x56280c147590 0x56280c50e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:37882/0 (socket says 172.21.15.19:37882) 2024-08-31T12:16:18.474 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.469+0000 7fd0ad12f640 1 -- 172.21.15.19:0/1437058021 learned_addr learned my addr 172.21.15.19:0/1437058021 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:18.474 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.469+0000 7fd0ae131640 1 --2- 172.21.15.19:0/1437058021 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56280c168270 0x56280c50eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:18.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.469+0000 7fd0ad930640 1 --2- 172.21.15.19:0/1437058021 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x56280c13f3b0 0x56280c54f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:18.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.469+0000 7fd0ad12f640 1 -- 172.21.15.19:0/1437058021 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x56280c13f3b0 msgr2=0x56280c54f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-31T12:16:18.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.469+0000 7fd0ad12f640 1 --2- 172.21.15.19:0/1437058021 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x56280c13f3b0 0x56280c54f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.469+0000 7fd0ad12f640 1 -- 172.21.15.19:0/1437058021 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56280c168270 msgr2=0x56280c50eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-31T12:16:18.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.469+0000 7fd0ad12f640 1 --2- 172.21.15.19:0/1437058021 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56280c168270 0x56280c50eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.469+0000 7fd0ad12f640 1 -- 172.21.15.19:0/1437058021 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56280c50ca20 con 0x56280c147590 2024-08-31T12:16:18.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.469+0000 7fd0ad930640 1 --2- 172.21.15.19:0/1437058021 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x56280c13f3b0 0x56280c54f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:16:18.476 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.469+0000 7fd0ad12f640 1 --2- 172.21.15.19:0/1437058021 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x56280c147590 0x56280c50e5f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd09807f3f0 tx=0x7fd09809e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc3fe3c83c0a91cf server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:18.476 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.469+0000 7fd08affd640 1 -- 172.21.15.19:0/1437058021 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd098048070 con 0x56280c147590 2024-08-31T12:16:18.476 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.470+0000 7fd08affd640 1 -- 172.21.15.19:0/1437058021 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd0980a8070 con 0x56280c147590 2024-08-31T12:16:18.476 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.470+0000 7fd08affd640 1 -- 172.21.15.19:0/1437058021 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd098082750 con 0x56280c147590 2024-08-31T12:16:18.476 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.471+0000 7fd0b6bda740 1 -- 172.21.15.19:0/1437058021 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x56280c147590 msgr2=0x56280c50e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:18.476 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.471+0000 7fd0b6bda740 1 --2- 172.21.15.19:0/1437058021 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x56280c147590 0x56280c50e5f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd09807f3f0 tx=0x7fd09809e000 comp rx=0 tx=0).stop 2024-08-31T12:16:18.476 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.471+0000 7fd0b6bda740 1 -- 172.21.15.19:0/1437058021 shutdown_connections 2024-08-31T12:16:18.477 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.471+0000 7fd0b6bda740 1 --2- 172.21.15.19:0/1437058021 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56280c168270 0x56280c50eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.477 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.471+0000 7fd0b6bda740 1 --2- 172.21.15.19:0/1437058021 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x56280c147590 0x56280c50e5f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.477 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.471+0000 7fd0b6bda740 1 --2- 172.21.15.19:0/1437058021 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x56280c13f3b0 0x56280c54f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.477 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.471+0000 7fd0b6bda740 1 -- 172.21.15.19:0/1437058021 >> 172.21.15.19:0/1437058021 conn(0x56280c15c3c0 msgr2=0x56280c50f6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:18.478 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.471+0000 7fd0b6bda740 1 -- 172.21.15.19:0/1437058021 shutdown_connections 2024-08-31T12:16:18.478 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.471+0000 7fd0b6bda740 1 -- 172.21.15.19:0/1437058021 wait complete. 2024-08-31T12:16:18.478 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.472+0000 7fd0b6bda740 1 -- start start 2024-08-31T12:16:18.478 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.472+0000 7fd0b6bda740 1 Processor -- start 2024-08-31T12:16:18.485 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.473+0000 7fd0b6bda740 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x56280c13f3b0 0x56280c50fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:18.485 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.473+0000 7fd0b6bda740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56280c147590 0x56280c5c58b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:18.485 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.473+0000 7fd0b6bda740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x56280c168270 0x56280c50e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:18.485 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.473+0000 7fd0b6bda740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x56280c5615e0 con 0x56280c168270 2024-08-31T12:16:18.485 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.473+0000 7fd0b6bda740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56280c549620 con 0x56280c147590 2024-08-31T12:16:18.486 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.473+0000 7fd0b6bda740 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x56280c541d30 con 0x56280c13f3b0 2024-08-31T12:16:18.486 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.473+0000 7fd0b6bda740 10 client.0 _ll_register_callbacks cb 0x7fd098085040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-31T12:16:18.486 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.473+0000 7fd0ae131640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x56280c168270 0x56280c50e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:18.486 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.473+0000 7fd0ae131640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x56280c168270 0x56280c50e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:37884/0 (socket says 172.21.15.19:37884) 2024-08-31T12:16:18.486 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.473+0000 7fd0ae131640 1 -- 172.21.15.19:0/938514560 learned_addr learned my addr 172.21.15.19:0/938514560 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:18.486 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.474+0000 7fd0ad930640 1 --2- 172.21.15.19:0/938514560 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x56280c13f3b0 0x56280c50fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:18.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.474+0000 7fd0ad12f640 1 --2- 172.21.15.19:0/938514560 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56280c147590 0x56280c5c58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:18.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.474+0000 7fd0ae131640 1 -- 172.21.15.19:0/938514560 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x56280c13f3b0 msgr2=0x56280c50fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-31T12:16:18.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.474+0000 7fd0ae131640 1 --2- 172.21.15.19:0/938514560 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x56280c13f3b0 0x56280c50fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.474+0000 7fd0ae131640 1 -- 172.21.15.19:0/938514560 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56280c147590 msgr2=0x56280c5c58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-31T12:16:18.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.474+0000 7fd0ae131640 1 --2- 172.21.15.19:0/938514560 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56280c147590 0x56280c5c58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.474+0000 7fd0ae131640 1 -- 172.21.15.19:0/938514560 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56280c50d7a0 con 0x56280c168270 2024-08-31T12:16:18.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.474+0000 7fd0ad930640 1 --2- 172.21.15.19:0/938514560 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x56280c13f3b0 0x56280c50fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:16:18.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.474+0000 7fd0ae131640 1 --2- 172.21.15.19:0/938514560 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x56280c168270 0x56280c50e4c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd0a0017e50 tx=0x7fd0a0015780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d6d04ebeb0a6c1 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:18.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.474+0000 7fd088ff9640 10 client.0 ms_handle_connect on v2:172.21.15.19:3300/0 2024-08-31T12:16:18.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.474+0000 7fd088ff9640 1 -- 172.21.15.19:0/938514560 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a0024060 con 0x56280c168270 2024-08-31T12:16:18.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.474+0000 7fd088ff9640 1 -- 172.21.15.19:0/938514560 <== mon.0 v2:172.21.15.19:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd0a001f3f0 con 0x56280c168270 2024-08-31T12:16:18.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.474+0000 7fd0b6bda740 1 -- 172.21.15.19:0/938514560 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0980a8070 con 0x56280c168270 2024-08-31T12:16:18.489 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.474+0000 7fd088ff9640 1 -- 172.21.15.19:0/938514560 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) ==== 24+0+0 (secure 0 0 0) 0x7fd0a002a050 con 0x56280c168270 2024-08-31T12:16:18.489 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.475+0000 7fd0b6bda740 10 client.4743 fetch_fsmap learned FSMap version 12 2024-08-31T12:16:18.489 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.475+0000 7fd0b6bda740 1 -- 172.21.15.19:0/938514560 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) -- 0x56280c5d2cd0 con 0x56280c168270 2024-08-31T12:16:18.489 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.475+0000 7fd088ff9640 1 -- 172.21.15.19:0/938514560 <== mon.0 v2:172.21.15.19:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd0980a8070 con 0x56280c168270 2024-08-31T12:16:18.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.475+0000 7fd088ff9640 1 -- 172.21.15.19:0/938514560 <== mon.0 v2:172.21.15.19:3300/0 5 ==== fsmap(e 12) ==== 4752+0+0 (secure 0 0 0) 0x7fd0a002d020 con 0x56280c168270 2024-08-31T12:16:18.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.475+0000 7fd0b6bda740 10 client.4743 fetch_fsmap finished waiting for FSMap version 12 2024-08-31T12:16:18.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.475+0000 7fd0b6bda740 10 client.4743 resolve_mds: resolved name 'f' to daemon mds.f 2024-08-31T12:16:18.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.475+0000 7fd0b6bda740 20 client.4743 populate_metadata read hostname 'smithi019' 2024-08-31T12:16:18.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.475+0000 7fd0b6bda740 1 --2- 172.21.15.19:0/938514560 >> [v2:172.21.15.43:6835/2103490885,v1:172.21.15.43:6837/2103490885] conn(0x56280c555900 0x56280c555f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:18.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.475+0000 7fd0b6bda740 4 client.4743 mds_command: new command op to 4398 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-31T12:16:18.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.475+0000 7fd0b6bda740 1 -- 172.21.15.19:0/938514560 --> [v2:172.21.15.43:6835/2103490885,v1:172.21.15.43:6837/2103490885] -- command(tid 0: {"prefix": "status"}) -- 0x56280c47e9f0 con 0x56280c555900 2024-08-31T12:16:18.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.475+0000 7fd0ad930640 1 --2- 172.21.15.19:0/938514560 >> [v2:172.21.15.43:6835/2103490885,v1:172.21.15.43:6837/2103490885] conn(0x56280c555900 0x56280c555f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:18.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.476+0000 7fd088ff9640 10 client.4743 ms_handle_connect on v2:172.21.15.43:6835/2103490885 2024-08-31T12:16:18.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.476+0000 7fd0ad930640 1 --2- 172.21.15.19:0/938514560 >> [v2:172.21.15.43:6835/2103490885,v1:172.21.15.43:6837/2103490885] conn(0x56280c555900 0x56280c555f20 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d052d32219fed401 server_cookie=60266b98742b8cce in_seq=0 out_seq=0 2024-08-31T12:16:18.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.476+0000 7fd088ff9640 1 -- 172.21.15.19:0/938514560 <== mds.2 v2:172.21.15.43:6835/2103490885 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x56280c47e9f0 con 0x56280c555900 2024-08-31T12:16:18.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.476+0000 7fd088ff9640 10 client.4743 handle_command_reply: tid=0 2024-08-31T12:16:18.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.476+0000 7fd0b6bda740 1 client.4743 shutdown 2024-08-31T12:16:18.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.476+0000 7fd0b6bda740 1 -- 172.21.15.19:0/938514560 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x56280c168270 msgr2=0x56280c50e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:18.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.476+0000 7fd0b6bda740 1 --2- 172.21.15.19:0/938514560 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x56280c168270 0x56280c50e4c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd0a0017e50 tx=0x7fd0a0015780 comp rx=0 tx=0).stop 2024-08-31T12:16:18.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.476+0000 7fd0b6bda740 1 -- 172.21.15.19:0/938514560 shutdown_connections 2024-08-31T12:16:18.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.476+0000 7fd0b6bda740 1 --2- 172.21.15.19:0/938514560 >> [v2:172.21.15.43:6835/2103490885,v1:172.21.15.43:6837/2103490885] conn(0x56280c555900 0x56280c555f20 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.493 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.476+0000 7fd0b6bda740 1 --2- 172.21.15.19:0/938514560 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x56280c168270 0x56280c50e4c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.493 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.476+0000 7fd0b6bda740 1 --2- 172.21.15.19:0/938514560 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56280c147590 0x56280c5c58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.493 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.476+0000 7fd0b6bda740 1 --2- 172.21.15.19:0/938514560 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x56280c13f3b0 0x56280c50fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.493 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.476+0000 7fd088ff9640 0 client.4743 ms_handle_reset on v2:172.21.15.43:6835/2103490885 2024-08-31T12:16:18.493 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.476+0000 7fd0b6bda740 1 -- 172.21.15.19:0/938514560 >> 172.21.15.19:0/938514560 conn(0x56280c15c3c0 msgr2=0x56280c5452c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:18.494 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.476+0000 7fd0b6bda740 1 -- 172.21.15.19:0/938514560 shutdown_connections 2024-08-31T12:16:18.494 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.476+0000 7fd0b6bda740 1 -- 172.21.15.19:0/938514560 wait complete. 2024-08-31T12:16:18.494 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.476+0000 7fd0b6bda740 20 client.4743 trim_cache size 0 max 16384 2024-08-31T12:16:18.494 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:16:18.494 INFO:teuthology.orchestra.run.smithi019.stdout: "cluster_fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:16:18.494 INFO:teuthology.orchestra.run.smithi019.stdout: "whoami": 2, 2024-08-31T12:16:18.494 INFO:teuthology.orchestra.run.smithi019.stdout: "id": 4398, 2024-08-31T12:16:18.495 INFO:teuthology.orchestra.run.smithi019.stdout: "want_state": "up:active", 2024-08-31T12:16:18.495 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "up:active", 2024-08-31T12:16:18.495 INFO:teuthology.orchestra.run.smithi019.stdout: "fs_name": "cephfs", 2024-08-31T12:16:18.495 INFO:teuthology.orchestra.run.smithi019.stdout: "rank_uptime": 10.115920709999999, 2024-08-31T12:16:18.495 INFO:teuthology.orchestra.run.smithi019.stdout: "mdsmap_epoch": 12, 2024-08-31T12:16:18.495 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_epoch": 21, 2024-08-31T12:16:18.495 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_epoch_barrier": 21, 2024-08-31T12:16:18.495 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 27.255834573000001 2024-08-31T12:16:18.495 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:16:18.496 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.477+0000 7fd0afb94640 1 -- 172.21.15.19:0/206375269 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd08c077a50 msgr2=0x7fd08c079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:18.496 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.477+0000 7fd0afb94640 1 --2- 172.21.15.19:0/206375269 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd08c077a50 0x7fd08c079f60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd0a4001720 tx=0x7fd0a4009040 comp rx=0 tx=0).stop 2024-08-31T12:16:18.497 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.477+0000 7fd0afb94640 1 -- 172.21.15.19:0/206375269 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd0a80c8a40 msgr2=0x7fd0a80cd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:18.497 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.477+0000 7fd0afb94640 1 --2- 172.21.15.19:0/206375269 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd0a80c8a40 0x7fd0a80cd320 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd09802ac30 tx=0x7fd098033df0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.497 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.477+0000 7fd0afb94640 1 -- 172.21.15.19:0/206375269 shutdown_connections 2024-08-31T12:16:18.497 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.477+0000 7fd0afb94640 1 --2- 172.21.15.19:0/206375269 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd08c077a50 0x7fd08c079f60 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.497 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.477+0000 7fd0afb94640 1 --2- 172.21.15.19:0/206375269 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd0a8155300 0x7fd0a80c6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.497 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.477+0000 7fd0afb94640 1 --2- 172.21.15.19:0/206375269 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd0a80c8a40 0x7fd0a80cd320 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.497 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.477+0000 7fd0afb94640 1 --2- 172.21.15.19:0/206375269 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd0a806e0e0 0x7fd0a81f5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.497 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.477+0000 7fd0afb94640 1 -- 172.21.15.19:0/206375269 >> 172.21.15.19:0/206375269 conn(0x7fd0a80690d0 msgr2=0x7fd0a8150e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:18.497 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.477+0000 7fd0afb94640 1 -- 172.21.15.19:0/206375269 shutdown_connections 2024-08-31T12:16:18.497 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.477+0000 7fd0afb94640 1 -- 172.21.15.19:0/206375269 wait complete. 2024-08-31T12:16:18.500 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-08-31T12:16:18.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.622+0000 7f0eca19f640 1 -- 172.21.15.19:0/358964181 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0ec415f350 msgr2=0x7f0ec40b59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:18.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.622+0000 7f0eca19f640 1 --2- 172.21.15.19:0/358964181 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0ec415f350 0x7f0ec40b59f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0eb8009830 tx=0x7f0eb802a480 comp rx=0 tx=0).stop 2024-08-31T12:16:18.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.622+0000 7f0eca19f640 1 -- 172.21.15.19:0/358964181 shutdown_connections 2024-08-31T12:16:18.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.622+0000 7f0eca19f640 1 --2- 172.21.15.19:0/358964181 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0ec415bbd0 0x7f0ec40c5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.622+0000 7f0eca19f640 1 --2- 172.21.15.19:0/358964181 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0ec415f350 0x7f0ec40b59f0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.622+0000 7f0eca19f640 1 -- 172.21.15.19:0/358964181 >> 172.21.15.19:0/358964181 conn(0x7f0ec406dba0 msgr2=0x7f0ec406e6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:18.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.622+0000 7f0eca19f640 1 -- 172.21.15.19:0/358964181 shutdown_connections 2024-08-31T12:16:18.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.622+0000 7f0eca19f640 1 -- 172.21.15.19:0/358964181 wait complete. 2024-08-31T12:16:18.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.624+0000 7f0eca19f640 1 Processor -- start 2024-08-31T12:16:18.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.624+0000 7f0eca19f640 1 -- start start 2024-08-31T12:16:18.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.626+0000 7f0eca19f640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ec40cb2a0 0x7f0ec41583c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:18.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.626+0000 7f0eca19f640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0ec415bbd0 0x7f0ec41f2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:18.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.626+0000 7f0eca19f640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0ec415f350 0x7f0ec40ca040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:18.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.626+0000 7f0eca19f640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec40b3db0 con 0x7f0ec415bbd0 2024-08-31T12:16:18.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.626+0000 7f0eca19f640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec40c6100 con 0x7f0ec40cb2a0 2024-08-31T12:16:18.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.626+0000 7f0eca19f640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f0ec40cba70 con 0x7f0ec415f350 2024-08-31T12:16:18.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.626+0000 7f0ec2ffd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0ec415bbd0 0x7f0ec41f2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:18.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.626+0000 7f0ec2ffd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0ec415bbd0 0x7f0ec41f2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:37892/0 (socket says 172.21.15.19:37892) 2024-08-31T12:16:18.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.626+0000 7f0ec2ffd640 1 -- 172.21.15.19:0/3952515220 learned_addr learned my addr 172.21.15.19:0/3952515220 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:18.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.626+0000 7f0ec3fff640 1 --2- 172.21.15.19:0/3952515220 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0ec415f350 0x7f0ec40ca040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:18.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.626+0000 7f0ec37fe640 1 --2- 172.21.15.19:0/3952515220 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ec40cb2a0 0x7f0ec41583c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:18.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.626+0000 7f0ec2ffd640 1 -- 172.21.15.19:0/3952515220 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0ec415f350 msgr2=0x7f0ec40ca040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:18.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.626+0000 7f0ec2ffd640 1 --2- 172.21.15.19:0/3952515220 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0ec415f350 0x7f0ec40ca040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.626+0000 7f0ec2ffd640 1 -- 172.21.15.19:0/3952515220 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ec40cb2a0 msgr2=0x7f0ec41583c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:18.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.626+0000 7f0ec2ffd640 1 --2- 172.21.15.19:0/3952515220 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ec40cb2a0 0x7f0ec41583c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.626+0000 7f0ec2ffd640 1 -- 172.21.15.19:0/3952515220 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ec40c7c20 con 0x7f0ec415bbd0 2024-08-31T12:16:18.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.626+0000 7f0ec2ffd640 1 --2- 172.21.15.19:0/3952515220 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0ec415bbd0 0x7f0ec41f2480 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0eb8000c00 tx=0x7f0eb8033c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:18.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.626+0000 7f0eabfff640 1 -- 172.21.15.19:0/3952515220 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb8005c90 con 0x7f0ec415bbd0 2024-08-31T12:16:18.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.626+0000 7f0eabfff640 1 -- 172.21.15.19:0/3952515220 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0eb8031e50 con 0x7f0ec415bbd0 2024-08-31T12:16:18.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.626+0000 7f0eca19f640 1 -- 172.21.15.19:0/3952515220 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ec41f29c0 con 0x7f0ec415bbd0 2024-08-31T12:16:18.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.626+0000 7f0eabfff640 1 -- 172.21.15.19:0/3952515220 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb80332f0 con 0x7f0ec415bbd0 2024-08-31T12:16:18.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.626+0000 7f0eca19f640 1 -- 172.21.15.19:0/3952515220 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ec40ca580 con 0x7f0ec415bbd0 2024-08-31T12:16:18.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.628+0000 7f0eabfff640 1 -- 172.21.15.19:0/3952515220 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f0eb8033490 con 0x7f0ec415bbd0 2024-08-31T12:16:18.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.629+0000 7f0eabfff640 1 --2- 172.21.15.19:0/3952515220 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0eb0077a80 0x7f0eb0079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:18.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.629+0000 7f0eabfff640 1 -- 172.21.15.19:0/3952515220 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0eb80baa90 con 0x7f0ec415bbd0 2024-08-31T12:16:18.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.629+0000 7f0ec37fe640 1 --2- 172.21.15.19:0/3952515220 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0eb0077a80 0x7f0eb0079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:18.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.630+0000 7f0ec37fe640 1 --2- 172.21.15.19:0/3952515220 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0eb0077a80 0x7f0eb0079f90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0eb4007fd0 tx=0x7f0eb4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:18.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.630+0000 7f0ed11ec740 1 Processor -- start 2024-08-31T12:16:18.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.630+0000 7f0ed11ec740 1 -- start start 2024-08-31T12:16:18.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.631+0000 7f0ed11ec740 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x55ee126ec270 0x55ee12ad6700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:18.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.631+0000 7f0ed11ec740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55ee126c33b0 0x55ee12a8ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:18.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.631+0000 7f0ed11ec740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee126cb590 0x55ee12a92070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:18.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.631+0000 7f0ed11ec740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x55ee12a05000 con 0x55ee126c33b0 2024-08-31T12:16:18.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.631+0000 7f0ed11ec740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ee129f4590 con 0x55ee126cb590 2024-08-31T12:16:18.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.631+0000 7f0ed11ec740 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x55ee129f8540 con 0x55ee126ec270 2024-08-31T12:16:18.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.631+0000 7f0ec2ffd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55ee126c33b0 0x55ee12a8ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:18.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.631+0000 7f0ec2ffd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55ee126c33b0 0x55ee12a8ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:37902/0 (socket says 172.21.15.19:37902) 2024-08-31T12:16:18.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.631+0000 7f0ec2ffd640 1 -- 172.21.15.19:0/361805574 learned_addr learned my addr 172.21.15.19:0/361805574 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:18.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.631+0000 7f0ec3fff640 1 --2- 172.21.15.19:0/361805574 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee126cb590 0x55ee12a92070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:18.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.631+0000 7f0ec37fe640 1 --2- 172.21.15.19:0/361805574 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x55ee126ec270 0x55ee12ad6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:18.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.631+0000 7f0ec2ffd640 1 -- 172.21.15.19:0/361805574 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x55ee126ec270 msgr2=0x55ee12ad6700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-31T12:16:18.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.631+0000 7f0ec2ffd640 1 --2- 172.21.15.19:0/361805574 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x55ee126ec270 0x55ee12ad6700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.631+0000 7f0ec2ffd640 1 -- 172.21.15.19:0/361805574 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee126cb590 msgr2=0x55ee12a92070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-31T12:16:18.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.631+0000 7f0ec2ffd640 1 --2- 172.21.15.19:0/361805574 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee126cb590 0x55ee12a92070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.631+0000 7f0ec2ffd640 1 -- 172.21.15.19:0/361805574 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee12757850 con 0x55ee126c33b0 2024-08-31T12:16:18.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.632+0000 7f0ec2ffd640 1 --2- 172.21.15.19:0/361805574 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55ee126c33b0 0x55ee12a8ce60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0eb8009780 tx=0x7f0eb8082900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df711451741909be server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:18.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.632+0000 7f0ea97fa640 1 -- 172.21.15.19:0/361805574 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb8083890 con 0x55ee126c33b0 2024-08-31T12:16:18.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.632+0000 7f0ea97fa640 1 -- 172.21.15.19:0/361805574 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0eb8083190 con 0x55ee126c33b0 2024-08-31T12:16:18.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.632+0000 7f0ea97fa640 1 -- 172.21.15.19:0/361805574 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb8083630 con 0x55ee126c33b0 2024-08-31T12:16:18.638 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.633+0000 7f0ed11ec740 1 -- 172.21.15.19:0/361805574 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55ee126c33b0 msgr2=0x55ee12a8ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:18.638 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.633+0000 7f0ed11ec740 1 --2- 172.21.15.19:0/361805574 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55ee126c33b0 0x55ee12a8ce60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0eb8009780 tx=0x7f0eb8082900 comp rx=0 tx=0).stop 2024-08-31T12:16:18.638 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.633+0000 7f0ed11ec740 1 -- 172.21.15.19:0/361805574 shutdown_connections 2024-08-31T12:16:18.638 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.633+0000 7f0ed11ec740 1 --2- 172.21.15.19:0/361805574 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee126cb590 0x55ee12a92070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.638 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.633+0000 7f0ed11ec740 1 --2- 172.21.15.19:0/361805574 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55ee126c33b0 0x55ee12a8ce60 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.638 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.633+0000 7f0ed11ec740 1 --2- 172.21.15.19:0/361805574 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x55ee126ec270 0x55ee12ad6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.638 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.633+0000 7f0ed11ec740 1 -- 172.21.15.19:0/361805574 >> 172.21.15.19:0/361805574 conn(0x55ee126e03c0 msgr2=0x55ee12ad5e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:18.638 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.633+0000 7f0ed11ec740 1 -- 172.21.15.19:0/361805574 shutdown_connections 2024-08-31T12:16:18.638 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.633+0000 7f0ed11ec740 1 -- 172.21.15.19:0/361805574 wait complete. 2024-08-31T12:16:18.639 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.634+0000 7f0ed11ec740 1 -- start start 2024-08-31T12:16:18.639 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.634+0000 7f0ed11ec740 1 Processor -- start 2024-08-31T12:16:18.639 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.635+0000 7f0ed11ec740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee126c33b0 0x55ee12ac2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:18.639 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.635+0000 7f0ed11ec740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55ee126cb590 0x55ee12ae5df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:18.639 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.635+0000 7f0ed11ec740 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x55ee126ec270 0x55ee12acbc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:18.639 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.635+0000 7f0ed11ec740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x55ee12766620 con 0x55ee126cb590 2024-08-31T12:16:18.639 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.635+0000 7f0ed11ec740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ee1297b620 con 0x55ee126c33b0 2024-08-31T12:16:18.639 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.635+0000 7f0ed11ec740 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x55ee12a05000 con 0x55ee126ec270 2024-08-31T12:16:18.639 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.635+0000 7f0ed11ec740 10 client.0 _ll_register_callbacks cb 0x55ee1299f0a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-31T12:16:18.639 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.635+0000 7f0ec2ffd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55ee126cb590 0x55ee12ae5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:18.639 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.635+0000 7f0ec2ffd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55ee126cb590 0x55ee12ae5df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:37908/0 (socket says 172.21.15.19:37908) 2024-08-31T12:16:18.639 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.635+0000 7f0ec2ffd640 1 -- 172.21.15.19:0/1199635080 learned_addr learned my addr 172.21.15.19:0/1199635080 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:18.640 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.635+0000 7f0ec3fff640 1 --2- 172.21.15.19:0/1199635080 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x55ee126ec270 0x55ee12acbc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:18.640 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.635+0000 7f0ec37fe640 1 --2- 172.21.15.19:0/1199635080 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee126c33b0 0x55ee12ac2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:18.640 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.635+0000 7f0ec2ffd640 1 -- 172.21.15.19:0/1199635080 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x55ee126ec270 msgr2=0x55ee12acbc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-31T12:16:18.640 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.635+0000 7f0ec2ffd640 1 --2- 172.21.15.19:0/1199635080 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x55ee126ec270 0x55ee12acbc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.640 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.635+0000 7f0ec2ffd640 1 -- 172.21.15.19:0/1199635080 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee126c33b0 msgr2=0x55ee12ac2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-31T12:16:18.640 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.635+0000 7f0ec2ffd640 1 --2- 172.21.15.19:0/1199635080 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee126c33b0 0x55ee12ac2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.640 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.635+0000 7f0ec2ffd640 1 -- 172.21.15.19:0/1199635080 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee127ce6b0 con 0x55ee126cb590 2024-08-31T12:16:18.640 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.635+0000 7f0ec37fe640 1 --2- 172.21.15.19:0/1199635080 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee126c33b0 0x55ee12ac2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:16:18.640 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.635+0000 7f0ec3fff640 1 --2- 172.21.15.19:0/1199635080 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x55ee126ec270 0x55ee12acbc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:16:18.640 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.635+0000 7f0ec2ffd640 1 --2- 172.21.15.19:0/1199635080 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55ee126cb590 0x55ee12ae5df0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0eb807f780 tx=0x7f0eb807f6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c717e94c494eb0ad server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:18.640 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.635+0000 7f0ea27fc640 10 client.0 ms_handle_connect on v2:172.21.15.19:3300/0 2024-08-31T12:16:18.640 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.636+0000 7f0ea27fc640 1 -- 172.21.15.19:0/1199635080 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb8083890 con 0x55ee126cb590 2024-08-31T12:16:18.640 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.636+0000 7f0ea27fc640 1 -- 172.21.15.19:0/1199635080 <== mon.0 v2:172.21.15.19:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0eb808bf90 con 0x55ee126cb590 2024-08-31T12:16:18.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.636+0000 7f0ed11ec740 1 -- 172.21.15.19:0/1199635080 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0eb8083190 con 0x55ee126cb590 2024-08-31T12:16:18.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.636+0000 7f0ea27fc640 1 -- 172.21.15.19:0/1199635080 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) ==== 24+0+0 (secure 0 0 0) 0x7f0eb80a3050 con 0x55ee126cb590 2024-08-31T12:16:18.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.636+0000 7f0ed11ec740 10 client.4758 fetch_fsmap learned FSMap version 12 2024-08-31T12:16:18.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.636+0000 7f0ed11ec740 1 -- 172.21.15.19:0/1199635080 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) -- 0x55ee12ac6d10 con 0x55ee126cb590 2024-08-31T12:16:18.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.636+0000 7f0ea27fc640 1 -- 172.21.15.19:0/1199635080 <== mon.0 v2:172.21.15.19:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0eb8083190 con 0x55ee126cb590 2024-08-31T12:16:18.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.636+0000 7f0ea27fc640 1 -- 172.21.15.19:0/1199635080 <== mon.0 v2:172.21.15.19:3300/0 5 ==== fsmap(e 12) ==== 4752+0+0 (secure 0 0 0) 0x7f0eb80b5020 con 0x55ee126cb590 2024-08-31T12:16:18.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.636+0000 7f0ed11ec740 10 client.4758 fetch_fsmap finished waiting for FSMap version 12 2024-08-31T12:16:18.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.636+0000 7f0ed11ec740 10 client.4758 resolve_mds: resolved name 'b' to daemon mds.b 2024-08-31T12:16:18.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.636+0000 7f0ed11ec740 20 client.4758 populate_metadata read hostname 'smithi019' 2024-08-31T12:16:18.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.636+0000 7f0ed11ec740 1 --2- 172.21.15.19:0/1199635080 >> [v2:172.21.15.43:6836/1196858061,v1:172.21.15.43:6839/1196858061] conn(0x55ee126aae40 0x55ee12ac52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:18.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.636+0000 7f0ed11ec740 4 client.4758 mds_command: new command op to 4426 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-31T12:16:18.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.636+0000 7f0ed11ec740 1 -- 172.21.15.19:0/1199635080 --> [v2:172.21.15.43:6836/1196858061,v1:172.21.15.43:6839/1196858061] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee12a06f50 con 0x55ee126aae40 2024-08-31T12:16:18.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.636+0000 7f0ec37fe640 1 --2- 172.21.15.19:0/1199635080 >> [v2:172.21.15.43:6836/1196858061,v1:172.21.15.43:6839/1196858061] conn(0x55ee126aae40 0x55ee12ac52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:18.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.637+0000 7f0ec37fe640 1 --2- 172.21.15.19:0/1199635080 >> [v2:172.21.15.43:6836/1196858061,v1:172.21.15.43:6839/1196858061] conn(0x55ee126aae40 0x55ee12ac52a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=414f20dfdeb328b1 server_cookie=7df5d67135cddac2 in_seq=0 out_seq=0 2024-08-31T12:16:18.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.637+0000 7f0ea27fc640 10 client.4758 ms_handle_connect on v2:172.21.15.43:6836/1196858061 2024-08-31T12:16:18.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.638+0000 7f0ea27fc640 1 -- 172.21.15.19:0/1199635080 <== mds.1 v2:172.21.15.43:6836/1196858061 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55ee12a06f50 con 0x55ee126aae40 2024-08-31T12:16:18.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.638+0000 7f0ea27fc640 10 client.4758 handle_command_reply: tid=0 2024-08-31T12:16:18.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.638+0000 7f0ed11ec740 1 client.4758 shutdown 2024-08-31T12:16:18.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.638+0000 7f0ed11ec740 1 -- 172.21.15.19:0/1199635080 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55ee126cb590 msgr2=0x55ee12ae5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:18.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.638+0000 7f0ed11ec740 1 --2- 172.21.15.19:0/1199635080 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55ee126cb590 0x55ee12ae5df0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0eb807f780 tx=0x7f0eb807f6b0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.638+0000 7f0ed11ec740 1 -- 172.21.15.19:0/1199635080 shutdown_connections 2024-08-31T12:16:18.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.638+0000 7f0ed11ec740 1 --2- 172.21.15.19:0/1199635080 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x55ee126ec270 0x55ee12acbc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.638+0000 7f0ed11ec740 1 --2- 172.21.15.19:0/1199635080 >> [v2:172.21.15.43:6836/1196858061,v1:172.21.15.43:6839/1196858061] conn(0x55ee126aae40 0x55ee12ac52a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.638+0000 7f0ed11ec740 1 --2- 172.21.15.19:0/1199635080 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55ee126cb590 0x55ee12ae5df0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.638+0000 7f0ed11ec740 1 --2- 172.21.15.19:0/1199635080 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee126c33b0 0x55ee12ac2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.638+0000 7f0ed11ec740 1 -- 172.21.15.19:0/1199635080 >> 172.21.15.19:0/1199635080 conn(0x55ee126e03c0 msgr2=0x55ee12ad5d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:18.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.638+0000 7f0ea27fc640 0 client.4758 ms_handle_reset on v2:172.21.15.43:6836/1196858061 2024-08-31T12:16:18.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.638+0000 7f0ed11ec740 1 -- 172.21.15.19:0/1199635080 shutdown_connections 2024-08-31T12:16:18.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.638+0000 7f0ed11ec740 1 -- 172.21.15.19:0/1199635080 wait complete. 2024-08-31T12:16:18.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.638+0000 7f0ed11ec740 20 client.4758 trim_cache size 0 max 16384 2024-08-31T12:16:18.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.657+0000 7f0ed11ec740 1 Processor -- start 2024-08-31T12:16:18.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.657+0000 7f0ed11ec740 1 -- start start 2024-08-31T12:16:18.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.658+0000 7f0ed11ec740 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x55ee126c33b0 0x55ee12a936a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:18.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.658+0000 7f0ed11ec740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee126cb590 0x55ee12a93be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:18.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.658+0000 7f0ed11ec740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55ee126ec270 0x55ee12ae2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:18.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.658+0000 7f0ed11ec740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec41529a0 con 0x55ee126ec270 2024-08-31T12:16:18.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.658+0000 7f0ed11ec740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ee12a05000 con 0x55ee126cb590 2024-08-31T12:16:18.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.658+0000 7f0ed11ec740 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x55ee12766620 con 0x55ee126c33b0 2024-08-31T12:16:18.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.658+0000 7f0ec3fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55ee126ec270 0x55ee12ae2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:18.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.658+0000 7f0ec3fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55ee126ec270 0x55ee12ae2150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:37910/0 (socket says 172.21.15.19:37910) 2024-08-31T12:16:18.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.658+0000 7f0ec3fff640 1 -- 172.21.15.19:0/4022341024 learned_addr learned my addr 172.21.15.19:0/4022341024 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:18.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.658+0000 7f0ec2ffd640 1 --2- 172.21.15.19:0/4022341024 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee126cb590 0x55ee12a93be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:18.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.658+0000 7f0ec37fe640 1 --2- 172.21.15.19:0/4022341024 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x55ee126c33b0 0x55ee12a936a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:18.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.658+0000 7f0ec3fff640 1 -- 172.21.15.19:0/4022341024 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x55ee126c33b0 msgr2=0x55ee12a936a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-31T12:16:18.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.658+0000 7f0ec3fff640 1 --2- 172.21.15.19:0/4022341024 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x55ee126c33b0 0x55ee12a936a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.658+0000 7f0ec3fff640 1 -- 172.21.15.19:0/4022341024 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee126cb590 msgr2=0x55ee12a93be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-31T12:16:18.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.658+0000 7f0ec3fff640 1 --2- 172.21.15.19:0/4022341024 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee126cb590 0x55ee12a93be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.659+0000 7f0ec3fff640 1 -- 172.21.15.19:0/4022341024 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee12a90ab0 con 0x55ee126ec270 2024-08-31T12:16:18.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.659+0000 7f0ec3fff640 1 --2- 172.21.15.19:0/4022341024 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55ee126ec270 0x55ee12ae2150 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc009f50 tx=0x7f0ebc012880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97b01687747040ce server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:18.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.659+0000 7f0ea8ff9640 1 -- 172.21.15.19:0/4022341024 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ebc004b20 con 0x55ee126ec270 2024-08-31T12:16:18.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.659+0000 7f0ea8ff9640 1 -- 172.21.15.19:0/4022341024 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0ebc004cc0 con 0x55ee126ec270 2024-08-31T12:16:18.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.659+0000 7f0ea8ff9640 1 -- 172.21.15.19:0/4022341024 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ebc005280 con 0x55ee126ec270 2024-08-31T12:16:18.665 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.660+0000 7f0ed11ec740 1 -- 172.21.15.19:0/4022341024 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55ee126ec270 msgr2=0x55ee12ae2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:18.666 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.660+0000 7f0ed11ec740 1 --2- 172.21.15.19:0/4022341024 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55ee126ec270 0x55ee12ae2150 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc009f50 tx=0x7f0ebc012880 comp rx=0 tx=0).stop 2024-08-31T12:16:18.666 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.660+0000 7f0ed11ec740 1 -- 172.21.15.19:0/4022341024 shutdown_connections 2024-08-31T12:16:18.666 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.660+0000 7f0ed11ec740 1 --2- 172.21.15.19:0/4022341024 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55ee126ec270 0x55ee12ae2150 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.666 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.660+0000 7f0ed11ec740 1 --2- 172.21.15.19:0/4022341024 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee126cb590 0x55ee12a93be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.666 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.660+0000 7f0ed11ec740 1 --2- 172.21.15.19:0/4022341024 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x55ee126c33b0 0x55ee12a936a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.666 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.660+0000 7f0ed11ec740 1 -- 172.21.15.19:0/4022341024 >> 172.21.15.19:0/4022341024 conn(0x55ee126e03c0 msgr2=0x55ee12ae5dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:18.666 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.660+0000 7f0ed11ec740 1 -- 172.21.15.19:0/4022341024 shutdown_connections 2024-08-31T12:16:18.666 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.660+0000 7f0ed11ec740 1 -- 172.21.15.19:0/4022341024 wait complete. 2024-08-31T12:16:18.666 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.660+0000 7f0ed11ec740 1 -- start start 2024-08-31T12:16:18.666 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.661+0000 7f0ed11ec740 1 Processor -- start 2024-08-31T12:16:18.666 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.662+0000 7f0ed11ec740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55ee126c33b0 0x55ee12a90f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:18.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.662+0000 7f0ed11ec740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee126cb590 0x55ee12b4c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:18.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.662+0000 7f0ed11ec740 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x55ee126ec270 0x55ee12b4cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:18.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.662+0000 7f0ed11ec740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x55ee12ac6330 con 0x55ee126c33b0 2024-08-31T12:16:18.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.662+0000 7f0ed11ec740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ee1297b620 con 0x55ee126cb590 2024-08-31T12:16:18.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.662+0000 7f0ed11ec740 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x55ee12a8c320 con 0x55ee126ec270 2024-08-31T12:16:18.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.662+0000 7f0ed11ec740 10 client.0 _ll_register_callbacks cb 0x7f0ebc009540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-31T12:16:18.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.662+0000 7f0ec37fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55ee126c33b0 0x55ee12a90f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:18.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.662+0000 7f0ec37fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55ee126c33b0 0x55ee12a90f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:37916/0 (socket says 172.21.15.19:37916) 2024-08-31T12:16:18.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.662+0000 7f0ec37fe640 1 -- 172.21.15.19:0/2900296966 learned_addr learned my addr 172.21.15.19:0/2900296966 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:18.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.662+0000 7f0ec3fff640 1 --2- 172.21.15.19:0/2900296966 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x55ee126ec270 0x55ee12b4cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:18.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.662+0000 7f0ec2ffd640 1 --2- 172.21.15.19:0/2900296966 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee126cb590 0x55ee12b4c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:18.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.662+0000 7f0ec37fe640 1 -- 172.21.15.19:0/2900296966 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x55ee126ec270 msgr2=0x55ee12b4cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-31T12:16:18.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.662+0000 7f0ec37fe640 1 --2- 172.21.15.19:0/2900296966 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x55ee126ec270 0x55ee12b4cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.662+0000 7f0ec37fe640 1 -- 172.21.15.19:0/2900296966 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee126cb590 msgr2=0x55ee12b4c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-31T12:16:18.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.662+0000 7f0ec37fe640 1 --2- 172.21.15.19:0/2900296966 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee126cb590 0x55ee12b4c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.662+0000 7f0ec37fe640 1 -- 172.21.15.19:0/2900296966 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee12ac9cb0 con 0x55ee126c33b0 2024-08-31T12:16:18.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.663+0000 7f0ec37fe640 1 --2- 172.21.15.19:0/2900296966 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55ee126c33b0 0x55ee12a90f90 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0eb8005eb0 tx=0x7f0eb4000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c228bfeb7a005439 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:18.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.663+0000 7f0ea27fc640 10 client.0 ms_handle_connect on v2:172.21.15.19:3300/0 2024-08-31T12:16:18.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.663+0000 7f0ea27fc640 1 -- 172.21.15.19:0/2900296966 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb402f070 con 0x55ee126c33b0 2024-08-31T12:16:18.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.663+0000 7f0ea27fc640 1 -- 172.21.15.19:0/2900296966 <== mon.0 v2:172.21.15.19:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0eb4010040 con 0x55ee126c33b0 2024-08-31T12:16:18.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.663+0000 7f0ed11ec740 1 -- 172.21.15.19:0/2900296966 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ebc004cc0 con 0x55ee126c33b0 2024-08-31T12:16:18.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.663+0000 7f0ea27fc640 1 -- 172.21.15.19:0/2900296966 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) ==== 24+0+0 (secure 0 0 0) 0x7f0eb403a050 con 0x55ee126c33b0 2024-08-31T12:16:18.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.663+0000 7f0ed11ec740 10 client.4764 fetch_fsmap learned FSMap version 12 2024-08-31T12:16:18.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.663+0000 7f0ed11ec740 1 -- 172.21.15.19:0/2900296966 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) -- 0x55ee12a8fbf0 con 0x55ee126c33b0 2024-08-31T12:16:18.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.663+0000 7f0ea27fc640 1 -- 172.21.15.19:0/2900296966 <== mon.0 v2:172.21.15.19:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0ebc004cc0 con 0x55ee126c33b0 2024-08-31T12:16:18.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.663+0000 7f0ea27fc640 1 -- 172.21.15.19:0/2900296966 <== mon.0 v2:172.21.15.19:3300/0 5 ==== fsmap(e 12) ==== 4752+0+0 (secure 0 0 0) 0x7f0eb403d020 con 0x55ee126c33b0 2024-08-31T12:16:18.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.663+0000 7f0ed11ec740 10 client.4764 fetch_fsmap finished waiting for FSMap version 12 2024-08-31T12:16:18.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.663+0000 7f0ed11ec740 10 client.4764 resolve_mds: resolved name 'b' to daemon mds.b 2024-08-31T12:16:18.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.663+0000 7f0ed11ec740 20 client.4764 populate_metadata read hostname 'smithi019' 2024-08-31T12:16:18.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.663+0000 7f0ed11ec740 1 --2- 172.21.15.19:0/2900296966 >> [v2:172.21.15.43:6836/1196858061,v1:172.21.15.43:6839/1196858061] conn(0x55ee12b536f0 0x55ee12aca7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:18.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.663+0000 7f0ed11ec740 4 client.4764 mds_command: new command op to 4426 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-31T12:16:18.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.663+0000 7f0ed11ec740 1 -- 172.21.15.19:0/2900296966 --> [v2:172.21.15.43:6836/1196858061,v1:172.21.15.43:6839/1196858061] -- command(tid 0: {"prefix": "status"}) -- 0x55ee12a029f0 con 0x55ee12b536f0 2024-08-31T12:16:18.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.663+0000 7f0ec2ffd640 1 --2- 172.21.15.19:0/2900296966 >> [v2:172.21.15.43:6836/1196858061,v1:172.21.15.43:6839/1196858061] conn(0x55ee12b536f0 0x55ee12aca7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:18.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.664+0000 7f0ea27fc640 10 client.4764 ms_handle_connect on v2:172.21.15.43:6836/1196858061 2024-08-31T12:16:18.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.664+0000 7f0ec2ffd640 1 --2- 172.21.15.19:0/2900296966 >> [v2:172.21.15.43:6836/1196858061,v1:172.21.15.43:6839/1196858061] conn(0x55ee12b536f0 0x55ee12aca7b0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=743940dae4be8fc4 server_cookie=9d91d8b72e10f8b7 in_seq=0 out_seq=0 2024-08-31T12:16:18.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.664+0000 7f0ea27fc640 1 -- 172.21.15.19:0/2900296966 <== mds.1 v2:172.21.15.43:6836/1196858061 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55ee12a029f0 con 0x55ee12b536f0 2024-08-31T12:16:18.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.664+0000 7f0ea27fc640 10 client.4764 handle_command_reply: tid=0 2024-08-31T12:16:18.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.664+0000 7f0ed11ec740 1 client.4764 shutdown 2024-08-31T12:16:18.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.664+0000 7f0ed11ec740 1 -- 172.21.15.19:0/2900296966 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55ee126c33b0 msgr2=0x55ee12a90f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:18.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.664+0000 7f0ed11ec740 1 --2- 172.21.15.19:0/2900296966 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55ee126c33b0 0x55ee12a90f90 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0eb8005eb0 tx=0x7f0eb4000ed0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.664+0000 7f0ed11ec740 1 -- 172.21.15.19:0/2900296966 shutdown_connections 2024-08-31T12:16:18.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.664+0000 7f0ed11ec740 1 --2- 172.21.15.19:0/2900296966 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x55ee126ec270 0x55ee12b4cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.664+0000 7f0ed11ec740 1 --2- 172.21.15.19:0/2900296966 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee126cb590 0x55ee12b4c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.664+0000 7f0ed11ec740 1 --2- 172.21.15.19:0/2900296966 >> [v2:172.21.15.43:6836/1196858061,v1:172.21.15.43:6839/1196858061] conn(0x55ee12b536f0 0x55ee12aca7b0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.664+0000 7f0ed11ec740 1 --2- 172.21.15.19:0/2900296966 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55ee126c33b0 0x55ee12a90f90 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.664+0000 7f0ed11ec740 1 -- 172.21.15.19:0/2900296966 >> 172.21.15.19:0/2900296966 conn(0x55ee126e03c0 msgr2=0x55ee12acb3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:18.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.664+0000 7f0ea27fc640 0 client.4764 ms_handle_reset on v2:172.21.15.43:6836/1196858061 2024-08-31T12:16:18.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.664+0000 7f0ed11ec740 1 -- 172.21.15.19:0/2900296966 shutdown_connections 2024-08-31T12:16:18.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.664+0000 7f0ed11ec740 1 -- 172.21.15.19:0/2900296966 wait complete. 2024-08-31T12:16:18.671 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.664+0000 7f0ed11ec740 20 client.4764 trim_cache size 0 max 16384 2024-08-31T12:16:18.671 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:16:18.671 INFO:teuthology.orchestra.run.smithi019.stdout: "cluster_fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:16:18.671 INFO:teuthology.orchestra.run.smithi019.stdout: "whoami": 1, 2024-08-31T12:16:18.671 INFO:teuthology.orchestra.run.smithi019.stdout: "id": 4426, 2024-08-31T12:16:18.671 INFO:teuthology.orchestra.run.smithi019.stdout: "want_state": "up:active", 2024-08-31T12:16:18.671 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "up:active", 2024-08-31T12:16:18.671 INFO:teuthology.orchestra.run.smithi019.stdout: "fs_name": "cephfs", 2024-08-31T12:16:18.671 INFO:teuthology.orchestra.run.smithi019.stdout: "rank_uptime": 16.342816235000001, 2024-08-31T12:16:18.671 INFO:teuthology.orchestra.run.smithi019.stdout: "mdsmap_epoch": 12, 2024-08-31T12:16:18.672 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_epoch": 21, 2024-08-31T12:16:18.672 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_epoch_barrier": 21, 2024-08-31T12:16:18.672 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 27.443767647000001 2024-08-31T12:16:18.672 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:16:18.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.665+0000 7f0eca19f640 1 -- 172.21.15.19:0/3952515220 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0eb0077a80 msgr2=0x7f0eb0079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:18.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.665+0000 7f0eca19f640 1 --2- 172.21.15.19:0/3952515220 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0eb0077a80 0x7f0eb0079f90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0eb4007fd0 tx=0x7f0eb4009040 comp rx=0 tx=0).stop 2024-08-31T12:16:18.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.665+0000 7f0eca19f640 1 -- 172.21.15.19:0/3952515220 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0ec415bbd0 msgr2=0x7f0ec41f2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:18.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.665+0000 7f0eca19f640 1 --2- 172.21.15.19:0/3952515220 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0ec415bbd0 0x7f0ec41f2480 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0eb8000c00 tx=0x7f0eb8033c90 comp rx=0 tx=0).stop 2024-08-31T12:16:18.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.665+0000 7f0eca19f640 1 -- 172.21.15.19:0/3952515220 shutdown_connections 2024-08-31T12:16:18.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.665+0000 7f0eca19f640 1 --2- 172.21.15.19:0/3952515220 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0eb0077a80 0x7f0eb0079f90 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.665+0000 7f0eca19f640 1 --2- 172.21.15.19:0/3952515220 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0ec415f350 0x7f0ec40ca040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.665+0000 7f0eca19f640 1 --2- 172.21.15.19:0/3952515220 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0ec415bbd0 0x7f0ec41f2480 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.665+0000 7f0eca19f640 1 --2- 172.21.15.19:0/3952515220 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ec40cb2a0 0x7f0ec41583c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.665+0000 7f0eca19f640 1 -- 172.21.15.19:0/3952515220 >> 172.21.15.19:0/3952515220 conn(0x7f0ec406dba0 msgr2=0x7f0ec4151560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:18.674 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.665+0000 7f0eca19f640 1 -- 172.21.15.19:0/3952515220 shutdown_connections 2024-08-31T12:16:18.674 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.665+0000 7f0eca19f640 1 -- 172.21.15.19:0/3952515220 wait complete. 2024-08-31T12:16:18.677 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-08-31T12:16:18.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.790+0000 7f2494aab640 1 -- 172.21.15.19:0/3172485207 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f24900c8a40 msgr2=0x7f24900c9850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:18.792 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.790+0000 7f2494aab640 1 --2- 172.21.15.19:0/3172485207 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f24900c8a40 0x7f24900c9850 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2488005fd0 tx=0x7f248802a710 comp rx=0 tx=0).stop 2024-08-31T12:16:18.792 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.790+0000 7f2494aab640 1 -- 172.21.15.19:0/3172485207 shutdown_connections 2024-08-31T12:16:18.792 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.790+0000 7f2494aab640 1 --2- 172.21.15.19:0/3172485207 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f24900c8a40 0x7f24900c9850 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.792 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.790+0000 7f2494aab640 1 -- 172.21.15.19:0/3172485207 >> 172.21.15.19:0/3172485207 conn(0x7f24900690d0 msgr2=0x7f24900d0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:18.792 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.790+0000 7f2494aab640 1 -- 172.21.15.19:0/3172485207 shutdown_connections 2024-08-31T12:16:18.793 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.790+0000 7f2494aab640 1 -- 172.21.15.19:0/3172485207 wait complete. 2024-08-31T12:16:18.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.792+0000 7f2494aab640 1 Processor -- start 2024-08-31T12:16:18.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.792+0000 7f2494aab640 1 -- start start 2024-08-31T12:16:18.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.793+0000 7f2494aab640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f249006e0e0 0x7f24900c5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:18.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.793+0000 7f2494aab640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f24900c8a40 0x7f249014e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:18.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.793+0000 7f2494aab640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2490155300 0x7f24901e5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:18.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.793+0000 7f2494aab640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f249015b250 con 0x7f249006e0e0 2024-08-31T12:16:18.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.793+0000 7f2494aab640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f249015af70 con 0x7f2490155300 2024-08-31T12:16:18.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.793+0000 7f2494aab640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f24900c8610 con 0x7f24900c8a40 2024-08-31T12:16:18.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.793+0000 7f248e59c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f249006e0e0 0x7f24900c5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:18.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.793+0000 7f248e59c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f249006e0e0 0x7f24900c5750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:37936/0 (socket says 172.21.15.19:37936) 2024-08-31T12:16:18.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.793+0000 7f248e59c640 1 -- 172.21.15.19:0/4089939587 learned_addr learned my addr 172.21.15.19:0/4089939587 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:18.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.793+0000 7f248dd9b640 1 --2- 172.21.15.19:0/4089939587 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f24900c8a40 0x7f249014e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:18.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.793+0000 7f248ed9d640 1 --2- 172.21.15.19:0/4089939587 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2490155300 0x7f24901e5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:18.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.793+0000 7f248e59c640 1 -- 172.21.15.19:0/4089939587 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f24900c8a40 msgr2=0x7f249014e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:18.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.793+0000 7f248e59c640 1 --2- 172.21.15.19:0/4089939587 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f24900c8a40 0x7f249014e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.793+0000 7f248e59c640 1 -- 172.21.15.19:0/4089939587 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2490155300 msgr2=0x7f24901e5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:18.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.793+0000 7f248e59c640 1 --2- 172.21.15.19:0/4089939587 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2490155300 0x7f24901e5e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.793+0000 7f248e59c640 1 -- 172.21.15.19:0/4089939587 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f249014ecf0 con 0x7f249006e0e0 2024-08-31T12:16:18.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.794+0000 7f248e59c640 1 --2- 172.21.15.19:0/4089939587 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f249006e0e0 0x7f24900c5750 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2488006010 tx=0x7f24880039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:18.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.794+0000 7f2486ffd640 1 -- 172.21.15.19:0/4089939587 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f248803e030 con 0x7f249006e0e0 2024-08-31T12:16:18.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.794+0000 7f2486ffd640 1 -- 172.21.15.19:0/4089939587 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f24880047a0 con 0x7f249006e0e0 2024-08-31T12:16:18.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.794+0000 7f2494aab640 1 -- 172.21.15.19:0/4089939587 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24900c4a20 con 0x7f249006e0e0 2024-08-31T12:16:18.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.794+0000 7f2486ffd640 1 -- 172.21.15.19:0/4089939587 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2488035040 con 0x7f249006e0e0 2024-08-31T12:16:18.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.794+0000 7f2494aab640 1 -- 172.21.15.19:0/4089939587 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24900cfed0 con 0x7f249006e0e0 2024-08-31T12:16:18.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.796+0000 7f2486ffd640 1 -- 172.21.15.19:0/4089939587 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f2488041030 con 0x7f249006e0e0 2024-08-31T12:16:18.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.796+0000 7f2486ffd640 1 --2- 172.21.15.19:0/4089939587 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2470077a50 0x7f2470079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:18.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.796+0000 7f2486ffd640 1 -- 172.21.15.19:0/4089939587 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f24880b9e10 con 0x7f249006e0e0 2024-08-31T12:16:18.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.796+0000 7f248dd9b640 1 --2- 172.21.15.19:0/4089939587 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2470077a50 0x7f2470079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:18.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.797+0000 7f248dd9b640 1 --2- 172.21.15.19:0/4089939587 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2470077a50 0x7f2470079f60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2478000f60 tx=0x7f2478009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:18.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.797+0000 7f249ba16740 1 Processor -- start 2024-08-31T12:16:18.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.797+0000 7f249ba16740 1 -- start start 2024-08-31T12:16:18.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.798+0000 7f249ba16740 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x55b412173270 0x55b41255d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:18.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.798+0000 7f249ba16740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b41214a3b0 0x55b412513e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:18.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.798+0000 7f249ba16740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b412152590 0x55b412519070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:18.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.798+0000 7f249ba16740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x55b41248c000 con 0x55b41214a3b0 2024-08-31T12:16:18.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.798+0000 7f249ba16740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55b41247b590 con 0x55b412152590 2024-08-31T12:16:18.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.798+0000 7f249ba16740 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x55b41247f540 con 0x55b412173270 2024-08-31T12:16:18.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.798+0000 7f248dd9b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b41214a3b0 0x55b412513e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:18.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.798+0000 7f248dd9b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b41214a3b0 0x55b412513e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:37940/0 (socket says 172.21.15.19:37940) 2024-08-31T12:16:18.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.798+0000 7f248dd9b640 1 -- 172.21.15.19:0/3878188308 learned_addr learned my addr 172.21.15.19:0/3878188308 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:18.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.799+0000 7f248e59c640 1 --2- 172.21.15.19:0/3878188308 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x55b412173270 0x55b41255d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:18.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.799+0000 7f248ed9d640 1 --2- 172.21.15.19:0/3878188308 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b412152590 0x55b412519070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:18.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.799+0000 7f248dd9b640 1 -- 172.21.15.19:0/3878188308 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x55b412173270 msgr2=0x55b41255d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-31T12:16:18.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.799+0000 7f248dd9b640 1 --2- 172.21.15.19:0/3878188308 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x55b412173270 0x55b41255d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.799+0000 7f248dd9b640 1 -- 172.21.15.19:0/3878188308 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b412152590 msgr2=0x55b412519070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-31T12:16:18.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.799+0000 7f248dd9b640 1 --2- 172.21.15.19:0/3878188308 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b412152590 0x55b412519070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.799+0000 7f248dd9b640 1 -- 172.21.15.19:0/3878188308 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4121de850 con 0x55b41214a3b0 2024-08-31T12:16:18.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.799+0000 7f248dd9b640 1 --2- 172.21.15.19:0/3878188308 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b41214a3b0 0x55b412513e60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f247800f990 tx=0x7f2478012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d892e614c636afb6 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:18.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.799+0000 7f246ffff640 1 -- 172.21.15.19:0/3878188308 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2478025070 con 0x55b41214a3b0 2024-08-31T12:16:18.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.799+0000 7f246ffff640 1 -- 172.21.15.19:0/3878188308 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f247801b050 con 0x55b41214a3b0 2024-08-31T12:16:18.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.799+0000 7f246ffff640 1 -- 172.21.15.19:0/3878188308 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2478014e20 con 0x55b41214a3b0 2024-08-31T12:16:18.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.800+0000 7f249ba16740 1 -- 172.21.15.19:0/3878188308 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b41214a3b0 msgr2=0x55b412513e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:18.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.800+0000 7f249ba16740 1 --2- 172.21.15.19:0/3878188308 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b41214a3b0 0x55b412513e60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f247800f990 tx=0x7f2478012000 comp rx=0 tx=0).stop 2024-08-31T12:16:18.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.800+0000 7f249ba16740 1 -- 172.21.15.19:0/3878188308 shutdown_connections 2024-08-31T12:16:18.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.800+0000 7f249ba16740 1 --2- 172.21.15.19:0/3878188308 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b412152590 0x55b412519070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.800+0000 7f249ba16740 1 --2- 172.21.15.19:0/3878188308 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b41214a3b0 0x55b412513e60 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.800+0000 7f249ba16740 1 --2- 172.21.15.19:0/3878188308 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x55b412173270 0x55b41255d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.800+0000 7f249ba16740 1 -- 172.21.15.19:0/3878188308 >> 172.21.15.19:0/3878188308 conn(0x55b4121673c0 msgr2=0x55b41255ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:18.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.800+0000 7f249ba16740 1 -- 172.21.15.19:0/3878188308 shutdown_connections 2024-08-31T12:16:18.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.800+0000 7f249ba16740 1 -- 172.21.15.19:0/3878188308 wait complete. 2024-08-31T12:16:18.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.801+0000 7f249ba16740 1 -- start start 2024-08-31T12:16:18.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.801+0000 7f249ba16740 1 Processor -- start 2024-08-31T12:16:18.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.802+0000 7f249ba16740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b41214a3b0 0x55b412551890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:18.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.802+0000 7f249ba16740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b412152590 0x55b412559610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:18.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.802+0000 7f249ba16740 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x55b412173270 0x55b412559b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:18.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.802+0000 7f249ba16740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x55b4121ed620 con 0x55b41214a3b0 2024-08-31T12:16:18.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.802+0000 7f249ba16740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55b412402620 con 0x55b412152590 2024-08-31T12:16:18.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.802+0000 7f249ba16740 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x55b41256cd70 con 0x55b412173270 2024-08-31T12:16:18.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.802+0000 7f249ba16740 10 client.0 _ll_register_callbacks cb 0x55b4124260a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-31T12:16:18.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.803+0000 7f248e59c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b41214a3b0 0x55b412551890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:18.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.803+0000 7f248e59c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b41214a3b0 0x55b412551890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:37952/0 (socket says 172.21.15.19:37952) 2024-08-31T12:16:18.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.803+0000 7f248e59c640 1 -- 172.21.15.19:0/3909990965 learned_addr learned my addr 172.21.15.19:0/3909990965 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:18.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.803+0000 7f248dd9b640 1 --2- 172.21.15.19:0/3909990965 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b412152590 0x55b412559610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:18.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.803+0000 7f248ed9d640 1 --2- 172.21.15.19:0/3909990965 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x55b412173270 0x55b412559b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:18.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.803+0000 7f248e59c640 1 -- 172.21.15.19:0/3909990965 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x55b412173270 msgr2=0x55b412559b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-31T12:16:18.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.803+0000 7f248e59c640 1 --2- 172.21.15.19:0/3909990965 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x55b412173270 0x55b412559b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.803+0000 7f248e59c640 1 -- 172.21.15.19:0/3909990965 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b412152590 msgr2=0x55b412559610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-31T12:16:18.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.803+0000 7f248e59c640 1 --2- 172.21.15.19:0/3909990965 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b412152590 0x55b412559610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.803+0000 7f248e59c640 1 -- 172.21.15.19:0/3909990965 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4122556b0 con 0x55b41214a3b0 2024-08-31T12:16:18.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.803+0000 7f248e59c640 1 --2- 172.21.15.19:0/3909990965 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b41214a3b0 0x55b412551890 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2488005820 tx=0x7f2488081fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e52664a7629e4a75 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:18.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.803+0000 7f246d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.19:3300/0 2024-08-31T12:16:18.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.803+0000 7f246d7fa640 1 -- 172.21.15.19:0/3909990965 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2488048040 con 0x55b41214a3b0 2024-08-31T12:16:18.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.803+0000 7f246d7fa640 1 -- 172.21.15.19:0/3909990965 <== mon.0 v2:172.21.15.19:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f248808b310 con 0x55b41214a3b0 2024-08-31T12:16:18.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.803+0000 7f249ba16740 1 -- 172.21.15.19:0/3909990965 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f247801b050 con 0x55b41214a3b0 2024-08-31T12:16:18.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.803+0000 7f246d7fa640 1 -- 172.21.15.19:0/3909990965 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) ==== 24+0+0 (secure 0 0 0) 0x7f24880892c0 con 0x55b41214a3b0 2024-08-31T12:16:18.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.803+0000 7f249ba16740 10 client.4779 fetch_fsmap learned FSMap version 12 2024-08-31T12:16:18.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.803+0000 7f249ba16740 1 -- 172.21.15.19:0/3909990965 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) -- 0x55b412549e80 con 0x55b41214a3b0 2024-08-31T12:16:18.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.803+0000 7f246d7fa640 1 -- 172.21.15.19:0/3909990965 <== mon.0 v2:172.21.15.19:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f247801b050 con 0x55b41214a3b0 2024-08-31T12:16:18.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.804+0000 7f246d7fa640 1 -- 172.21.15.19:0/3909990965 <== mon.0 v2:172.21.15.19:3300/0 5 ==== fsmap(e 12) ==== 4752+0+0 (secure 0 0 0) 0x7f2488081820 con 0x55b41214a3b0 2024-08-31T12:16:18.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.804+0000 7f249ba16740 10 client.4779 fetch_fsmap finished waiting for FSMap version 12 2024-08-31T12:16:18.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.804+0000 7f249ba16740 10 client.4779 resolve_mds: resolved name 'd' to daemon mds.d 2024-08-31T12:16:18.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.804+0000 7f249ba16740 20 client.4779 populate_metadata read hostname 'smithi019' 2024-08-31T12:16:18.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.804+0000 7f249ba16740 1 --2- 172.21.15.19:0/3909990965 >> [v2:172.21.15.43:6834/3981040655,v1:172.21.15.43:6838/3981040655] conn(0x55b412131e40 0x55b412577c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:18.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.804+0000 7f249ba16740 4 client.4779 mds_command: new command op to 4429 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-08-31T12:16:18.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.804+0000 7f249ba16740 1 -- 172.21.15.19:0/3909990965 --> [v2:172.21.15.43:6834/3981040655,v1:172.21.15.43:6838/3981040655] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b41248df50 con 0x55b412131e40 2024-08-31T12:16:18.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.804+0000 7f248dd9b640 1 --2- 172.21.15.19:0/3909990965 >> [v2:172.21.15.43:6834/3981040655,v1:172.21.15.43:6838/3981040655] conn(0x55b412131e40 0x55b412577c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:18.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.804+0000 7f248dd9b640 1 --2- 172.21.15.19:0/3909990965 >> [v2:172.21.15.43:6834/3981040655,v1:172.21.15.43:6838/3981040655] conn(0x55b412131e40 0x55b412577c80 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6bda3afdd972983 server_cookie=fdcde09c3776d937 in_seq=0 out_seq=0 2024-08-31T12:16:18.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.804+0000 7f246d7fa640 10 client.4779 ms_handle_connect on v2:172.21.15.43:6834/3981040655 2024-08-31T12:16:18.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.805+0000 7f246d7fa640 1 -- 172.21.15.19:0/3909990965 <== mds.0 v2:172.21.15.43:6834/3981040655 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55b41248df50 con 0x55b412131e40 2024-08-31T12:16:18.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.805+0000 7f246d7fa640 10 client.4779 handle_command_reply: tid=0 2024-08-31T12:16:18.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.805+0000 7f249ba16740 1 client.4779 shutdown 2024-08-31T12:16:18.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.805+0000 7f249ba16740 1 -- 172.21.15.19:0/3909990965 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b41214a3b0 msgr2=0x55b412551890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:18.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.805+0000 7f249ba16740 1 --2- 172.21.15.19:0/3909990965 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b41214a3b0 0x55b412551890 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2488005820 tx=0x7f2488081fd0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.806+0000 7f249ba16740 1 -- 172.21.15.19:0/3909990965 shutdown_connections 2024-08-31T12:16:18.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.806+0000 7f249ba16740 1 --2- 172.21.15.19:0/3909990965 >> [v2:172.21.15.43:6834/3981040655,v1:172.21.15.43:6838/3981040655] conn(0x55b412131e40 0x55b412577c80 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.806+0000 7f249ba16740 1 --2- 172.21.15.19:0/3909990965 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x55b412173270 0x55b412559b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.806+0000 7f249ba16740 1 --2- 172.21.15.19:0/3909990965 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b412152590 0x55b412559610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.806+0000 7f249ba16740 1 --2- 172.21.15.19:0/3909990965 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b41214a3b0 0x55b412551890 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.806+0000 7f246d7fa640 0 client.4779 ms_handle_reset on v2:172.21.15.43:6834/3981040655 2024-08-31T12:16:18.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.806+0000 7f249ba16740 1 -- 172.21.15.19:0/3909990965 >> 172.21.15.19:0/3909990965 conn(0x55b4121673c0 msgr2=0x55b41251a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:18.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.806+0000 7f249ba16740 1 -- 172.21.15.19:0/3909990965 shutdown_connections 2024-08-31T12:16:18.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.806+0000 7f249ba16740 1 -- 172.21.15.19:0/3909990965 wait complete. 2024-08-31T12:16:18.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.806+0000 7f249ba16740 20 client.4779 trim_cache size 0 max 16384 2024-08-31T12:16:18.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.824+0000 7f249ba16740 1 Processor -- start 2024-08-31T12:16:18.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.824+0000 7f249ba16740 1 -- start start 2024-08-31T12:16:18.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.825+0000 7f249ba16740 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x55b41214a3b0 0x55b4125595b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:18.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.825+0000 7f249ba16740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b412152590 0x55b412548a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:18.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.825+0000 7f249ba16740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b412173270 0x55b412548f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:18.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.825+0000 7f249ba16740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f24900ceb60 con 0x55b412173270 2024-08-31T12:16:18.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.825+0000 7f249ba16740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55b41254dc30 con 0x55b412152590 2024-08-31T12:16:18.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.825+0000 7f249ba16740 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x55b412551f10 con 0x55b41214a3b0 2024-08-31T12:16:18.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.826+0000 7f248ed9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b412173270 0x55b412548f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:18.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.826+0000 7f248ed9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b412173270 0x55b412548f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:37968/0 (socket says 172.21.15.19:37968) 2024-08-31T12:16:18.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.826+0000 7f248ed9d640 1 -- 172.21.15.19:0/1286893817 learned_addr learned my addr 172.21.15.19:0/1286893817 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:18.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.826+0000 7f248dd9b640 1 --2- 172.21.15.19:0/1286893817 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b412152590 0x55b412548a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:18.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.826+0000 7f248e59c640 1 --2- 172.21.15.19:0/1286893817 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x55b41214a3b0 0x55b4125595b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:18.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.826+0000 7f248ed9d640 1 -- 172.21.15.19:0/1286893817 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x55b41214a3b0 msgr2=0x55b4125595b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-31T12:16:18.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.826+0000 7f248ed9d640 1 --2- 172.21.15.19:0/1286893817 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x55b41214a3b0 0x55b4125595b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.826+0000 7f248ed9d640 1 -- 172.21.15.19:0/1286893817 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b412152590 msgr2=0x55b412548a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-31T12:16:18.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.826+0000 7f248ed9d640 1 --2- 172.21.15.19:0/1286893817 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b412152590 0x55b412548a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.826+0000 7f248ed9d640 1 -- 172.21.15.19:0/1286893817 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b412517a20 con 0x55b412173270 2024-08-31T12:16:18.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.826+0000 7f248dd9b640 1 --2- 172.21.15.19:0/1286893817 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b412152590 0x55b412548a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-31T12:16:18.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.826+0000 7f248e59c640 1 --2- 172.21.15.19:0/1286893817 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x55b41214a3b0 0x55b4125595b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-31T12:16:18.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.826+0000 7f248ed9d640 1 --2- 172.21.15.19:0/1286893817 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b412173270 0x55b412548f80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2480001ff0 tx=0x7f2480001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89f511d0c8f730d2 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:18.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.826+0000 7f246f7fe640 1 -- 172.21.15.19:0/1286893817 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2480015e60 con 0x55b412173270 2024-08-31T12:16:18.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.826+0000 7f246f7fe640 1 -- 172.21.15.19:0/1286893817 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2480003c70 con 0x55b412173270 2024-08-31T12:16:18.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.826+0000 7f246f7fe640 1 -- 172.21.15.19:0/1286893817 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2480005330 con 0x55b412173270 2024-08-31T12:16:18.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.827+0000 7f249ba16740 1 -- 172.21.15.19:0/1286893817 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b412173270 msgr2=0x55b412548f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:18.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.827+0000 7f249ba16740 1 --2- 172.21.15.19:0/1286893817 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b412173270 0x55b412548f80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2480001ff0 tx=0x7f2480001630 comp rx=0 tx=0).stop 2024-08-31T12:16:18.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.827+0000 7f249ba16740 1 -- 172.21.15.19:0/1286893817 shutdown_connections 2024-08-31T12:16:18.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.827+0000 7f249ba16740 1 --2- 172.21.15.19:0/1286893817 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b412173270 0x55b412548f80 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.827+0000 7f249ba16740 1 --2- 172.21.15.19:0/1286893817 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b412152590 0x55b412548a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.827+0000 7f249ba16740 1 --2- 172.21.15.19:0/1286893817 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x55b41214a3b0 0x55b4125595b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.827+0000 7f249ba16740 1 -- 172.21.15.19:0/1286893817 >> 172.21.15.19:0/1286893817 conn(0x55b4121673c0 msgr2=0x55b412556350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:18.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.827+0000 7f249ba16740 1 -- 172.21.15.19:0/1286893817 shutdown_connections 2024-08-31T12:16:18.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.827+0000 7f249ba16740 1 -- 172.21.15.19:0/1286893817 wait complete. 2024-08-31T12:16:18.834 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.828+0000 7f249ba16740 1 -- start start 2024-08-31T12:16:18.834 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.828+0000 7f249ba16740 1 Processor -- start 2024-08-31T12:16:18.834 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.829+0000 7f249ba16740 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x55b41214a3b0 0x55b412570ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:18.834 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.829+0000 7f249ba16740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b412152590 0x55b412571730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:18.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.829+0000 7f249ba16740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b412173270 0x55b41256e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:18.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.829+0000 7f249ba16740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x55b412552ee0 con 0x55b412152590 2024-08-31T12:16:18.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.829+0000 7f249ba16740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55b412552890 con 0x55b412173270 2024-08-31T12:16:18.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.829+0000 7f249ba16740 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x55b412557cd0 con 0x55b41214a3b0 2024-08-31T12:16:18.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.829+0000 7f249ba16740 10 client.0 _ll_register_callbacks cb 0x7f2488009610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-08-31T12:16:18.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.829+0000 7f248dd9b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b412152590 0x55b412571730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:18.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.829+0000 7f248dd9b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b412152590 0x55b412571730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:37974/0 (socket says 172.21.15.19:37974) 2024-08-31T12:16:18.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.829+0000 7f248dd9b640 1 -- 172.21.15.19:0/3565589895 learned_addr learned my addr 172.21.15.19:0/3565589895 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:18.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.829+0000 7f248ed9d640 1 --2- 172.21.15.19:0/3565589895 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b412173270 0x55b41256e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:18.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.829+0000 7f248e59c640 1 --2- 172.21.15.19:0/3565589895 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x55b41214a3b0 0x55b412570ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:18.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.830+0000 7f248dd9b640 1 -- 172.21.15.19:0/3565589895 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x55b41214a3b0 msgr2=0x55b412570ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-31T12:16:18.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.830+0000 7f248dd9b640 1 --2- 172.21.15.19:0/3565589895 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x55b41214a3b0 0x55b412570ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.830+0000 7f248dd9b640 1 -- 172.21.15.19:0/3565589895 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b412173270 msgr2=0x55b41256e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-31T12:16:18.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.830+0000 7f248dd9b640 1 --2- 172.21.15.19:0/3565589895 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b412173270 0x55b41256e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.830+0000 7f248dd9b640 1 -- 172.21.15.19:0/3565589895 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b41254a630 con 0x55b412152590 2024-08-31T12:16:18.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.830+0000 7f248dd9b640 1 --2- 172.21.15.19:0/3565589895 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b412152590 0x55b412571730 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2478023040 tx=0x7f247800f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afca71a668afcf23 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:18.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.830+0000 7f246d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.19:3300/0 2024-08-31T12:16:18.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.830+0000 7f246d7fa640 1 -- 172.21.15.19:0/3565589895 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2478025040 con 0x55b412152590 2024-08-31T12:16:18.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.830+0000 7f246d7fa640 1 -- 172.21.15.19:0/3565589895 <== mon.0 v2:172.21.15.19:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2478021ac0 con 0x55b412152590 2024-08-31T12:16:18.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.830+0000 7f249ba16740 1 -- 172.21.15.19:0/3565589895 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2480003c70 con 0x55b412152590 2024-08-31T12:16:18.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.830+0000 7f246d7fa640 1 -- 172.21.15.19:0/3565589895 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) ==== 24+0+0 (secure 0 0 0) 0x7f2478038050 con 0x55b412152590 2024-08-31T12:16:18.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.830+0000 7f249ba16740 10 client.4785 fetch_fsmap learned FSMap version 12 2024-08-31T12:16:18.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.830+0000 7f249ba16740 1 -- 172.21.15.19:0/3565589895 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) -- 0x55b412573220 con 0x55b412152590 2024-08-31T12:16:18.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.830+0000 7f246d7fa640 1 -- 172.21.15.19:0/3565589895 <== mon.0 v2:172.21.15.19:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2480003c70 con 0x55b412152590 2024-08-31T12:16:18.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.830+0000 7f246d7fa640 1 -- 172.21.15.19:0/3565589895 <== mon.0 v2:172.21.15.19:3300/0 5 ==== fsmap(e 12) ==== 4752+0+0 (secure 0 0 0) 0x7f247803b020 con 0x55b412152590 2024-08-31T12:16:18.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.830+0000 7f249ba16740 10 client.4785 fetch_fsmap finished waiting for FSMap version 12 2024-08-31T12:16:18.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.830+0000 7f249ba16740 10 client.4785 resolve_mds: resolved name 'd' to daemon mds.d 2024-08-31T12:16:18.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.830+0000 7f249ba16740 20 client.4785 populate_metadata read hostname 'smithi019' 2024-08-31T12:16:18.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.830+0000 7f249ba16740 1 --2- 172.21.15.19:0/3565589895 >> [v2:172.21.15.43:6834/3981040655,v1:172.21.15.43:6838/3981040655] conn(0x55b4125d3ea0 0x55b4125d42b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:18.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.830+0000 7f249ba16740 4 client.4785 mds_command: new command op to 4429 tid=0 multi_id=0 [{"prefix": "status"}] 2024-08-31T12:16:18.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.830+0000 7f249ba16740 1 -- 172.21.15.19:0/3565589895 --> [v2:172.21.15.43:6834/3981040655,v1:172.21.15.43:6838/3981040655] -- command(tid 0: {"prefix": "status"}) -- 0x55b41254b500 con 0x55b4125d3ea0 2024-08-31T12:16:18.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.831+0000 7f248e59c640 1 --2- 172.21.15.19:0/3565589895 >> [v2:172.21.15.43:6834/3981040655,v1:172.21.15.43:6838/3981040655] conn(0x55b4125d3ea0 0x55b4125d42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:18.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.831+0000 7f246d7fa640 10 client.4785 ms_handle_connect on v2:172.21.15.43:6834/3981040655 2024-08-31T12:16:18.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.831+0000 7f248e59c640 1 --2- 172.21.15.19:0/3565589895 >> [v2:172.21.15.43:6834/3981040655,v1:172.21.15.43:6838/3981040655] conn(0x55b4125d3ea0 0x55b4125d42b0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d28047d8fddc3823 server_cookie=8a04dcda626dffc7 in_seq=0 out_seq=0 2024-08-31T12:16:18.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.831+0000 7f246d7fa640 1 -- 172.21.15.19:0/3565589895 <== mds.0 v2:172.21.15.43:6834/3981040655 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x55b41254b500 con 0x55b4125d3ea0 2024-08-31T12:16:18.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.831+0000 7f246d7fa640 10 client.4785 handle_command_reply: tid=0 2024-08-31T12:16:18.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.831+0000 7f249ba16740 1 client.4785 shutdown 2024-08-31T12:16:18.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.831+0000 7f249ba16740 1 -- 172.21.15.19:0/3565589895 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b412152590 msgr2=0x55b412571730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:18.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.831+0000 7f249ba16740 1 --2- 172.21.15.19:0/3565589895 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b412152590 0x55b412571730 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2478023040 tx=0x7f247800f610 comp rx=0 tx=0).stop 2024-08-31T12:16:18.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.831+0000 7f249ba16740 1 -- 172.21.15.19:0/3565589895 shutdown_connections 2024-08-31T12:16:18.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.831+0000 7f249ba16740 1 --2- 172.21.15.19:0/3565589895 >> [v2:172.21.15.43:6834/3981040655,v1:172.21.15.43:6838/3981040655] conn(0x55b4125d3ea0 0x55b4125d42b0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.831+0000 7f249ba16740 1 --2- 172.21.15.19:0/3565589895 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b412173270 0x55b41256e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.831+0000 7f249ba16740 1 --2- 172.21.15.19:0/3565589895 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b412152590 0x55b412571730 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.831+0000 7f249ba16740 1 --2- 172.21.15.19:0/3565589895 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x55b41214a3b0 0x55b412570ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.831+0000 7f246d7fa640 0 client.4785 ms_handle_reset on v2:172.21.15.43:6834/3981040655 2024-08-31T12:16:18.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.831+0000 7f249ba16740 1 -- 172.21.15.19:0/3565589895 >> 172.21.15.19:0/3565589895 conn(0x55b4121673c0 msgr2=0x55b412550c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:18.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.832+0000 7f249ba16740 1 -- 172.21.15.19:0/3565589895 shutdown_connections 2024-08-31T12:16:18.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.832+0000 7f249ba16740 1 -- 172.21.15.19:0/3565589895 wait complete. 2024-08-31T12:16:18.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.832+0000 7f249ba16740 20 client.4785 trim_cache size 0 max 16384 2024-08-31T12:16:18.839 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:16:18.839 INFO:teuthology.orchestra.run.smithi019.stdout: "cluster_fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:16:18.839 INFO:teuthology.orchestra.run.smithi019.stdout: "whoami": 0, 2024-08-31T12:16:18.839 INFO:teuthology.orchestra.run.smithi019.stdout: "id": 4429, 2024-08-31T12:16:18.839 INFO:teuthology.orchestra.run.smithi019.stdout: "want_state": "up:active", 2024-08-31T12:16:18.839 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "up:active", 2024-08-31T12:16:18.839 INFO:teuthology.orchestra.run.smithi019.stdout: "fs_name": "cephfs", 2024-08-31T12:16:18.839 INFO:teuthology.orchestra.run.smithi019.stdout: "rank_uptime": 21.538127407000001, 2024-08-31T12:16:18.839 INFO:teuthology.orchestra.run.smithi019.stdout: "mdsmap_epoch": 12, 2024-08-31T12:16:18.839 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_epoch": 21, 2024-08-31T12:16:18.840 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_epoch_barrier": 21, 2024-08-31T12:16:18.840 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 27.611047738 2024-08-31T12:16:18.840 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:16:18.841 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.832+0000 7f2494aab640 1 -- 172.21.15.19:0/4089939587 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2470077a50 msgr2=0x7f2470079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:18.841 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.832+0000 7f2494aab640 1 --2- 172.21.15.19:0/4089939587 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2470077a50 0x7f2470079f60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2478000f60 tx=0x7f2478009040 comp rx=0 tx=0).stop 2024-08-31T12:16:18.841 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.832+0000 7f2494aab640 1 -- 172.21.15.19:0/4089939587 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f249006e0e0 msgr2=0x7f24900c5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:18.841 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.832+0000 7f2494aab640 1 --2- 172.21.15.19:0/4089939587 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f249006e0e0 0x7f24900c5750 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2488006010 tx=0x7f24880039e0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.841 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.832+0000 7f2494aab640 1 -- 172.21.15.19:0/4089939587 shutdown_connections 2024-08-31T12:16:18.841 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.832+0000 7f2494aab640 1 --2- 172.21.15.19:0/4089939587 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2470077a50 0x7f2470079f60 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.841 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.832+0000 7f2494aab640 1 --2- 172.21.15.19:0/4089939587 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2490155300 0x7f24901e5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.841 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.832+0000 7f2494aab640 1 --2- 172.21.15.19:0/4089939587 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f24900c8a40 0x7f249014e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.841 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.832+0000 7f2494aab640 1 --2- 172.21.15.19:0/4089939587 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f249006e0e0 0x7f24900c5750 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:18.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.832+0000 7f2494aab640 1 -- 172.21.15.19:0/4089939587 >> 172.21.15.19:0/4089939587 conn(0x7f24900690d0 msgr2=0x7f24901596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:18.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.832+0000 7f2494aab640 1 -- 172.21.15.19:0/4089939587 shutdown_connections 2024-08-31T12:16:18.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:18.832+0000 7f2494aab640 1 -- 172.21.15.19:0/4089939587 wait complete. 2024-08-31T12:16:18.844 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-08-31T12:16:18.869 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-08-31T12:16:18.869 INFO:tasks.ceph_fuse:config is {'client.0': {}} 2024-08-31T12:16:18.870 INFO:tasks.ceph_fuse:client.0 config is {} 2024-08-31T12:16:18.870 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-08-31T12:16:18.870 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-08-31T12:16:18.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-31T12:16:18.870 DEBUG:teuthology.orchestra.run.smithi019:> ip netns list 2024-08-31T12:16:18.886 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-31T12:16:18.886 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip link delete ceph-brx 2024-08-31T12:16:18.952 INFO:teuthology.orchestra.run.smithi019.stderr:Cannot find device "ceph-brx" 2024-08-31T12:16:18.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T12:16:18.953 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-08-31T12:16:18.953 DEBUG:tasks.ceph_fuse:passing mntargs=[] 2024-08-31T12:16:18.954 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-08-31T12:16:19.108 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.107+0000 7f9238d9d640 1 -- 172.21.15.19:0/1140853113 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f923406dbf0 msgr2=0x7f923406e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:19.109 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.107+0000 7f9238d9d640 1 --2- 172.21.15.19:0/1140853113 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f923406dbf0 0x7f923406e000 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9234072dc0 tx=0x7f922c02a680 comp rx=0 tx=0).stop 2024-08-31T12:16:19.109 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.107+0000 7f9238d9d640 1 -- 172.21.15.19:0/1140853113 shutdown_connections 2024-08-31T12:16:19.109 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.107+0000 7f9238d9d640 1 --2- 172.21.15.19:0/1140853113 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f923406dbf0 0x7f923406e000 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:19.109 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.107+0000 7f9238d9d640 1 --2- 172.21.15.19:0/1140853113 >> v2:172.21.15.43:3300/0 conn(0x7f92340c4710 0x7f923414e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:19.109 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.107+0000 7f9238d9d640 1 -- 172.21.15.19:0/1140853113 >> 172.21.15.19:0/1140853113 conn(0x7f92340718e0 msgr2=0x7f9234071d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:19.109 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.107+0000 7f9238d9d640 1 -- 172.21.15.19:0/1140853113 shutdown_connections 2024-08-31T12:16:19.109 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.107+0000 7f9238d9d640 1 -- 172.21.15.19:0/1140853113 wait complete. 2024-08-31T12:16:19.110 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.109+0000 7f9238d9d640 1 Processor -- start 2024-08-31T12:16:19.110 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.109+0000 7f9238d9d640 1 -- start start 2024-08-31T12:16:19.112 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.110+0000 7f9238d9d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f923406dbf0 0x7f92341f3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:19.112 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.110+0000 7f9238d9d640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f92340c4710 0x7f9234156130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:19.112 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.110+0000 7f9238d9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f92340cb070 0x7f9234158d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:19.112 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.110+0000 7f9238d9d640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f92340c6a20 con 0x7f92340cb070 2024-08-31T12:16:19.112 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.110+0000 7f9238d9d640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f92340c6740 con 0x7f92340c4710 2024-08-31T12:16:19.112 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.110+0000 7f9238d9d640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f92340c6d00 con 0x7f923406dbf0 2024-08-31T12:16:19.112 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.110+0000 7f9233fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f92340cb070 0x7f9234158d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:19.113 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.110+0000 7f9233fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f92340cb070 0x7f9234158d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:37986/0 (socket says 172.21.15.19:37986) 2024-08-31T12:16:19.113 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.110+0000 7f9233fff640 1 -- 172.21.15.19:0/1484967381 learned_addr learned my addr 172.21.15.19:0/1484967381 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:19.113 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.110+0000 7f9232ffd640 1 --2- 172.21.15.19:0/1484967381 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f92340c4710 0x7f9234156130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:19.113 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.110+0000 7f92337fe640 1 --2- 172.21.15.19:0/1484967381 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f923406dbf0 0x7f92341f3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:19.113 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.110+0000 7f9233fff640 1 -- 172.21.15.19:0/1484967381 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f923406dbf0 msgr2=0x7f92341f3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:19.113 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.110+0000 7f9233fff640 1 --2- 172.21.15.19:0/1484967381 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f923406dbf0 0x7f92341f3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:19.113 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.110+0000 7f9233fff640 1 -- 172.21.15.19:0/1484967381 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f92340c4710 msgr2=0x7f9234156130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:19.113 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.110+0000 7f9233fff640 1 --2- 172.21.15.19:0/1484967381 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f92340c4710 0x7f9234156130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:19.113 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.110+0000 7f9233fff640 1 -- 172.21.15.19:0/1484967381 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92341f35c0 con 0x7f92340cb070 2024-08-31T12:16:19.113 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.111+0000 7f9233fff640 1 --2- 172.21.15.19:0/1484967381 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f92340cb070 0x7f9234158d80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f922c009230 tx=0x7f922c003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:19.113 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.111+0000 7f921bfff640 1 -- 172.21.15.19:0/1484967381 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f922c03e030 con 0x7f92340cb070 2024-08-31T12:16:19.113 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.111+0000 7f921bfff640 1 -- 172.21.15.19:0/1484967381 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f922c003d10 con 0x7f92340cb070 2024-08-31T12:16:19.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.111+0000 7f9238d9d640 1 -- 172.21.15.19:0/1484967381 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92340c9460 con 0x7f92340cb070 2024-08-31T12:16:19.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.111+0000 7f921bfff640 1 -- 172.21.15.19:0/1484967381 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f922c035040 con 0x7f92340cb070 2024-08-31T12:16:19.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.111+0000 7f9238d9d640 1 -- 172.21.15.19:0/1484967381 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92340c9a20 con 0x7f92340cb070 2024-08-31T12:16:19.115 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.112+0000 7f9238d9d640 1 -- 172.21.15.19:0/1484967381 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92341f2d70 con 0x7f92340cb070 2024-08-31T12:16:19.115 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.113+0000 7f921bfff640 1 -- 172.21.15.19:0/1484967381 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f922c041030 con 0x7f92340cb070 2024-08-31T12:16:19.115 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.113+0000 7f921bfff640 1 --2- 172.21.15.19:0/1484967381 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9220077a50 0x7f9220079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:19.115 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.113+0000 7f921bfff640 1 -- 172.21.15.19:0/1484967381 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f922c0ba0e0 con 0x7f92340cb070 2024-08-31T12:16:19.115 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.113+0000 7f92337fe640 1 --2- 172.21.15.19:0/1484967381 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9220077a50 0x7f9220079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:19.115 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.114+0000 7f92337fe640 1 --2- 172.21.15.19:0/1484967381 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9220077a50 0x7f9220079f60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9224001ce0 tx=0x7f9224009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:19.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.118+0000 7f921bfff640 1 -- 172.21.15.19:0/1484967381 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f922c082e60 con 0x7f92340cb070 2024-08-31T12:16:19.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.340+0000 7f9238d9d640 1 -- 172.21.15.19:0/1484967381 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f923406e880 con 0x7f92340cb070 2024-08-31T12:16:19.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.340+0000 7f921bfff640 1 -- 172.21.15.19:0/1484967381 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v12) ==== 53+0+73 (secure 0 0 0) 0x7f922c08b310 con 0x7f92340cb070 2024-08-31T12:16:19.341 INFO:teuthology.orchestra.run.smithi019.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-08-31T12:16:19.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.342+0000 7f9238d9d640 1 -- 172.21.15.19:0/1484967381 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9220077a50 msgr2=0x7f9220079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:19.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.342+0000 7f9238d9d640 1 --2- 172.21.15.19:0/1484967381 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9220077a50 0x7f9220079f60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9224001ce0 tx=0x7f9224009040 comp rx=0 tx=0).stop 2024-08-31T12:16:19.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.342+0000 7f9238d9d640 1 -- 172.21.15.19:0/1484967381 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f92340cb070 msgr2=0x7f9234158d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:19.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.342+0000 7f9238d9d640 1 --2- 172.21.15.19:0/1484967381 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f92340cb070 0x7f9234158d80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f922c009230 tx=0x7f922c003820 comp rx=0 tx=0).stop 2024-08-31T12:16:19.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.342+0000 7f9238d9d640 1 -- 172.21.15.19:0/1484967381 shutdown_connections 2024-08-31T12:16:19.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.342+0000 7f9238d9d640 1 --2- 172.21.15.19:0/1484967381 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9220077a50 0x7f9220079f60 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:19.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.342+0000 7f9238d9d640 1 --2- 172.21.15.19:0/1484967381 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f92340cb070 0x7f9234158d80 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:19.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.342+0000 7f9238d9d640 1 --2- 172.21.15.19:0/1484967381 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f92340c4710 0x7f9234156130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:19.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.342+0000 7f9238d9d640 1 --2- 172.21.15.19:0/1484967381 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f923406dbf0 0x7f92341f3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:19.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.342+0000 7f9238d9d640 1 -- 172.21.15.19:0/1484967381 >> 172.21.15.19:0/1484967381 conn(0x7f92340718e0 msgr2=0x7f923414a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:19.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.342+0000 7f9238d9d640 1 -- 172.21.15.19:0/1484967381 shutdown_connections 2024-08-31T12:16:19.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19.342+0000 7f9238d9d640 1 -- 172.21.15.19:0/1484967381 wait complete. 2024-08-31T12:16:19.354 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-08-31T12:16:19.354 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-08-31T12:16:19.354 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi019.front.sepia.ceph.com 2024-08-31T12:16:19.354 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-08-31T12:16:19.354 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-08-31T12:16:19.354 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-08-31T12:16:19.354 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-08-31T12:16:19.355 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-08-31T12:16:19.355 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-08-31T12:16:19.355 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-31T12:16:19.355 DEBUG:teuthology.orchestra.run.smithi019:> ip addr 2024-08-31T12:16:19.370 INFO:teuthology.orchestra.run.smithi019.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-08-31T12:16:19.370 INFO:teuthology.orchestra.run.smithi019.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-08-31T12:16:19.371 INFO:teuthology.orchestra.run.smithi019.stdout: inet 127.0.0.1/8 scope host lo 2024-08-31T12:16:19.371 INFO:teuthology.orchestra.run.smithi019.stdout: valid_lft forever preferred_lft forever 2024-08-31T12:16:19.371 INFO:teuthology.orchestra.run.smithi019.stdout: inet6 ::1/128 scope host 2024-08-31T12:16:19.371 INFO:teuthology.orchestra.run.smithi019.stdout: valid_lft forever preferred_lft forever 2024-08-31T12:16:19.371 INFO:teuthology.orchestra.run.smithi019.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-31T12:16:19.371 INFO:teuthology.orchestra.run.smithi019.stdout: link/ether 0c:c4:7a:6b:fd:12 brd ff:ff:ff:ff:ff:ff 2024-08-31T12:16:19.371 INFO:teuthology.orchestra.run.smithi019.stdout: altname enp7s0f0 2024-08-31T12:16:19.371 INFO:teuthology.orchestra.run.smithi019.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-31T12:16:19.371 INFO:teuthology.orchestra.run.smithi019.stdout: link/ether 0c:c4:7a:6b:fd:13 brd ff:ff:ff:ff:ff:ff 2024-08-31T12:16:19.371 INFO:teuthology.orchestra.run.smithi019.stdout: altname enp7s0f1 2024-08-31T12:16:19.371 INFO:teuthology.orchestra.run.smithi019.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-08-31T12:16:19.371 INFO:teuthology.orchestra.run.smithi019.stdout: link/ether 0c:c4:7a:bd:16:1e brd ff:ff:ff:ff:ff:ff 2024-08-31T12:16:19.372 INFO:teuthology.orchestra.run.smithi019.stdout: altname enp1s0f0 2024-08-31T12:16:19.372 INFO:teuthology.orchestra.run.smithi019.stdout: inet 172.21.15.19/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-08-31T12:16:19.372 INFO:teuthology.orchestra.run.smithi019.stdout: valid_lft 42840sec preferred_lft 42840sec 2024-08-31T12:16:19.372 INFO:teuthology.orchestra.run.smithi019.stdout: inet6 fe80::ffb2:8deb:7d77:650e/64 scope link noprefixroute 2024-08-31T12:16:19.372 INFO:teuthology.orchestra.run.smithi019.stdout: valid_lft forever preferred_lft forever 2024-08-31T12:16:19.372 INFO:teuthology.orchestra.run.smithi019.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-08-31T12:16:19.372 INFO:teuthology.orchestra.run.smithi019.stdout: link/ether 0c:c4:7a:bd:16:1f brd ff:ff:ff:ff:ff:ff 2024-08-31T12:16:19.372 INFO:teuthology.orchestra.run.smithi019.stdout: altname enp1s0f1 2024-08-31T12:16:19.372 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-08-31T12:16:19.373 DEBUG:teuthology.orchestra.run.smithi019:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-31T12:16:19.373 DEBUG:teuthology.orchestra.run.smithi019:> set -e 2024-08-31T12:16:19.373 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip link add name ceph-brx type bridge 2024-08-31T12:16:19.373 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip addr flush dev ceph-brx 2024-08-31T12:16:19.373 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip link set ceph-brx up 2024-08-31T12:16:19.373 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-08-31T12:16:19.373 DEBUG:teuthology.orchestra.run.smithi019:> ') 2024-08-31T12:16:19.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-31T12:16:19.559 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-31T12:16:19.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-31T12:16:19.566 DEBUG:teuthology.orchestra.run.smithi019:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-08-31T12:16:19.591 INFO:teuthology.orchestra.run.smithi019.stdout:1 2024-08-31T12:16:19.592 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-31T12:16:19.593 DEBUG:teuthology.orchestra.run.smithi019:> route 2024-08-31T12:16:19.665 INFO:teuthology.orchestra.run.smithi019.stdout:Kernel IP routing table 2024-08-31T12:16:19.665 INFO:teuthology.orchestra.run.smithi019.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-08-31T12:16:19.665 INFO:teuthology.orchestra.run.smithi019.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-08-31T12:16:19.665 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-08-31T12:16:19.665 INFO:teuthology.orchestra.run.smithi019.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-08-31T12:16:19.666 DEBUG:teuthology.orchestra.run.smithi019:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-31T12:16:19.666 DEBUG:teuthology.orchestra.run.smithi019:> set -e 2024-08-31T12:16:19.666 DEBUG:teuthology.orchestra.run.smithi019:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-08-31T12:16:19.666 DEBUG:teuthology.orchestra.run.smithi019:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-08-31T12:16:19.666 DEBUG:teuthology.orchestra.run.smithi019:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-08-31T12:16:19.666 DEBUG:teuthology.orchestra.run.smithi019:> ') 2024-08-31T12:16:19.754 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-31T12:16:19.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-31T12:16:19.932 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-31T12:16:19.932 DEBUG:teuthology.orchestra.run.smithi019:> ip netns list 2024-08-31T12:16:19.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-31T12:16:19.948 DEBUG:teuthology.orchestra.run.smithi019:> ip netns list-id 2024-08-31T12:16:20.003 DEBUG:teuthology.orchestra.run.smithi019:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-31T12:16:20.003 DEBUG:teuthology.orchestra.run.smithi019:> set -e 2024-08-31T12:16:20.003 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-08-31T12:16:20.003 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-08-31T12:16:20.003 DEBUG:teuthology.orchestra.run.smithi019:> ') 2024-08-31T12:16:20.090 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-31T12:16:20.113 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-31T12:16:20.119 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-08-31T12:16:20.119 DEBUG:teuthology.orchestra.run.smithi019:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-31T12:16:20.120 DEBUG:teuthology.orchestra.run.smithi019:> set -e 2024-08-31T12:16:20.120 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-08-31T12:16:20.120 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-08-31T12:16:20.120 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-08-31T12:16:20.120 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-08-31T12:16:20.120 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-08-31T12:16:20.120 DEBUG:teuthology.orchestra.run.smithi019:> ') 2024-08-31T12:16:20.168 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-31T12:16:20.264 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-31T12:16:20.270 DEBUG:teuthology.orchestra.run.smithi019:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-08-31T12:16:20.270 DEBUG:teuthology.orchestra.run.smithi019:> set -e 2024-08-31T12:16:20.270 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip link set brx.0 up 2024-08-31T12:16:20.270 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip link set dev brx.0 master ceph-brx 2024-08-31T12:16:20.270 DEBUG:teuthology.orchestra.run.smithi019:> ') 2024-08-31T12:16:20.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-08-31T12:16:20.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-31T12:16:20.347 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-08-31T12:16:20.347 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-31T12:16:20.347 DEBUG:teuthology.orchestra.run.smithi019:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-08-31T12:16:20.376 INFO:teuthology.orchestra.run.smithi019.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-08-31T12:16:20.376 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-08-31T12:16:20.376 DEBUG:teuthology.orchestra.run.smithi019:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-08-31T12:16:20.431 DEBUG:teuthology.orchestra.run.smithi019:> sudo modprobe fuse 2024-08-31T12:16:20.498 DEBUG:teuthology.orchestra.run.smithi019:> cat /proc/self/mounts | awk '{print $2}' 2024-08-31T12:16:20.553 INFO:teuthology.orchestra.run.smithi019.stdout:/proc 2024-08-31T12:16:20.554 INFO:teuthology.orchestra.run.smithi019.stdout:/sys 2024-08-31T12:16:20.554 INFO:teuthology.orchestra.run.smithi019.stdout:/dev 2024-08-31T12:16:20.554 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/kernel/security 2024-08-31T12:16:20.554 INFO:teuthology.orchestra.run.smithi019.stdout:/dev/shm 2024-08-31T12:16:20.554 INFO:teuthology.orchestra.run.smithi019.stdout:/dev/pts 2024-08-31T12:16:20.554 INFO:teuthology.orchestra.run.smithi019.stdout:/run 2024-08-31T12:16:20.554 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/fs/cgroup 2024-08-31T12:16:20.554 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/fs/pstore 2024-08-31T12:16:20.554 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/fs/bpf 2024-08-31T12:16:20.554 INFO:teuthology.orchestra.run.smithi019.stdout:/ 2024-08-31T12:16:20.554 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/fs/selinux 2024-08-31T12:16:20.555 INFO:teuthology.orchestra.run.smithi019.stdout:/proc/sys/fs/binfmt_misc 2024-08-31T12:16:20.555 INFO:teuthology.orchestra.run.smithi019.stdout:/dev/hugepages 2024-08-31T12:16:20.555 INFO:teuthology.orchestra.run.smithi019.stdout:/dev/mqueue 2024-08-31T12:16:20.555 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/kernel/debug 2024-08-31T12:16:20.555 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/kernel/tracing 2024-08-31T12:16:20.555 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/kernel/config 2024-08-31T12:16:20.555 INFO:teuthology.orchestra.run.smithi019.stdout:/run/credentials/systemd-sysusers.service 2024-08-31T12:16:20.555 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/fs/fuse/connections 2024-08-31T12:16:20.555 INFO:teuthology.orchestra.run.smithi019.stdout:/run/credentials/systemd-sysctl.service 2024-08-31T12:16:20.555 INFO:teuthology.orchestra.run.smithi019.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-08-31T12:16:20.555 INFO:teuthology.orchestra.run.smithi019.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-08-31T12:16:20.555 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/nfs/rpc_pipefs 2024-08-31T12:16:20.555 INFO:teuthology.orchestra.run.smithi019.stdout:/run/user/1000 2024-08-31T12:16:20.556 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/ceph 2024-08-31T12:16:20.556 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/ceph/osd/ceph-0 2024-08-31T12:16:20.556 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/ceph/osd/ceph-1 2024-08-31T12:16:20.556 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/ceph/osd/ceph-2 2024-08-31T12:16:20.556 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/ceph/osd/ceph-3 2024-08-31T12:16:20.556 INFO:teuthology.orchestra.run.smithi019.stdout:/run/netns 2024-08-31T12:16:20.556 INFO:teuthology.orchestra.run.smithi019.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-08-31T12:16:20.556 INFO:teuthology.orchestra.run.smithi019.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-08-31T12:16:20.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-31T12:16:20.556 DEBUG:teuthology.orchestra.run.smithi019:> ls /sys/fs/fuse/connections 2024-08-31T12:16:20.609 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-08-31T12:16:20.609 DEBUG:teuthology.orchestra.run.smithi019:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-08-31T12:16:20.651 DEBUG:teuthology.orchestra.run.smithi019:> sudo modprobe fuse 2024-08-31T12:16:20.676 DEBUG:teuthology.orchestra.run.smithi019:> cat /proc/self/mounts | awk '{print $2}' 2024-08-31T12:16:20.733 INFO:teuthology.orchestra.run.smithi019.stdout:/proc 2024-08-31T12:16:20.733 INFO:teuthology.orchestra.run.smithi019.stdout:/sys 2024-08-31T12:16:20.733 INFO:teuthology.orchestra.run.smithi019.stdout:/dev 2024-08-31T12:16:20.733 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/kernel/security 2024-08-31T12:16:20.733 INFO:teuthology.orchestra.run.smithi019.stdout:/dev/shm 2024-08-31T12:16:20.733 INFO:teuthology.orchestra.run.smithi019.stdout:/dev/pts 2024-08-31T12:16:20.733 INFO:teuthology.orchestra.run.smithi019.stdout:/run 2024-08-31T12:16:20.733 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/fs/cgroup 2024-08-31T12:16:20.733 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/fs/pstore 2024-08-31T12:16:20.733 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/fs/bpf 2024-08-31T12:16:20.733 INFO:teuthology.orchestra.run.smithi019.stdout:/ 2024-08-31T12:16:20.734 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/fs/selinux 2024-08-31T12:16:20.734 INFO:teuthology.orchestra.run.smithi019.stdout:/proc/sys/fs/binfmt_misc 2024-08-31T12:16:20.734 INFO:teuthology.orchestra.run.smithi019.stdout:/dev/hugepages 2024-08-31T12:16:20.734 INFO:teuthology.orchestra.run.smithi019.stdout:/dev/mqueue 2024-08-31T12:16:20.734 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/kernel/debug 2024-08-31T12:16:20.734 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/kernel/tracing 2024-08-31T12:16:20.734 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/kernel/config 2024-08-31T12:16:20.734 INFO:teuthology.orchestra.run.smithi019.stdout:/run/credentials/systemd-sysusers.service 2024-08-31T12:16:20.734 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/fs/fuse/connections 2024-08-31T12:16:20.734 INFO:teuthology.orchestra.run.smithi019.stdout:/run/credentials/systemd-sysctl.service 2024-08-31T12:16:20.734 INFO:teuthology.orchestra.run.smithi019.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-08-31T12:16:20.734 INFO:teuthology.orchestra.run.smithi019.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-08-31T12:16:20.734 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/nfs/rpc_pipefs 2024-08-31T12:16:20.734 INFO:teuthology.orchestra.run.smithi019.stdout:/run/user/1000 2024-08-31T12:16:20.735 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/ceph 2024-08-31T12:16:20.735 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/ceph/osd/ceph-0 2024-08-31T12:16:20.735 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/ceph/osd/ceph-1 2024-08-31T12:16:20.735 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/ceph/osd/ceph-2 2024-08-31T12:16:20.735 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/ceph/osd/ceph-3 2024-08-31T12:16:20.735 INFO:teuthology.orchestra.run.smithi019.stdout:/run/netns 2024-08-31T12:16:20.735 INFO:teuthology.orchestra.run.smithi019.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-08-31T12:16:20.735 INFO:teuthology.orchestra.run.smithi019.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-08-31T12:16:20.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-31T12:16:20.735 DEBUG:teuthology.orchestra.run.smithi019:> ls /sys/fs/fuse/connections 2024-08-31T12:16:20.756 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi019.stderr:2024-08-31T12:16:20.755+0000 7f6733747480 -1 init, newargv = 0x7f6718001270 newargc=15 2024-08-31T12:16:20.756 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi019.stderr:ceph-fuse[33817]: starting ceph client 2024-08-31T12:16:20.762 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi019.stderr:ceph-fuse[33817]: starting fuse 2024-08-31T12:16:20.771 INFO:teuthology.orchestra.run.smithi019.stdout:44 2024-08-31T12:16:20.771 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [44] 2024-08-31T12:16:20.772 DEBUG:teuthology.orchestra.run.smithi019:> sudo stdin-killer -- python3 -c ' 2024-08-31T12:16:20.772 DEBUG:teuthology.orchestra.run.smithi019:> import glob 2024-08-31T12:16:20.772 DEBUG:teuthology.orchestra.run.smithi019:> import re 2024-08-31T12:16:20.772 DEBUG:teuthology.orchestra.run.smithi019:> import os 2024-08-31T12:16:20.772 DEBUG:teuthology.orchestra.run.smithi019:> import subprocess 2024-08-31T12:16:20.772 DEBUG:teuthology.orchestra.run.smithi019:> 2024-08-31T12:16:20.772 DEBUG:teuthology.orchestra.run.smithi019:> def _find_admin_socket(client_name): 2024-08-31T12:16:20.772 DEBUG:teuthology.orchestra.run.smithi019:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-08-31T12:16:20.772 DEBUG:teuthology.orchestra.run.smithi019:> files = glob.glob(asok_path) 2024-08-31T12:16:20.772 DEBUG:teuthology.orchestra.run.smithi019:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-08-31T12:16:20.772 DEBUG:teuthology.orchestra.run.smithi019:> 2024-08-31T12:16:20.772 DEBUG:teuthology.orchestra.run.smithi019:> # Given a non-glob path, it better be there 2024-08-31T12:16:20.772 DEBUG:teuthology.orchestra.run.smithi019:> if "*" not in asok_path: 2024-08-31T12:16:20.773 DEBUG:teuthology.orchestra.run.smithi019:> assert(len(files) == 1) 2024-08-31T12:16:20.773 DEBUG:teuthology.orchestra.run.smithi019:> return files[0] 2024-08-31T12:16:20.773 DEBUG:teuthology.orchestra.run.smithi019:> 2024-08-31T12:16:20.773 DEBUG:teuthology.orchestra.run.smithi019:> for f in files: 2024-08-31T12:16:20.773 DEBUG:teuthology.orchestra.run.smithi019:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-08-31T12:16:20.773 DEBUG:teuthology.orchestra.run.smithi019:> if os.path.exists("/proc/{0}".format(pid)): 2024-08-31T12:16:20.773 DEBUG:teuthology.orchestra.run.smithi019:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-08-31T12:16:20.773 DEBUG:teuthology.orchestra.run.smithi019:> contents = proc_f.read() 2024-08-31T12:16:20.773 DEBUG:teuthology.orchestra.run.smithi019:> if mountpoint in contents: 2024-08-31T12:16:20.773 DEBUG:teuthology.orchestra.run.smithi019:> return f 2024-08-31T12:16:20.773 DEBUG:teuthology.orchestra.run.smithi019:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-08-31T12:16:20.773 DEBUG:teuthology.orchestra.run.smithi019:> 2024-08-31T12:16:20.773 DEBUG:teuthology.orchestra.run.smithi019:> print(_find_admin_socket("client.0")) 2024-08-31T12:16:20.773 DEBUG:teuthology.orchestra.run.smithi019:> ' 2024-08-31T12:16:20.890 INFO:teuthology.orchestra.run.smithi019.stdout:/var/run/ceph/ceph-client.0.33817.asok 2024-08-31T12:16:20.893 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-08-31T12:16:20.900 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.33817.asok 2024-08-31T12:16:20.900 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-31T12:16:20.900 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.33817.asok status 2024-08-31T12:16:21.043 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:16:21.043 INFO:teuthology.orchestra.run.smithi019.stdout: "metadata": { 2024-08-31T12:16:21.043 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph_sha1": "d19e6118f7015f0047264006cb0c873528eb3ae6", 2024-08-31T12:16:21.043 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph_version": "ceph version 19.3.0-4560-gd19e6118 (d19e6118f7015f0047264006cb0c873528eb3ae6) squid (dev)", 2024-08-31T12:16:21.043 INFO:teuthology.orchestra.run.smithi019.stdout: "entity_id": "0", 2024-08-31T12:16:21.043 INFO:teuthology.orchestra.run.smithi019.stdout: "hostname": "smithi019", 2024-08-31T12:16:21.043 INFO:teuthology.orchestra.run.smithi019.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-08-31T12:16:21.043 INFO:teuthology.orchestra.run.smithi019.stdout: "pid": "33817", 2024-08-31T12:16:21.043 INFO:teuthology.orchestra.run.smithi019.stdout: "root": "/" 2024-08-31T12:16:21.043 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:21.043 INFO:teuthology.orchestra.run.smithi019.stdout: "dentry_count": 0, 2024-08-31T12:16:21.043 INFO:teuthology.orchestra.run.smithi019.stdout: "dentry_pinned_count": 0, 2024-08-31T12:16:21.044 INFO:teuthology.orchestra.run.smithi019.stdout: "id": 4800, 2024-08-31T12:16:21.044 INFO:teuthology.orchestra.run.smithi019.stdout: "inst": { 2024-08-31T12:16:21.044 INFO:teuthology.orchestra.run.smithi019.stdout: "name": { 2024-08-31T12:16:21.044 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "client", 2024-08-31T12:16:21.044 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 4800 2024-08-31T12:16:21.044 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:21.044 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": { 2024-08-31T12:16:21.044 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:16:21.044 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "192.168.0.1:0", 2024-08-31T12:16:21.044 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 969923281 2024-08-31T12:16:21.044 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:21.044 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:21.044 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": { 2024-08-31T12:16:21.044 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:16:21.045 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "192.168.0.1:0", 2024-08-31T12:16:21.045 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 969923281 2024-08-31T12:16:21.045 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:21.045 INFO:teuthology.orchestra.run.smithi019.stdout: "inst_str": "client.4800 192.168.0.1:0/969923281", 2024-08-31T12:16:21.045 INFO:teuthology.orchestra.run.smithi019.stdout: "addr_str": "192.168.0.1:0/969923281", 2024-08-31T12:16:21.045 INFO:teuthology.orchestra.run.smithi019.stdout: "inode_count": 1, 2024-08-31T12:16:21.045 INFO:teuthology.orchestra.run.smithi019.stdout: "mds_epoch": 12, 2024-08-31T12:16:21.045 INFO:teuthology.orchestra.run.smithi019.stdout: "osd_epoch": 21, 2024-08-31T12:16:21.045 INFO:teuthology.orchestra.run.smithi019.stdout: "osd_epoch_barrier": 0, 2024-08-31T12:16:21.045 INFO:teuthology.orchestra.run.smithi019.stdout: "blocklisted": false, 2024-08-31T12:16:21.045 INFO:teuthology.orchestra.run.smithi019.stdout: "fs_name": "cephfs" 2024-08-31T12:16:21.045 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:16:21.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-31T12:16:21.051 DEBUG:teuthology.orchestra.run.smithi019:> stat --file-system '--printf=%T 2024-08-31T12:16:21.051 DEBUG:teuthology.orchestra.run.smithi019:> ' -- /home/ubuntu/cephtest/mnt.0 2024-08-31T12:16:21.067 INFO:teuthology.orchestra.run.smithi019.stdout:fuseblk 2024-08-31T12:16:21.067 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-08-31T12:16:21.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-31T12:16:21.067 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-08-31T12:16:21.137 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-08-31T12:16:21.150 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-08-31T12:16:21.150 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-31T12:16:21.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.290+0000 7f0570f39640 1 -- 172.21.15.19:0/501353265 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f056c0ca770 msgr2=0x7f056c0d0cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:21.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.290+0000 7f0570f39640 1 --2- 172.21.15.19:0/501353265 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f056c0ca770 0x7f056c0d0cd0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f056c08e3f0 tx=0x7f056002a680 comp rx=0 tx=0).stop 2024-08-31T12:16:21.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.290+0000 7f0570f39640 1 -- 172.21.15.19:0/501353265 shutdown_connections 2024-08-31T12:16:21.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.290+0000 7f0570f39640 1 --2- 172.21.15.19:0/501353265 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f056c0ca770 0x7f056c0d0cd0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:21.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.290+0000 7f0570f39640 1 --2- 172.21.15.19:0/501353265 >> v2:172.21.15.43:3300/0 conn(0x7f056c0cc620 0x7f056c0cdab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:21.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.290+0000 7f0570f39640 1 -- 172.21.15.19:0/501353265 >> 172.21.15.19:0/501353265 conn(0x7f056c06e6e0 msgr2=0x7f056c06eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:21.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.291+0000 7f0570f39640 1 -- 172.21.15.19:0/501353265 shutdown_connections 2024-08-31T12:16:21.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.291+0000 7f0570f39640 1 -- 172.21.15.19:0/501353265 wait complete. 2024-08-31T12:16:21.293 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.292+0000 7f0570f39640 1 Processor -- start 2024-08-31T12:16:21.293 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.292+0000 7f0570f39640 1 -- start start 2024-08-31T12:16:21.295 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.294+0000 7f0570f39640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f056c0ca770 0x7f056c0ad8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:21.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.294+0000 7f0570f39640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f056c0cc620 0x7f056c0c64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:21.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.294+0000 7f0570f39640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f056c0d1210 0x7f056c157370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:21.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.294+0000 7f0570f39640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f056c15d090 con 0x7f056c0cc620 2024-08-31T12:16:21.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.294+0000 7f0570f39640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f056c15cdb0 con 0x7f056c0d1210 2024-08-31T12:16:21.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.294+0000 7f0570f39640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f056c0cc320 con 0x7f056c0ca770 2024-08-31T12:16:21.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.294+0000 7f0569d9b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f056c0cc620 0x7f056c0c64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:21.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.294+0000 7f0569d9b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f056c0cc620 0x7f056c0c64a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:38008/0 (socket says 172.21.15.19:38008) 2024-08-31T12:16:21.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.294+0000 7f0569d9b640 1 -- 172.21.15.19:0/3292970710 learned_addr learned my addr 172.21.15.19:0/3292970710 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:21.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.294+0000 7f056a59c640 1 --2- 172.21.15.19:0/3292970710 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f056c0ca770 0x7f056c0ad8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:21.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.294+0000 7f056ad9d640 1 --2- 172.21.15.19:0/3292970710 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f056c0d1210 0x7f056c157370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:21.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.294+0000 7f0569d9b640 1 -- 172.21.15.19:0/3292970710 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f056c0ca770 msgr2=0x7f056c0ad8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:21.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.294+0000 7f0569d9b640 1 --2- 172.21.15.19:0/3292970710 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f056c0ca770 0x7f056c0ad8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:21.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.294+0000 7f0569d9b640 1 -- 172.21.15.19:0/3292970710 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f056c0d1210 msgr2=0x7f056c157370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:21.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.294+0000 7f0569d9b640 1 --2- 172.21.15.19:0/3292970710 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f056c0d1210 0x7f056c157370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:21.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.294+0000 7f0569d9b640 1 -- 172.21.15.19:0/3292970710 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f056c0ade30 con 0x7f056c0cc620 2024-08-31T12:16:21.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.294+0000 7f0569d9b640 1 --2- 172.21.15.19:0/3292970710 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f056c0cc620 0x7f056c0c64a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f05600099c0 tx=0x7f05600038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:21.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.294+0000 7f055affd640 1 -- 172.21.15.19:0/3292970710 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f056003e030 con 0x7f056c0cc620 2024-08-31T12:16:21.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.294+0000 7f0570f39640 1 -- 172.21.15.19:0/3292970710 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f056c0c69e0 con 0x7f056c0cc620 2024-08-31T12:16:21.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.294+0000 7f055affd640 1 -- 172.21.15.19:0/3292970710 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0560003db0 con 0x7f056c0cc620 2024-08-31T12:16:21.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.294+0000 7f055affd640 1 -- 172.21.15.19:0/3292970710 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0560035040 con 0x7f056c0cc620 2024-08-31T12:16:21.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.295+0000 7f0570f39640 1 -- 172.21.15.19:0/3292970710 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f056c1f1a20 con 0x7f056c0cc620 2024-08-31T12:16:21.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.296+0000 7f0570f39640 1 -- 172.21.15.19:0/3292970710 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f056c0c7c40 con 0x7f056c0cc620 2024-08-31T12:16:21.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.296+0000 7f055affd640 1 -- 172.21.15.19:0/3292970710 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f0560041030 con 0x7f056c0cc620 2024-08-31T12:16:21.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.297+0000 7f055affd640 1 --2- 172.21.15.19:0/3292970710 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f054c077a50 0x7f054c079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:21.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.297+0000 7f055affd640 1 -- 172.21.15.19:0/3292970710 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f05600ba130 con 0x7f056c0cc620 2024-08-31T12:16:21.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.297+0000 7f056a59c640 1 --2- 172.21.15.19:0/3292970710 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f054c077a50 0x7f054c079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:21.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.297+0000 7f056a59c640 1 --2- 172.21.15.19:0/3292970710 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f054c077a50 0x7f054c079f60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f055c009ac0 tx=0x7f055c005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:21.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.301+0000 7f055affd640 1 -- 172.21.15.19:0/3292970710 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f056c0c7c40 con 0x7f056c0cc620 2024-08-31T12:16:21.489 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.488+0000 7f0570f39640 1 -- 172.21.15.19:0/3292970710 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f056c158c50 con 0x7f056c0cc620 2024-08-31T12:16:21.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.489+0000 7f055affd640 1 -- 172.21.15.19:0/3292970710 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16617 (secure 0 0 0) 0x7f056c158c50 con 0x7f056c0cc620 2024-08-31T12:16:21.490 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:16:21.490 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":21,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","created":"2024-08-31T12:15:32.084309+0000","modified":"2024-08-31T12:15:58.292413+0000","last_up_change":"2024-08-31T12:15:45.213688+0000","last_in_change":"2024-08-31T12:15:35.545202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-31T12:15:44.508197+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-31T12:15:54.512584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-31T12:15:55.618251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b80d6a3f-7e4e-4d4d-b1b4-0df71c9176ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6800","nonce":680447146},{"type":"v1","addr":"172.21.15.19:6801","nonce":680447146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":680447146},{"type":"v1","addr":"172.21.15.19:6803","nonce":680447146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":680447146},{"type":"v1","addr":"172.21.15.19:6807","nonce":680447146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":680447146},{"type":"v1","addr":"172.21.15.19:6805","nonce":680447146}]},"public_addr":"172.21.15.19:6801/680447146","cluster_addr":"172.21.15.19:6803/680447146","heartbeat_back_addr":"172.21.15.19:6807/680447146","heartbeat_front_addr":"172.21.15.19:6805/680447146","state":["exists","up"]},{"osd":1,"uuid":"6f986366-82a6-4133-9f2e-43f84c7b03bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":1522610034},{"type":"v1","addr":"172.21.15.19:6817","nonce":1522610034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":1522610034},{"type":"v1","addr":"172.21.15.19:6819","nonce":1522610034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":1522610034},{"type":"v1","addr":"172.21.15.19:6823","nonce":1522610034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":1522610034},{"type":"v1","addr":"172.21.15.19:6821","nonce":1522610034}]},"public_addr":"172.21.15.19:6817/1522610034","cluster_addr":"172.21.15.19:6819/1522610034","heartbeat_back_addr":"172.21.15.19:6823/1522610034","heartbeat_front_addr":"172.21.15.19:6821/1522610034","state":["exists","up"]},{"osd":2,"uuid":"ce8c2068-845c-4cbb-a4d1-077375d605ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6809","nonce":2332368824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6811","nonce":2332368824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6815","nonce":2332368824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6813","nonce":2332368824}]},"public_addr":"172.21.15.19:6809/2332368824","cluster_addr":"172.21.15.19:6811/2332368824","heartbeat_back_addr":"172.21.15.19:6815/2332368824","heartbeat_front_addr":"172.21.15.19:6813/2332368824","state":["exists","up"]},{"osd":3,"uuid":"b1cb0ef4-aa9b-421b-a94e-82de0c46e5cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":3785550598},{"type":"v1","addr":"172.21.15.19:6825","nonce":3785550598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":3785550598},{"type":"v1","addr":"172.21.15.19:6827","nonce":3785550598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":3785550598},{"type":"v1","addr":"172.21.15.19:6831","nonce":3785550598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":3785550598},{"type":"v1","addr":"172.21.15.19:6829","nonce":3785550598}]},"public_addr":"172.21.15.19:6825/3785550598","cluster_addr":"172.21.15.19:6827/3785550598","heartbeat_back_addr":"172.21.15.19:6831/3785550598","heartbeat_front_addr":"172.21.15.19:6829/3785550598","state":["exists","up"]},{"osd":4,"uuid":"825b58ec-a6da-4515-b6a5-81debe327011","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":1985519349},{"type":"v1","addr":"172.21.15.43:6819","nonce":1985519349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":1985519349},{"type":"v1","addr":"172.21.15.43:6821","nonce":1985519349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":1985519349},{"type":"v1","addr":"172.21.15.43:6825","nonce":1985519349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":1985519349},{"type":"v1","addr":"172.21.15.43:6823","nonce":1985519349}]},"public_addr":"172.21.15.43:6819/1985519349","cluster_addr":"172.21.15.43:6821/1985519349","heartbeat_back_addr":"172.21.15.43:6825/1985519349","heartbeat_front_addr":"172.21.15.43:6823/1985519349","state":["exists","up"]},{"osd":5,"uuid":"7f34e537-5997-40e4-92e7-2570507160c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":4271971521},{"type":"v1","addr":"172.21.15.43:6803","nonce":4271971521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":4271971521},{"type":"v1","addr":"172.21.15.43:6805","nonce":4271971521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":4271971521},{"type":"v1","addr":"172.21.15.43:6809","nonce":4271971521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":4271971521},{"type":"v1","addr":"172.21.15.43:6807","nonce":4271971521}]},"public_addr":"172.21.15.43:6803/4271971521","cluster_addr":"172.21.15.43:6805/4271971521","heartbeat_back_addr":"172.21.15.43:6809/4271971521","heartbeat_front_addr":"172.21.15.43:6807/4271971521","state":["exists","up"]},{"osd":6,"uuid":"195dd8ad-640a-4fa3-9ee4-06e4172e0464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2863384694},{"type":"v1","addr":"172.21.15.43:6827","nonce":2863384694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2863384694},{"type":"v1","addr":"172.21.15.43:6829","nonce":2863384694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":2863384694},{"type":"v1","addr":"172.21.15.43:6833","nonce":2863384694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2863384694},{"type":"v1","addr":"172.21.15.43:6831","nonce":2863384694}]},"public_addr":"172.21.15.43:6827/2863384694","cluster_addr":"172.21.15.43:6829/2863384694","heartbeat_back_addr":"172.21.15.43:6833/2863384694","heartbeat_front_addr":"172.21.15.43:6831/2863384694","state":["exists","up"]},{"osd":7,"uuid":"4645661f-f430-4f05-bdd1-69138dcd1bbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":792572234},{"type":"v1","addr":"172.21.15.43:6811","nonce":792572234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":792572234},{"type":"v1","addr":"172.21.15.43:6813","nonce":792572234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":792572234},{"type":"v1","addr":"172.21.15.43:6817","nonce":792572234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":792572234},{"type":"v1","addr":"172.21.15.43:6815","nonce":792572234}]},"public_addr":"172.21.15.43:6811/792572234","cluster_addr":"172.21.15.43:6813/792572234","heartbeat_back_addr":"172.21.15.43:6817/792572234","heartbeat_front_addr":"172.21.15.43:6815/792572234","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.156485+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.578126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.075440+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.639867+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.543602+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:41.919592+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.615557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:41.813337+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-31T12:16:21.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.491+0000 7f0570f39640 1 -- 172.21.15.19:0/3292970710 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f054c077a50 msgr2=0x7f054c079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:21.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.491+0000 7f0570f39640 1 --2- 172.21.15.19:0/3292970710 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f054c077a50 0x7f054c079f60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f055c009ac0 tx=0x7f055c005f90 comp rx=0 tx=0).stop 2024-08-31T12:16:21.493 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.491+0000 7f0570f39640 1 -- 172.21.15.19:0/3292970710 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f056c0cc620 msgr2=0x7f056c0c64a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:21.493 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.491+0000 7f0570f39640 1 --2- 172.21.15.19:0/3292970710 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f056c0cc620 0x7f056c0c64a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f05600099c0 tx=0x7f05600038a0 comp rx=0 tx=0).stop 2024-08-31T12:16:21.493 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.491+0000 7f0570f39640 1 -- 172.21.15.19:0/3292970710 shutdown_connections 2024-08-31T12:16:21.493 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.491+0000 7f0570f39640 1 --2- 172.21.15.19:0/3292970710 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f054c077a50 0x7f054c079f60 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:21.493 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.491+0000 7f0570f39640 1 --2- 172.21.15.19:0/3292970710 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f056c0d1210 0x7f056c157370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:21.493 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.491+0000 7f0570f39640 1 --2- 172.21.15.19:0/3292970710 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f056c0cc620 0x7f056c0c64a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:21.493 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.491+0000 7f0570f39640 1 --2- 172.21.15.19:0/3292970710 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f056c0ca770 0x7f056c0ad8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:21.493 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.491+0000 7f0570f39640 1 -- 172.21.15.19:0/3292970710 >> 172.21.15.19:0/3292970710 conn(0x7f056c06e6e0 msgr2=0x7f056c15d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:21.493 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.491+0000 7f0570f39640 1 -- 172.21.15.19:0/3292970710 shutdown_connections 2024-08-31T12:16:21.493 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.491+0000 7f0570f39640 1 -- 172.21.15.19:0/3292970710 wait complete. 2024-08-31T12:16:21.503 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-08-31T12:15:44.508197+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-08-31T12:15:54.512584+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-08-31T12:15:55.618251+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-08-31T12:16:21.504 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-08-31T12:16:21.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.620+0000 7ffb44684640 1 -- 172.21.15.19:0/1985459096 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffb28003a00 msgr2=0x7ffb28003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:21.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.620+0000 7ffb44684640 1 --2- 172.21.15.19:0/1985459096 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffb28003a00 0x7ffb28003e70 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ffb3800c290 tx=0x7ffb28005830 comp rx=0 tx=0).stop 2024-08-31T12:16:21.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.620+0000 7ffb42c21640 1 -- 172.21.15.19:0/1985459096 reap_dead start 2024-08-31T12:16:21.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.620+0000 7ffb44684640 1 -- 172.21.15.19:0/1985459096 shutdown_connections 2024-08-31T12:16:21.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.620+0000 7ffb44684640 1 -- 172.21.15.19:0/1985459096 >> 172.21.15.19:0/1985459096 conn(0x7ffb3c074210 msgr2=0x7ffb3c074620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:21.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.620+0000 7ffb44684640 1 -- 172.21.15.19:0/1985459096 shutdown_connections 2024-08-31T12:16:21.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.620+0000 7ffb44684640 1 -- 172.21.15.19:0/1985459096 wait complete. 2024-08-31T12:16:21.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.622+0000 7ffb44684640 1 Processor -- start 2024-08-31T12:16:21.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.622+0000 7ffb44684640 1 -- start start 2024-08-31T12:16:21.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.623+0000 7ffb44684640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffb28003a00 0x7ffb3c15cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:21.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.623+0000 7ffb44684640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb3c0cf7b0 0x7ffb3c14f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:21.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.623+0000 7ffb44684640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffb3c0cfbc0 0x7ffb3c14f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:21.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.623+0000 7ffb44684640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7ffb3c0cc130 con 0x7ffb28003a00 2024-08-31T12:16:21.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.623+0000 7ffb44684640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ffb3c0d1370 con 0x7ffb3c0cf7b0 2024-08-31T12:16:21.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.623+0000 7ffb44684640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7ffb3c152170 con 0x7ffb3c0cfbc0 2024-08-31T12:16:21.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.623+0000 7ffb42420640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffb28003a00 0x7ffb3c15cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:21.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.623+0000 7ffb42420640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffb28003a00 0x7ffb3c15cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:38022/0 (socket says 172.21.15.19:38022) 2024-08-31T12:16:21.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.623+0000 7ffb42420640 1 -- 172.21.15.19:0/772374489 learned_addr learned my addr 172.21.15.19:0/772374489 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:21.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.623+0000 7ffb41c1f640 1 --2- 172.21.15.19:0/772374489 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb3c0cf7b0 0x7ffb3c14f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:21.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.623+0000 7ffb42c21640 1 --2- 172.21.15.19:0/772374489 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffb3c0cfbc0 0x7ffb3c14f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:21.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.624+0000 7ffb42420640 1 -- 172.21.15.19:0/772374489 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffb3c0cfbc0 msgr2=0x7ffb3c14f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:21.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.624+0000 7ffb42420640 1 --2- 172.21.15.19:0/772374489 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffb3c0cfbc0 0x7ffb3c14f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:21.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.624+0000 7ffb42420640 1 -- 172.21.15.19:0/772374489 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb3c0cf7b0 msgr2=0x7ffb3c14f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:21.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.624+0000 7ffb42420640 1 --2- 172.21.15.19:0/772374489 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb3c0cf7b0 0x7ffb3c14f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:21.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.624+0000 7ffb42420640 1 -- 172.21.15.19:0/772374489 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb3c15d040 con 0x7ffb28003a00 2024-08-31T12:16:21.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.624+0000 7ffb42420640 1 --2- 172.21.15.19:0/772374489 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffb28003a00 0x7ffb3c15cb00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ffb38000fb0 tx=0x7ffb38001d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:21.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.624+0000 7ffb32ffd640 1 -- 172.21.15.19:0/772374489 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb38042070 con 0x7ffb28003a00 2024-08-31T12:16:21.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.624+0000 7ffb32ffd640 1 -- 172.21.15.19:0/772374489 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffb38003920 con 0x7ffb28003a00 2024-08-31T12:16:21.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.624+0000 7ffb44684640 1 -- 172.21.15.19:0/772374489 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb3c15da50 con 0x7ffb28003a00 2024-08-31T12:16:21.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.624+0000 7ffb32ffd640 1 -- 172.21.15.19:0/772374489 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb38039070 con 0x7ffb28003a00 2024-08-31T12:16:21.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.624+0000 7ffb44684640 1 -- 172.21.15.19:0/772374489 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb3c15d710 con 0x7ffb28003a00 2024-08-31T12:16:21.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.625+0000 7ffb44684640 1 -- 172.21.15.19:0/772374489 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb3c155680 con 0x7ffb28003a00 2024-08-31T12:16:21.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.626+0000 7ffb32ffd640 1 -- 172.21.15.19:0/772374489 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7ffb3802c500 con 0x7ffb28003a00 2024-08-31T12:16:21.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.626+0000 7ffb32ffd640 1 --2- 172.21.15.19:0/772374489 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ffb20077a80 0x7ffb20079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:21.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.627+0000 7ffb32ffd640 1 -- 172.21.15.19:0/772374489 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ffb380b9400 con 0x7ffb28003a00 2024-08-31T12:16:21.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.627+0000 7ffb41c1f640 1 --2- 172.21.15.19:0/772374489 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ffb20077a80 0x7ffb20079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:21.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.627+0000 7ffb41c1f640 1 --2- 172.21.15.19:0/772374489 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ffb20077a80 0x7ffb20079f90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c0d0380 tx=0x7ffb2c005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:21.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.631+0000 7ffb32ffd640 1 -- 172.21.15.19:0/772374489 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7ffb38030030 con 0x7ffb28003a00 2024-08-31T12:16:21.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.815+0000 7ffb44684640 1 -- 172.21.15.19:0/772374489 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ffb3c1589e0 con 0x7ffb28003a00 2024-08-31T12:16:21.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.815+0000 7ffb32ffd640 1 -- 172.21.15.19:0/772374489 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7ffb38081b60 con 0x7ffb28003a00 2024-08-31T12:16:21.816 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 1 2024-08-31T12:16:21.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.817+0000 7ffb44684640 1 -- 172.21.15.19:0/772374489 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ffb20077a80 msgr2=0x7ffb20079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:21.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.817+0000 7ffb44684640 1 --2- 172.21.15.19:0/772374489 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ffb20077a80 0x7ffb20079f90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c0d0380 tx=0x7ffb2c005f90 comp rx=0 tx=0).stop 2024-08-31T12:16:21.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.817+0000 7ffb44684640 1 -- 172.21.15.19:0/772374489 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffb28003a00 msgr2=0x7ffb3c15cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:21.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.817+0000 7ffb44684640 1 --2- 172.21.15.19:0/772374489 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffb28003a00 0x7ffb3c15cb00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ffb38000fb0 tx=0x7ffb38001d20 comp rx=0 tx=0).stop 2024-08-31T12:16:21.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.817+0000 7ffb44684640 1 -- 172.21.15.19:0/772374489 shutdown_connections 2024-08-31T12:16:21.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.817+0000 7ffb44684640 1 --2- 172.21.15.19:0/772374489 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ffb20077a80 0x7ffb20079f90 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:21.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.817+0000 7ffb44684640 1 --2- 172.21.15.19:0/772374489 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffb3c0cfbc0 0x7ffb3c14f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:21.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.817+0000 7ffb44684640 1 --2- 172.21.15.19:0/772374489 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb3c0cf7b0 0x7ffb3c14f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:21.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.817+0000 7ffb44684640 1 --2- 172.21.15.19:0/772374489 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffb28003a00 0x7ffb3c15cb00 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:21.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.817+0000 7ffb44684640 1 -- 172.21.15.19:0/772374489 >> 172.21.15.19:0/772374489 conn(0x7ffb3c074210 msgr2=0x7ffb3c14bda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:21.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.817+0000 7ffb44684640 1 -- 172.21.15.19:0/772374489 shutdown_connections 2024-08-31T12:16:21.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.817+0000 7ffb44684640 1 -- 172.21.15.19:0/772374489 wait complete. 2024-08-31T12:16:21.829 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-08-31T12:16:21.946 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.945+0000 7ffa6bebd640 1 -- 172.21.15.19:0/2726139243 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffa640d0f60 msgr2=0x7ffa64163a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:21.947 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.945+0000 7ffa6bebd640 1 --2- 172.21.15.19:0/2726139243 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffa640d0f60 0x7ffa64163a30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ffa58009860 tx=0x7ffa5802a5c0 comp rx=0 tx=0).stop 2024-08-31T12:16:21.947 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.945+0000 7ffa6bebd640 1 -- 172.21.15.19:0/2726139243 shutdown_connections 2024-08-31T12:16:21.947 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.945+0000 7ffa6bebd640 1 --2- 172.21.15.19:0/2726139243 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffa640d0f60 0x7ffa64163a30 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:21.947 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.945+0000 7ffa6bebd640 1 -- 172.21.15.19:0/2726139243 >> 172.21.15.19:0/2726139243 conn(0x7ffa64074210 msgr2=0x7ffa64074620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:21.947 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.946+0000 7ffa6bebd640 1 -- 172.21.15.19:0/2726139243 shutdown_connections 2024-08-31T12:16:21.947 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.946+0000 7ffa6bebd640 1 -- 172.21.15.19:0/2726139243 wait complete. 2024-08-31T12:16:21.948 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.947+0000 7ffa6bebd640 1 Processor -- start 2024-08-31T12:16:21.948 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.947+0000 7ffa6bebd640 1 -- start start 2024-08-31T12:16:21.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.948+0000 7ffa6bebd640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffa640d0b50 0x7ffa6415a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:21.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.948+0000 7ffa6bebd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffa640d0f60 0x7ffa641f5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:21.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.948+0000 7ffa6bebd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffa64163f70 0x7ffa641f5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:21.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.948+0000 7ffa6bebd640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7ffa640cc130 con 0x7ffa640d0f60 2024-08-31T12:16:21.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.948+0000 7ffa6bebd640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ffa640d1370 con 0x7ffa640d0b50 2024-08-31T12:16:21.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.948+0000 7ffa6bebd640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7ffa64152170 con 0x7ffa64163f70 2024-08-31T12:16:21.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.949+0000 7ffa69458640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffa640d0f60 0x7ffa641f5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:21.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.949+0000 7ffa69458640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffa640d0f60 0x7ffa641f5550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:38038/0 (socket says 172.21.15.19:38038) 2024-08-31T12:16:21.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.949+0000 7ffa69458640 1 -- 172.21.15.19:0/3641510725 learned_addr learned my addr 172.21.15.19:0/3641510725 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:21.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.949+0000 7ffa69c59640 1 --2- 172.21.15.19:0/3641510725 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffa640d0b50 0x7ffa6415a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:21.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.949+0000 7ffa6a45a640 1 --2- 172.21.15.19:0/3641510725 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffa64163f70 0x7ffa641f5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:21.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.949+0000 7ffa69458640 1 -- 172.21.15.19:0/3641510725 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffa64163f70 msgr2=0x7ffa641f5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:21.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.949+0000 7ffa69458640 1 --2- 172.21.15.19:0/3641510725 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffa64163f70 0x7ffa641f5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:21.952 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.949+0000 7ffa69458640 1 -- 172.21.15.19:0/3641510725 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffa640d0b50 msgr2=0x7ffa6415a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:21.952 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.949+0000 7ffa69458640 1 --2- 172.21.15.19:0/3641510725 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffa640d0b50 0x7ffa6415a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:21.952 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.949+0000 7ffa69458640 1 -- 172.21.15.19:0/3641510725 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa6414d360 con 0x7ffa640d0f60 2024-08-31T12:16:21.952 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.949+0000 7ffa69458640 1 --2- 172.21.15.19:0/3641510725 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffa640d0f60 0x7ffa641f5550 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ffa5802abb0 tx=0x7ffa58003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:21.952 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.949+0000 7ffa627fc640 1 -- 172.21.15.19:0/3641510725 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa58005c90 con 0x7ffa640d0f60 2024-08-31T12:16:21.952 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.949+0000 7ffa627fc640 1 -- 172.21.15.19:0/3641510725 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffa58003e00 con 0x7ffa640d0f60 2024-08-31T12:16:21.952 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.949+0000 7ffa6bebd640 1 -- 172.21.15.19:0/3641510725 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa641f5080 con 0x7ffa640d0f60 2024-08-31T12:16:21.952 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.949+0000 7ffa627fc640 1 -- 172.21.15.19:0/3641510725 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa58005260 con 0x7ffa640d0f60 2024-08-31T12:16:21.952 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.949+0000 7ffa6bebd640 1 -- 172.21.15.19:0/3641510725 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa641f5fd0 con 0x7ffa640d0f60 2024-08-31T12:16:21.953 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.950+0000 7ffa6bebd640 1 -- 172.21.15.19:0/3641510725 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa64155680 con 0x7ffa640d0f60 2024-08-31T12:16:21.953 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.951+0000 7ffa627fc640 1 -- 172.21.15.19:0/3641510725 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7ffa58045020 con 0x7ffa640d0f60 2024-08-31T12:16:21.953 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.952+0000 7ffa627fc640 1 --2- 172.21.15.19:0/3641510725 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ffa50077a80 0x7ffa50079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:21.953 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.952+0000 7ffa627fc640 1 -- 172.21.15.19:0/3641510725 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ffa580b9cb0 con 0x7ffa640d0f60 2024-08-31T12:16:21.953 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.952+0000 7ffa69c59640 1 --2- 172.21.15.19:0/3641510725 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ffa50077a80 0x7ffa50079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:21.953 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.953+0000 7ffa69c59640 1 --2- 172.21.15.19:0/3641510725 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ffa50077a80 0x7ffa50079f90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ffa54001740 tx=0x7ffa54001aa0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:21.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:21.957+0000 7ffa627fc640 1 -- 172.21.15.19:0/3641510725 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7ffa58082340 con 0x7ffa640d0f60 2024-08-31T12:16:22.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.138+0000 7ffa6bebd640 1 -- 172.21.15.19:0/3641510725 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ffa640b33a0 con 0x7ffa640d0f60 2024-08-31T12:16:22.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.139+0000 7ffa627fc640 1 -- 172.21.15.19:0/3641510725 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7ffa58086b00 con 0x7ffa640d0f60 2024-08-31T12:16:22.140 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 32 2024-08-31T12:16:22.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.141+0000 7ffa6bebd640 1 -- 172.21.15.19:0/3641510725 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ffa50077a80 msgr2=0x7ffa50079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:22.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.141+0000 7ffa6bebd640 1 --2- 172.21.15.19:0/3641510725 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ffa50077a80 0x7ffa50079f90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ffa54001740 tx=0x7ffa54001aa0 comp rx=0 tx=0).stop 2024-08-31T12:16:22.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.141+0000 7ffa6bebd640 1 -- 172.21.15.19:0/3641510725 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffa640d0f60 msgr2=0x7ffa641f5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:22.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.141+0000 7ffa6bebd640 1 --2- 172.21.15.19:0/3641510725 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffa640d0f60 0x7ffa641f5550 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ffa5802abb0 tx=0x7ffa58003c10 comp rx=0 tx=0).stop 2024-08-31T12:16:22.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.141+0000 7ffa6bebd640 1 -- 172.21.15.19:0/3641510725 shutdown_connections 2024-08-31T12:16:22.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.141+0000 7ffa6bebd640 1 --2- 172.21.15.19:0/3641510725 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ffa50077a80 0x7ffa50079f90 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:22.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.141+0000 7ffa6bebd640 1 --2- 172.21.15.19:0/3641510725 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffa64163f70 0x7ffa641f5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:22.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.141+0000 7ffa6bebd640 1 --2- 172.21.15.19:0/3641510725 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffa640d0f60 0x7ffa641f5550 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:22.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.141+0000 7ffa6bebd640 1 --2- 172.21.15.19:0/3641510725 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffa640d0b50 0x7ffa6415a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:22.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.141+0000 7ffa6bebd640 1 -- 172.21.15.19:0/3641510725 >> 172.21.15.19:0/3641510725 conn(0x7ffa64074210 msgr2=0x7ffa64159300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:22.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.141+0000 7ffa6bebd640 1 -- 172.21.15.19:0/3641510725 shutdown_connections 2024-08-31T12:16:22.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.141+0000 7ffa6bebd640 1 -- 172.21.15.19:0/3641510725 wait complete. 2024-08-31T12:16:22.153 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-08-31T12:16:22.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.270+0000 7fbfea2b4640 1 -- 172.21.15.19:0/3343704222 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbfe40d0f60 msgr2=0x7fbfe4163a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:22.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.270+0000 7fbfea2b4640 1 --2- 172.21.15.19:0/3343704222 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbfe40d0f60 0x7fbfe4163a30 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fbfe40b4b80 tx=0x7fbfd802a680 comp rx=0 tx=0).stop 2024-08-31T12:16:22.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.270+0000 7fbfea2b4640 1 -- 172.21.15.19:0/3343704222 shutdown_connections 2024-08-31T12:16:22.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.270+0000 7fbfea2b4640 1 --2- 172.21.15.19:0/3343704222 >> v2:172.21.15.43:3300/0 conn(0x7fbfe4163f70 0x7fbfe4166390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:22.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.270+0000 7fbfea2b4640 1 --2- 172.21.15.19:0/3343704222 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbfe40d0f60 0x7fbfe4163a30 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:22.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.270+0000 7fbfea2b4640 1 -- 172.21.15.19:0/3343704222 >> 172.21.15.19:0/3343704222 conn(0x7fbfe4074210 msgr2=0x7fbfe4074620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:22.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.270+0000 7fbfea2b4640 1 -- 172.21.15.19:0/3343704222 shutdown_connections 2024-08-31T12:16:22.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.270+0000 7fbfea2b4640 1 -- 172.21.15.19:0/3343704222 wait complete. 2024-08-31T12:16:22.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.271+0000 7fbfea2b4640 1 Processor -- start 2024-08-31T12:16:22.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.272+0000 7fbfea2b4640 1 -- start start 2024-08-31T12:16:22.275 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.273+0000 7fbfea2b4640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbfe40d0b50 0x7fbfe41524a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:22.275 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.273+0000 7fbfea2b4640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbfe40d0f60 0x7fbfe414e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:22.275 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.273+0000 7fbfea2b4640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbfe4163f70 0x7fbfe4149560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:22.275 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.273+0000 7fbfea2b4640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe40cc130 con 0x7fbfe4163f70 2024-08-31T12:16:22.275 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.273+0000 7fbfea2b4640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe40d1370 con 0x7fbfe40d0b50 2024-08-31T12:16:22.275 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.273+0000 7fbfea2b4640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fbfe4152170 con 0x7fbfe40d0f60 2024-08-31T12:16:22.275 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.273+0000 7fbfe8851640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbfe4163f70 0x7fbfe4149560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:22.275 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.273+0000 7fbfe8851640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbfe4163f70 0x7fbfe4149560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:38066/0 (socket says 172.21.15.19:38066) 2024-08-31T12:16:22.275 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.273+0000 7fbfe8851640 1 -- 172.21.15.19:0/1503164801 learned_addr learned my addr 172.21.15.19:0/1503164801 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:22.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.273+0000 7fbfe3fff640 1 --2- 172.21.15.19:0/1503164801 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbfe40d0b50 0x7fbfe41524a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:22.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.273+0000 7fbfe37fe640 1 --2- 172.21.15.19:0/1503164801 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbfe40d0f60 0x7fbfe414e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:22.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.273+0000 7fbfe8851640 1 -- 172.21.15.19:0/1503164801 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbfe40d0f60 msgr2=0x7fbfe414e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:22.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.273+0000 7fbfe8851640 1 --2- 172.21.15.19:0/1503164801 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbfe40d0f60 0x7fbfe414e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:22.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.273+0000 7fbfe8851640 1 -- 172.21.15.19:0/1503164801 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbfe40d0b50 msgr2=0x7fbfe41524a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:22.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.273+0000 7fbfe8851640 1 --2- 172.21.15.19:0/1503164801 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbfe40d0b50 0x7fbfe41524a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:22.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.273+0000 7fbfe8851640 1 -- 172.21.15.19:0/1503164801 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfe414e570 con 0x7fbfe4163f70 2024-08-31T12:16:22.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.273+0000 7fbfe3fff640 1 --2- 172.21.15.19:0/1503164801 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbfe40d0b50 0x7fbfe41524a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:16:22.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.273+0000 7fbfe37fe640 1 --2- 172.21.15.19:0/1503164801 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbfe40d0f60 0x7fbfe414e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:16:22.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.273+0000 7fbfe8851640 1 --2- 172.21.15.19:0/1503164801 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbfe4163f70 0x7fbfe4149560 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc00ea70 tx=0x7fbfdc00ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:22.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.274+0000 7fbfe0ff9640 1 -- 172.21.15.19:0/1503164801 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc00ce40 con 0x7fbfe4163f70 2024-08-31T12:16:22.277 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.274+0000 7fbfe0ff9640 1 -- 172.21.15.19:0/1503164801 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbfdc004c10 con 0x7fbfe4163f70 2024-08-31T12:16:22.277 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.274+0000 7fbfea2b4640 1 -- 172.21.15.19:0/1503164801 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfe4151bc0 con 0x7fbfe4163f70 2024-08-31T12:16:22.277 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.274+0000 7fbfe0ff9640 1 -- 172.21.15.19:0/1503164801 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc005020 con 0x7fbfe4163f70 2024-08-31T12:16:22.277 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.274+0000 7fbfea2b4640 1 -- 172.21.15.19:0/1503164801 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfe415d7c0 con 0x7fbfe4163f70 2024-08-31T12:16:22.278 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.275+0000 7fbfea2b4640 1 -- 172.21.15.19:0/1503164801 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfe4155680 con 0x7fbfe4163f70 2024-08-31T12:16:22.278 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.275+0000 7fbfe0ff9640 1 -- 172.21.15.19:0/1503164801 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fbfdc0051c0 con 0x7fbfe4163f70 2024-08-31T12:16:22.278 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.276+0000 7fbfe0ff9640 1 --2- 172.21.15.19:0/1503164801 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fbfd0077a80 0x7fbfd0079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:22.278 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.276+0000 7fbfe0ff9640 1 -- 172.21.15.19:0/1503164801 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fbfdc09a3d0 con 0x7fbfe4163f70 2024-08-31T12:16:22.278 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.276+0000 7fbfe3fff640 1 --2- 172.21.15.19:0/1503164801 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fbfd0077a80 0x7fbfd0079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:22.278 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.276+0000 7fbfe3fff640 1 --2- 172.21.15.19:0/1503164801 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fbfd0077a80 0x7fbfd0079f90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4007fd0 tx=0x7fbfd4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:22.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.280+0000 7fbfe0ff9640 1 -- 172.21.15.19:0/1503164801 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fbfdc0631d0 con 0x7fbfe4163f70 2024-08-31T12:16:22.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.466+0000 7fbfea2b4640 1 -- 172.21.15.19:0/1503164801 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fbfe40b33a0 con 0x7fbfe4163f70 2024-08-31T12:16:22.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.466+0000 7fbfe0ff9640 1 -- 172.21.15.19:0/1503164801 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7fbfdc062ad0 con 0x7fbfe4163f70 2024-08-31T12:16:22.467 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 64 2024-08-31T12:16:22.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.468+0000 7fbfea2b4640 1 -- 172.21.15.19:0/1503164801 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fbfd0077a80 msgr2=0x7fbfd0079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:22.470 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.468+0000 7fbfea2b4640 1 --2- 172.21.15.19:0/1503164801 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fbfd0077a80 0x7fbfd0079f90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4007fd0 tx=0x7fbfd4009040 comp rx=0 tx=0).stop 2024-08-31T12:16:22.470 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.468+0000 7fbfea2b4640 1 -- 172.21.15.19:0/1503164801 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbfe4163f70 msgr2=0x7fbfe4149560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:22.470 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.468+0000 7fbfea2b4640 1 --2- 172.21.15.19:0/1503164801 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbfe4163f70 0x7fbfe4149560 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc00ea70 tx=0x7fbfdc00ef30 comp rx=0 tx=0).stop 2024-08-31T12:16:22.470 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.468+0000 7fbfea2b4640 1 -- 172.21.15.19:0/1503164801 shutdown_connections 2024-08-31T12:16:22.470 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.468+0000 7fbfea2b4640 1 --2- 172.21.15.19:0/1503164801 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fbfd0077a80 0x7fbfd0079f90 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:22.470 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.468+0000 7fbfea2b4640 1 --2- 172.21.15.19:0/1503164801 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbfe4163f70 0x7fbfe4149560 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:22.470 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.468+0000 7fbfea2b4640 1 --2- 172.21.15.19:0/1503164801 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbfe40d0f60 0x7fbfe414e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:22.470 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.468+0000 7fbfea2b4640 1 --2- 172.21.15.19:0/1503164801 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbfe40d0b50 0x7fbfe41524a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:22.470 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.468+0000 7fbfea2b4640 1 -- 172.21.15.19:0/1503164801 >> 172.21.15.19:0/1503164801 conn(0x7fbfe4074210 msgr2=0x7fbfe415a330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:22.470 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.468+0000 7fbfea2b4640 1 -- 172.21.15.19:0/1503164801 shutdown_connections 2024-08-31T12:16:22.470 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.468+0000 7fbfea2b4640 1 -- 172.21.15.19:0/1503164801 wait complete. 2024-08-31T12:16:22.480 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-08-31T12:16:22.481 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-31T12:16:22.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.595+0000 7f7fca779640 1 -- 172.21.15.19:0/1821743327 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7fc415bd90 msgr2=0x7f7fc40cbeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:22.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.595+0000 7f7fca779640 1 --2- 172.21.15.19:0/1821743327 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7fc415bd90 0x7f7fc40cbeb0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc005fd0 tx=0x7f7fbc02aa10 comp rx=0 tx=0).stop 2024-08-31T12:16:22.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.595+0000 7f7fca779640 1 -- 172.21.15.19:0/1821743327 shutdown_connections 2024-08-31T12:16:22.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.595+0000 7f7fca779640 1 --2- 172.21.15.19:0/1821743327 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7fc415bd90 0x7f7fc40cbeb0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:22.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.595+0000 7f7fca779640 1 --2- 172.21.15.19:0/1821743327 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7fc40cca10 0x7f7fc40c75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:22.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.595+0000 7f7fca779640 1 -- 172.21.15.19:0/1821743327 >> 172.21.15.19:0/1821743327 conn(0x7f7fc406e6e0 msgr2=0x7f7fc406eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:22.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.595+0000 7f7fca779640 1 -- 172.21.15.19:0/1821743327 shutdown_connections 2024-08-31T12:16:22.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.595+0000 7f7fca779640 1 -- 172.21.15.19:0/1821743327 wait complete. 2024-08-31T12:16:22.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.596+0000 7f7fca779640 1 Processor -- start 2024-08-31T12:16:22.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.597+0000 7f7fca779640 1 -- start start 2024-08-31T12:16:22.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.598+0000 7f7fca779640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7fc40cca10 0x7f7fc4153a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:22.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.598+0000 7f7fca779640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7fc4155360 0x7f7fc41f2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:22.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.598+0000 7f7fca779640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7fc415bd90 0x7f7fc40c6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:22.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.598+0000 7f7fca779640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f7fc40d0c50 con 0x7f7fc40cca10 2024-08-31T12:16:22.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.598+0000 7f7fca779640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7fc415b5d0 con 0x7f7fc4155360 2024-08-31T12:16:22.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.598+0000 7f7fca779640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f7fc40d0dd0 con 0x7f7fc415bd90 2024-08-31T12:16:22.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.598+0000 7f7fc3fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7fc40cca10 0x7f7fc4153a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:22.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.598+0000 7f7fc3fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7fc40cca10 0x7f7fc4153a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:38078/0 (socket says 172.21.15.19:38078) 2024-08-31T12:16:22.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.598+0000 7f7fc3fff640 1 -- 172.21.15.19:0/1147421039 learned_addr learned my addr 172.21.15.19:0/1147421039 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:22.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.598+0000 7f7fc37fe640 1 --2- 172.21.15.19:0/1147421039 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7fc4155360 0x7f7fc41f2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:22.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.598+0000 7f7fc8d16640 1 --2- 172.21.15.19:0/1147421039 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7fc415bd90 0x7f7fc40c6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:22.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.598+0000 7f7fc3fff640 1 -- 172.21.15.19:0/1147421039 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7fc415bd90 msgr2=0x7f7fc40c6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:22.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.598+0000 7f7fc3fff640 1 --2- 172.21.15.19:0/1147421039 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7fc415bd90 0x7f7fc40c6060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:22.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.598+0000 7f7fc3fff640 1 -- 172.21.15.19:0/1147421039 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7fc4155360 msgr2=0x7f7fc41f2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:22.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.598+0000 7f7fc3fff640 1 --2- 172.21.15.19:0/1147421039 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7fc4155360 0x7f7fc41f2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:22.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.598+0000 7f7fc3fff640 1 -- 172.21.15.19:0/1147421039 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fc41f2f80 con 0x7f7fc40cca10 2024-08-31T12:16:22.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.599+0000 7f7fc3fff640 1 --2- 172.21.15.19:0/1147421039 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7fc40cca10 0x7f7fc4153a50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7fb400c3a0 tx=0x7f7fb400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:22.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.599+0000 7f7fc0ff9640 1 -- 172.21.15.19:0/1147421039 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb40052d0 con 0x7f7fc40cca10 2024-08-31T12:16:22.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.599+0000 7f7fc0ff9640 1 -- 172.21.15.19:0/1147421039 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7fb4007020 con 0x7f7fc40cca10 2024-08-31T12:16:22.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.599+0000 7f7fca779640 1 -- 172.21.15.19:0/1147421039 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fc41e6ec0 con 0x7f7fc40cca10 2024-08-31T12:16:22.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.599+0000 7f7fc0ff9640 1 -- 172.21.15.19:0/1147421039 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb4007470 con 0x7f7fc40cca10 2024-08-31T12:16:22.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.599+0000 7f7fca779640 1 -- 172.21.15.19:0/1147421039 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fc41522d0 con 0x7f7fc40cca10 2024-08-31T12:16:22.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.600+0000 7f7fca779640 1 -- 172.21.15.19:0/1147421039 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fc41520e0 con 0x7f7fc40cca10 2024-08-31T12:16:22.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.601+0000 7f7fc0ff9640 1 -- 172.21.15.19:0/1147421039 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f7fb40067d0 con 0x7f7fc40cca10 2024-08-31T12:16:22.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.601+0000 7f7fc0ff9640 1 --2- 172.21.15.19:0/1147421039 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7fb0077a50 0x7f7fb0079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:22.603 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.601+0000 7f7fc0ff9640 1 -- 172.21.15.19:0/1147421039 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f7fb4098ad0 con 0x7f7fc40cca10 2024-08-31T12:16:22.603 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.601+0000 7f7fc37fe640 1 --2- 172.21.15.19:0/1147421039 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7fb0077a50 0x7f7fb0079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:22.603 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.602+0000 7f7fc37fe640 1 --2- 172.21.15.19:0/1147421039 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7fb0077a50 0x7f7fb0079f60 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7fc41e4020 tx=0x7f7fb800a250 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:22.606 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.606+0000 7f7fc0ff9640 1 -- 172.21.15.19:0/1147421039 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f7fb4061160 con 0x7f7fc40cca10 2024-08-31T12:16:22.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.787+0000 7f7fca779640 1 -- 172.21.15.19:0/1147421039 --> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7fc40c75b0 con 0x7f7fb0077a50 2024-08-31T12:16:22.792 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.792+0000 7f7fc0ff9640 1 -- 172.21.15.19:0/1147421039 <== mgr.4100 v2:172.21.15.43:6800/2824336655 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251905 (secure 0 0 0) 0x7f7fc40c75b0 con 0x7f7fb0077a50 2024-08-31T12:16:22.795 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:16:22.796 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-08-31T12:16:22.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.795+0000 7f7fca779640 1 -- 172.21.15.19:0/1147421039 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7fb0077a50 msgr2=0x7f7fb0079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:22.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.795+0000 7f7fca779640 1 --2- 172.21.15.19:0/1147421039 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7fb0077a50 0x7f7fb0079f60 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7fc41e4020 tx=0x7f7fb800a250 comp rx=0 tx=0).stop 2024-08-31T12:16:22.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.795+0000 7f7fca779640 1 -- 172.21.15.19:0/1147421039 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7fc40cca10 msgr2=0x7f7fc4153a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:22.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.795+0000 7f7fca779640 1 --2- 172.21.15.19:0/1147421039 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7fc40cca10 0x7f7fc4153a50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7fb400c3a0 tx=0x7f7fb400c860 comp rx=0 tx=0).stop 2024-08-31T12:16:22.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.795+0000 7f7fca779640 1 -- 172.21.15.19:0/1147421039 shutdown_connections 2024-08-31T12:16:22.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.795+0000 7f7fca779640 1 --2- 172.21.15.19:0/1147421039 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7fb0077a50 0x7f7fb0079f60 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:22.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.795+0000 7f7fca779640 1 --2- 172.21.15.19:0/1147421039 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7fc415bd90 0x7f7fc40c6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:22.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.795+0000 7f7fca779640 1 --2- 172.21.15.19:0/1147421039 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7fc4155360 0x7f7fc41f2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:22.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.795+0000 7f7fca779640 1 --2- 172.21.15.19:0/1147421039 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7fc40cca10 0x7f7fc4153a50 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:22.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.795+0000 7f7fca779640 1 -- 172.21.15.19:0/1147421039 >> 172.21.15.19:0/1147421039 conn(0x7f7fc406e6e0 msgr2=0x7f7fc40691f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:22.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.795+0000 7f7fca779640 1 -- 172.21.15.19:0/1147421039 shutdown_connections 2024-08-31T12:16:22.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.795+0000 7f7fca779640 1 -- 172.21.15.19:0/1147421039 wait complete. 2024-08-31T12:16:22.807 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-08-31T12:16:22.497453+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596031,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":227,"num_write_kb":1667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":182,"ondisk_log_size":182,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":252424,"kb_used_data":2952,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754722296,"statfs":{"total":773094113280,"available":772835631104,"internally_reserved":0,"allocated":3022848,"data_stored":1775988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3231,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003066"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306730+0000","last_change":"2024-08-31T12:15:57.306560+0000","last_active":"2024-08-31T12:15:58.306730+0000","last_peered":"2024-08-31T12:15:58.306730+0000","last_clean":"2024-08-31T12:15:58.306730+0000","last_became_active":"2024-08-31T12:15:57.306148+0000","last_became_peered":"2024-08-31T12:15:57.306148+0000","last_unstale":"2024-08-31T12:15:58.306730+0000","last_undegraded":"2024-08-31T12:15:58.306730+0000","last_fullsized":"2024-08-31T12:15:58.306730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:35:53.831742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300935+0000","last_change":"2024-08-31T12:15:57.307314+0000","last_active":"2024-08-31T12:15:58.300935+0000","last_peered":"2024-08-31T12:15:58.300935+0000","last_clean":"2024-08-31T12:15:58.300935+0000","last_became_active":"2024-08-31T12:15:57.306940+0000","last_became_peered":"2024-08-31T12:15:57.306940+0000","last_unstale":"2024-08-31T12:15:58.300935+0000","last_undegraded":"2024-08-31T12:15:58.300935+0000","last_fullsized":"2024-08-31T12:15:58.300935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:38:30.321214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937267+0000","last_change":"2024-08-31T12:15:57.309372+0000","last_active":"2024-08-31T12:15:58.937267+0000","last_peered":"2024-08-31T12:15:58.937267+0000","last_clean":"2024-08-31T12:15:58.937267+0000","last_became_active":"2024-08-31T12:15:57.305795+0000","last_became_peered":"2024-08-31T12:15:57.305795+0000","last_unstale":"2024-08-31T12:15:58.937267+0000","last_undegraded":"2024-08-31T12:15:58.937267+0000","last_fullsized":"2024-08-31T12:15:58.937267+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:37:45.891394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302363+0000","last_change":"2024-08-31T12:15:57.307339+0000","last_active":"2024-08-31T12:15:58.302363+0000","last_peered":"2024-08-31T12:15:58.302363+0000","last_clean":"2024-08-31T12:15:58.302363+0000","last_became_active":"2024-08-31T12:15:57.306957+0000","last_became_peered":"2024-08-31T12:15:57.306957+0000","last_unstale":"2024-08-31T12:15:58.302363+0000","last_undegraded":"2024-08-31T12:15:58.302363+0000","last_fullsized":"2024-08-31T12:15:58.302363+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:43:33.705099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306757+0000","last_change":"2024-08-31T12:15:57.317492+0000","last_active":"2024-08-31T12:15:58.306757+0000","last_peered":"2024-08-31T12:15:58.306757+0000","last_clean":"2024-08-31T12:15:58.306757+0000","last_became_active":"2024-08-31T12:15:57.317079+0000","last_became_peered":"2024-08-31T12:15:57.317079+0000","last_unstale":"2024-08-31T12:15:58.306757+0000","last_undegraded":"2024-08-31T12:15:58.306757+0000","last_fullsized":"2024-08-31T12:15:58.306757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:44:40.712699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302342+0000","last_change":"2024-08-31T12:15:57.307505+0000","last_active":"2024-08-31T12:15:58.302342+0000","last_peered":"2024-08-31T12:15:58.302342+0000","last_clean":"2024-08-31T12:15:58.302342+0000","last_became_active":"2024-08-31T12:15:57.307010+0000","last_became_peered":"2024-08-31T12:15:57.307010+0000","last_unstale":"2024-08-31T12:15:58.302342+0000","last_undegraded":"2024-08-31T12:15:58.302342+0000","last_fullsized":"2024-08-31T12:15:58.302342+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:39:31.100565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306247+0000","last_change":"2024-08-31T12:15:57.316892+0000","last_active":"2024-08-31T12:15:58.306247+0000","last_peered":"2024-08-31T12:15:58.306247+0000","last_clean":"2024-08-31T12:15:58.306247+0000","last_became_active":"2024-08-31T12:15:57.316504+0000","last_became_peered":"2024-08-31T12:15:57.316504+0000","last_unstale":"2024-08-31T12:15:58.306247+0000","last_undegraded":"2024-08-31T12:15:58.306247+0000","last_fullsized":"2024-08-31T12:15:58.306247+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:05:48.614918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937976+0000","last_change":"2024-08-31T12:15:57.309224+0000","last_active":"2024-08-31T12:15:58.937976+0000","last_peered":"2024-08-31T12:15:58.937976+0000","last_clean":"2024-08-31T12:15:58.937976+0000","last_became_active":"2024-08-31T12:15:57.308914+0000","last_became_peered":"2024-08-31T12:15:57.308914+0000","last_unstale":"2024-08-31T12:15:58.937976+0000","last_undegraded":"2024-08-31T12:15:58.937976+0000","last_fullsized":"2024-08-31T12:15:58.937976+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:37:02.435678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306762+0000","last_change":"2024-08-31T12:15:57.316723+0000","last_active":"2024-08-31T12:15:58.306762+0000","last_peered":"2024-08-31T12:15:58.306762+0000","last_clean":"2024-08-31T12:15:58.306762+0000","last_became_active":"2024-08-31T12:15:57.316354+0000","last_became_peered":"2024-08-31T12:15:57.316354+0000","last_unstale":"2024-08-31T12:15:58.306762+0000","last_undegraded":"2024-08-31T12:15:58.306762+0000","last_fullsized":"2024-08-31T12:15:58.306762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:31:19.860950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302888+0000","last_change":"2024-08-31T12:15:57.311161+0000","last_active":"2024-08-31T12:15:58.302888+0000","last_peered":"2024-08-31T12:15:58.302888+0000","last_clean":"2024-08-31T12:15:58.302888+0000","last_became_active":"2024-08-31T12:15:57.310807+0000","last_became_peered":"2024-08-31T12:15:57.310807+0000","last_unstale":"2024-08-31T12:15:58.302888+0000","last_undegraded":"2024-08-31T12:15:58.302888+0000","last_fullsized":"2024-08-31T12:15:58.302888+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:56:03.280339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302118+0000","last_change":"2024-08-31T12:15:57.300930+0000","last_active":"2024-08-31T12:15:58.302118+0000","last_peered":"2024-08-31T12:15:58.302118+0000","last_clean":"2024-08-31T12:15:58.302118+0000","last_became_active":"2024-08-31T12:15:57.300610+0000","last_became_peered":"2024-08-31T12:15:57.300610+0000","last_unstale":"2024-08-31T12:15:58.302118+0000","last_undegraded":"2024-08-31T12:15:58.302118+0000","last_fullsized":"2024-08-31T12:15:58.302118+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:42:39.731742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936435+0000","last_change":"2024-08-31T12:15:57.303151+0000","last_active":"2024-08-31T12:15:58.936435+0000","last_peered":"2024-08-31T12:15:58.936435+0000","last_clean":"2024-08-31T12:15:58.936435+0000","last_became_active":"2024-08-31T12:15:57.302622+0000","last_became_peered":"2024-08-31T12:15:57.302622+0000","last_unstale":"2024-08-31T12:15:58.936435+0000","last_undegraded":"2024-08-31T12:15:58.936435+0000","last_fullsized":"2024-08-31T12:15:58.936435+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:49:18.493629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937309+0000","last_change":"2024-08-31T12:15:57.305676+0000","last_active":"2024-08-31T12:15:58.937309+0000","last_peered":"2024-08-31T12:15:58.937309+0000","last_clean":"2024-08-31T12:15:58.937309+0000","last_became_active":"2024-08-31T12:15:57.305321+0000","last_became_peered":"2024-08-31T12:15:57.305321+0000","last_unstale":"2024-08-31T12:15:58.937309+0000","last_undegraded":"2024-08-31T12:15:58.937309+0000","last_fullsized":"2024-08-31T12:15:58.937309+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:42:39.731742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936688+0000","last_change":"2024-08-31T12:15:57.309671+0000","last_active":"2024-08-31T12:15:58.936688+0000","last_peered":"2024-08-31T12:15:58.936688+0000","last_clean":"2024-08-31T12:15:58.936688+0000","last_became_active":"2024-08-31T12:15:57.307013+0000","last_became_peered":"2024-08-31T12:15:57.307013+0000","last_unstale":"2024-08-31T12:15:58.936688+0000","last_undegraded":"2024-08-31T12:15:58.936688+0000","last_fullsized":"2024-08-31T12:15:58.936688+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:18:55.451164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300929+0000","last_change":"2024-08-31T12:15:57.298150+0000","last_active":"2024-08-31T12:15:58.300929+0000","last_peered":"2024-08-31T12:15:58.300929+0000","last_clean":"2024-08-31T12:15:58.300929+0000","last_became_active":"2024-08-31T12:15:57.297182+0000","last_became_peered":"2024-08-31T12:15:57.297182+0000","last_unstale":"2024-08-31T12:15:58.300929+0000","last_undegraded":"2024-08-31T12:15:58.300929+0000","last_fullsized":"2024-08-31T12:15:58.300929+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:16:42.625396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302762+0000","last_change":"2024-08-31T12:15:57.314224+0000","last_active":"2024-08-31T12:15:58.302762+0000","last_peered":"2024-08-31T12:15:58.302762+0000","last_clean":"2024-08-31T12:15:58.302762+0000","last_became_active":"2024-08-31T12:15:57.313887+0000","last_became_peered":"2024-08-31T12:15:57.313887+0000","last_unstale":"2024-08-31T12:15:58.302762+0000","last_undegraded":"2024-08-31T12:15:58.302762+0000","last_fullsized":"2024-08-31T12:15:58.302762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:54:45.720621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303640+0000","last_change":"2024-08-31T12:15:57.308899+0000","last_active":"2024-08-31T12:15:58.303640+0000","last_peered":"2024-08-31T12:15:58.303640+0000","last_clean":"2024-08-31T12:15:58.303640+0000","last_became_active":"2024-08-31T12:15:57.308388+0000","last_became_peered":"2024-08-31T12:15:57.308388+0000","last_unstale":"2024-08-31T12:15:58.303640+0000","last_undegraded":"2024-08-31T12:15:58.303640+0000","last_fullsized":"2024-08-31T12:15:58.303640+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:54:03.871840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936410+0000","last_change":"2024-08-31T12:15:57.300201+0000","last_active":"2024-08-31T12:15:58.936410+0000","last_peered":"2024-08-31T12:15:58.936410+0000","last_clean":"2024-08-31T12:15:58.936410+0000","last_became_active":"2024-08-31T12:15:57.299826+0000","last_became_peered":"2024-08-31T12:15:57.299826+0000","last_unstale":"2024-08-31T12:15:58.936410+0000","last_undegraded":"2024-08-31T12:15:58.936410+0000","last_fullsized":"2024-08-31T12:15:58.936410+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:14:34.488102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.307053+0000","last_change":"2024-08-31T12:15:57.317606+0000","last_active":"2024-08-31T12:15:58.307053+0000","last_peered":"2024-08-31T12:15:58.307053+0000","last_clean":"2024-08-31T12:15:58.307053+0000","last_became_active":"2024-08-31T12:15:57.317185+0000","last_became_peered":"2024-08-31T12:15:57.317185+0000","last_unstale":"2024-08-31T12:15:58.307053+0000","last_undegraded":"2024-08-31T12:15:58.307053+0000","last_fullsized":"2024-08-31T12:15:58.307053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:25:46.346490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302171+0000","last_change":"2024-08-31T12:15:57.298778+0000","last_active":"2024-08-31T12:15:58.302171+0000","last_peered":"2024-08-31T12:15:58.302171+0000","last_clean":"2024-08-31T12:15:58.302171+0000","last_became_active":"2024-08-31T12:15:57.298430+0000","last_became_peered":"2024-08-31T12:15:57.298430+0000","last_unstale":"2024-08-31T12:15:58.302171+0000","last_undegraded":"2024-08-31T12:15:58.302171+0000","last_fullsized":"2024-08-31T12:15:58.302171+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:22:13.435300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937190+0000","last_change":"2024-08-31T12:15:57.317176+0000","last_active":"2024-08-31T12:15:58.937190+0000","last_peered":"2024-08-31T12:15:58.937190+0000","last_clean":"2024-08-31T12:15:58.937190+0000","last_became_active":"2024-08-31T12:15:57.316870+0000","last_became_peered":"2024-08-31T12:15:57.316870+0000","last_unstale":"2024-08-31T12:15:58.937190+0000","last_undegraded":"2024-08-31T12:15:58.937190+0000","last_fullsized":"2024-08-31T12:15:58.937190+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:43:12.703971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302636+0000","last_change":"2024-08-31T12:15:57.308907+0000","last_active":"2024-08-31T12:15:58.302636+0000","last_peered":"2024-08-31T12:15:58.302636+0000","last_clean":"2024-08-31T12:15:58.302636+0000","last_became_active":"2024-08-31T12:15:57.308459+0000","last_became_peered":"2024-08-31T12:15:57.308459+0000","last_unstale":"2024-08-31T12:15:58.302636+0000","last_undegraded":"2024-08-31T12:15:58.302636+0000","last_fullsized":"2024-08-31T12:15:58.302636+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:16:46.648912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937760+0000","last_change":"2024-08-31T12:15:57.304032+0000","last_active":"2024-08-31T12:15:58.937760+0000","last_peered":"2024-08-31T12:15:58.937760+0000","last_clean":"2024-08-31T12:15:58.937760+0000","last_became_active":"2024-08-31T12:15:57.303401+0000","last_became_peered":"2024-08-31T12:15:57.303401+0000","last_unstale":"2024-08-31T12:15:58.937760+0000","last_undegraded":"2024-08-31T12:15:58.937760+0000","last_fullsized":"2024-08-31T12:15:58.937760+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:56:03.280339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303013+0000","last_change":"2024-08-31T12:15:57.305412+0000","last_active":"2024-08-31T12:15:58.303013+0000","last_peered":"2024-08-31T12:15:58.303013+0000","last_clean":"2024-08-31T12:15:58.303013+0000","last_became_active":"2024-08-31T12:15:57.305075+0000","last_became_peered":"2024-08-31T12:15:57.305075+0000","last_unstale":"2024-08-31T12:15:58.303013+0000","last_undegraded":"2024-08-31T12:15:58.303013+0000","last_fullsized":"2024-08-31T12:15:58.303013+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:30:13.715505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937828+0000","last_change":"2024-08-31T12:15:57.300179+0000","last_active":"2024-08-31T12:15:58.937828+0000","last_peered":"2024-08-31T12:15:58.937828+0000","last_clean":"2024-08-31T12:15:58.937828+0000","last_became_active":"2024-08-31T12:15:57.299861+0000","last_became_peered":"2024-08-31T12:15:57.299861+0000","last_unstale":"2024-08-31T12:15:58.937828+0000","last_undegraded":"2024-08-31T12:15:58.937828+0000","last_fullsized":"2024-08-31T12:15:58.937828+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:47:51.267409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303001+0000","last_change":"2024-08-31T12:15:57.308519+0000","last_active":"2024-08-31T12:15:58.303001+0000","last_peered":"2024-08-31T12:15:58.303001+0000","last_clean":"2024-08-31T12:15:58.303001+0000","last_became_active":"2024-08-31T12:15:57.308189+0000","last_became_peered":"2024-08-31T12:15:57.308189+0000","last_unstale":"2024-08-31T12:15:58.303001+0000","last_undegraded":"2024-08-31T12:15:58.303001+0000","last_fullsized":"2024-08-31T12:15:58.303001+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:37:02.435678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300666+0000","last_change":"2024-08-31T12:15:57.308321+0000","last_active":"2024-08-31T12:15:58.300666+0000","last_peered":"2024-08-31T12:15:58.300666+0000","last_clean":"2024-08-31T12:15:58.300666+0000","last_became_active":"2024-08-31T12:15:57.304514+0000","last_became_peered":"2024-08-31T12:15:57.304514+0000","last_unstale":"2024-08-31T12:15:58.300666+0000","last_undegraded":"2024-08-31T12:15:58.300666+0000","last_fullsized":"2024-08-31T12:15:58.300666+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:09:49.617826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936830+0000","last_change":"2024-08-31T12:15:57.299268+0000","last_active":"2024-08-31T12:15:58.936830+0000","last_peered":"2024-08-31T12:15:58.936830+0000","last_clean":"2024-08-31T12:15:58.936830+0000","last_became_active":"2024-08-31T12:15:57.298766+0000","last_became_peered":"2024-08-31T12:15:57.298766+0000","last_unstale":"2024-08-31T12:15:58.936830+0000","last_undegraded":"2024-08-31T12:15:58.936830+0000","last_fullsized":"2024-08-31T12:15:58.936830+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:37:45.891394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937525+0000","last_change":"2024-08-31T12:15:57.310121+0000","last_active":"2024-08-31T12:15:58.937525+0000","last_peered":"2024-08-31T12:15:58.937525+0000","last_clean":"2024-08-31T12:15:58.937525+0000","last_became_active":"2024-08-31T12:15:57.309440+0000","last_became_peered":"2024-08-31T12:15:57.309440+0000","last_unstale":"2024-08-31T12:15:58.937525+0000","last_undegraded":"2024-08-31T12:15:58.937525+0000","last_fullsized":"2024-08-31T12:15:58.937525+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:53:19.523811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302889+0000","last_change":"2024-08-31T12:15:57.306214+0000","last_active":"2024-08-31T12:15:58.302889+0000","last_peered":"2024-08-31T12:15:58.302889+0000","last_clean":"2024-08-31T12:15:58.302889+0000","last_became_active":"2024-08-31T12:15:57.305878+0000","last_became_peered":"2024-08-31T12:15:57.305878+0000","last_unstale":"2024-08-31T12:15:58.302889+0000","last_undegraded":"2024-08-31T12:15:58.302889+0000","last_fullsized":"2024-08-31T12:15:58.302889+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:05:48.614918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.938282+0000","last_change":"2024-08-31T12:15:57.307376+0000","last_active":"2024-08-31T12:15:58.938282+0000","last_peered":"2024-08-31T12:15:58.938282+0000","last_clean":"2024-08-31T12:15:58.938282+0000","last_became_active":"2024-08-31T12:15:57.307056+0000","last_became_peered":"2024-08-31T12:15:57.307056+0000","last_unstale":"2024-08-31T12:15:58.938282+0000","last_undegraded":"2024-08-31T12:15:58.938282+0000","last_fullsized":"2024-08-31T12:15:58.938282+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:52:18.004608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302319+0000","last_change":"2024-08-31T12:15:57.313798+0000","last_active":"2024-08-31T12:15:58.302319+0000","last_peered":"2024-08-31T12:15:58.302319+0000","last_clean":"2024-08-31T12:15:58.302319+0000","last_became_active":"2024-08-31T12:15:57.313463+0000","last_became_peered":"2024-08-31T12:15:57.313463+0000","last_unstale":"2024-08-31T12:15:58.302319+0000","last_undegraded":"2024-08-31T12:15:58.302319+0000","last_fullsized":"2024-08-31T12:15:58.302319+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:47:51.267409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936240+0000","last_change":"2024-08-31T12:15:57.293350+0000","last_active":"2024-08-31T12:15:58.936240+0000","last_peered":"2024-08-31T12:15:58.936240+0000","last_clean":"2024-08-31T12:15:58.936240+0000","last_became_active":"2024-08-31T12:15:57.292981+0000","last_became_peered":"2024-08-31T12:15:57.292981+0000","last_unstale":"2024-08-31T12:15:58.936240+0000","last_undegraded":"2024-08-31T12:15:58.936240+0000","last_fullsized":"2024-08-31T12:15:58.936240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:52:00.312013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'6","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:07.349665+0000","last_change":"2024-08-31T12:15:56.293497+0000","last_active":"2024-08-31T12:16:07.349665+0000","last_peered":"2024-08-31T12:16:07.349665+0000","last_clean":"2024-08-31T12:16:07.349665+0000","last_became_active":"2024-08-31T12:15:56.293164+0000","last_became_peered":"2024-08-31T12:15:56.293164+0000","last_unstale":"2024-08-31T12:16:07.349665+0000","last_undegraded":"2024-08-31T12:16:07.349665+0000","last_fullsized":"2024-08-31T12:16:07.349665+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:43:17.443433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1063,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937814+0000","last_change":"2024-08-31T12:15:57.312738+0000","last_active":"2024-08-31T12:15:58.937814+0000","last_peered":"2024-08-31T12:15:58.937814+0000","last_clean":"2024-08-31T12:15:58.937814+0000","last_became_active":"2024-08-31T12:15:57.312360+0000","last_became_peered":"2024-08-31T12:15:57.312360+0000","last_unstale":"2024-08-31T12:15:58.937814+0000","last_undegraded":"2024-08-31T12:15:58.937814+0000","last_fullsized":"2024-08-31T12:15:58.937814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:22:13.435300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'3","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:13.370679+0000","last_change":"2024-08-31T12:15:56.283526+0000","last_active":"2024-08-31T12:16:13.370679+0000","last_peered":"2024-08-31T12:16:13.370679+0000","last_clean":"2024-08-31T12:16:13.370679+0000","last_became_active":"2024-08-31T12:15:56.283147+0000","last_became_peered":"2024-08-31T12:15:56.283147+0000","last_unstale":"2024-08-31T12:16:13.370679+0000","last_undegraded":"2024-08-31T12:16:13.370679+0000","last_fullsized":"2024-08-31T12:16:13.370679+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:51:59.306912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"21'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:13.369457+0000","last_change":"2024-08-31T12:15:56.291170+0000","last_active":"2024-08-31T12:16:13.369457+0000","last_peered":"2024-08-31T12:16:13.369457+0000","last_clean":"2024-08-31T12:16:13.369457+0000","last_became_active":"2024-08-31T12:15:56.290835+0000","last_became_peered":"2024-08-31T12:15:56.290835+0000","last_unstale":"2024-08-31T12:16:13.369457+0000","last_undegraded":"2024-08-31T12:16:13.369457+0000","last_fullsized":"2024-08-31T12:16:13.369457+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:15:51.561831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.938053+0000","last_change":"2024-08-31T12:15:57.308183+0000","last_active":"2024-08-31T12:15:58.938053+0000","last_peered":"2024-08-31T12:15:58.938053+0000","last_clean":"2024-08-31T12:15:58.938053+0000","last_became_active":"2024-08-31T12:15:57.306962+0000","last_became_peered":"2024-08-31T12:15:57.306962+0000","last_unstale":"2024-08-31T12:15:58.938053+0000","last_undegraded":"2024-08-31T12:15:58.938053+0000","last_fullsized":"2024-08-31T12:15:58.938053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:01:18.188264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"21'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:00.127998+0000","last_change":"2024-08-31T12:15:56.288126+0000","last_active":"2024-08-31T12:16:00.127998+0000","last_peered":"2024-08-31T12:16:00.127998+0000","last_clean":"2024-08-31T12:16:00.127998+0000","last_became_active":"2024-08-31T12:15:56.287552+0000","last_became_peered":"2024-08-31T12:15:56.287552+0000","last_unstale":"2024-08-31T12:16:00.127998+0000","last_undegraded":"2024-08-31T12:16:00.127998+0000","last_fullsized":"2024-08-31T12:16:00.127998+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:54:29.988079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300810+0000","last_change":"2024-08-31T12:15:57.305029+0000","last_active":"2024-08-31T12:15:58.300810+0000","last_peered":"2024-08-31T12:15:58.300810+0000","last_clean":"2024-08-31T12:15:58.300810+0000","last_became_active":"2024-08-31T12:15:57.304700+0000","last_became_peered":"2024-08-31T12:15:57.304700+0000","last_unstale":"2024-08-31T12:15:58.300810+0000","last_undegraded":"2024-08-31T12:15:58.300810+0000","last_fullsized":"2024-08-31T12:15:58.300810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:58:17.482204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"21'3","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:07.332783+0000","last_change":"2024-08-31T12:15:56.284907+0000","last_active":"2024-08-31T12:16:07.332783+0000","last_peered":"2024-08-31T12:16:07.332783+0000","last_clean":"2024-08-31T12:16:07.332783+0000","last_became_active":"2024-08-31T12:15:56.284558+0000","last_became_peered":"2024-08-31T12:15:56.284558+0000","last_unstale":"2024-08-31T12:16:07.332783+0000","last_undegraded":"2024-08-31T12:16:07.332783+0000","last_fullsized":"2024-08-31T12:16:07.332783+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:29:53.117724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303538+0000","last_change":"2024-08-31T12:15:57.308679+0000","last_active":"2024-08-31T12:15:58.303538+0000","last_peered":"2024-08-31T12:15:58.303538+0000","last_clean":"2024-08-31T12:15:58.303538+0000","last_became_active":"2024-08-31T12:15:57.308260+0000","last_became_peered":"2024-08-31T12:15:57.308260+0000","last_unstale":"2024-08-31T12:15:58.303538+0000","last_undegraded":"2024-08-31T12:15:58.303538+0000","last_fullsized":"2024-08-31T12:15:58.303538+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:31:19.860950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"21'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:07.332438+0000","last_change":"2024-08-31T12:15:56.290763+0000","last_active":"2024-08-31T12:16:07.332438+0000","last_peered":"2024-08-31T12:16:07.332438+0000","last_clean":"2024-08-31T12:16:07.332438+0000","last_became_active":"2024-08-31T12:15:56.290400+0000","last_became_peered":"2024-08-31T12:15:56.290400+0000","last_unstale":"2024-08-31T12:16:07.332438+0000","last_undegraded":"2024-08-31T12:16:07.332438+0000","last_fullsized":"2024-08-31T12:16:07.332438+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:53:08.929860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.938122+0000","last_change":"2024-08-31T12:15:57.311657+0000","last_active":"2024-08-31T12:15:58.938122+0000","last_peered":"2024-08-31T12:15:58.938122+0000","last_clean":"2024-08-31T12:15:58.938122+0000","last_became_active":"2024-08-31T12:15:57.310564+0000","last_became_peered":"2024-08-31T12:15:57.310564+0000","last_unstale":"2024-08-31T12:15:58.938122+0000","last_undegraded":"2024-08-31T12:15:58.938122+0000","last_fullsized":"2024-08-31T12:15:58.938122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:37:02.435678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'4","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:02.304146+0000","last_change":"2024-08-31T12:15:56.286792+0000","last_active":"2024-08-31T12:16:02.304146+0000","last_peered":"2024-08-31T12:16:02.304146+0000","last_clean":"2024-08-31T12:16:02.304146+0000","last_became_active":"2024-08-31T12:15:56.286458+0000","last_became_peered":"2024-08-31T12:15:56.286458+0000","last_unstale":"2024-08-31T12:16:02.304146+0000","last_undegraded":"2024-08-31T12:16:02.304146+0000","last_fullsized":"2024-08-31T12:16:02.304146+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:49:17.488528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936603+0000","last_change":"2024-08-31T12:15:57.309485+0000","last_active":"2024-08-31T12:15:58.936603+0000","last_peered":"2024-08-31T12:15:58.936603+0000","last_clean":"2024-08-31T12:15:58.936603+0000","last_became_active":"2024-08-31T12:15:57.309102+0000","last_became_peered":"2024-08-31T12:15:57.309102+0000","last_unstale":"2024-08-31T12:15:58.936603+0000","last_undegraded":"2024-08-31T12:15:58.936603+0000","last_fullsized":"2024-08-31T12:15:58.936603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:17:34.692296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'8","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:07.331373+0000","last_change":"2024-08-31T12:15:56.289460+0000","last_active":"2024-08-31T12:16:07.331373+0000","last_peered":"2024-08-31T12:16:07.331373+0000","last_clean":"2024-08-31T12:16:07.331373+0000","last_became_active":"2024-08-31T12:15:56.289096+0000","last_became_peered":"2024-08-31T12:15:56.289096+0000","last_unstale":"2024-08-31T12:16:07.331373+0000","last_undegraded":"2024-08-31T12:16:07.331373+0000","last_fullsized":"2024-08-31T12:16:07.331373+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:06:34.313772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306865+0000","last_change":"2024-08-31T12:15:57.317968+0000","last_active":"2024-08-31T12:15:58.306865+0000","last_peered":"2024-08-31T12:15:58.306865+0000","last_clean":"2024-08-31T12:15:58.306865+0000","last_became_active":"2024-08-31T12:15:57.317632+0000","last_became_peered":"2024-08-31T12:15:57.317632+0000","last_unstale":"2024-08-31T12:15:58.306865+0000","last_undegraded":"2024-08-31T12:15:58.306865+0000","last_fullsized":"2024-08-31T12:15:58.306865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:22:02.569106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'6","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:13.370534+0000","last_change":"2024-08-31T12:15:56.284014+0000","last_active":"2024-08-31T12:16:13.370534+0000","last_peered":"2024-08-31T12:16:13.370534+0000","last_clean":"2024-08-31T12:16:13.370534+0000","last_became_active":"2024-08-31T12:15:56.283675+0000","last_became_peered":"2024-08-31T12:15:56.283675+0000","last_unstale":"2024-08-31T12:16:13.370534+0000","last_undegraded":"2024-08-31T12:16:13.370534+0000","last_fullsized":"2024-08-31T12:16:13.370534+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:51:59.306912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937018+0000","last_change":"2024-08-31T12:15:57.312440+0000","last_active":"2024-08-31T12:15:58.937018+0000","last_peered":"2024-08-31T12:15:58.937018+0000","last_clean":"2024-08-31T12:15:58.937018+0000","last_became_active":"2024-08-31T12:15:57.312007+0000","last_became_peered":"2024-08-31T12:15:57.312007+0000","last_unstale":"2024-08-31T12:15:58.937018+0000","last_undegraded":"2024-08-31T12:15:58.937018+0000","last_fullsized":"2024-08-31T12:15:58.937018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:23:43.553625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"21'3","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:07.345066+0000","last_change":"2024-08-31T12:15:56.290968+0000","last_active":"2024-08-31T12:16:07.345066+0000","last_peered":"2024-08-31T12:16:07.345066+0000","last_clean":"2024-08-31T12:16:07.345066+0000","last_became_active":"2024-08-31T12:15:56.290603+0000","last_became_peered":"2024-08-31T12:15:56.290603+0000","last_unstale":"2024-08-31T12:16:07.345066+0000","last_undegraded":"2024-08-31T12:16:07.345066+0000","last_fullsized":"2024-08-31T12:16:07.345066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:50:23.215239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.307024+0000","last_change":"2024-08-31T12:15:57.312790+0000","last_active":"2024-08-31T12:15:58.307024+0000","last_peered":"2024-08-31T12:15:58.307024+0000","last_clean":"2024-08-31T12:15:58.307024+0000","last_became_active":"2024-08-31T12:15:57.312383+0000","last_became_peered":"2024-08-31T12:15:57.312383+0000","last_unstale":"2024-08-31T12:15:58.307024+0000","last_undegraded":"2024-08-31T12:15:58.307024+0000","last_fullsized":"2024-08-31T12:15:58.307024+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:20:41.042375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"21'3","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:13.369033+0000","last_change":"2024-08-31T12:15:56.289795+0000","last_active":"2024-08-31T12:16:13.369033+0000","last_peered":"2024-08-31T12:16:13.369033+0000","last_clean":"2024-08-31T12:16:13.369033+0000","last_became_active":"2024-08-31T12:15:56.289477+0000","last_became_peered":"2024-08-31T12:15:56.289477+0000","last_unstale":"2024-08-31T12:16:13.369033+0000","last_undegraded":"2024-08-31T12:16:13.369033+0000","last_fullsized":"2024-08-31T12:16:13.369033+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:15:51.561831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302374+0000","last_change":"2024-08-31T12:15:57.312865+0000","last_active":"2024-08-31T12:15:58.302374+0000","last_peered":"2024-08-31T12:15:58.302374+0000","last_clean":"2024-08-31T12:15:58.302374+0000","last_became_active":"2024-08-31T12:15:57.312462+0000","last_became_peered":"2024-08-31T12:15:57.312462+0000","last_unstale":"2024-08-31T12:15:58.302374+0000","last_undegraded":"2024-08-31T12:15:58.302374+0000","last_fullsized":"2024-08-31T12:15:58.302374+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:16:42.625396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'15","reported_seq":30,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:13.375856+0000","last_change":"2024-08-31T12:15:56.285412+0000","last_active":"2024-08-31T12:16:13.375856+0000","last_peered":"2024-08-31T12:16:13.375856+0000","last_clean":"2024-08-31T12:16:13.375856+0000","last_became_active":"2024-08-31T12:15:56.285108+0000","last_became_peered":"2024-08-31T12:15:56.285108+0000","last_unstale":"2024-08-31T12:16:13.375856+0000","last_undegraded":"2024-08-31T12:16:13.375856+0000","last_fullsized":"2024-08-31T12:16:13.375856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:29:45.580957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306407+0000","last_change":"2024-08-31T12:15:57.311701+0000","last_active":"2024-08-31T12:15:58.306407+0000","last_peered":"2024-08-31T12:15:58.306407+0000","last_clean":"2024-08-31T12:15:58.306407+0000","last_became_active":"2024-08-31T12:15:57.311368+0000","last_became_peered":"2024-08-31T12:15:57.311368+0000","last_unstale":"2024-08-31T12:15:58.306407+0000","last_undegraded":"2024-08-31T12:15:58.306407+0000","last_fullsized":"2024-08-31T12:15:58.306407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:30:13.715505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:02.303735+0000","last_change":"2024-08-31T12:15:56.289979+0000","last_active":"2024-08-31T12:16:02.303735+0000","last_peered":"2024-08-31T12:16:02.303735+0000","last_clean":"2024-08-31T12:16:02.303735+0000","last_became_active":"2024-08-31T12:15:56.288289+0000","last_became_peered":"2024-08-31T12:15:56.288289+0000","last_unstale":"2024-08-31T12:16:02.303735+0000","last_undegraded":"2024-08-31T12:16:02.303735+0000","last_fullsized":"2024-08-31T12:16:02.303735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:46:55.057087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306825+0000","last_change":"2024-08-31T12:15:57.311580+0000","last_active":"2024-08-31T12:15:58.306825+0000","last_peered":"2024-08-31T12:15:58.306825+0000","last_clean":"2024-08-31T12:15:58.306825+0000","last_became_active":"2024-08-31T12:15:57.311323+0000","last_became_peered":"2024-08-31T12:15:57.311323+0000","last_unstale":"2024-08-31T12:15:58.306825+0000","last_undegraded":"2024-08-31T12:15:58.306825+0000","last_fullsized":"2024-08-31T12:15:58.306825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:54:03.871840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:02.304157+0000","last_change":"2024-08-31T12:15:56.290496+0000","last_active":"2024-08-31T12:16:02.304157+0000","last_peered":"2024-08-31T12:16:02.304157+0000","last_clean":"2024-08-31T12:16:02.304157+0000","last_became_active":"2024-08-31T12:15:56.290196+0000","last_became_peered":"2024-08-31T12:15:56.290196+0000","last_unstale":"2024-08-31T12:16:02.304157+0000","last_undegraded":"2024-08-31T12:16:02.304157+0000","last_fullsized":"2024-08-31T12:16:02.304157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:52:20.055757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306986+0000","last_change":"2024-08-31T12:15:57.313317+0000","last_active":"2024-08-31T12:15:58.306986+0000","last_peered":"2024-08-31T12:15:58.306986+0000","last_clean":"2024-08-31T12:15:58.306986+0000","last_became_active":"2024-08-31T12:15:57.312145+0000","last_became_peered":"2024-08-31T12:15:57.312145+0000","last_unstale":"2024-08-31T12:15:58.306986+0000","last_undegraded":"2024-08-31T12:15:58.306986+0000","last_fullsized":"2024-08-31T12:15:58.306986+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:09:31.904512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300816+0000","last_change":"2024-08-31T12:15:56.281809+0000","last_active":"2024-08-31T12:15:58.300816+0000","last_peered":"2024-08-31T12:15:58.300816+0000","last_clean":"2024-08-31T12:15:58.300816+0000","last_became_active":"2024-08-31T12:15:56.281368+0000","last_became_peered":"2024-08-31T12:15:56.281368+0000","last_unstale":"2024-08-31T12:15:58.300816+0000","last_undegraded":"2024-08-31T12:15:58.300816+0000","last_fullsized":"2024-08-31T12:15:58.300816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:17:33.687196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303702+0000","last_change":"2024-08-31T12:15:57.305874+0000","last_active":"2024-08-31T12:15:58.303702+0000","last_peered":"2024-08-31T12:15:58.303702+0000","last_clean":"2024-08-31T12:15:58.303702+0000","last_became_active":"2024-08-31T12:15:57.305507+0000","last_became_peered":"2024-08-31T12:15:57.305507+0000","last_unstale":"2024-08-31T12:15:58.303702+0000","last_undegraded":"2024-08-31T12:15:58.303702+0000","last_fullsized":"2024-08-31T12:15:58.303702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:22:02.569106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"21'3","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:13.372823+0000","last_change":"2024-08-31T12:15:56.291851+0000","last_active":"2024-08-31T12:16:13.372823+0000","last_peered":"2024-08-31T12:16:13.372823+0000","last_clean":"2024-08-31T12:16:13.372823+0000","last_became_active":"2024-08-31T12:15:56.291446+0000","last_became_peered":"2024-08-31T12:15:56.291446+0000","last_unstale":"2024-08-31T12:16:13.372823+0000","last_undegraded":"2024-08-31T12:16:13.372823+0000","last_fullsized":"2024-08-31T12:16:13.372823+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:54:29.988079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306538+0000","last_change":"2024-08-31T12:15:57.316154+0000","last_active":"2024-08-31T12:15:58.306538+0000","last_peered":"2024-08-31T12:15:58.306538+0000","last_clean":"2024-08-31T12:15:58.306538+0000","last_became_active":"2024-08-31T12:15:57.315830+0000","last_became_peered":"2024-08-31T12:15:57.315830+0000","last_unstale":"2024-08-31T12:15:58.306538+0000","last_undegraded":"2024-08-31T12:15:58.306538+0000","last_fullsized":"2024-08-31T12:15:58.306538+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:05:53.039520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":111,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306756+0000","last_change":"2024-08-31T12:15:47.084912+0000","last_active":"2024-08-31T12:15:58.306756+0000","last_peered":"2024-08-31T12:15:58.306756+0000","last_clean":"2024-08-31T12:15:58.306756+0000","last_became_active":"2024-08-31T12:15:47.083771+0000","last_became_peered":"2024-08-31T12:15:47.083771+0000","last_unstale":"2024-08-31T12:15:58.306756+0000","last_undegraded":"2024-08-31T12:15:58.306756+0000","last_fullsized":"2024-08-31T12:15:58.306756+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:45.213688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:45.213688+0000","last_clean_scrub_stamp":"2024-08-31T12:15:45.213688+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:25:24.827530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"21'3","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:07.336972+0000","last_change":"2024-08-31T12:15:56.285355+0000","last_active":"2024-08-31T12:16:07.336972+0000","last_peered":"2024-08-31T12:16:07.336972+0000","last_clean":"2024-08-31T12:16:07.336972+0000","last_became_active":"2024-08-31T12:15:56.284470+0000","last_became_peered":"2024-08-31T12:15:56.284470+0000","last_unstale":"2024-08-31T12:16:07.336972+0000","last_undegraded":"2024-08-31T12:16:07.336972+0000","last_fullsized":"2024-08-31T12:16:07.336972+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:52:20.055757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936697+0000","last_change":"2024-08-31T12:15:57.309039+0000","last_active":"2024-08-31T12:15:58.936697+0000","last_peered":"2024-08-31T12:15:58.936697+0000","last_clean":"2024-08-31T12:15:58.936697+0000","last_became_active":"2024-08-31T12:15:57.308721+0000","last_became_peered":"2024-08-31T12:15:57.308721+0000","last_unstale":"2024-08-31T12:15:58.936697+0000","last_undegraded":"2024-08-31T12:15:58.936697+0000","last_fullsized":"2024-08-31T12:15:58.936697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:43:12.703971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"21'3","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:13.370868+0000","last_change":"2024-08-31T12:15:56.293221+0000","last_active":"2024-08-31T12:16:13.370868+0000","last_peered":"2024-08-31T12:16:13.370868+0000","last_clean":"2024-08-31T12:16:13.370868+0000","last_became_active":"2024-08-31T12:15:56.292910+0000","last_became_peered":"2024-08-31T12:15:56.292910+0000","last_unstale":"2024-08-31T12:16:13.370868+0000","last_undegraded":"2024-08-31T12:16:13.370868+0000","last_fullsized":"2024-08-31T12:16:13.370868+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:14:33.483002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937638+0000","last_change":"2024-08-31T12:15:57.307266+0000","last_active":"2024-08-31T12:15:58.937638+0000","last_peered":"2024-08-31T12:15:58.937638+0000","last_clean":"2024-08-31T12:15:58.937638+0000","last_became_active":"2024-08-31T12:15:57.306529+0000","last_became_peered":"2024-08-31T12:15:57.306529+0000","last_unstale":"2024-08-31T12:15:58.937638+0000","last_undegraded":"2024-08-31T12:15:58.937638+0000","last_fullsized":"2024-08-31T12:15:58.937638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:42:39.731742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"21'4","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:13.373114+0000","last_change":"2024-08-31T12:15:56.291822+0000","last_active":"2024-08-31T12:16:13.373114+0000","last_peered":"2024-08-31T12:16:13.373114+0000","last_clean":"2024-08-31T12:16:13.373114+0000","last_became_active":"2024-08-31T12:15:56.291064+0000","last_became_peered":"2024-08-31T12:15:56.291064+0000","last_unstale":"2024-08-31T12:16:13.373114+0000","last_undegraded":"2024-08-31T12:16:13.373114+0000","last_fullsized":"2024-08-31T12:16:13.373114+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:53:08.929860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300606+0000","last_change":"2024-08-31T12:15:57.299212+0000","last_active":"2024-08-31T12:15:58.300606+0000","last_peered":"2024-08-31T12:15:58.300606+0000","last_clean":"2024-08-31T12:15:58.300606+0000","last_became_active":"2024-08-31T12:15:57.298892+0000","last_became_peered":"2024-08-31T12:15:57.298892+0000","last_unstale":"2024-08-31T12:15:58.300606+0000","last_undegraded":"2024-08-31T12:15:58.300606+0000","last_fullsized":"2024-08-31T12:15:58.300606+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:53:19.523811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"21'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:00.128075+0000","last_change":"2024-08-31T12:15:56.286720+0000","last_active":"2024-08-31T12:16:00.128075+0000","last_peered":"2024-08-31T12:16:00.128075+0000","last_clean":"2024-08-31T12:16:00.128075+0000","last_became_active":"2024-08-31T12:15:56.285648+0000","last_became_peered":"2024-08-31T12:15:56.285648+0000","last_unstale":"2024-08-31T12:16:00.128075+0000","last_undegraded":"2024-08-31T12:16:00.128075+0000","last_fullsized":"2024-08-31T12:16:00.128075+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:39:31.946792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936342+0000","last_change":"2024-08-31T12:15:57.311785+0000","last_active":"2024-08-31T12:15:58.936342+0000","last_peered":"2024-08-31T12:15:58.936342+0000","last_clean":"2024-08-31T12:15:58.936342+0000","last_became_active":"2024-08-31T12:15:57.311469+0000","last_became_peered":"2024-08-31T12:15:57.311469+0000","last_unstale":"2024-08-31T12:15:58.936342+0000","last_undegraded":"2024-08-31T12:15:58.936342+0000","last_fullsized":"2024-08-31T12:15:58.936342+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:29:54.122825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300691+0000","last_change":"2024-08-31T12:15:56.284302+0000","last_active":"2024-08-31T12:15:58.300691+0000","last_peered":"2024-08-31T12:15:58.300691+0000","last_clean":"2024-08-31T12:15:58.300691+0000","last_became_active":"2024-08-31T12:15:56.283979+0000","last_became_peered":"2024-08-31T12:15:56.283979+0000","last_unstale":"2024-08-31T12:15:58.300691+0000","last_undegraded":"2024-08-31T12:15:58.300691+0000","last_fullsized":"2024-08-31T12:15:58.300691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:51:59.306912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303594+0000","last_change":"2024-08-31T12:15:57.305383+0000","last_active":"2024-08-31T12:15:58.303594+0000","last_peered":"2024-08-31T12:15:58.303594+0000","last_clean":"2024-08-31T12:15:58.303594+0000","last_became_active":"2024-08-31T12:15:57.305048+0000","last_became_peered":"2024-08-31T12:15:57.305048+0000","last_unstale":"2024-08-31T12:15:58.303594+0000","last_undegraded":"2024-08-31T12:15:58.303594+0000","last_fullsized":"2024-08-31T12:15:58.303594+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:34:56.946695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"21'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:00.127631+0000","last_change":"2024-08-31T12:15:56.297648+0000","last_active":"2024-08-31T12:16:00.127631+0000","last_peered":"2024-08-31T12:16:00.127631+0000","last_clean":"2024-08-31T12:16:00.127631+0000","last_became_active":"2024-08-31T12:15:56.297306+0000","last_became_peered":"2024-08-31T12:15:56.297306+0000","last_unstale":"2024-08-31T12:16:00.127631+0000","last_undegraded":"2024-08-31T12:16:00.127631+0000","last_fullsized":"2024-08-31T12:16:00.127631+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:29:54.693819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300681+0000","last_change":"2024-08-31T12:15:57.297230+0000","last_active":"2024-08-31T12:15:58.300681+0000","last_peered":"2024-08-31T12:15:58.300681+0000","last_clean":"2024-08-31T12:15:58.300681+0000","last_became_active":"2024-08-31T12:15:57.296892+0000","last_became_peered":"2024-08-31T12:15:57.296892+0000","last_unstale":"2024-08-31T12:15:58.300681+0000","last_undegraded":"2024-08-31T12:15:58.300681+0000","last_fullsized":"2024-08-31T12:15:58.300681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:42:36.033586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'3","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:13.369276+0000","last_change":"2024-08-31T12:15:56.286504+0000","last_active":"2024-08-31T12:16:13.369276+0000","last_peered":"2024-08-31T12:16:13.369276+0000","last_clean":"2024-08-31T12:16:13.369276+0000","last_became_active":"2024-08-31T12:15:56.286173+0000","last_became_peered":"2024-08-31T12:15:56.286173+0000","last_unstale":"2024-08-31T12:16:13.369276+0000","last_undegraded":"2024-08-31T12:16:13.369276+0000","last_fullsized":"2024-08-31T12:16:13.369276+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:06:34.313772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.938135+0000","last_change":"2024-08-31T12:15:57.308013+0000","last_active":"2024-08-31T12:15:58.938135+0000","last_peered":"2024-08-31T12:15:58.938135+0000","last_clean":"2024-08-31T12:15:58.938135+0000","last_became_active":"2024-08-31T12:15:57.307701+0000","last_became_peered":"2024-08-31T12:15:57.307701+0000","last_unstale":"2024-08-31T12:15:58.938135+0000","last_undegraded":"2024-08-31T12:15:58.938135+0000","last_fullsized":"2024-08-31T12:15:58.938135+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:06:15.161589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'4","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:07.331216+0000","last_change":"2024-08-31T12:15:56.294852+0000","last_active":"2024-08-31T12:16:07.331216+0000","last_peered":"2024-08-31T12:16:07.331216+0000","last_clean":"2024-08-31T12:16:07.331216+0000","last_became_active":"2024-08-31T12:15:56.294496+0000","last_became_peered":"2024-08-31T12:15:56.294496+0000","last_unstale":"2024-08-31T12:16:07.331216+0000","last_undegraded":"2024-08-31T12:16:07.331216+0000","last_fullsized":"2024-08-31T12:16:07.331216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:49:17.488528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937973+0000","last_change":"2024-08-31T12:15:57.305202+0000","last_active":"2024-08-31T12:15:58.937973+0000","last_peered":"2024-08-31T12:15:58.937973+0000","last_clean":"2024-08-31T12:15:58.937973+0000","last_became_active":"2024-08-31T12:15:57.304602+0000","last_became_peered":"2024-08-31T12:15:57.304602+0000","last_unstale":"2024-08-31T12:15:58.937973+0000","last_undegraded":"2024-08-31T12:15:58.937973+0000","last_fullsized":"2024-08-31T12:15:58.937973+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:56:03.280339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"21'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:07.335145+0000","last_change":"2024-08-31T12:15:56.297542+0000","last_active":"2024-08-31T12:16:07.335145+0000","last_peered":"2024-08-31T12:16:07.335145+0000","last_clean":"2024-08-31T12:16:07.335145+0000","last_became_active":"2024-08-31T12:15:56.297168+0000","last_became_peered":"2024-08-31T12:15:56.297168+0000","last_unstale":"2024-08-31T12:16:07.335145+0000","last_undegraded":"2024-08-31T12:16:07.335145+0000","last_fullsized":"2024-08-31T12:16:07.335145+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:56:41.924772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936311+0000","last_change":"2024-08-31T12:15:57.298845+0000","last_active":"2024-08-31T12:15:58.936311+0000","last_peered":"2024-08-31T12:15:58.936311+0000","last_clean":"2024-08-31T12:15:58.936311+0000","last_became_active":"2024-08-31T12:15:57.298520+0000","last_became_peered":"2024-08-31T12:15:57.298520+0000","last_unstale":"2024-08-31T12:15:58.936311+0000","last_undegraded":"2024-08-31T12:15:58.936311+0000","last_fullsized":"2024-08-31T12:15:58.936311+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:06:35.318873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"21'3","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:07.335708+0000","last_change":"2024-08-31T12:15:56.290475+0000","last_active":"2024-08-31T12:16:07.335708+0000","last_peered":"2024-08-31T12:16:07.335708+0000","last_clean":"2024-08-31T12:16:07.335708+0000","last_became_active":"2024-08-31T12:15:56.289960+0000","last_became_peered":"2024-08-31T12:15:56.289960+0000","last_unstale":"2024-08-31T12:16:07.335708+0000","last_undegraded":"2024-08-31T12:16:07.335708+0000","last_fullsized":"2024-08-31T12:16:07.335708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:29:53.117724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.938261+0000","last_change":"2024-08-31T12:15:57.301460+0000","last_active":"2024-08-31T12:15:58.938261+0000","last_peered":"2024-08-31T12:15:58.938261+0000","last_clean":"2024-08-31T12:15:58.938261+0000","last_became_active":"2024-08-31T12:15:57.301157+0000","last_became_peered":"2024-08-31T12:15:57.301157+0000","last_unstale":"2024-08-31T12:15:58.938261+0000","last_undegraded":"2024-08-31T12:15:58.938261+0000","last_fullsized":"2024-08-31T12:15:58.938261+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:22:52.173578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"21'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.301045+0000","last_change":"2024-08-31T12:15:56.289720+0000","last_active":"2024-08-31T12:15:58.301045+0000","last_peered":"2024-08-31T12:15:58.301045+0000","last_clean":"2024-08-31T12:15:58.301045+0000","last_became_active":"2024-08-31T12:15:56.289332+0000","last_became_peered":"2024-08-31T12:15:56.289332+0000","last_unstale":"2024-08-31T12:15:58.301045+0000","last_undegraded":"2024-08-31T12:15:58.301045+0000","last_fullsized":"2024-08-31T12:15:58.301045+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:15:51.561831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937371+0000","last_change":"2024-08-31T12:15:57.313802+0000","last_active":"2024-08-31T12:15:58.937371+0000","last_peered":"2024-08-31T12:15:58.937371+0000","last_clean":"2024-08-31T12:15:58.937371+0000","last_became_active":"2024-08-31T12:15:57.313428+0000","last_became_peered":"2024-08-31T12:15:57.313428+0000","last_unstale":"2024-08-31T12:15:58.937371+0000","last_undegraded":"2024-08-31T12:15:58.937371+0000","last_fullsized":"2024-08-31T12:15:58.937371+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:22:13.435300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'4","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:07.330625+0000","last_change":"2024-08-31T12:15:56.296105+0000","last_active":"2024-08-31T12:16:07.330625+0000","last_peered":"2024-08-31T12:16:07.330625+0000","last_clean":"2024-08-31T12:16:07.330625+0000","last_became_active":"2024-08-31T12:15:56.295524+0000","last_became_peered":"2024-08-31T12:15:56.295524+0000","last_unstale":"2024-08-31T12:16:07.330625+0000","last_undegraded":"2024-08-31T12:16:07.330625+0000","last_fullsized":"2024-08-31T12:16:07.330625+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:41:55.482044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.301683+0000","last_change":"2024-08-31T12:15:57.300155+0000","last_active":"2024-08-31T12:15:58.301683+0000","last_peered":"2024-08-31T12:15:58.301683+0000","last_clean":"2024-08-31T12:15:58.301683+0000","last_became_active":"2024-08-31T12:15:57.299436+0000","last_became_peered":"2024-08-31T12:15:57.299436+0000","last_unstale":"2024-08-31T12:15:58.301683+0000","last_undegraded":"2024-08-31T12:15:58.301683+0000","last_fullsized":"2024-08-31T12:15:58.301683+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:37:45.891394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303455+0000","last_change":"2024-08-31T12:15:57.300529+0000","last_active":"2024-08-31T12:15:58.303455+0000","last_peered":"2024-08-31T12:15:58.303455+0000","last_clean":"2024-08-31T12:15:58.303455+0000","last_became_active":"2024-08-31T12:15:57.300159+0000","last_became_peered":"2024-08-31T12:15:57.300159+0000","last_unstale":"2024-08-31T12:15:58.303455+0000","last_undegraded":"2024-08-31T12:15:58.303455+0000","last_fullsized":"2024-08-31T12:15:58.303455+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:35:53.831742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:02.304978+0000","last_change":"2024-08-31T12:15:56.297791+0000","last_active":"2024-08-31T12:16:02.304978+0000","last_peered":"2024-08-31T12:16:02.304978+0000","last_clean":"2024-08-31T12:16:02.304978+0000","last_became_active":"2024-08-31T12:15:56.296976+0000","last_became_peered":"2024-08-31T12:15:56.296976+0000","last_unstale":"2024-08-31T12:16:02.304978+0000","last_undegraded":"2024-08-31T12:16:02.304978+0000","last_fullsized":"2024-08-31T12:16:02.304978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:07:54.963050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303405+0000","last_change":"2024-08-31T12:15:57.301167+0000","last_active":"2024-08-31T12:15:58.303405+0000","last_peered":"2024-08-31T12:15:58.303405+0000","last_clean":"2024-08-31T12:15:58.303405+0000","last_became_active":"2024-08-31T12:15:57.300646+0000","last_became_peered":"2024-08-31T12:15:57.300646+0000","last_unstale":"2024-08-31T12:15:58.303405+0000","last_undegraded":"2024-08-31T12:15:58.303405+0000","last_fullsized":"2024-08-31T12:15:58.303405+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:05:53.039520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'4","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:13.369827+0000","last_change":"2024-08-31T12:15:56.295139+0000","last_active":"2024-08-31T12:16:13.369827+0000","last_peered":"2024-08-31T12:16:13.369827+0000","last_clean":"2024-08-31T12:16:13.369827+0000","last_became_active":"2024-08-31T12:15:56.294560+0000","last_became_peered":"2024-08-31T12:15:56.294560+0000","last_unstale":"2024-08-31T12:16:13.369827+0000","last_undegraded":"2024-08-31T12:16:13.369827+0000","last_fullsized":"2024-08-31T12:16:13.369827+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:52:16.999507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937898+0000","last_change":"2024-08-31T12:15:57.311074+0000","last_active":"2024-08-31T12:15:58.937898+0000","last_peered":"2024-08-31T12:15:58.937898+0000","last_clean":"2024-08-31T12:15:58.937898+0000","last_became_active":"2024-08-31T12:15:57.310262+0000","last_became_peered":"2024-08-31T12:15:57.310262+0000","last_unstale":"2024-08-31T12:15:58.937898+0000","last_undegraded":"2024-08-31T12:15:58.937898+0000","last_fullsized":"2024-08-31T12:15:58.937898+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:47:51.267409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:02.303000+0000","last_change":"2024-08-31T12:15:56.285428+0000","last_active":"2024-08-31T12:16:02.303000+0000","last_peered":"2024-08-31T12:16:02.303000+0000","last_clean":"2024-08-31T12:16:02.303000+0000","last_became_active":"2024-08-31T12:15:56.284884+0000","last_became_peered":"2024-08-31T12:15:56.284884+0000","last_unstale":"2024-08-31T12:16:02.303000+0000","last_undegraded":"2024-08-31T12:16:02.303000+0000","last_fullsized":"2024-08-31T12:16:02.303000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:49:25.181432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937174+0000","last_change":"2024-08-31T12:15:57.305772+0000","last_active":"2024-08-31T12:15:58.937174+0000","last_peered":"2024-08-31T12:15:58.937174+0000","last_clean":"2024-08-31T12:15:58.937174+0000","last_became_active":"2024-08-31T12:15:57.305409+0000","last_became_peered":"2024-08-31T12:15:57.305409+0000","last_unstale":"2024-08-31T12:15:58.937174+0000","last_undegraded":"2024-08-31T12:15:58.937174+0000","last_fullsized":"2024-08-31T12:15:58.937174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:53:19.523811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'9","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:13.370923+0000","last_change":"2024-08-31T12:15:56.294197+0000","last_active":"2024-08-31T12:16:13.370923+0000","last_peered":"2024-08-31T12:16:13.370923+0000","last_clean":"2024-08-31T12:16:13.370923+0000","last_became_active":"2024-08-31T12:15:56.293864+0000","last_became_peered":"2024-08-31T12:15:56.293864+0000","last_unstale":"2024-08-31T12:16:13.370923+0000","last_undegraded":"2024-08-31T12:16:13.370923+0000","last_fullsized":"2024-08-31T12:16:13.370923+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:43:17.443433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5663,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":11326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509451,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35568,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336272,"statfs":{"total":96636764160,"available":96600342528,"internally_reserved":0,"allocated":229376,"data_stored":75391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509450,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":233472,"data_stored":75425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509451,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":233472,"data_stored":74045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509451,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":819200,"data_stored":664129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542154,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36148,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335692,"statfs":{"total":96636764160,"available":96599748608,"internally_reserved":0,"allocated":823296,"data_stored":664785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542156,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35568,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336272,"statfs":{"total":96636764160,"available":96600342528,"internally_reserved":0,"allocated":229376,"data_stored":74479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542154,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35564,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336276,"statfs":{"total":96636764160,"available":96600346624,"internally_reserved":0,"allocated":225280,"data_stored":73815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":73919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-31T12:16:22.812 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-31T12:16:22.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.960+0000 7fcea528d640 1 -- 172.21.15.19:0/2742151228 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcea00cc620 msgr2=0x7fcea00cdab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:22.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.960+0000 7fcea528d640 1 --2- 172.21.15.19:0/2742151228 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcea00cc620 0x7fcea00cdab0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fcea006cb90 tx=0x7fce9002a5d0 comp rx=0 tx=0).stop 2024-08-31T12:16:22.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.961+0000 7fcea528d640 1 -- 172.21.15.19:0/2742151228 shutdown_connections 2024-08-31T12:16:22.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.961+0000 7fcea528d640 1 --2- 172.21.15.19:0/2742151228 >> v2:172.21.15.43:3300/0 conn(0x7fcea00ca770 0x7fcea00d0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:22.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.961+0000 7fcea528d640 1 --2- 172.21.15.19:0/2742151228 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcea00cc620 0x7fcea00cdab0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:22.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.961+0000 7fcea528d640 1 -- 172.21.15.19:0/2742151228 >> 172.21.15.19:0/2742151228 conn(0x7fcea006e6e0 msgr2=0x7fcea006eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:22.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.961+0000 7fcea528d640 1 -- 172.21.15.19:0/2742151228 shutdown_connections 2024-08-31T12:16:22.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.961+0000 7fcea528d640 1 -- 172.21.15.19:0/2742151228 wait complete. 2024-08-31T12:16:22.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.962+0000 7fcea528d640 1 Processor -- start 2024-08-31T12:16:22.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.963+0000 7fcea528d640 1 -- start start 2024-08-31T12:16:22.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.964+0000 7fcea528d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcea00ca770 0x7fcea0151840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:22.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.964+0000 7fcea528d640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcea00cc620 0x7fcea01f1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:22.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.964+0000 7fcea528d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcea00d1210 0x7fcea00c4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:22.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.964+0000 7fcea528d640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fcea015d090 con 0x7fcea00d1210 2024-08-31T12:16:22.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.964+0000 7fcea528d640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fcea015cdb0 con 0x7fcea00cc620 2024-08-31T12:16:22.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.964+0000 7fcea528d640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fcea00cc320 con 0x7fcea00ca770 2024-08-31T12:16:22.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.964+0000 7fce9f7fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcea00d1210 0x7fcea00c4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:22.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.964+0000 7fce9f7fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcea00d1210 0x7fcea00c4f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:38096/0 (socket says 172.21.15.19:38096) 2024-08-31T12:16:22.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.964+0000 7fce9f7fe640 1 -- 172.21.15.19:0/3354308275 learned_addr learned my addr 172.21.15.19:0/3354308275 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:22.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.964+0000 7fce9e7fc640 1 --2- 172.21.15.19:0/3354308275 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcea00cc620 0x7fcea01f1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:22.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.964+0000 7fce9effd640 1 --2- 172.21.15.19:0/3354308275 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcea00ca770 0x7fcea0151840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:22.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.964+0000 7fce9f7fe640 1 -- 172.21.15.19:0/3354308275 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcea00ca770 msgr2=0x7fcea0151840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:22.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.964+0000 7fce9f7fe640 1 --2- 172.21.15.19:0/3354308275 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcea00ca770 0x7fcea0151840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:22.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.964+0000 7fce9f7fe640 1 -- 172.21.15.19:0/3354308275 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcea00cc620 msgr2=0x7fcea01f1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:22.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.964+0000 7fce9f7fe640 1 --2- 172.21.15.19:0/3354308275 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcea00cc620 0x7fcea01f1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:22.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.964+0000 7fce9f7fe640 1 -- 172.21.15.19:0/3354308275 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcea01f2240 con 0x7fcea00d1210 2024-08-31T12:16:22.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.964+0000 7fce9e7fc640 1 --2- 172.21.15.19:0/3354308275 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcea00cc620 0x7fcea01f1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:16:22.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.965+0000 7fce9f7fe640 1 --2- 172.21.15.19:0/3354308275 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcea00d1210 0x7fcea00c4f60 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fce9800c760 tx=0x7fce9800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:22.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.965+0000 7fce877fe640 1 -- 172.21.15.19:0/3354308275 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce98003da0 con 0x7fcea00d1210 2024-08-31T12:16:22.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.965+0000 7fce877fe640 1 -- 172.21.15.19:0/3354308275 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fce9800ce10 con 0x7fcea00d1210 2024-08-31T12:16:22.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.965+0000 7fcea528d640 1 -- 172.21.15.19:0/3354308275 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcea01506a0 con 0x7fcea00d1210 2024-08-31T12:16:22.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.965+0000 7fce877fe640 1 -- 172.21.15.19:0/3354308275 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce980052d0 con 0x7fcea00d1210 2024-08-31T12:16:22.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.965+0000 7fcea528d640 1 -- 172.21.15.19:0/3354308275 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcea00c5910 con 0x7fcea00d1210 2024-08-31T12:16:22.969 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.966+0000 7fcea528d640 1 -- 172.21.15.19:0/3354308275 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcea00d0100 con 0x7fcea00d1210 2024-08-31T12:16:22.969 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.967+0000 7fce877fe640 1 -- 172.21.15.19:0/3354308275 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fce98005470 con 0x7fcea00d1210 2024-08-31T12:16:22.969 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.967+0000 7fce877fe640 1 --2- 172.21.15.19:0/3354308275 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fce8c077a50 0x7fce8c079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:22.969 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.967+0000 7fce877fe640 1 -- 172.21.15.19:0/3354308275 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fce980994a0 con 0x7fcea00d1210 2024-08-31T12:16:22.969 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.967+0000 7fce9effd640 1 --2- 172.21.15.19:0/3354308275 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fce8c077a50 0x7fce8c079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:22.969 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.968+0000 7fce9effd640 1 --2- 172.21.15.19:0/3354308275 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fce8c077a50 0x7fce8c079f60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fce90034000 tx=0x7fce90004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:22.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:22.972+0000 7fce877fe640 1 -- 172.21.15.19:0/3354308275 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fcea00d0100 con 0x7fcea00d1210 2024-08-31T12:16:23.155 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.154+0000 7fcea528d640 1 -- 172.21.15.19:0/3354308275 --> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcea01f19e0 con 0x7fce8c077a50 2024-08-31T12:16:23.159 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.159+0000 7fce877fe640 1 -- 172.21.15.19:0/3354308275 <== mgr.4100 v2:172.21.15.43:6800/2824336655 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251905 (secure 0 0 0) 0x7fcea01f19e0 con 0x7fce8c077a50 2024-08-31T12:16:23.162 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:16:23.163 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-08-31T12:16:23.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.162+0000 7fcea528d640 1 -- 172.21.15.19:0/3354308275 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fce8c077a50 msgr2=0x7fce8c079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:23.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.162+0000 7fcea528d640 1 --2- 172.21.15.19:0/3354308275 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fce8c077a50 0x7fce8c079f60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fce90034000 tx=0x7fce90004730 comp rx=0 tx=0).stop 2024-08-31T12:16:23.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.162+0000 7fcea528d640 1 -- 172.21.15.19:0/3354308275 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcea00d1210 msgr2=0x7fcea00c4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:23.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.162+0000 7fcea528d640 1 --2- 172.21.15.19:0/3354308275 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcea00d1210 0x7fcea00c4f60 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fce9800c760 tx=0x7fce9800cc20 comp rx=0 tx=0).stop 2024-08-31T12:16:23.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.162+0000 7fcea528d640 1 -- 172.21.15.19:0/3354308275 shutdown_connections 2024-08-31T12:16:23.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.162+0000 7fcea528d640 1 --2- 172.21.15.19:0/3354308275 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fce8c077a50 0x7fce8c079f60 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:23.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.162+0000 7fcea528d640 1 --2- 172.21.15.19:0/3354308275 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcea00d1210 0x7fcea00c4f60 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:23.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.162+0000 7fcea528d640 1 --2- 172.21.15.19:0/3354308275 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcea00cc620 0x7fcea01f1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:23.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.162+0000 7fcea528d640 1 --2- 172.21.15.19:0/3354308275 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcea00ca770 0x7fcea0151840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:23.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.162+0000 7fcea528d640 1 -- 172.21.15.19:0/3354308275 >> 172.21.15.19:0/3354308275 conn(0x7fcea006e6e0 msgr2=0x7fcea00cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:23.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.162+0000 7fcea528d640 1 -- 172.21.15.19:0/3354308275 shutdown_connections 2024-08-31T12:16:23.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.162+0000 7fcea528d640 1 -- 172.21.15.19:0/3354308275 wait complete. 2024-08-31T12:16:23.174 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-08-31T12:16:22.497453+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596031,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":227,"num_write_kb":1667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":182,"ondisk_log_size":182,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":252424,"kb_used_data":2952,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754722296,"statfs":{"total":773094113280,"available":772835631104,"internally_reserved":0,"allocated":3022848,"data_stored":1775988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3231,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003066"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306730+0000","last_change":"2024-08-31T12:15:57.306560+0000","last_active":"2024-08-31T12:15:58.306730+0000","last_peered":"2024-08-31T12:15:58.306730+0000","last_clean":"2024-08-31T12:15:58.306730+0000","last_became_active":"2024-08-31T12:15:57.306148+0000","last_became_peered":"2024-08-31T12:15:57.306148+0000","last_unstale":"2024-08-31T12:15:58.306730+0000","last_undegraded":"2024-08-31T12:15:58.306730+0000","last_fullsized":"2024-08-31T12:15:58.306730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:35:53.831742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300935+0000","last_change":"2024-08-31T12:15:57.307314+0000","last_active":"2024-08-31T12:15:58.300935+0000","last_peered":"2024-08-31T12:15:58.300935+0000","last_clean":"2024-08-31T12:15:58.300935+0000","last_became_active":"2024-08-31T12:15:57.306940+0000","last_became_peered":"2024-08-31T12:15:57.306940+0000","last_unstale":"2024-08-31T12:15:58.300935+0000","last_undegraded":"2024-08-31T12:15:58.300935+0000","last_fullsized":"2024-08-31T12:15:58.300935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:38:30.321214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937267+0000","last_change":"2024-08-31T12:15:57.309372+0000","last_active":"2024-08-31T12:15:58.937267+0000","last_peered":"2024-08-31T12:15:58.937267+0000","last_clean":"2024-08-31T12:15:58.937267+0000","last_became_active":"2024-08-31T12:15:57.305795+0000","last_became_peered":"2024-08-31T12:15:57.305795+0000","last_unstale":"2024-08-31T12:15:58.937267+0000","last_undegraded":"2024-08-31T12:15:58.937267+0000","last_fullsized":"2024-08-31T12:15:58.937267+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:37:45.891394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302363+0000","last_change":"2024-08-31T12:15:57.307339+0000","last_active":"2024-08-31T12:15:58.302363+0000","last_peered":"2024-08-31T12:15:58.302363+0000","last_clean":"2024-08-31T12:15:58.302363+0000","last_became_active":"2024-08-31T12:15:57.306957+0000","last_became_peered":"2024-08-31T12:15:57.306957+0000","last_unstale":"2024-08-31T12:15:58.302363+0000","last_undegraded":"2024-08-31T12:15:58.302363+0000","last_fullsized":"2024-08-31T12:15:58.302363+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:43:33.705099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306757+0000","last_change":"2024-08-31T12:15:57.317492+0000","last_active":"2024-08-31T12:15:58.306757+0000","last_peered":"2024-08-31T12:15:58.306757+0000","last_clean":"2024-08-31T12:15:58.306757+0000","last_became_active":"2024-08-31T12:15:57.317079+0000","last_became_peered":"2024-08-31T12:15:57.317079+0000","last_unstale":"2024-08-31T12:15:58.306757+0000","last_undegraded":"2024-08-31T12:15:58.306757+0000","last_fullsized":"2024-08-31T12:15:58.306757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:44:40.712699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302342+0000","last_change":"2024-08-31T12:15:57.307505+0000","last_active":"2024-08-31T12:15:58.302342+0000","last_peered":"2024-08-31T12:15:58.302342+0000","last_clean":"2024-08-31T12:15:58.302342+0000","last_became_active":"2024-08-31T12:15:57.307010+0000","last_became_peered":"2024-08-31T12:15:57.307010+0000","last_unstale":"2024-08-31T12:15:58.302342+0000","last_undegraded":"2024-08-31T12:15:58.302342+0000","last_fullsized":"2024-08-31T12:15:58.302342+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:39:31.100565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306247+0000","last_change":"2024-08-31T12:15:57.316892+0000","last_active":"2024-08-31T12:15:58.306247+0000","last_peered":"2024-08-31T12:15:58.306247+0000","last_clean":"2024-08-31T12:15:58.306247+0000","last_became_active":"2024-08-31T12:15:57.316504+0000","last_became_peered":"2024-08-31T12:15:57.316504+0000","last_unstale":"2024-08-31T12:15:58.306247+0000","last_undegraded":"2024-08-31T12:15:58.306247+0000","last_fullsized":"2024-08-31T12:15:58.306247+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:05:48.614918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937976+0000","last_change":"2024-08-31T12:15:57.309224+0000","last_active":"2024-08-31T12:15:58.937976+0000","last_peered":"2024-08-31T12:15:58.937976+0000","last_clean":"2024-08-31T12:15:58.937976+0000","last_became_active":"2024-08-31T12:15:57.308914+0000","last_became_peered":"2024-08-31T12:15:57.308914+0000","last_unstale":"2024-08-31T12:15:58.937976+0000","last_undegraded":"2024-08-31T12:15:58.937976+0000","last_fullsized":"2024-08-31T12:15:58.937976+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:37:02.435678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306762+0000","last_change":"2024-08-31T12:15:57.316723+0000","last_active":"2024-08-31T12:15:58.306762+0000","last_peered":"2024-08-31T12:15:58.306762+0000","last_clean":"2024-08-31T12:15:58.306762+0000","last_became_active":"2024-08-31T12:15:57.316354+0000","last_became_peered":"2024-08-31T12:15:57.316354+0000","last_unstale":"2024-08-31T12:15:58.306762+0000","last_undegraded":"2024-08-31T12:15:58.306762+0000","last_fullsized":"2024-08-31T12:15:58.306762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:31:19.860950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302888+0000","last_change":"2024-08-31T12:15:57.311161+0000","last_active":"2024-08-31T12:15:58.302888+0000","last_peered":"2024-08-31T12:15:58.302888+0000","last_clean":"2024-08-31T12:15:58.302888+0000","last_became_active":"2024-08-31T12:15:57.310807+0000","last_became_peered":"2024-08-31T12:15:57.310807+0000","last_unstale":"2024-08-31T12:15:58.302888+0000","last_undegraded":"2024-08-31T12:15:58.302888+0000","last_fullsized":"2024-08-31T12:15:58.302888+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:56:03.280339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302118+0000","last_change":"2024-08-31T12:15:57.300930+0000","last_active":"2024-08-31T12:15:58.302118+0000","last_peered":"2024-08-31T12:15:58.302118+0000","last_clean":"2024-08-31T12:15:58.302118+0000","last_became_active":"2024-08-31T12:15:57.300610+0000","last_became_peered":"2024-08-31T12:15:57.300610+0000","last_unstale":"2024-08-31T12:15:58.302118+0000","last_undegraded":"2024-08-31T12:15:58.302118+0000","last_fullsized":"2024-08-31T12:15:58.302118+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:42:39.731742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936435+0000","last_change":"2024-08-31T12:15:57.303151+0000","last_active":"2024-08-31T12:15:58.936435+0000","last_peered":"2024-08-31T12:15:58.936435+0000","last_clean":"2024-08-31T12:15:58.936435+0000","last_became_active":"2024-08-31T12:15:57.302622+0000","last_became_peered":"2024-08-31T12:15:57.302622+0000","last_unstale":"2024-08-31T12:15:58.936435+0000","last_undegraded":"2024-08-31T12:15:58.936435+0000","last_fullsized":"2024-08-31T12:15:58.936435+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:49:18.493629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937309+0000","last_change":"2024-08-31T12:15:57.305676+0000","last_active":"2024-08-31T12:15:58.937309+0000","last_peered":"2024-08-31T12:15:58.937309+0000","last_clean":"2024-08-31T12:15:58.937309+0000","last_became_active":"2024-08-31T12:15:57.305321+0000","last_became_peered":"2024-08-31T12:15:57.305321+0000","last_unstale":"2024-08-31T12:15:58.937309+0000","last_undegraded":"2024-08-31T12:15:58.937309+0000","last_fullsized":"2024-08-31T12:15:58.937309+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:42:39.731742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936688+0000","last_change":"2024-08-31T12:15:57.309671+0000","last_active":"2024-08-31T12:15:58.936688+0000","last_peered":"2024-08-31T12:15:58.936688+0000","last_clean":"2024-08-31T12:15:58.936688+0000","last_became_active":"2024-08-31T12:15:57.307013+0000","last_became_peered":"2024-08-31T12:15:57.307013+0000","last_unstale":"2024-08-31T12:15:58.936688+0000","last_undegraded":"2024-08-31T12:15:58.936688+0000","last_fullsized":"2024-08-31T12:15:58.936688+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:18:55.451164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300929+0000","last_change":"2024-08-31T12:15:57.298150+0000","last_active":"2024-08-31T12:15:58.300929+0000","last_peered":"2024-08-31T12:15:58.300929+0000","last_clean":"2024-08-31T12:15:58.300929+0000","last_became_active":"2024-08-31T12:15:57.297182+0000","last_became_peered":"2024-08-31T12:15:57.297182+0000","last_unstale":"2024-08-31T12:15:58.300929+0000","last_undegraded":"2024-08-31T12:15:58.300929+0000","last_fullsized":"2024-08-31T12:15:58.300929+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:16:42.625396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302762+0000","last_change":"2024-08-31T12:15:57.314224+0000","last_active":"2024-08-31T12:15:58.302762+0000","last_peered":"2024-08-31T12:15:58.302762+0000","last_clean":"2024-08-31T12:15:58.302762+0000","last_became_active":"2024-08-31T12:15:57.313887+0000","last_became_peered":"2024-08-31T12:15:57.313887+0000","last_unstale":"2024-08-31T12:15:58.302762+0000","last_undegraded":"2024-08-31T12:15:58.302762+0000","last_fullsized":"2024-08-31T12:15:58.302762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:54:45.720621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303640+0000","last_change":"2024-08-31T12:15:57.308899+0000","last_active":"2024-08-31T12:15:58.303640+0000","last_peered":"2024-08-31T12:15:58.303640+0000","last_clean":"2024-08-31T12:15:58.303640+0000","last_became_active":"2024-08-31T12:15:57.308388+0000","last_became_peered":"2024-08-31T12:15:57.308388+0000","last_unstale":"2024-08-31T12:15:58.303640+0000","last_undegraded":"2024-08-31T12:15:58.303640+0000","last_fullsized":"2024-08-31T12:15:58.303640+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:54:03.871840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936410+0000","last_change":"2024-08-31T12:15:57.300201+0000","last_active":"2024-08-31T12:15:58.936410+0000","last_peered":"2024-08-31T12:15:58.936410+0000","last_clean":"2024-08-31T12:15:58.936410+0000","last_became_active":"2024-08-31T12:15:57.299826+0000","last_became_peered":"2024-08-31T12:15:57.299826+0000","last_unstale":"2024-08-31T12:15:58.936410+0000","last_undegraded":"2024-08-31T12:15:58.936410+0000","last_fullsized":"2024-08-31T12:15:58.936410+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:14:34.488102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.307053+0000","last_change":"2024-08-31T12:15:57.317606+0000","last_active":"2024-08-31T12:15:58.307053+0000","last_peered":"2024-08-31T12:15:58.307053+0000","last_clean":"2024-08-31T12:15:58.307053+0000","last_became_active":"2024-08-31T12:15:57.317185+0000","last_became_peered":"2024-08-31T12:15:57.317185+0000","last_unstale":"2024-08-31T12:15:58.307053+0000","last_undegraded":"2024-08-31T12:15:58.307053+0000","last_fullsized":"2024-08-31T12:15:58.307053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:25:46.346490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302171+0000","last_change":"2024-08-31T12:15:57.298778+0000","last_active":"2024-08-31T12:15:58.302171+0000","last_peered":"2024-08-31T12:15:58.302171+0000","last_clean":"2024-08-31T12:15:58.302171+0000","last_became_active":"2024-08-31T12:15:57.298430+0000","last_became_peered":"2024-08-31T12:15:57.298430+0000","last_unstale":"2024-08-31T12:15:58.302171+0000","last_undegraded":"2024-08-31T12:15:58.302171+0000","last_fullsized":"2024-08-31T12:15:58.302171+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:22:13.435300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937190+0000","last_change":"2024-08-31T12:15:57.317176+0000","last_active":"2024-08-31T12:15:58.937190+0000","last_peered":"2024-08-31T12:15:58.937190+0000","last_clean":"2024-08-31T12:15:58.937190+0000","last_became_active":"2024-08-31T12:15:57.316870+0000","last_became_peered":"2024-08-31T12:15:57.316870+0000","last_unstale":"2024-08-31T12:15:58.937190+0000","last_undegraded":"2024-08-31T12:15:58.937190+0000","last_fullsized":"2024-08-31T12:15:58.937190+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:43:12.703971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302636+0000","last_change":"2024-08-31T12:15:57.308907+0000","last_active":"2024-08-31T12:15:58.302636+0000","last_peered":"2024-08-31T12:15:58.302636+0000","last_clean":"2024-08-31T12:15:58.302636+0000","last_became_active":"2024-08-31T12:15:57.308459+0000","last_became_peered":"2024-08-31T12:15:57.308459+0000","last_unstale":"2024-08-31T12:15:58.302636+0000","last_undegraded":"2024-08-31T12:15:58.302636+0000","last_fullsized":"2024-08-31T12:15:58.302636+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:16:46.648912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937760+0000","last_change":"2024-08-31T12:15:57.304032+0000","last_active":"2024-08-31T12:15:58.937760+0000","last_peered":"2024-08-31T12:15:58.937760+0000","last_clean":"2024-08-31T12:15:58.937760+0000","last_became_active":"2024-08-31T12:15:57.303401+0000","last_became_peered":"2024-08-31T12:15:57.303401+0000","last_unstale":"2024-08-31T12:15:58.937760+0000","last_undegraded":"2024-08-31T12:15:58.937760+0000","last_fullsized":"2024-08-31T12:15:58.937760+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:56:03.280339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303013+0000","last_change":"2024-08-31T12:15:57.305412+0000","last_active":"2024-08-31T12:15:58.303013+0000","last_peered":"2024-08-31T12:15:58.303013+0000","last_clean":"2024-08-31T12:15:58.303013+0000","last_became_active":"2024-08-31T12:15:57.305075+0000","last_became_peered":"2024-08-31T12:15:57.305075+0000","last_unstale":"2024-08-31T12:15:58.303013+0000","last_undegraded":"2024-08-31T12:15:58.303013+0000","last_fullsized":"2024-08-31T12:15:58.303013+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:30:13.715505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937828+0000","last_change":"2024-08-31T12:15:57.300179+0000","last_active":"2024-08-31T12:15:58.937828+0000","last_peered":"2024-08-31T12:15:58.937828+0000","last_clean":"2024-08-31T12:15:58.937828+0000","last_became_active":"2024-08-31T12:15:57.299861+0000","last_became_peered":"2024-08-31T12:15:57.299861+0000","last_unstale":"2024-08-31T12:15:58.937828+0000","last_undegraded":"2024-08-31T12:15:58.937828+0000","last_fullsized":"2024-08-31T12:15:58.937828+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:47:51.267409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303001+0000","last_change":"2024-08-31T12:15:57.308519+0000","last_active":"2024-08-31T12:15:58.303001+0000","last_peered":"2024-08-31T12:15:58.303001+0000","last_clean":"2024-08-31T12:15:58.303001+0000","last_became_active":"2024-08-31T12:15:57.308189+0000","last_became_peered":"2024-08-31T12:15:57.308189+0000","last_unstale":"2024-08-31T12:15:58.303001+0000","last_undegraded":"2024-08-31T12:15:58.303001+0000","last_fullsized":"2024-08-31T12:15:58.303001+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:37:02.435678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300666+0000","last_change":"2024-08-31T12:15:57.308321+0000","last_active":"2024-08-31T12:15:58.300666+0000","last_peered":"2024-08-31T12:15:58.300666+0000","last_clean":"2024-08-31T12:15:58.300666+0000","last_became_active":"2024-08-31T12:15:57.304514+0000","last_became_peered":"2024-08-31T12:15:57.304514+0000","last_unstale":"2024-08-31T12:15:58.300666+0000","last_undegraded":"2024-08-31T12:15:58.300666+0000","last_fullsized":"2024-08-31T12:15:58.300666+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:09:49.617826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936830+0000","last_change":"2024-08-31T12:15:57.299268+0000","last_active":"2024-08-31T12:15:58.936830+0000","last_peered":"2024-08-31T12:15:58.936830+0000","last_clean":"2024-08-31T12:15:58.936830+0000","last_became_active":"2024-08-31T12:15:57.298766+0000","last_became_peered":"2024-08-31T12:15:57.298766+0000","last_unstale":"2024-08-31T12:15:58.936830+0000","last_undegraded":"2024-08-31T12:15:58.936830+0000","last_fullsized":"2024-08-31T12:15:58.936830+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:37:45.891394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937525+0000","last_change":"2024-08-31T12:15:57.310121+0000","last_active":"2024-08-31T12:15:58.937525+0000","last_peered":"2024-08-31T12:15:58.937525+0000","last_clean":"2024-08-31T12:15:58.937525+0000","last_became_active":"2024-08-31T12:15:57.309440+0000","last_became_peered":"2024-08-31T12:15:57.309440+0000","last_unstale":"2024-08-31T12:15:58.937525+0000","last_undegraded":"2024-08-31T12:15:58.937525+0000","last_fullsized":"2024-08-31T12:15:58.937525+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:53:19.523811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302889+0000","last_change":"2024-08-31T12:15:57.306214+0000","last_active":"2024-08-31T12:15:58.302889+0000","last_peered":"2024-08-31T12:15:58.302889+0000","last_clean":"2024-08-31T12:15:58.302889+0000","last_became_active":"2024-08-31T12:15:57.305878+0000","last_became_peered":"2024-08-31T12:15:57.305878+0000","last_unstale":"2024-08-31T12:15:58.302889+0000","last_undegraded":"2024-08-31T12:15:58.302889+0000","last_fullsized":"2024-08-31T12:15:58.302889+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:05:48.614918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.938282+0000","last_change":"2024-08-31T12:15:57.307376+0000","last_active":"2024-08-31T12:15:58.938282+0000","last_peered":"2024-08-31T12:15:58.938282+0000","last_clean":"2024-08-31T12:15:58.938282+0000","last_became_active":"2024-08-31T12:15:57.307056+0000","last_became_peered":"2024-08-31T12:15:57.307056+0000","last_unstale":"2024-08-31T12:15:58.938282+0000","last_undegraded":"2024-08-31T12:15:58.938282+0000","last_fullsized":"2024-08-31T12:15:58.938282+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:52:18.004608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302319+0000","last_change":"2024-08-31T12:15:57.313798+0000","last_active":"2024-08-31T12:15:58.302319+0000","last_peered":"2024-08-31T12:15:58.302319+0000","last_clean":"2024-08-31T12:15:58.302319+0000","last_became_active":"2024-08-31T12:15:57.313463+0000","last_became_peered":"2024-08-31T12:15:57.313463+0000","last_unstale":"2024-08-31T12:15:58.302319+0000","last_undegraded":"2024-08-31T12:15:58.302319+0000","last_fullsized":"2024-08-31T12:15:58.302319+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:47:51.267409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936240+0000","last_change":"2024-08-31T12:15:57.293350+0000","last_active":"2024-08-31T12:15:58.936240+0000","last_peered":"2024-08-31T12:15:58.936240+0000","last_clean":"2024-08-31T12:15:58.936240+0000","last_became_active":"2024-08-31T12:15:57.292981+0000","last_became_peered":"2024-08-31T12:15:57.292981+0000","last_unstale":"2024-08-31T12:15:58.936240+0000","last_undegraded":"2024-08-31T12:15:58.936240+0000","last_fullsized":"2024-08-31T12:15:58.936240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:52:00.312013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'6","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:07.349665+0000","last_change":"2024-08-31T12:15:56.293497+0000","last_active":"2024-08-31T12:16:07.349665+0000","last_peered":"2024-08-31T12:16:07.349665+0000","last_clean":"2024-08-31T12:16:07.349665+0000","last_became_active":"2024-08-31T12:15:56.293164+0000","last_became_peered":"2024-08-31T12:15:56.293164+0000","last_unstale":"2024-08-31T12:16:07.349665+0000","last_undegraded":"2024-08-31T12:16:07.349665+0000","last_fullsized":"2024-08-31T12:16:07.349665+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:43:17.443433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1063,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937814+0000","last_change":"2024-08-31T12:15:57.312738+0000","last_active":"2024-08-31T12:15:58.937814+0000","last_peered":"2024-08-31T12:15:58.937814+0000","last_clean":"2024-08-31T12:15:58.937814+0000","last_became_active":"2024-08-31T12:15:57.312360+0000","last_became_peered":"2024-08-31T12:15:57.312360+0000","last_unstale":"2024-08-31T12:15:58.937814+0000","last_undegraded":"2024-08-31T12:15:58.937814+0000","last_fullsized":"2024-08-31T12:15:58.937814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:22:13.435300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'3","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:13.370679+0000","last_change":"2024-08-31T12:15:56.283526+0000","last_active":"2024-08-31T12:16:13.370679+0000","last_peered":"2024-08-31T12:16:13.370679+0000","last_clean":"2024-08-31T12:16:13.370679+0000","last_became_active":"2024-08-31T12:15:56.283147+0000","last_became_peered":"2024-08-31T12:15:56.283147+0000","last_unstale":"2024-08-31T12:16:13.370679+0000","last_undegraded":"2024-08-31T12:16:13.370679+0000","last_fullsized":"2024-08-31T12:16:13.370679+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:51:59.306912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"21'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:13.369457+0000","last_change":"2024-08-31T12:15:56.291170+0000","last_active":"2024-08-31T12:16:13.369457+0000","last_peered":"2024-08-31T12:16:13.369457+0000","last_clean":"2024-08-31T12:16:13.369457+0000","last_became_active":"2024-08-31T12:15:56.290835+0000","last_became_peered":"2024-08-31T12:15:56.290835+0000","last_unstale":"2024-08-31T12:16:13.369457+0000","last_undegraded":"2024-08-31T12:16:13.369457+0000","last_fullsized":"2024-08-31T12:16:13.369457+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:15:51.561831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.938053+0000","last_change":"2024-08-31T12:15:57.308183+0000","last_active":"2024-08-31T12:15:58.938053+0000","last_peered":"2024-08-31T12:15:58.938053+0000","last_clean":"2024-08-31T12:15:58.938053+0000","last_became_active":"2024-08-31T12:15:57.306962+0000","last_became_peered":"2024-08-31T12:15:57.306962+0000","last_unstale":"2024-08-31T12:15:58.938053+0000","last_undegraded":"2024-08-31T12:15:58.938053+0000","last_fullsized":"2024-08-31T12:15:58.938053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:01:18.188264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"21'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:00.127998+0000","last_change":"2024-08-31T12:15:56.288126+0000","last_active":"2024-08-31T12:16:00.127998+0000","last_peered":"2024-08-31T12:16:00.127998+0000","last_clean":"2024-08-31T12:16:00.127998+0000","last_became_active":"2024-08-31T12:15:56.287552+0000","last_became_peered":"2024-08-31T12:15:56.287552+0000","last_unstale":"2024-08-31T12:16:00.127998+0000","last_undegraded":"2024-08-31T12:16:00.127998+0000","last_fullsized":"2024-08-31T12:16:00.127998+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:54:29.988079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300810+0000","last_change":"2024-08-31T12:15:57.305029+0000","last_active":"2024-08-31T12:15:58.300810+0000","last_peered":"2024-08-31T12:15:58.300810+0000","last_clean":"2024-08-31T12:15:58.300810+0000","last_became_active":"2024-08-31T12:15:57.304700+0000","last_became_peered":"2024-08-31T12:15:57.304700+0000","last_unstale":"2024-08-31T12:15:58.300810+0000","last_undegraded":"2024-08-31T12:15:58.300810+0000","last_fullsized":"2024-08-31T12:15:58.300810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:58:17.482204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"21'3","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:07.332783+0000","last_change":"2024-08-31T12:15:56.284907+0000","last_active":"2024-08-31T12:16:07.332783+0000","last_peered":"2024-08-31T12:16:07.332783+0000","last_clean":"2024-08-31T12:16:07.332783+0000","last_became_active":"2024-08-31T12:15:56.284558+0000","last_became_peered":"2024-08-31T12:15:56.284558+0000","last_unstale":"2024-08-31T12:16:07.332783+0000","last_undegraded":"2024-08-31T12:16:07.332783+0000","last_fullsized":"2024-08-31T12:16:07.332783+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:29:53.117724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303538+0000","last_change":"2024-08-31T12:15:57.308679+0000","last_active":"2024-08-31T12:15:58.303538+0000","last_peered":"2024-08-31T12:15:58.303538+0000","last_clean":"2024-08-31T12:15:58.303538+0000","last_became_active":"2024-08-31T12:15:57.308260+0000","last_became_peered":"2024-08-31T12:15:57.308260+0000","last_unstale":"2024-08-31T12:15:58.303538+0000","last_undegraded":"2024-08-31T12:15:58.303538+0000","last_fullsized":"2024-08-31T12:15:58.303538+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:31:19.860950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"21'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:07.332438+0000","last_change":"2024-08-31T12:15:56.290763+0000","last_active":"2024-08-31T12:16:07.332438+0000","last_peered":"2024-08-31T12:16:07.332438+0000","last_clean":"2024-08-31T12:16:07.332438+0000","last_became_active":"2024-08-31T12:15:56.290400+0000","last_became_peered":"2024-08-31T12:15:56.290400+0000","last_unstale":"2024-08-31T12:16:07.332438+0000","last_undegraded":"2024-08-31T12:16:07.332438+0000","last_fullsized":"2024-08-31T12:16:07.332438+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:53:08.929860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.938122+0000","last_change":"2024-08-31T12:15:57.311657+0000","last_active":"2024-08-31T12:15:58.938122+0000","last_peered":"2024-08-31T12:15:58.938122+0000","last_clean":"2024-08-31T12:15:58.938122+0000","last_became_active":"2024-08-31T12:15:57.310564+0000","last_became_peered":"2024-08-31T12:15:57.310564+0000","last_unstale":"2024-08-31T12:15:58.938122+0000","last_undegraded":"2024-08-31T12:15:58.938122+0000","last_fullsized":"2024-08-31T12:15:58.938122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:37:02.435678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'4","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:02.304146+0000","last_change":"2024-08-31T12:15:56.286792+0000","last_active":"2024-08-31T12:16:02.304146+0000","last_peered":"2024-08-31T12:16:02.304146+0000","last_clean":"2024-08-31T12:16:02.304146+0000","last_became_active":"2024-08-31T12:15:56.286458+0000","last_became_peered":"2024-08-31T12:15:56.286458+0000","last_unstale":"2024-08-31T12:16:02.304146+0000","last_undegraded":"2024-08-31T12:16:02.304146+0000","last_fullsized":"2024-08-31T12:16:02.304146+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:49:17.488528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936603+0000","last_change":"2024-08-31T12:15:57.309485+0000","last_active":"2024-08-31T12:15:58.936603+0000","last_peered":"2024-08-31T12:15:58.936603+0000","last_clean":"2024-08-31T12:15:58.936603+0000","last_became_active":"2024-08-31T12:15:57.309102+0000","last_became_peered":"2024-08-31T12:15:57.309102+0000","last_unstale":"2024-08-31T12:15:58.936603+0000","last_undegraded":"2024-08-31T12:15:58.936603+0000","last_fullsized":"2024-08-31T12:15:58.936603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:17:34.692296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'8","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:07.331373+0000","last_change":"2024-08-31T12:15:56.289460+0000","last_active":"2024-08-31T12:16:07.331373+0000","last_peered":"2024-08-31T12:16:07.331373+0000","last_clean":"2024-08-31T12:16:07.331373+0000","last_became_active":"2024-08-31T12:15:56.289096+0000","last_became_peered":"2024-08-31T12:15:56.289096+0000","last_unstale":"2024-08-31T12:16:07.331373+0000","last_undegraded":"2024-08-31T12:16:07.331373+0000","last_fullsized":"2024-08-31T12:16:07.331373+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:06:34.313772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306865+0000","last_change":"2024-08-31T12:15:57.317968+0000","last_active":"2024-08-31T12:15:58.306865+0000","last_peered":"2024-08-31T12:15:58.306865+0000","last_clean":"2024-08-31T12:15:58.306865+0000","last_became_active":"2024-08-31T12:15:57.317632+0000","last_became_peered":"2024-08-31T12:15:57.317632+0000","last_unstale":"2024-08-31T12:15:58.306865+0000","last_undegraded":"2024-08-31T12:15:58.306865+0000","last_fullsized":"2024-08-31T12:15:58.306865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:22:02.569106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'6","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:13.370534+0000","last_change":"2024-08-31T12:15:56.284014+0000","last_active":"2024-08-31T12:16:13.370534+0000","last_peered":"2024-08-31T12:16:13.370534+0000","last_clean":"2024-08-31T12:16:13.370534+0000","last_became_active":"2024-08-31T12:15:56.283675+0000","last_became_peered":"2024-08-31T12:15:56.283675+0000","last_unstale":"2024-08-31T12:16:13.370534+0000","last_undegraded":"2024-08-31T12:16:13.370534+0000","last_fullsized":"2024-08-31T12:16:13.370534+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:51:59.306912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937018+0000","last_change":"2024-08-31T12:15:57.312440+0000","last_active":"2024-08-31T12:15:58.937018+0000","last_peered":"2024-08-31T12:15:58.937018+0000","last_clean":"2024-08-31T12:15:58.937018+0000","last_became_active":"2024-08-31T12:15:57.312007+0000","last_became_peered":"2024-08-31T12:15:57.312007+0000","last_unstale":"2024-08-31T12:15:58.937018+0000","last_undegraded":"2024-08-31T12:15:58.937018+0000","last_fullsized":"2024-08-31T12:15:58.937018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:23:43.553625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"21'3","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:07.345066+0000","last_change":"2024-08-31T12:15:56.290968+0000","last_active":"2024-08-31T12:16:07.345066+0000","last_peered":"2024-08-31T12:16:07.345066+0000","last_clean":"2024-08-31T12:16:07.345066+0000","last_became_active":"2024-08-31T12:15:56.290603+0000","last_became_peered":"2024-08-31T12:15:56.290603+0000","last_unstale":"2024-08-31T12:16:07.345066+0000","last_undegraded":"2024-08-31T12:16:07.345066+0000","last_fullsized":"2024-08-31T12:16:07.345066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:50:23.215239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.307024+0000","last_change":"2024-08-31T12:15:57.312790+0000","last_active":"2024-08-31T12:15:58.307024+0000","last_peered":"2024-08-31T12:15:58.307024+0000","last_clean":"2024-08-31T12:15:58.307024+0000","last_became_active":"2024-08-31T12:15:57.312383+0000","last_became_peered":"2024-08-31T12:15:57.312383+0000","last_unstale":"2024-08-31T12:15:58.307024+0000","last_undegraded":"2024-08-31T12:15:58.307024+0000","last_fullsized":"2024-08-31T12:15:58.307024+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:20:41.042375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"21'3","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:13.369033+0000","last_change":"2024-08-31T12:15:56.289795+0000","last_active":"2024-08-31T12:16:13.369033+0000","last_peered":"2024-08-31T12:16:13.369033+0000","last_clean":"2024-08-31T12:16:13.369033+0000","last_became_active":"2024-08-31T12:15:56.289477+0000","last_became_peered":"2024-08-31T12:15:56.289477+0000","last_unstale":"2024-08-31T12:16:13.369033+0000","last_undegraded":"2024-08-31T12:16:13.369033+0000","last_fullsized":"2024-08-31T12:16:13.369033+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:15:51.561831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.302374+0000","last_change":"2024-08-31T12:15:57.312865+0000","last_active":"2024-08-31T12:15:58.302374+0000","last_peered":"2024-08-31T12:15:58.302374+0000","last_clean":"2024-08-31T12:15:58.302374+0000","last_became_active":"2024-08-31T12:15:57.312462+0000","last_became_peered":"2024-08-31T12:15:57.312462+0000","last_unstale":"2024-08-31T12:15:58.302374+0000","last_undegraded":"2024-08-31T12:15:58.302374+0000","last_fullsized":"2024-08-31T12:15:58.302374+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:16:42.625396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'15","reported_seq":30,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:13.375856+0000","last_change":"2024-08-31T12:15:56.285412+0000","last_active":"2024-08-31T12:16:13.375856+0000","last_peered":"2024-08-31T12:16:13.375856+0000","last_clean":"2024-08-31T12:16:13.375856+0000","last_became_active":"2024-08-31T12:15:56.285108+0000","last_became_peered":"2024-08-31T12:15:56.285108+0000","last_unstale":"2024-08-31T12:16:13.375856+0000","last_undegraded":"2024-08-31T12:16:13.375856+0000","last_fullsized":"2024-08-31T12:16:13.375856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:29:45.580957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306407+0000","last_change":"2024-08-31T12:15:57.311701+0000","last_active":"2024-08-31T12:15:58.306407+0000","last_peered":"2024-08-31T12:15:58.306407+0000","last_clean":"2024-08-31T12:15:58.306407+0000","last_became_active":"2024-08-31T12:15:57.311368+0000","last_became_peered":"2024-08-31T12:15:57.311368+0000","last_unstale":"2024-08-31T12:15:58.306407+0000","last_undegraded":"2024-08-31T12:15:58.306407+0000","last_fullsized":"2024-08-31T12:15:58.306407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:30:13.715505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:02.303735+0000","last_change":"2024-08-31T12:15:56.289979+0000","last_active":"2024-08-31T12:16:02.303735+0000","last_peered":"2024-08-31T12:16:02.303735+0000","last_clean":"2024-08-31T12:16:02.303735+0000","last_became_active":"2024-08-31T12:15:56.288289+0000","last_became_peered":"2024-08-31T12:15:56.288289+0000","last_unstale":"2024-08-31T12:16:02.303735+0000","last_undegraded":"2024-08-31T12:16:02.303735+0000","last_fullsized":"2024-08-31T12:16:02.303735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:46:55.057087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306825+0000","last_change":"2024-08-31T12:15:57.311580+0000","last_active":"2024-08-31T12:15:58.306825+0000","last_peered":"2024-08-31T12:15:58.306825+0000","last_clean":"2024-08-31T12:15:58.306825+0000","last_became_active":"2024-08-31T12:15:57.311323+0000","last_became_peered":"2024-08-31T12:15:57.311323+0000","last_unstale":"2024-08-31T12:15:58.306825+0000","last_undegraded":"2024-08-31T12:15:58.306825+0000","last_fullsized":"2024-08-31T12:15:58.306825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:54:03.871840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:02.304157+0000","last_change":"2024-08-31T12:15:56.290496+0000","last_active":"2024-08-31T12:16:02.304157+0000","last_peered":"2024-08-31T12:16:02.304157+0000","last_clean":"2024-08-31T12:16:02.304157+0000","last_became_active":"2024-08-31T12:15:56.290196+0000","last_became_peered":"2024-08-31T12:15:56.290196+0000","last_unstale":"2024-08-31T12:16:02.304157+0000","last_undegraded":"2024-08-31T12:16:02.304157+0000","last_fullsized":"2024-08-31T12:16:02.304157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:52:20.055757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306986+0000","last_change":"2024-08-31T12:15:57.313317+0000","last_active":"2024-08-31T12:15:58.306986+0000","last_peered":"2024-08-31T12:15:58.306986+0000","last_clean":"2024-08-31T12:15:58.306986+0000","last_became_active":"2024-08-31T12:15:57.312145+0000","last_became_peered":"2024-08-31T12:15:57.312145+0000","last_unstale":"2024-08-31T12:15:58.306986+0000","last_undegraded":"2024-08-31T12:15:58.306986+0000","last_fullsized":"2024-08-31T12:15:58.306986+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:09:31.904512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300816+0000","last_change":"2024-08-31T12:15:56.281809+0000","last_active":"2024-08-31T12:15:58.300816+0000","last_peered":"2024-08-31T12:15:58.300816+0000","last_clean":"2024-08-31T12:15:58.300816+0000","last_became_active":"2024-08-31T12:15:56.281368+0000","last_became_peered":"2024-08-31T12:15:56.281368+0000","last_unstale":"2024-08-31T12:15:58.300816+0000","last_undegraded":"2024-08-31T12:15:58.300816+0000","last_fullsized":"2024-08-31T12:15:58.300816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:17:33.687196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303702+0000","last_change":"2024-08-31T12:15:57.305874+0000","last_active":"2024-08-31T12:15:58.303702+0000","last_peered":"2024-08-31T12:15:58.303702+0000","last_clean":"2024-08-31T12:15:58.303702+0000","last_became_active":"2024-08-31T12:15:57.305507+0000","last_became_peered":"2024-08-31T12:15:57.305507+0000","last_unstale":"2024-08-31T12:15:58.303702+0000","last_undegraded":"2024-08-31T12:15:58.303702+0000","last_fullsized":"2024-08-31T12:15:58.303702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:22:02.569106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"21'3","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:13.372823+0000","last_change":"2024-08-31T12:15:56.291851+0000","last_active":"2024-08-31T12:16:13.372823+0000","last_peered":"2024-08-31T12:16:13.372823+0000","last_clean":"2024-08-31T12:16:13.372823+0000","last_became_active":"2024-08-31T12:15:56.291446+0000","last_became_peered":"2024-08-31T12:15:56.291446+0000","last_unstale":"2024-08-31T12:16:13.372823+0000","last_undegraded":"2024-08-31T12:16:13.372823+0000","last_fullsized":"2024-08-31T12:16:13.372823+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:54:29.988079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306538+0000","last_change":"2024-08-31T12:15:57.316154+0000","last_active":"2024-08-31T12:15:58.306538+0000","last_peered":"2024-08-31T12:15:58.306538+0000","last_clean":"2024-08-31T12:15:58.306538+0000","last_became_active":"2024-08-31T12:15:57.315830+0000","last_became_peered":"2024-08-31T12:15:57.315830+0000","last_unstale":"2024-08-31T12:15:58.306538+0000","last_undegraded":"2024-08-31T12:15:58.306538+0000","last_fullsized":"2024-08-31T12:15:58.306538+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:05:53.039520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":111,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.306756+0000","last_change":"2024-08-31T12:15:47.084912+0000","last_active":"2024-08-31T12:15:58.306756+0000","last_peered":"2024-08-31T12:15:58.306756+0000","last_clean":"2024-08-31T12:15:58.306756+0000","last_became_active":"2024-08-31T12:15:47.083771+0000","last_became_peered":"2024-08-31T12:15:47.083771+0000","last_unstale":"2024-08-31T12:15:58.306756+0000","last_undegraded":"2024-08-31T12:15:58.306756+0000","last_fullsized":"2024-08-31T12:15:58.306756+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:45.213688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:45.213688+0000","last_clean_scrub_stamp":"2024-08-31T12:15:45.213688+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:25:24.827530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"21'3","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:07.336972+0000","last_change":"2024-08-31T12:15:56.285355+0000","last_active":"2024-08-31T12:16:07.336972+0000","last_peered":"2024-08-31T12:16:07.336972+0000","last_clean":"2024-08-31T12:16:07.336972+0000","last_became_active":"2024-08-31T12:15:56.284470+0000","last_became_peered":"2024-08-31T12:15:56.284470+0000","last_unstale":"2024-08-31T12:16:07.336972+0000","last_undegraded":"2024-08-31T12:16:07.336972+0000","last_fullsized":"2024-08-31T12:16:07.336972+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:52:20.055757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936697+0000","last_change":"2024-08-31T12:15:57.309039+0000","last_active":"2024-08-31T12:15:58.936697+0000","last_peered":"2024-08-31T12:15:58.936697+0000","last_clean":"2024-08-31T12:15:58.936697+0000","last_became_active":"2024-08-31T12:15:57.308721+0000","last_became_peered":"2024-08-31T12:15:57.308721+0000","last_unstale":"2024-08-31T12:15:58.936697+0000","last_undegraded":"2024-08-31T12:15:58.936697+0000","last_fullsized":"2024-08-31T12:15:58.936697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:43:12.703971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"21'3","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:13.370868+0000","last_change":"2024-08-31T12:15:56.293221+0000","last_active":"2024-08-31T12:16:13.370868+0000","last_peered":"2024-08-31T12:16:13.370868+0000","last_clean":"2024-08-31T12:16:13.370868+0000","last_became_active":"2024-08-31T12:15:56.292910+0000","last_became_peered":"2024-08-31T12:15:56.292910+0000","last_unstale":"2024-08-31T12:16:13.370868+0000","last_undegraded":"2024-08-31T12:16:13.370868+0000","last_fullsized":"2024-08-31T12:16:13.370868+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:14:33.483002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937638+0000","last_change":"2024-08-31T12:15:57.307266+0000","last_active":"2024-08-31T12:15:58.937638+0000","last_peered":"2024-08-31T12:15:58.937638+0000","last_clean":"2024-08-31T12:15:58.937638+0000","last_became_active":"2024-08-31T12:15:57.306529+0000","last_became_peered":"2024-08-31T12:15:57.306529+0000","last_unstale":"2024-08-31T12:15:58.937638+0000","last_undegraded":"2024-08-31T12:15:58.937638+0000","last_fullsized":"2024-08-31T12:15:58.937638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:42:39.731742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"21'4","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:13.373114+0000","last_change":"2024-08-31T12:15:56.291822+0000","last_active":"2024-08-31T12:16:13.373114+0000","last_peered":"2024-08-31T12:16:13.373114+0000","last_clean":"2024-08-31T12:16:13.373114+0000","last_became_active":"2024-08-31T12:15:56.291064+0000","last_became_peered":"2024-08-31T12:15:56.291064+0000","last_unstale":"2024-08-31T12:16:13.373114+0000","last_undegraded":"2024-08-31T12:16:13.373114+0000","last_fullsized":"2024-08-31T12:16:13.373114+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:53:08.929860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300606+0000","last_change":"2024-08-31T12:15:57.299212+0000","last_active":"2024-08-31T12:15:58.300606+0000","last_peered":"2024-08-31T12:15:58.300606+0000","last_clean":"2024-08-31T12:15:58.300606+0000","last_became_active":"2024-08-31T12:15:57.298892+0000","last_became_peered":"2024-08-31T12:15:57.298892+0000","last_unstale":"2024-08-31T12:15:58.300606+0000","last_undegraded":"2024-08-31T12:15:58.300606+0000","last_fullsized":"2024-08-31T12:15:58.300606+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:53:19.523811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"21'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:00.128075+0000","last_change":"2024-08-31T12:15:56.286720+0000","last_active":"2024-08-31T12:16:00.128075+0000","last_peered":"2024-08-31T12:16:00.128075+0000","last_clean":"2024-08-31T12:16:00.128075+0000","last_became_active":"2024-08-31T12:15:56.285648+0000","last_became_peered":"2024-08-31T12:15:56.285648+0000","last_unstale":"2024-08-31T12:16:00.128075+0000","last_undegraded":"2024-08-31T12:16:00.128075+0000","last_fullsized":"2024-08-31T12:16:00.128075+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:39:31.946792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936342+0000","last_change":"2024-08-31T12:15:57.311785+0000","last_active":"2024-08-31T12:15:58.936342+0000","last_peered":"2024-08-31T12:15:58.936342+0000","last_clean":"2024-08-31T12:15:58.936342+0000","last_became_active":"2024-08-31T12:15:57.311469+0000","last_became_peered":"2024-08-31T12:15:57.311469+0000","last_unstale":"2024-08-31T12:15:58.936342+0000","last_undegraded":"2024-08-31T12:15:58.936342+0000","last_fullsized":"2024-08-31T12:15:58.936342+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:29:54.122825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300691+0000","last_change":"2024-08-31T12:15:56.284302+0000","last_active":"2024-08-31T12:15:58.300691+0000","last_peered":"2024-08-31T12:15:58.300691+0000","last_clean":"2024-08-31T12:15:58.300691+0000","last_became_active":"2024-08-31T12:15:56.283979+0000","last_became_peered":"2024-08-31T12:15:56.283979+0000","last_unstale":"2024-08-31T12:15:58.300691+0000","last_undegraded":"2024-08-31T12:15:58.300691+0000","last_fullsized":"2024-08-31T12:15:58.300691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:51:59.306912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303594+0000","last_change":"2024-08-31T12:15:57.305383+0000","last_active":"2024-08-31T12:15:58.303594+0000","last_peered":"2024-08-31T12:15:58.303594+0000","last_clean":"2024-08-31T12:15:58.303594+0000","last_became_active":"2024-08-31T12:15:57.305048+0000","last_became_peered":"2024-08-31T12:15:57.305048+0000","last_unstale":"2024-08-31T12:15:58.303594+0000","last_undegraded":"2024-08-31T12:15:58.303594+0000","last_fullsized":"2024-08-31T12:15:58.303594+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:34:56.946695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"21'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:00.127631+0000","last_change":"2024-08-31T12:15:56.297648+0000","last_active":"2024-08-31T12:16:00.127631+0000","last_peered":"2024-08-31T12:16:00.127631+0000","last_clean":"2024-08-31T12:16:00.127631+0000","last_became_active":"2024-08-31T12:15:56.297306+0000","last_became_peered":"2024-08-31T12:15:56.297306+0000","last_unstale":"2024-08-31T12:16:00.127631+0000","last_undegraded":"2024-08-31T12:16:00.127631+0000","last_fullsized":"2024-08-31T12:16:00.127631+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:29:54.693819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.300681+0000","last_change":"2024-08-31T12:15:57.297230+0000","last_active":"2024-08-31T12:15:58.300681+0000","last_peered":"2024-08-31T12:15:58.300681+0000","last_clean":"2024-08-31T12:15:58.300681+0000","last_became_active":"2024-08-31T12:15:57.296892+0000","last_became_peered":"2024-08-31T12:15:57.296892+0000","last_unstale":"2024-08-31T12:15:58.300681+0000","last_undegraded":"2024-08-31T12:15:58.300681+0000","last_fullsized":"2024-08-31T12:15:58.300681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:42:36.033586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'3","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:13.369276+0000","last_change":"2024-08-31T12:15:56.286504+0000","last_active":"2024-08-31T12:16:13.369276+0000","last_peered":"2024-08-31T12:16:13.369276+0000","last_clean":"2024-08-31T12:16:13.369276+0000","last_became_active":"2024-08-31T12:15:56.286173+0000","last_became_peered":"2024-08-31T12:15:56.286173+0000","last_unstale":"2024-08-31T12:16:13.369276+0000","last_undegraded":"2024-08-31T12:16:13.369276+0000","last_fullsized":"2024-08-31T12:16:13.369276+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:06:34.313772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.938135+0000","last_change":"2024-08-31T12:15:57.308013+0000","last_active":"2024-08-31T12:15:58.938135+0000","last_peered":"2024-08-31T12:15:58.938135+0000","last_clean":"2024-08-31T12:15:58.938135+0000","last_became_active":"2024-08-31T12:15:57.307701+0000","last_became_peered":"2024-08-31T12:15:57.307701+0000","last_unstale":"2024-08-31T12:15:58.938135+0000","last_undegraded":"2024-08-31T12:15:58.938135+0000","last_fullsized":"2024-08-31T12:15:58.938135+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:06:15.161589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'4","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:07.331216+0000","last_change":"2024-08-31T12:15:56.294852+0000","last_active":"2024-08-31T12:16:07.331216+0000","last_peered":"2024-08-31T12:16:07.331216+0000","last_clean":"2024-08-31T12:16:07.331216+0000","last_became_active":"2024-08-31T12:15:56.294496+0000","last_became_peered":"2024-08-31T12:15:56.294496+0000","last_unstale":"2024-08-31T12:16:07.331216+0000","last_undegraded":"2024-08-31T12:16:07.331216+0000","last_fullsized":"2024-08-31T12:16:07.331216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:49:17.488528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937973+0000","last_change":"2024-08-31T12:15:57.305202+0000","last_active":"2024-08-31T12:15:58.937973+0000","last_peered":"2024-08-31T12:15:58.937973+0000","last_clean":"2024-08-31T12:15:58.937973+0000","last_became_active":"2024-08-31T12:15:57.304602+0000","last_became_peered":"2024-08-31T12:15:57.304602+0000","last_unstale":"2024-08-31T12:15:58.937973+0000","last_undegraded":"2024-08-31T12:15:58.937973+0000","last_fullsized":"2024-08-31T12:15:58.937973+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:56:03.280339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"21'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:07.335145+0000","last_change":"2024-08-31T12:15:56.297542+0000","last_active":"2024-08-31T12:16:07.335145+0000","last_peered":"2024-08-31T12:16:07.335145+0000","last_clean":"2024-08-31T12:16:07.335145+0000","last_became_active":"2024-08-31T12:15:56.297168+0000","last_became_peered":"2024-08-31T12:15:56.297168+0000","last_unstale":"2024-08-31T12:16:07.335145+0000","last_undegraded":"2024-08-31T12:16:07.335145+0000","last_fullsized":"2024-08-31T12:16:07.335145+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:56:41.924772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.936311+0000","last_change":"2024-08-31T12:15:57.298845+0000","last_active":"2024-08-31T12:15:58.936311+0000","last_peered":"2024-08-31T12:15:58.936311+0000","last_clean":"2024-08-31T12:15:58.936311+0000","last_became_active":"2024-08-31T12:15:57.298520+0000","last_became_peered":"2024-08-31T12:15:57.298520+0000","last_unstale":"2024-08-31T12:15:58.936311+0000","last_undegraded":"2024-08-31T12:15:58.936311+0000","last_fullsized":"2024-08-31T12:15:58.936311+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:06:35.318873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"21'3","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:07.335708+0000","last_change":"2024-08-31T12:15:56.290475+0000","last_active":"2024-08-31T12:16:07.335708+0000","last_peered":"2024-08-31T12:16:07.335708+0000","last_clean":"2024-08-31T12:16:07.335708+0000","last_became_active":"2024-08-31T12:15:56.289960+0000","last_became_peered":"2024-08-31T12:15:56.289960+0000","last_unstale":"2024-08-31T12:16:07.335708+0000","last_undegraded":"2024-08-31T12:16:07.335708+0000","last_fullsized":"2024-08-31T12:16:07.335708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:29:53.117724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.938261+0000","last_change":"2024-08-31T12:15:57.301460+0000","last_active":"2024-08-31T12:15:58.938261+0000","last_peered":"2024-08-31T12:15:58.938261+0000","last_clean":"2024-08-31T12:15:58.938261+0000","last_became_active":"2024-08-31T12:15:57.301157+0000","last_became_peered":"2024-08-31T12:15:57.301157+0000","last_unstale":"2024-08-31T12:15:58.938261+0000","last_undegraded":"2024-08-31T12:15:58.938261+0000","last_fullsized":"2024-08-31T12:15:58.938261+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:22:52.173578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"21'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.301045+0000","last_change":"2024-08-31T12:15:56.289720+0000","last_active":"2024-08-31T12:15:58.301045+0000","last_peered":"2024-08-31T12:15:58.301045+0000","last_clean":"2024-08-31T12:15:58.301045+0000","last_became_active":"2024-08-31T12:15:56.289332+0000","last_became_peered":"2024-08-31T12:15:56.289332+0000","last_unstale":"2024-08-31T12:15:58.301045+0000","last_undegraded":"2024-08-31T12:15:58.301045+0000","last_fullsized":"2024-08-31T12:15:58.301045+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:15:51.561831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937371+0000","last_change":"2024-08-31T12:15:57.313802+0000","last_active":"2024-08-31T12:15:58.937371+0000","last_peered":"2024-08-31T12:15:58.937371+0000","last_clean":"2024-08-31T12:15:58.937371+0000","last_became_active":"2024-08-31T12:15:57.313428+0000","last_became_peered":"2024-08-31T12:15:57.313428+0000","last_unstale":"2024-08-31T12:15:58.937371+0000","last_undegraded":"2024-08-31T12:15:58.937371+0000","last_fullsized":"2024-08-31T12:15:58.937371+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:22:13.435300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'4","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:07.330625+0000","last_change":"2024-08-31T12:15:56.296105+0000","last_active":"2024-08-31T12:16:07.330625+0000","last_peered":"2024-08-31T12:16:07.330625+0000","last_clean":"2024-08-31T12:16:07.330625+0000","last_became_active":"2024-08-31T12:15:56.295524+0000","last_became_peered":"2024-08-31T12:15:56.295524+0000","last_unstale":"2024-08-31T12:16:07.330625+0000","last_undegraded":"2024-08-31T12:16:07.330625+0000","last_fullsized":"2024-08-31T12:16:07.330625+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:41:55.482044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.301683+0000","last_change":"2024-08-31T12:15:57.300155+0000","last_active":"2024-08-31T12:15:58.301683+0000","last_peered":"2024-08-31T12:15:58.301683+0000","last_clean":"2024-08-31T12:15:58.301683+0000","last_became_active":"2024-08-31T12:15:57.299436+0000","last_became_peered":"2024-08-31T12:15:57.299436+0000","last_unstale":"2024-08-31T12:15:58.301683+0000","last_undegraded":"2024-08-31T12:15:58.301683+0000","last_fullsized":"2024-08-31T12:15:58.301683+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:37:45.891394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303455+0000","last_change":"2024-08-31T12:15:57.300529+0000","last_active":"2024-08-31T12:15:58.303455+0000","last_peered":"2024-08-31T12:15:58.303455+0000","last_clean":"2024-08-31T12:15:58.303455+0000","last_became_active":"2024-08-31T12:15:57.300159+0000","last_became_peered":"2024-08-31T12:15:57.300159+0000","last_unstale":"2024-08-31T12:15:58.303455+0000","last_undegraded":"2024-08-31T12:15:58.303455+0000","last_fullsized":"2024-08-31T12:15:58.303455+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:35:53.831742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:02.304978+0000","last_change":"2024-08-31T12:15:56.297791+0000","last_active":"2024-08-31T12:16:02.304978+0000","last_peered":"2024-08-31T12:16:02.304978+0000","last_clean":"2024-08-31T12:16:02.304978+0000","last_became_active":"2024-08-31T12:15:56.296976+0000","last_became_peered":"2024-08-31T12:15:56.296976+0000","last_unstale":"2024-08-31T12:16:02.304978+0000","last_undegraded":"2024-08-31T12:16:02.304978+0000","last_fullsized":"2024-08-31T12:16:02.304978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:07:54.963050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.303405+0000","last_change":"2024-08-31T12:15:57.301167+0000","last_active":"2024-08-31T12:15:58.303405+0000","last_peered":"2024-08-31T12:15:58.303405+0000","last_clean":"2024-08-31T12:15:58.303405+0000","last_became_active":"2024-08-31T12:15:57.300646+0000","last_became_peered":"2024-08-31T12:15:57.300646+0000","last_unstale":"2024-08-31T12:15:58.303405+0000","last_undegraded":"2024-08-31T12:15:58.303405+0000","last_fullsized":"2024-08-31T12:15:58.303405+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:05:53.039520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'4","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:13.369827+0000","last_change":"2024-08-31T12:15:56.295139+0000","last_active":"2024-08-31T12:16:13.369827+0000","last_peered":"2024-08-31T12:16:13.369827+0000","last_clean":"2024-08-31T12:16:13.369827+0000","last_became_active":"2024-08-31T12:15:56.294560+0000","last_became_peered":"2024-08-31T12:15:56.294560+0000","last_unstale":"2024-08-31T12:16:13.369827+0000","last_undegraded":"2024-08-31T12:16:13.369827+0000","last_fullsized":"2024-08-31T12:16:13.369827+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:52:16.999507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937898+0000","last_change":"2024-08-31T12:15:57.311074+0000","last_active":"2024-08-31T12:15:58.937898+0000","last_peered":"2024-08-31T12:15:58.937898+0000","last_clean":"2024-08-31T12:15:58.937898+0000","last_became_active":"2024-08-31T12:15:57.310262+0000","last_became_peered":"2024-08-31T12:15:57.310262+0000","last_unstale":"2024-08-31T12:15:58.937898+0000","last_undegraded":"2024-08-31T12:15:58.937898+0000","last_fullsized":"2024-08-31T12:15:58.937898+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:47:51.267409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:02.303000+0000","last_change":"2024-08-31T12:15:56.285428+0000","last_active":"2024-08-31T12:16:02.303000+0000","last_peered":"2024-08-31T12:16:02.303000+0000","last_clean":"2024-08-31T12:16:02.303000+0000","last_became_active":"2024-08-31T12:15:56.284884+0000","last_became_peered":"2024-08-31T12:15:56.284884+0000","last_unstale":"2024-08-31T12:16:02.303000+0000","last_undegraded":"2024-08-31T12:16:02.303000+0000","last_fullsized":"2024-08-31T12:16:02.303000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:49:25.181432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:15:58.937174+0000","last_change":"2024-08-31T12:15:57.305772+0000","last_active":"2024-08-31T12:15:58.937174+0000","last_peered":"2024-08-31T12:15:58.937174+0000","last_clean":"2024-08-31T12:15:58.937174+0000","last_became_active":"2024-08-31T12:15:57.305409+0000","last_became_peered":"2024-08-31T12:15:57.305409+0000","last_unstale":"2024-08-31T12:15:58.937174+0000","last_undegraded":"2024-08-31T12:15:58.937174+0000","last_fullsized":"2024-08-31T12:15:58.937174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:53:19.523811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'9","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-08-31T12:16:13.370923+0000","last_change":"2024-08-31T12:15:56.294197+0000","last_active":"2024-08-31T12:16:13.370923+0000","last_peered":"2024-08-31T12:16:13.370923+0000","last_clean":"2024-08-31T12:16:13.370923+0000","last_became_active":"2024-08-31T12:15:56.293864+0000","last_became_peered":"2024-08-31T12:15:56.293864+0000","last_unstale":"2024-08-31T12:16:13.370923+0000","last_undegraded":"2024-08-31T12:16:13.370923+0000","last_fullsized":"2024-08-31T12:16:13.370923+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:43:17.443433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5663,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":11326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509451,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35568,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336272,"statfs":{"total":96636764160,"available":96600342528,"internally_reserved":0,"allocated":229376,"data_stored":75391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509450,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":233472,"data_stored":75425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509451,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":233472,"data_stored":74045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509451,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":819200,"data_stored":664129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542154,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36148,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335692,"statfs":{"total":96636764160,"available":96599748608,"internally_reserved":0,"allocated":823296,"data_stored":664785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542156,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35568,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336272,"statfs":{"total":96636764160,"available":96600342528,"internally_reserved":0,"allocated":229376,"data_stored":74479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542154,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35564,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336276,"statfs":{"total":96636764160,"available":96600346624,"internally_reserved":0,"allocated":225280,"data_stored":73815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":73919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-31T12:16:23.178 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-08-31T12:16:23.178 DEBUG:tasks.mon_thrash:Yielding 2024-08-31T12:16:23.178 INFO:teuthology.run_tasks:Running task workunit... 2024-08-31T12:16:23.191 INFO:tasks.workunit:Pulling workunits from ref d19e6118f7015f0047264006cb0c873528eb3ae6 2024-08-31T12:16:23.191 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-08-31T12:16:23.191 INFO:tasks.workunit:timeout=3h 2024-08-31T12:16:23.192 INFO:tasks.workunit:cleanup=True 2024-08-31T12:16:23.192 DEBUG:teuthology.orchestra.run.smithi019:> stat -- /home/ubuntu/cephtest/mnt.0 2024-08-31T12:16:23.192 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-31T12:16:23.231 INFO:teuthology.orchestra.run.smithi019.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-08-31T12:16:23.232 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-08-31T12:16:23.232 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 2ch/44d Inode: 1 Links: 2 2024-08-31T12:16:23.232 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-31T12:16:23.232 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:fusefs_t:s0 2024-08-31T12:16:23.232 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-08-31T12:16:23.232 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-08-31 12:16:02.295688585 +0000 2024-08-31T12:16:23.232 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-08-31 12:16:21.131433103 +0000 2024-08-31T12:16:23.232 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: - 2024-08-31T12:16:23.232 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-08-31T12:16:23.232 DEBUG:teuthology.orchestra.run.smithi019:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-08-31T12:16:23.304 DEBUG:teuthology.orchestra.run.smithi019:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout d19e6118f7015f0047264006cb0c873528eb3ae6 2024-08-31T12:16:23.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.329+0000 7f54d0684640 1 -- 172.21.15.19:0/1009994158 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f54c80c5440 msgr2=0x7f54c8074860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:23.331 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.329+0000 7f54d0684640 1 --2- 172.21.15.19:0/1009994158 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f54c80c5440 0x7f54c8074860 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f54c4009830 tx=0x7f54c402a5c0 comp rx=0 tx=0).stop 2024-08-31T12:16:23.331 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.329+0000 7f54d0684640 1 -- 172.21.15.19:0/1009994158 shutdown_connections 2024-08-31T12:16:23.331 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.329+0000 7f54d0684640 1 --2- 172.21.15.19:0/1009994158 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f54c80c5440 0x7f54c8074860 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:23.331 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.329+0000 7f54d0684640 1 -- 172.21.15.19:0/1009994158 >> 172.21.15.19:0/1009994158 conn(0x7f54c80ce0b0 msgr2=0x7f54c80c9f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:23.331 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.329+0000 7f54d0684640 1 -- 172.21.15.19:0/1009994158 shutdown_connections 2024-08-31T12:16:23.331 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.329+0000 7f54d0684640 1 -- 172.21.15.19:0/1009994158 wait complete. 2024-08-31T12:16:23.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.331+0000 7f54d0684640 1 Processor -- start 2024-08-31T12:16:23.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.331+0000 7f54d0684640 1 -- start start 2024-08-31T12:16:23.334 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.332+0000 7f54d0684640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54c80c5440 0x7f54c8154960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:23.334 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.332+0000 7f54d0684640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f54c80c7e60 0x7f54c8155460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:23.334 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.332+0000 7f54d0684640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f54c8158010 0x7f54c814d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:23.335 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.332+0000 7f54d0684640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f54c8074da0 con 0x7f54c80c7e60 2024-08-31T12:16:23.335 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.332+0000 7f54d0684640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f54c80c76e0 con 0x7f54c80c5440 2024-08-31T12:16:23.335 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.332+0000 7f54d0684640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f54c8151950 con 0x7f54c8158010 2024-08-31T12:16:23.335 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.332+0000 7f54cdc1f640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f54c80c7e60 0x7f54c8155460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:23.335 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.332+0000 7f54cdc1f640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f54c80c7e60 0x7f54c8155460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:38116/0 (socket says 172.21.15.19:38116) 2024-08-31T12:16:23.335 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.332+0000 7f54cdc1f640 1 -- 172.21.15.19:0/1416223929 learned_addr learned my addr 172.21.15.19:0/1416223929 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:23.335 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.332+0000 7f54ce420640 1 --2- 172.21.15.19:0/1416223929 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54c80c5440 0x7f54c8154960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:23.335 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.333+0000 7f54cec21640 1 --2- 172.21.15.19:0/1416223929 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f54c8158010 0x7f54c814d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:23.335 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.333+0000 7f54cdc1f640 1 -- 172.21.15.19:0/1416223929 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f54c8158010 msgr2=0x7f54c814d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:23.335 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.333+0000 7f54cdc1f640 1 --2- 172.21.15.19:0/1416223929 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f54c8158010 0x7f54c814d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:23.335 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.333+0000 7f54cdc1f640 1 -- 172.21.15.19:0/1416223929 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54c80c5440 msgr2=0x7f54c8154960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:23.335 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.333+0000 7f54cdc1f640 1 --2- 172.21.15.19:0/1416223929 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54c80c5440 0x7f54c8154960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:23.336 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.333+0000 7f54cdc1f640 1 -- 172.21.15.19:0/1416223929 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54c8148bc0 con 0x7f54c80c7e60 2024-08-31T12:16:23.336 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.333+0000 7f54cdc1f640 1 --2- 172.21.15.19:0/1416223929 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f54c80c7e60 0x7f54c8155460 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f54b800c3a0 tx=0x7f54b800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:23.336 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.333+0000 7f54beffd640 1 -- 172.21.15.19:0/1416223929 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b8017070 con 0x7f54c80c7e60 2024-08-31T12:16:23.336 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.333+0000 7f54beffd640 1 -- 172.21.15.19:0/1416223929 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f54b80053c0 con 0x7f54c80c7e60 2024-08-31T12:16:23.336 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.333+0000 7f54beffd640 1 -- 172.21.15.19:0/1416223929 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b8003850 con 0x7f54c80c7e60 2024-08-31T12:16:23.336 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.333+0000 7f54d0684640 1 -- 172.21.15.19:0/1416223929 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54c81488e0 con 0x7f54c80c7e60 2024-08-31T12:16:23.336 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.333+0000 7f54d0684640 1 -- 172.21.15.19:0/1416223929 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54c81ef8d0 con 0x7f54c80c7e60 2024-08-31T12:16:23.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.334+0000 7f54d0684640 1 -- 172.21.15.19:0/1416223929 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54c8071370 con 0x7f54c80c7e60 2024-08-31T12:16:23.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.335+0000 7f54beffd640 1 -- 172.21.15.19:0/1416223929 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f54b801a070 con 0x7f54c80c7e60 2024-08-31T12:16:23.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.336+0000 7f54beffd640 1 --2- 172.21.15.19:0/1416223929 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f54ac077a50 0x7f54ac079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:23.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.336+0000 7f54beffd640 1 -- 172.21.15.19:0/1416223929 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f54b8098880 con 0x7f54c80c7e60 2024-08-31T12:16:23.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.336+0000 7f54ce420640 1 --2- 172.21.15.19:0/1416223929 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f54ac077a50 0x7f54ac079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:23.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.336+0000 7f54ce420640 1 --2- 172.21.15.19:0/1416223929 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f54ac077a50 0x7f54ac079f60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f54c4034000 tx=0x7f54c4004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:23.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.341+0000 7f54beffd640 1 -- 172.21.15.19:0/1416223929 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f54c8071370 con 0x7f54c80c7e60 2024-08-31T12:16:23.347 INFO:tasks.workunit.client.0.smithi019.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-08-31T12:16:23.572 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.571+0000 7f54d0684640 1 -- 172.21.15.19:0/1416223929 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f54c8074860 con 0x7f54c80c7e60 2024-08-31T12:16:23.572 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.572+0000 7f54beffd640 1 -- 172.21.15.19:0/1416223929 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+6901 (secure 0 0 0) 0x7f54c8148bc0 con 0x7f54c80c7e60 2024-08-31T12:16:23.573 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:16:23.573 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":12,"btime":"2024-08-31T12:16:13:439874+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4371,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6833/4215834946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":4215834946},{"type":"v1","addr":"172.21.15.19:6833","nonce":4215834946}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/543344732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":543344732},{"type":"v1","addr":"172.21.15.19:6835","nonce":543344732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4380,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6837/699077015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6836","nonce":699077015},{"type":"v1","addr":"172.21.15.19:6837","nonce":699077015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-31T12:15:56.652936+0000","modified":"2024-08-31T12:16:13.439859+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4429,"mds_1":4426,"mds_2":4398},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4398":{"gid":4398,"name":"f","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.43:6837/2103490885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6835","nonce":2103490885},{"type":"v1","addr":"172.21.15.43:6837","nonce":2103490885}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4426":{"gid":4426,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.43:6839/1196858061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1196858061},{"type":"v1","addr":"172.21.15.43:6839","nonce":1196858061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4429":{"gid":4429,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.43:6838/3981040655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":3981040655},{"type":"v1","addr":"172.21.15.43:6838","nonce":3981040655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4429,"qdb_cluster":[4398,4426,4429]},"id":1}]} 2024-08-31T12:16:23.573 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 12 2024-08-31T12:16:23.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.574+0000 7f54d0684640 1 -- 172.21.15.19:0/1416223929 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f54ac077a50 msgr2=0x7f54ac079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:23.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.574+0000 7f54d0684640 1 --2- 172.21.15.19:0/1416223929 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f54ac077a50 0x7f54ac079f60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f54c4034000 tx=0x7f54c4004730 comp rx=0 tx=0).stop 2024-08-31T12:16:23.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.574+0000 7f54d0684640 1 -- 172.21.15.19:0/1416223929 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f54c80c7e60 msgr2=0x7f54c8155460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:23.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.574+0000 7f54d0684640 1 --2- 172.21.15.19:0/1416223929 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f54c80c7e60 0x7f54c8155460 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f54b800c3a0 tx=0x7f54b800c860 comp rx=0 tx=0).stop 2024-08-31T12:16:23.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.574+0000 7f54d0684640 1 -- 172.21.15.19:0/1416223929 shutdown_connections 2024-08-31T12:16:23.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.574+0000 7f54d0684640 1 --2- 172.21.15.19:0/1416223929 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f54ac077a50 0x7f54ac079f60 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:23.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.574+0000 7f54d0684640 1 --2- 172.21.15.19:0/1416223929 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f54c8158010 0x7f54c814d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:23.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.574+0000 7f54d0684640 1 --2- 172.21.15.19:0/1416223929 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f54c80c7e60 0x7f54c8155460 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:23.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.574+0000 7f54d0684640 1 --2- 172.21.15.19:0/1416223929 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54c80c5440 0x7f54c8154960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:23.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.574+0000 7f54d0684640 1 -- 172.21.15.19:0/1416223929 >> 172.21.15.19:0/1416223929 conn(0x7f54c80ce0b0 msgr2=0x7f54c8071770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:23.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.574+0000 7f54d0684640 1 -- 172.21.15.19:0/1416223929 shutdown_connections 2024-08-31T12:16:23.576 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.574+0000 7f54d0684640 1 -- 172.21.15.19:0/1416223929 wait complete. 2024-08-31T12:16:23.586 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-08-31T12:16:23.586 INFO:tasks.mon_thrash.mon_thrasher:seed: 1725106583, revive delay: 20.0, thrash delay: 10.0 thrash many: False, maintain quorum: True store thrash: False, probability: 50 freeze mon: prob 10.0 duration 15.0 2024-08-31T12:16:23.586 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-08-31T12:16:23.586 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:16:23.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.738+0000 7f46a9fad640 1 -- 172.21.15.19:0/425770183 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46a40ccbb0 msgr2=0x7f46a40cb280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:23.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.738+0000 7f46a9fad640 1 --2- 172.21.15.19:0/425770183 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46a40ccbb0 0x7f46a40cb280 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4694005fd0 tx=0x7f469402a670 comp rx=0 tx=0).stop 2024-08-31T12:16:23.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.738+0000 7f46a9fad640 1 -- 172.21.15.19:0/425770183 shutdown_connections 2024-08-31T12:16:23.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.738+0000 7f46a9fad640 1 --2- 172.21.15.19:0/425770183 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f46a40cc070 0x7f46a40cc480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:23.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.738+0000 7f46a9fad640 1 --2- 172.21.15.19:0/425770183 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46a40ccbb0 0x7f46a40cb280 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:23.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.738+0000 7f46a9fad640 1 -- 172.21.15.19:0/425770183 >> 172.21.15.19:0/425770183 conn(0x7f46a4074810 msgr2=0x7f46a4068aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:23.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.738+0000 7f46a9fad640 1 -- 172.21.15.19:0/425770183 shutdown_connections 2024-08-31T12:16:23.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.738+0000 7f46a9fad640 1 -- 172.21.15.19:0/425770183 wait complete. 2024-08-31T12:16:23.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.740+0000 7f46a9fad640 1 Processor -- start 2024-08-31T12:16:23.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.740+0000 7f46a9fad640 1 -- start start 2024-08-31T12:16:23.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.741+0000 7f46a9fad640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f46a40cc070 0x7f46a40c7b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:23.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.741+0000 7f46a9fad640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f46a40ccbb0 0x7f46a41f23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:23.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.741+0000 7f46a9fad640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46a415b120 0x7f46a40c6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:23.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.741+0000 7f46a9fad640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f46a41557e0 con 0x7f46a40ccbb0 2024-08-31T12:16:23.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.741+0000 7f46a9fad640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f46a4155660 con 0x7f46a415b120 2024-08-31T12:16:23.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.741+0000 7f46a9fad640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f46a406e1a0 con 0x7f46a40cc070 2024-08-31T12:16:23.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.741+0000 7f46a2ffd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f46a40ccbb0 0x7f46a41f23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:23.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.741+0000 7f46a2ffd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f46a40ccbb0 0x7f46a41f23f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:38122/0 (socket says 172.21.15.19:38122) 2024-08-31T12:16:23.744 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.741+0000 7f46a2ffd640 1 -- 172.21.15.19:0/3186616779 learned_addr learned my addr 172.21.15.19:0/3186616779 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:23.744 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.741+0000 7f46a3fff640 1 --2- 172.21.15.19:0/3186616779 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46a415b120 0x7f46a40c6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:23.744 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.741+0000 7f46a37fe640 1 --2- 172.21.15.19:0/3186616779 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f46a40cc070 0x7f46a40c7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:23.744 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.741+0000 7f46a2ffd640 1 -- 172.21.15.19:0/3186616779 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f46a40cc070 msgr2=0x7f46a40c7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:23.744 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.741+0000 7f46a2ffd640 1 --2- 172.21.15.19:0/3186616779 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f46a40cc070 0x7f46a40c7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:23.744 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.741+0000 7f46a2ffd640 1 -- 172.21.15.19:0/3186616779 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46a415b120 msgr2=0x7f46a40c6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:23.744 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.741+0000 7f46a2ffd640 1 --2- 172.21.15.19:0/3186616779 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46a415b120 0x7f46a40c6450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:23.744 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.741+0000 7f46a2ffd640 1 -- 172.21.15.19:0/3186616779 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46a41f2930 con 0x7f46a40ccbb0 2024-08-31T12:16:23.744 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.741+0000 7f46a3fff640 1 --2- 172.21.15.19:0/3186616779 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46a415b120 0x7f46a40c6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:16:23.744 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.741+0000 7f46a37fe640 1 --2- 172.21.15.19:0/3186616779 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f46a40cc070 0x7f46a40c7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:16:23.744 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.742+0000 7f46a2ffd640 1 --2- 172.21.15.19:0/3186616779 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f46a40ccbb0 0x7f46a41f23f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f469800ea20 tx=0x7f469800eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:23.744 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.742+0000 7f468bfff640 1 -- 172.21.15.19:0/3186616779 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f469800cd50 con 0x7f46a40ccbb0 2024-08-31T12:16:23.744 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.742+0000 7f468bfff640 1 -- 172.21.15.19:0/3186616779 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4698004020 con 0x7f46a40ccbb0 2024-08-31T12:16:23.745 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.742+0000 7f468bfff640 1 -- 172.21.15.19:0/3186616779 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4698004450 con 0x7f46a40ccbb0 2024-08-31T12:16:23.745 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.742+0000 7f46a9fad640 1 -- 172.21.15.19:0/3186616779 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46a41566f0 con 0x7f46a40ccbb0 2024-08-31T12:16:23.745 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.742+0000 7f46a9fad640 1 -- 172.21.15.19:0/3186616779 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46a4155cc0 con 0x7f46a40ccbb0 2024-08-31T12:16:23.745 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.743+0000 7f46a9fad640 1 -- 172.21.15.19:0/3186616779 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46a40cb280 con 0x7f46a40ccbb0 2024-08-31T12:16:23.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.744+0000 7f468bfff640 1 -- 172.21.15.19:0/3186616779 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f46980035f0 con 0x7f46a40ccbb0 2024-08-31T12:16:23.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.744+0000 7f468bfff640 1 --2- 172.21.15.19:0/3186616779 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4690077a80 0x7f4690079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:23.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.744+0000 7f468bfff640 1 -- 172.21.15.19:0/3186616779 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f4698099f70 con 0x7f46a40ccbb0 2024-08-31T12:16:23.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.744+0000 7f46a37fe640 1 --2- 172.21.15.19:0/3186616779 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4690077a80 0x7f4690079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:23.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.745+0000 7f46a37fe640 1 --2- 172.21.15.19:0/3186616779 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4690077a80 0x7f4690079f90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4694033000 tx=0x7f4694004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:23.749 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:23.749+0000 7f468bfff640 1 -- 172.21.15.19:0/3186616779 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f4698062cd0 con 0x7f46a40ccbb0 2024-08-31T12:16:24.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.006+0000 7f46a9fad640 1 -- 172.21.15.19:0/3186616779 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f46a40b5a10 con 0x7f46a40ccbb0 2024-08-31T12:16:24.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.006+0000 7f468bfff640 1 -- 172.21.15.19:0/3186616779 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f46980625d0 con 0x7f46a40ccbb0 2024-08-31T12:16:24.007 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":51,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:16:24.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.008+0000 7f46a9fad640 1 -- 172.21.15.19:0/3186616779 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4690077a80 msgr2=0x7f4690079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:24.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.008+0000 7f46a9fad640 1 --2- 172.21.15.19:0/3186616779 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4690077a80 0x7f4690079f90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4694033000 tx=0x7f4694004730 comp rx=0 tx=0).stop 2024-08-31T12:16:24.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.008+0000 7f46a9fad640 1 -- 172.21.15.19:0/3186616779 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f46a40ccbb0 msgr2=0x7f46a41f23f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:24.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.008+0000 7f46a9fad640 1 --2- 172.21.15.19:0/3186616779 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f46a40ccbb0 0x7f46a41f23f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f469800ea20 tx=0x7f469800eee0 comp rx=0 tx=0).stop 2024-08-31T12:16:24.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.008+0000 7f46a9fad640 1 -- 172.21.15.19:0/3186616779 shutdown_connections 2024-08-31T12:16:24.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.008+0000 7f46a9fad640 1 --2- 172.21.15.19:0/3186616779 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4690077a80 0x7f4690079f90 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:24.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.008+0000 7f46a9fad640 1 --2- 172.21.15.19:0/3186616779 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46a415b120 0x7f46a40c6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:24.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.008+0000 7f46a9fad640 1 --2- 172.21.15.19:0/3186616779 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f46a40ccbb0 0x7f46a41f23f0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:24.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.008+0000 7f46a9fad640 1 --2- 172.21.15.19:0/3186616779 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f46a40cc070 0x7f46a40c7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:24.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.008+0000 7f46a9fad640 1 -- 172.21.15.19:0/3186616779 >> 172.21.15.19:0/3186616779 conn(0x7f46a4074810 msgr2=0x7f46a4150270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:24.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.008+0000 7f46a9fad640 1 -- 172.21.15.19:0/3186616779 shutdown_connections 2024-08-31T12:16:24.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.008+0000 7f46a9fad640 1 -- 172.21.15.19:0/3186616779 wait complete. 2024-08-31T12:16:24.020 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-08-31T12:16:24.020 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-08-31T12:16:24.020 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-08-31T12:16:24.020 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-31T12:16:24.135 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.134+0000 7f9d079a8640 1 -- 172.21.15.19:0/737840127 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9d00155300 msgr2=0x7f9d0006d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:24.135 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.134+0000 7f9d079a8640 1 --2- 172.21.15.19:0/737840127 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9d00155300 0x7f9d0006d000 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9cf8009830 tx=0x7f9cf802a530 comp rx=0 tx=0).stop 2024-08-31T12:16:24.135 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.134+0000 7f9d079a8640 1 -- 172.21.15.19:0/737840127 shutdown_connections 2024-08-31T12:16:24.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.134+0000 7f9d079a8640 1 --2- 172.21.15.19:0/737840127 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9d00155300 0x7f9d0006d000 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:24.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.134+0000 7f9d079a8640 1 --2- 172.21.15.19:0/737840127 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9d0006e0e0 0x7f9d000c9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:24.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.134+0000 7f9d079a8640 1 -- 172.21.15.19:0/737840127 >> 172.21.15.19:0/737840127 conn(0x7f9d000690d0 msgr2=0x7f9d000d0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:24.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.134+0000 7f9d079a8640 1 -- 172.21.15.19:0/737840127 shutdown_connections 2024-08-31T12:16:24.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.134+0000 7f9d079a8640 1 -- 172.21.15.19:0/737840127 wait complete. 2024-08-31T12:16:24.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.136+0000 7f9d079a8640 1 Processor -- start 2024-08-31T12:16:24.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.136+0000 7f9d079a8640 1 -- start start 2024-08-31T12:16:24.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.137+0000 7f9d079a8640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9d0006e0e0 0x7f9d00150770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:24.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.137+0000 7f9d079a8640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9d000c8a40 0x7f9d001e6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:24.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.137+0000 7f9d079a8640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9d00155300 0x7f9d000c2540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:24.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.137+0000 7f9d079a8640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f9d000c8330 con 0x7f9d00155300 2024-08-31T12:16:24.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.137+0000 7f9d079a8640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9d0015b250 con 0x7f9d0006e0e0 2024-08-31T12:16:24.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.137+0000 7f9d079a8640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f9d000c8610 con 0x7f9d000c8a40 2024-08-31T12:16:24.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.137+0000 7f9d05f45640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9d00155300 0x7f9d000c2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:24.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.137+0000 7f9d05f45640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9d00155300 0x7f9d000c2540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:38132/0 (socket says 172.21.15.19:38132) 2024-08-31T12:16:24.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.137+0000 7f9d05f45640 1 -- 172.21.15.19:0/241017735 learned_addr learned my addr 172.21.15.19:0/241017735 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:24.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.137+0000 7f9d05744640 1 --2- 172.21.15.19:0/241017735 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9d0006e0e0 0x7f9d00150770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:24.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.138+0000 7f9d04f43640 1 --2- 172.21.15.19:0/241017735 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9d000c8a40 0x7f9d001e6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:24.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.138+0000 7f9d05f45640 1 -- 172.21.15.19:0/241017735 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9d000c8a40 msgr2=0x7f9d001e6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:24.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.138+0000 7f9d05f45640 1 --2- 172.21.15.19:0/241017735 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9d000c8a40 0x7f9d001e6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:24.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.138+0000 7f9d05f45640 1 -- 172.21.15.19:0/241017735 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9d0006e0e0 msgr2=0x7f9d00150770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:24.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.138+0000 7f9d05f45640 1 --2- 172.21.15.19:0/241017735 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9d0006e0e0 0x7f9d00150770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:24.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.138+0000 7f9d05f45640 1 -- 172.21.15.19:0/241017735 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d0014ea20 con 0x7f9d00155300 2024-08-31T12:16:24.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.138+0000 7f9d05f45640 1 --2- 172.21.15.19:0/241017735 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9d00155300 0x7f9d000c2540 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9cf802ab20 tx=0x7f9cf8003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:24.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.138+0000 7f9cfdffb640 1 -- 172.21.15.19:0/241017735 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf8005c90 con 0x7f9d00155300 2024-08-31T12:16:24.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.138+0000 7f9cfdffb640 1 -- 172.21.15.19:0/241017735 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9cf80317c0 con 0x7f9d00155300 2024-08-31T12:16:24.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.138+0000 7f9cfdffb640 1 -- 172.21.15.19:0/241017735 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf8031bf0 con 0x7f9d00155300 2024-08-31T12:16:24.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.138+0000 7f9d079a8640 1 -- 172.21.15.19:0/241017735 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d0014e4a0 con 0x7f9d00155300 2024-08-31T12:16:24.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.138+0000 7f9d079a8640 1 -- 172.21.15.19:0/241017735 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d000c5ce0 con 0x7f9d00155300 2024-08-31T12:16:24.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.139+0000 7f9d079a8640 1 --2- 172.21.15.19:0/241017735 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9d0014d720 0x7f9d000c2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:24.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.139+0000 7f9d079a8640 1 -- 172.21.15.19:0/241017735 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f9d0014fa10 con 0x7f9d0014d720 2024-08-31T12:16:24.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.139+0000 7f9d05744640 1 --2- 172.21.15.19:0/241017735 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9d0014d720 0x7f9d000c2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:24.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.140+0000 7f9d05744640 1 -- 172.21.15.19:0/241017735 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9d000c2fc0 con 0x7f9d0014d720 2024-08-31T12:16:24.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.140+0000 7f9d05744640 1 --2- 172.21.15.19:0/241017735 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9d0014d720 0x7f9d000c2a80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9cf000fae0 tx=0x7f9cf0001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:24.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.140+0000 7f9cfdffb640 1 -- 172.21.15.19:0/241017735 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf0003b80 con 0x7f9d0014d720 2024-08-31T12:16:24.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.140+0000 7f9cfdffb640 1 -- 172.21.15.19:0/241017735 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9cf0018070 con 0x7f9d0014d720 2024-08-31T12:16:24.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.140+0000 7f9cfdffb640 1 -- 172.21.15.19:0/241017735 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9d0014d720 msgr2=0x7f9d000c2a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:24.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.140+0000 7f9cfdffb640 1 --2- 172.21.15.19:0/241017735 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9d0014d720 0x7f9d000c2a80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9cf000fae0 tx=0x7f9cf0001a80 comp rx=0 tx=0).stop 2024-08-31T12:16:24.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.140+0000 7f9cfdffb640 1 -- 172.21.15.19:0/241017735 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9d0014d720 msgr2=0x7f9d000c2a80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:16:24.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.140+0000 7f9cfdffb640 1 --2- 172.21.15.19:0/241017735 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9d0014d720 0x7f9d000c2a80 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:24.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.140+0000 7f9cfdffb640 1 -- 172.21.15.19:0/241017735 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f9cf803b410 con 0x7f9d00155300 2024-08-31T12:16:24.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.141+0000 7f9cfdffb640 1 --2- 172.21.15.19:0/241017735 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9cec0774f0 0x7f9cec079a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:24.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.141+0000 7f9cfdffb640 1 -- 172.21.15.19:0/241017735 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9cf80ba7e0 con 0x7f9d00155300 2024-08-31T12:16:24.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.141+0000 7f9d05744640 1 --2- 172.21.15.19:0/241017735 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9cec0774f0 0x7f9cec079a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:24.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.142+0000 7f9d05744640 1 --2- 172.21.15.19:0/241017735 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9cec0774f0 0x7f9cec079a00 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9cf000fae0 tx=0x7f9cf0005000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:24.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.147+0000 7f9d079a8640 1 --2- 172.21.15.19:0/241017735 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9d001f4860 0x7f9d001f4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:24.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.147+0000 7f9d079a8640 1 -- 172.21.15.19:0/241017735 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f9d000c9850 con 0x7f9d001f4860 2024-08-31T12:16:24.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.147+0000 7f9d04f43640 1 --2- 172.21.15.19:0/241017735 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9d001f4860 0x7f9d001f4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:24.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.147+0000 7f9d04f43640 1 -- 172.21.15.19:0/241017735 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9d000ccf20 con 0x7f9d001f4860 2024-08-31T12:16:24.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.147+0000 7f9d04f43640 1 --2- 172.21.15.19:0/241017735 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9d001f4860 0x7f9d001f4c70 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9d00158a60 tx=0x7f9cf4011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:24.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.147+0000 7f9cfdffb640 1 -- 172.21.15.19:0/241017735 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf400f980 con 0x7f9d001f4860 2024-08-31T12:16:24.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.148+0000 7f9cfdffb640 1 -- 172.21.15.19:0/241017735 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f9cf400f980 con 0x7f9d001f4860 2024-08-31T12:16:24.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.148+0000 7f9cfdffb640 1 -- 172.21.15.19:0/241017735 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9d001f4860 msgr2=0x7f9d001f4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:24.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.148+0000 7f9cfdffb640 1 --2- 172.21.15.19:0/241017735 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9d001f4860 0x7f9d001f4c70 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9d00158a60 tx=0x7f9cf4011e70 comp rx=0 tx=0).stop 2024-08-31T12:16:24.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.148+0000 7f9cfdffb640 1 -- 172.21.15.19:0/241017735 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9d001f4860 msgr2=0x7f9d001f4c70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:16:24.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.148+0000 7f9cfdffb640 1 --2- 172.21.15.19:0/241017735 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9d001f4860 0x7f9d001f4c70 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:24.149 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:16:24.149 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:16:24.150 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:16:24.150 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "leader", 2024-08-31T12:16:24.150 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 4, 2024-08-31T12:16:24.150 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 52284, 2024-08-31T12:16:24.150 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:16:24.150 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:16:24.150 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:16:24.150 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:16:24.150 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:16:24.150 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 52, 2024-08-31T12:16:24.150 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:16:24.150 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:16:24.150 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:16:24.151 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:16:24.151 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:16:24.151 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:16:24.151 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:16:24.151 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:16:24.151 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:16:24.151 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:16:24.151 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:16:24.151 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:16:24.151 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:16:24.151 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:16:24.151 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:16:24.151 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:16:24.151 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:16:24.152 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:16:24.152 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:16:24.152 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:16:24.152 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:16:24.152 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:16:24.152 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:16:24.152 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:16:24.152 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:16:24.152 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:16:24.152 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:16:24.152 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:16:24.152 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:16:24.152 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:24.152 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:16:24.153 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:16:24.153 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:16:24.153 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:16:24.153 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:16:24.153 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:16:24.153 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:16:24.153 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:16:24.153 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:16:24.153 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:16:24.153 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:16:24.153 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:16:24.153 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:16:24.153 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:16:24.154 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:16:24.154 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:16:24.154 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:16:24.154 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:16:24.154 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:16:24.154 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:16:24.154 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:16:24.154 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:16:24.154 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:16:24.154 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:16:24.154 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:16:24.154 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:16:24.154 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:16:24.154 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:16:24.155 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:16:24.155 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:16:24.155 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:24.155 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:16:24.155 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:24.155 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:16:24.155 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:16:24.155 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:16:24.155 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:16:24.155 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:24.155 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:16:24.155 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:16:24.155 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:16:24.155 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:24.156 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:24.156 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:16:24.156 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:16:24.156 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:16:24.156 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:24.156 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:16:24.156 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:24.156 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:16:24.156 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:16:24.156 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:16:24.156 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:16:24.156 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:16:24.157 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:24.157 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:24.157 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:16:24.157 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:16:24.157 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:16:24.157 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:16:24.157 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:24.157 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:16:24.157 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:16:24.157 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:16:24.157 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:24.157 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:24.157 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:16:24.157 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:16:24.158 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:16:24.158 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:24.158 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:16:24.158 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:24.158 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:16:24.158 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:16:24.158 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:16:24.158 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:16:24.158 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:16:24.158 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:24.158 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:24.158 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:16:24.158 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:16:24.158 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:16:24.159 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:16:24.159 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:24.159 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:16:24.159 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:16:24.159 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:16:24.159 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:24.159 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:24.159 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:16:24.159 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:16:24.159 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:16:24.159 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:24.159 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:16:24.159 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:24.159 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:16:24.160 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:16:24.160 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:16:24.160 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:16:24.160 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:16:24.160 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:24.160 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:16:24.160 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:24.160 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:16:24.160 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:16:24.160 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:24.160 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:16:24.160 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:16:24.160 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:16:24.160 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:24.161 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:16:24.161 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:16:24.161 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:24.161 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:16:24.161 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:16:24.161 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 4 2024-08-31T12:16:24.161 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:24.161 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:16:24.161 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:16:24.161 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:24.161 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:16:24.161 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:16:24.161 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 4 2024-08-31T12:16:24.161 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:24.161 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:16:24.162 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:16:24.162 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:24.162 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:16:24.162 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:16:24.162 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 3 2024-08-31T12:16:24.162 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:24.162 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:16:24.162 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:24.162 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:16:24.162 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:16:24.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.148+0000 7f9d079a8640 1 -- 172.21.15.19:0/241017735 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9cec0774f0 msgr2=0x7f9cec079a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:24.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.148+0000 7f9d079a8640 1 --2- 172.21.15.19:0/241017735 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9cec0774f0 0x7f9cec079a00 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9cf000fae0 tx=0x7f9cf0005000 comp rx=0 tx=0).stop 2024-08-31T12:16:24.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.148+0000 7f9d079a8640 1 -- 172.21.15.19:0/241017735 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9d00155300 msgr2=0x7f9d000c2540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:24.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.148+0000 7f9d079a8640 1 --2- 172.21.15.19:0/241017735 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9d00155300 0x7f9d000c2540 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9cf802ab20 tx=0x7f9cf8003c10 comp rx=0 tx=0).stop 2024-08-31T12:16:24.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.148+0000 7f9d05f45640 1 -- 172.21.15.19:0/241017735 reap_dead start 2024-08-31T12:16:24.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.148+0000 7f9d05f45640 1 -- 172.21.15.19:0/241017735 reap_dead start 2024-08-31T12:16:24.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.148+0000 7f9d079a8640 1 -- 172.21.15.19:0/241017735 shutdown_connections 2024-08-31T12:16:24.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.148+0000 7f9d079a8640 1 -- 172.21.15.19:0/241017735 >> 172.21.15.19:0/241017735 conn(0x7f9d000690d0 msgr2=0x7f9d0015a090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:24.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.148+0000 7f9d079a8640 1 -- 172.21.15.19:0/241017735 shutdown_connections 2024-08-31T12:16:24.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.148+0000 7f9d079a8640 1 -- 172.21.15.19:0/241017735 wait complete. 2024-08-31T12:16:24.164 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:16:24.165 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-08-31T12:16:24.165 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-31T12:16:24.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.279+0000 7f4cd3a15640 1 -- 172.21.15.19:0/3796032181 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4cb8005250 msgr2=0x7f4cb800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:24.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.279+0000 7f4cd3a15640 1 --2- 172.21.15.19:0/3796032181 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4cb8005250 0x7f4cb800b9c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f4cc4009ed0 tx=0x7f4cb8005830 comp rx=0 tx=0).stop 2024-08-31T12:16:24.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.279+0000 7f4cd3a15640 1 -- 172.21.15.19:0/3796032181 shutdown_connections 2024-08-31T12:16:24.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.279+0000 7f4cd3a15640 1 --2- 172.21.15.19:0/3796032181 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4cb8005250 0x7f4cb800b9c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:24.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.279+0000 7f4cd3a15640 1 -- 172.21.15.19:0/3796032181 >> 172.21.15.19:0/3796032181 conn(0x7f4ccc0690d0 msgr2=0x7f4ccc0d0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:24.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.279+0000 7f4cd3a15640 1 -- 172.21.15.19:0/3796032181 shutdown_connections 2024-08-31T12:16:24.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.279+0000 7f4cd3a15640 1 -- 172.21.15.19:0/3796032181 wait complete. 2024-08-31T12:16:24.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.281+0000 7f4cd3a15640 1 Processor -- start 2024-08-31T12:16:24.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.281+0000 7f4cd3a15640 1 -- start start 2024-08-31T12:16:24.284 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.282+0000 7f4cd3a15640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4cb8005250 0x7f4ccc0c4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:24.284 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.282+0000 7f4cd3a15640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ccc1f11d0 0x7f4ccc0c4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:24.284 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.282+0000 7f4cd3a15640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4ccc15d690 0x7f4ccc0c25d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:24.284 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.282+0000 7f4cd3a15640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f4ccc15b250 con 0x7f4ccc15d690 2024-08-31T12:16:24.284 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.282+0000 7f4cd3a15640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4ccc15af70 con 0x7f4ccc1f11d0 2024-08-31T12:16:24.284 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.282+0000 7f4cd3a15640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f4ccc0c8610 con 0x7f4cb8005250 2024-08-31T12:16:24.284 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.282+0000 7f4cd1fb2640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4ccc15d690 0x7f4ccc0c25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:24.285 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.282+0000 7f4cd1fb2640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4ccc15d690 0x7f4ccc0c25d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:38158/0 (socket says 172.21.15.19:38158) 2024-08-31T12:16:24.285 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.282+0000 7f4cd1fb2640 1 -- 172.21.15.19:0/1832103900 learned_addr learned my addr 172.21.15.19:0/1832103900 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:24.285 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.282+0000 7f4cd17b1640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4cb8005250 0x7f4ccc0c4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:24.285 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.282+0000 7f4cd0fb0640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ccc1f11d0 0x7f4ccc0c4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:24.285 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.282+0000 7f4cd1fb2640 1 -- 172.21.15.19:0/1832103900 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4cb8005250 msgr2=0x7f4ccc0c4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:24.285 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.282+0000 7f4cd1fb2640 1 --2- 172.21.15.19:0/1832103900 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4cb8005250 0x7f4ccc0c4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:24.285 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.282+0000 7f4cd1fb2640 1 -- 172.21.15.19:0/1832103900 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ccc1f11d0 msgr2=0x7f4ccc0c4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:24.285 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.282+0000 7f4cd1fb2640 1 --2- 172.21.15.19:0/1832103900 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ccc1f11d0 0x7f4ccc0c4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:24.285 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.283+0000 7f4cd1fb2640 1 -- 172.21.15.19:0/1832103900 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ccc0c4c70 con 0x7f4ccc15d690 2024-08-31T12:16:24.285 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.283+0000 7f4cd1fb2640 1 --2- 172.21.15.19:0/1832103900 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4ccc15d690 0x7f4ccc0c25d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4cc4009e20 tx=0x7f4cc4034c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:24.285 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.283+0000 7f4cc1ffb640 1 -- 172.21.15.19:0/1832103900 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cc4041070 con 0x7f4ccc15d690 2024-08-31T12:16:24.285 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.283+0000 7f4cc1ffb640 1 -- 172.21.15.19:0/1832103900 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4cc40325f0 con 0x7f4ccc15d690 2024-08-31T12:16:24.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.283+0000 7f4cd3a15640 1 -- 172.21.15.19:0/1832103900 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ccc0c7290 con 0x7f4ccc15d690 2024-08-31T12:16:24.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.283+0000 7f4cc1ffb640 1 -- 172.21.15.19:0/1832103900 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cc4009ad0 con 0x7f4ccc15d690 2024-08-31T12:16:24.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.283+0000 7f4cd3a15640 1 -- 172.21.15.19:0/1832103900 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ccc0cd5c0 con 0x7f4ccc15d690 2024-08-31T12:16:24.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.284+0000 7f4cd3a15640 1 --2- 172.21.15.19:0/1832103900 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ccc15daa0 0x7f4ccc0c2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:24.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.284+0000 7f4cd3a15640 1 -- 172.21.15.19:0/1832103900 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4ccc0cd790 con 0x7f4ccc15daa0 2024-08-31T12:16:24.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.284+0000 7f4cd17b1640 1 --2- 172.21.15.19:0/1832103900 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ccc15daa0 0x7f4ccc0c2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:24.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.285+0000 7f4cd17b1640 1 -- 172.21.15.19:0/1832103900 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4ccc15aa50 con 0x7f4ccc15daa0 2024-08-31T12:16:24.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.285+0000 7f4cc1ffb640 1 -- 172.21.15.19:0/1832103900 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f4cc4032a40 con 0x7f4ccc15d690 2024-08-31T12:16:24.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.285+0000 7f4cd17b1640 1 --2- 172.21.15.19:0/1832103900 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ccc15daa0 0x7f4ccc0c2b10 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4cc8006650 tx=0x7f4cc80128d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:24.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.285+0000 7f4cc1ffb640 1 --2- 172.21.15.19:0/1832103900 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4cb0077a80 0x7f4cb0079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:24.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.285+0000 7f4cc1ffb640 1 -- 172.21.15.19:0/1832103900 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f4cc40b9cb0 con 0x7f4ccc15d690 2024-08-31T12:16:24.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.285+0000 7f4cd0fb0640 1 --2- 172.21.15.19:0/1832103900 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4cb0077a80 0x7f4cb0079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:24.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.285+0000 7f4cc1ffb640 1 -- 172.21.15.19:0/1832103900 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cc800f890 con 0x7f4ccc15daa0 2024-08-31T12:16:24.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.285+0000 7f4cc1ffb640 1 -- 172.21.15.19:0/1832103900 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4cc800fa30 con 0x7f4ccc15daa0 2024-08-31T12:16:24.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.286+0000 7f4cc1ffb640 1 -- 172.21.15.19:0/1832103900 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ccc15daa0 msgr2=0x7f4ccc0c2b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:24.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.286+0000 7f4cc1ffb640 1 --2- 172.21.15.19:0/1832103900 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ccc15daa0 0x7f4ccc0c2b10 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4cc8006650 tx=0x7f4cc80128d0 comp rx=0 tx=0).stop 2024-08-31T12:16:24.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.286+0000 7f4cc1ffb640 1 -- 172.21.15.19:0/1832103900 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ccc15daa0 msgr2=0x7f4ccc0c2b10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:16:24.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.286+0000 7f4cc1ffb640 1 --2- 172.21.15.19:0/1832103900 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ccc15daa0 0x7f4ccc0c2b10 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:24.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.286+0000 7f4cd0fb0640 1 --2- 172.21.15.19:0/1832103900 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4cb0077a80 0x7f4cb0079f90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc1eb840 tx=0x7f4cbc001110 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:24.293 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.292+0000 7f4cd3a15640 1 --2- 172.21.15.19:0/1832103900 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ccc1f15e0 0x7f4ccc0cf690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:24.293 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.292+0000 7f4cd3a15640 1 -- 172.21.15.19:0/1832103900 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4ccc1553e0 con 0x7f4ccc1f15e0 2024-08-31T12:16:24.293 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.292+0000 7f4cd17b1640 1 --2- 172.21.15.19:0/1832103900 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ccc1f15e0 0x7f4ccc0cf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:24.293 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.292+0000 7f4cd17b1640 1 -- 172.21.15.19:0/1832103900 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4ccc06d000 con 0x7f4ccc1f15e0 2024-08-31T12:16:24.293 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.293+0000 7f4cd17b1640 1 --2- 172.21.15.19:0/1832103900 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ccc1f15e0 0x7f4ccc0cf690 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4cc8006650 tx=0x7f4cc8001120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:24.294 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.293+0000 7f4cc1ffb640 1 -- 172.21.15.19:0/1832103900 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cc800f890 con 0x7f4ccc1f15e0 2024-08-31T12:16:24.294 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.293+0000 7f4cc1ffb640 1 -- 172.21.15.19:0/1832103900 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5041 (secure 0 0 0) 0x7f4cc801b640 con 0x7f4ccc1f15e0 2024-08-31T12:16:24.294 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.293+0000 7f4cc1ffb640 1 -- 172.21.15.19:0/1832103900 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ccc1f15e0 msgr2=0x7f4ccc0cf690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:24.294 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.293+0000 7f4cc1ffb640 1 --2- 172.21.15.19:0/1832103900 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ccc1f15e0 0x7f4ccc0cf690 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4cc8006650 tx=0x7f4cc8001120 comp rx=0 tx=0).stop 2024-08-31T12:16:24.294 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.293+0000 7f4cc1ffb640 1 -- 172.21.15.19:0/1832103900 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ccc1f15e0 msgr2=0x7f4ccc0cf690 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:16:24.295 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.293+0000 7f4cc1ffb640 1 --2- 172.21.15.19:0/1832103900 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ccc1f15e0 0x7f4ccc0cf690 secure :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4cc8006650 tx=0x7f4cc8001120 comp rx=0 tx=0).stop 2024-08-31T12:16:24.295 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:16:24.295 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:16:24.295 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:16:24.295 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:16:24.295 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 4, 2024-08-31T12:16:24.295 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 52389, 2024-08-31T12:16:24.295 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:16:24.295 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:16:24.295 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:16:24.296 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:16:24.296 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:16:24.296 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 52, 2024-08-31T12:16:24.296 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:16:24.296 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:16:24.296 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:16:24.296 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:16:24.296 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:16:24.296 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:16:24.296 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:16:24.296 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:16:24.296 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:16:24.296 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:16:24.296 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:16:24.297 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:16:24.297 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:16:24.297 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:16:24.297 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:16:24.297 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:16:24.297 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:16:24.297 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:16:24.297 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:16:24.297 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:16:24.297 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:16:24.297 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:16:24.297 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:16:24.297 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:16:24.298 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:16:24.298 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:16:24.298 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:16:24.298 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:16:24.298 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:16:24.298 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:24.298 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:16:24.298 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:16:24.298 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:16:24.298 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:16:24.298 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:16:24.298 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:16:24.298 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:16:24.298 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:16:24.299 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:16:24.299 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:16:24.299 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:16:24.299 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:16:24.299 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:16:24.299 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:16:24.299 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:16:24.299 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:16:24.299 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:16:24.299 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:16:24.299 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:16:24.299 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:16:24.299 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:16:24.300 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:16:24.300 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:16:24.300 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:16:24.300 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:16:24.300 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:16:24.300 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:16:24.300 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:16:24.300 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:16:24.300 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:16:24.300 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:24.300 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:16:24.300 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:24.300 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:16:24.300 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:16:24.300 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:16:24.301 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:16:24.301 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:24.301 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:16:24.301 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:16:24.301 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:16:24.301 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:24.301 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:24.301 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:16:24.301 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:16:24.301 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:16:24.301 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:24.301 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:16:24.301 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:24.301 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:16:24.302 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:16:24.302 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:16:24.302 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:16:24.302 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:16:24.302 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:24.302 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:24.302 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:16:24.302 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:16:24.302 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:16:24.302 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:16:24.302 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:24.302 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:16:24.302 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:16:24.302 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:16:24.303 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:24.303 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:24.303 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:16:24.303 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:16:24.303 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:16:24.303 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:24.303 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:16:24.303 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:24.303 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:16:24.303 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:16:24.303 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:16:24.303 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:16:24.303 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:16:24.303 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:24.304 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:24.304 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:16:24.304 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:16:24.304 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:16:24.304 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:16:24.304 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:24.304 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:16:24.304 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:16:24.304 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:16:24.304 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:24.304 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:24.304 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:16:24.304 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:16:24.305 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:16:24.305 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:24.305 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:16:24.305 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:24.305 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:16:24.305 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:16:24.305 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:16:24.305 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:16:24.305 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:16:24.305 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:24.305 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:16:24.305 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:24.305 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:16:24.306 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:16:24.306 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:24.306 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:16:24.306 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:16:24.306 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:16:24.306 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:24.306 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:16:24.306 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:16:24.306 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:24.306 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:16:24.306 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:16:24.306 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:16:24.306 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:24.306 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:16:24.307 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:16:24.307 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:24.307 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:16:24.307 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:16:24.307 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 3 2024-08-31T12:16:24.307 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:24.307 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:16:24.307 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:16:24.307 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:24.307 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:16:24.307 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:16:24.307 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 4 2024-08-31T12:16:24.307 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:24.308 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:16:24.308 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:16:24.308 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:24.308 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:16:24.308 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:16:24.308 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:16:24.308 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:24.308 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:16:24.308 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:24.308 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:16:24.308 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:16:24.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.293+0000 7f4cd3a15640 1 -- 172.21.15.19:0/1832103900 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4cb0077a80 msgr2=0x7f4cb0079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:24.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.293+0000 7f4cd3a15640 1 --2- 172.21.15.19:0/1832103900 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4cb0077a80 0x7f4cb0079f90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc1eb840 tx=0x7f4cbc001110 comp rx=0 tx=0).stop 2024-08-31T12:16:24.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.293+0000 7f4cd3a15640 1 -- 172.21.15.19:0/1832103900 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4ccc15d690 msgr2=0x7f4ccc0c25d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:24.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.293+0000 7f4cd3a15640 1 --2- 172.21.15.19:0/1832103900 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4ccc15d690 0x7f4ccc0c25d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4cc4009e20 tx=0x7f4cc4034c20 comp rx=0 tx=0).stop 2024-08-31T12:16:24.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.293+0000 7f4cd1fb2640 1 -- 172.21.15.19:0/1832103900 reap_dead start 2024-08-31T12:16:24.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.293+0000 7f4cd1fb2640 1 -- 172.21.15.19:0/1832103900 reap_dead start 2024-08-31T12:16:24.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.293+0000 7f4cd3a15640 1 -- 172.21.15.19:0/1832103900 shutdown_connections 2024-08-31T12:16:24.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.294+0000 7f4cd3a15640 1 -- 172.21.15.19:0/1832103900 >> 172.21.15.19:0/1832103900 conn(0x7f4ccc0690d0 msgr2=0x7f4ccc15ee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:24.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.294+0000 7f4cd3a15640 1 -- 172.21.15.19:0/1832103900 shutdown_connections 2024-08-31T12:16:24.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.294+0000 7f4cd3a15640 1 -- 172.21.15.19:0/1832103900 wait complete. 2024-08-31T12:16:24.311 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:16:24.311 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-08-31T12:16:24.311 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-31T12:16:24.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.424+0000 7f5a103e4640 1 -- 172.21.15.19:0/3629942898 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5a0815bbd0 msgr2=0x7f5a080c5bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:24.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.424+0000 7f5a103e4640 1 --2- 172.21.15.19:0/3629942898 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5a0815bbd0 0x7f5a080c5bc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f5a080cbed0 tx=0x7f5a0002a970 comp rx=0 tx=0).stop 2024-08-31T12:16:24.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.424+0000 7f5a103e4640 1 -- 172.21.15.19:0/3629942898 shutdown_connections 2024-08-31T12:16:24.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.424+0000 7f5a103e4640 1 --2- 172.21.15.19:0/3629942898 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5a0815bbd0 0x7f5a080c5bc0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:24.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.424+0000 7f5a103e4640 1 --2- 172.21.15.19:0/3629942898 >> v2:172.21.15.43:3300/0 conn(0x7f5a080cb2a0 0x7f5a080b59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:24.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.424+0000 7f5a103e4640 1 -- 172.21.15.19:0/3629942898 >> 172.21.15.19:0/3629942898 conn(0x7f5a0806dba0 msgr2=0x7f5a0806e6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:24.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.424+0000 7f5a103e4640 1 -- 172.21.15.19:0/3629942898 shutdown_connections 2024-08-31T12:16:24.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.424+0000 7f5a103e4640 1 -- 172.21.15.19:0/3629942898 wait complete. 2024-08-31T12:16:24.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.426+0000 7f5a103e4640 1 Processor -- start 2024-08-31T12:16:24.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.426+0000 7f5a103e4640 1 -- start start 2024-08-31T12:16:24.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.427+0000 7f5a103e4640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a080cb2a0 0x7f5a08156810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:24.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.427+0000 7f5a103e4640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5a0815bbd0 0x7f5a08157ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:24.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.427+0000 7f5a103e4640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a0815f350 0x7f5a081f4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:24.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.427+0000 7f5a103e4640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f5a080b3db0 con 0x7f5a0815bbd0 2024-08-31T12:16:24.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.427+0000 7f5a103e4640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5a080c6100 con 0x7f5a0815f350 2024-08-31T12:16:24.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.427+0000 7f5a103e4640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f5a080cba70 con 0x7f5a080cb2a0 2024-08-31T12:16:24.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.427+0000 7f5a0d97f640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5a0815bbd0 0x7f5a08157ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:24.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.427+0000 7f5a0d97f640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5a0815bbd0 0x7f5a08157ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:38188/0 (socket says 172.21.15.19:38188) 2024-08-31T12:16:24.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.427+0000 7f5a0d97f640 1 -- 172.21.15.19:0/3610358293 learned_addr learned my addr 172.21.15.19:0/3610358293 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:24.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.427+0000 7f5a0e981640 1 --2- 172.21.15.19:0/3610358293 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a0815f350 0x7f5a081f4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:24.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.427+0000 7f5a0e180640 1 --2- 172.21.15.19:0/3610358293 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a080cb2a0 0x7f5a08156810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:24.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.428+0000 7f5a0d97f640 1 -- 172.21.15.19:0/3610358293 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a080cb2a0 msgr2=0x7f5a08156810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:24.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.428+0000 7f5a0d97f640 1 --2- 172.21.15.19:0/3610358293 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a080cb2a0 0x7f5a08156810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:24.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.428+0000 7f5a0d97f640 1 -- 172.21.15.19:0/3610358293 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a0815f350 msgr2=0x7f5a081f4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:24.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.428+0000 7f5a0d97f640 1 --2- 172.21.15.19:0/3610358293 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a0815f350 0x7f5a081f4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:24.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.428+0000 7f5a0d97f640 1 -- 172.21.15.19:0/3610358293 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a08156d50 con 0x7f5a0815bbd0 2024-08-31T12:16:24.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.428+0000 7f5a0d97f640 1 --2- 172.21.15.19:0/3610358293 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5a0815bbd0 0x7f5a08157ff0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f59f800b410 tx=0x7f59f800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:24.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.428+0000 7f59feffd640 1 -- 172.21.15.19:0/3610358293 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f8015020 con 0x7f5a0815bbd0 2024-08-31T12:16:24.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.428+0000 7f59feffd640 1 -- 172.21.15.19:0/3610358293 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f59f8011920 con 0x7f5a0815bbd0 2024-08-31T12:16:24.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.428+0000 7f5a103e4640 1 -- 172.21.15.19:0/3610358293 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a08159790 con 0x7f5a0815bbd0 2024-08-31T12:16:24.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.428+0000 7f59feffd640 1 -- 172.21.15.19:0/3610358293 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f8011d70 con 0x7f5a0815bbd0 2024-08-31T12:16:24.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.428+0000 7f5a103e4640 1 -- 172.21.15.19:0/3610358293 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a080c9aa0 con 0x7f5a0815bbd0 2024-08-31T12:16:24.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.429+0000 7f5a103e4640 1 --2- 172.21.15.19:0/3610358293 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a0814f180 0x7f5a0814f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:24.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.429+0000 7f5a103e4640 1 -- 172.21.15.19:0/3610358293 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f5a08157220 con 0x7f5a0814f180 2024-08-31T12:16:24.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.429+0000 7f5a0e180640 1 --2- 172.21.15.19:0/3610358293 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a0814f180 0x7f5a0814f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:24.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.429+0000 7f5a0e180640 1 -- 172.21.15.19:0/3610358293 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5a08156460 con 0x7f5a0814f180 2024-08-31T12:16:24.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.430+0000 7f5a0e180640 1 --2- 172.21.15.19:0/3610358293 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a0814f180 0x7f5a0814f590 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5a04003ed0 tx=0x7f5a0400b9b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:24.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.430+0000 7f59feffd640 1 -- 172.21.15.19:0/3610358293 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a04005220 con 0x7f5a0814f180 2024-08-31T12:16:24.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.430+0000 7f59feffd640 1 -- 172.21.15.19:0/3610358293 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f59f8018020 con 0x7f5a0815bbd0 2024-08-31T12:16:24.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.430+0000 7f59feffd640 1 --2- 172.21.15.19:0/3610358293 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f59ec077950 0x7f59ec079e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:24.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.430+0000 7f59feffd640 1 -- 172.21.15.19:0/3610358293 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f59f80995f0 con 0x7f5a0815bbd0 2024-08-31T12:16:24.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.431+0000 7f5a0e981640 1 --2- 172.21.15.19:0/3610358293 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f59ec077950 0x7f59ec079e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:24.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.431+0000 7f59feffd640 1 -- 172.21.15.19:0/3610358293 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5a040053c0 con 0x7f5a0814f180 2024-08-31T12:16:24.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.431+0000 7f59feffd640 1 -- 172.21.15.19:0/3610358293 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a0814f180 msgr2=0x7f5a0814f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:24.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.431+0000 7f59feffd640 1 --2- 172.21.15.19:0/3610358293 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a0814f180 0x7f5a0814f590 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5a04003ed0 tx=0x7f5a0400b9b0 comp rx=0 tx=0).stop 2024-08-31T12:16:24.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.431+0000 7f59feffd640 1 -- 172.21.15.19:0/3610358293 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a0814f180 msgr2=0x7f5a0814f590 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:16:24.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.431+0000 7f59feffd640 1 --2- 172.21.15.19:0/3610358293 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a0814f180 0x7f5a0814f590 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:24.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.431+0000 7f5a0e981640 1 --2- 172.21.15.19:0/3610358293 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f59ec077950 0x7f59ec079e60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5a00035000 tx=0x7f5a0003c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:24.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.437+0000 7f5a103e4640 1 --2- 172.21.15.19:0/3610358293 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a081f13e0 0x7f5a081f17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:24.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.437+0000 7f5a103e4640 1 -- 172.21.15.19:0/3610358293 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f5a080cad50 con 0x7f5a081f13e0 2024-08-31T12:16:24.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.437+0000 7f5a0e180640 1 --2- 172.21.15.19:0/3610358293 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a081f13e0 0x7f5a081f17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:24.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.437+0000 7f5a0e180640 1 -- 172.21.15.19:0/3610358293 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5a080cbf30 con 0x7f5a081f13e0 2024-08-31T12:16:24.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.438+0000 7f5a0e180640 1 --2- 172.21.15.19:0/3610358293 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a081f13e0 0x7f5a081f17f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5a04019000 tx=0x7f5a040046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:24.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.438+0000 7f59feffd640 1 -- 172.21.15.19:0/3610358293 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a040108b0 con 0x7f5a081f13e0 2024-08-31T12:16:24.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.438+0000 7f59feffd640 1 -- 172.21.15.19:0/3610358293 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4874 (secure 0 0 0) 0x7f5a040108b0 con 0x7f5a081f13e0 2024-08-31T12:16:24.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.438+0000 7f59feffd640 1 -- 172.21.15.19:0/3610358293 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a081f13e0 msgr2=0x7f5a081f17f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:24.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.438+0000 7f59feffd640 1 --2- 172.21.15.19:0/3610358293 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a081f13e0 0x7f5a081f17f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5a04019000 tx=0x7f5a040046c0 comp rx=0 tx=0).stop 2024-08-31T12:16:24.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.438+0000 7f59feffd640 1 -- 172.21.15.19:0/3610358293 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a081f13e0 msgr2=0x7f5a081f17f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:16:24.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.438+0000 7f59feffd640 1 --2- 172.21.15.19:0/3610358293 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a081f13e0 0x7f5a081f17f0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:24.440 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:16:24.440 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:16:24.440 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:16:24.440 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:16:24.440 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 4, 2024-08-31T12:16:24.440 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 52533, 2024-08-31T12:16:24.440 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:16:24.440 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:16:24.440 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:16:24.441 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:16:24.441 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:16:24.441 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 52, 2024-08-31T12:16:24.441 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:16:24.441 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:16:24.441 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:16:24.441 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:16:24.441 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:16:24.441 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:16:24.441 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:16:24.441 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:16:24.441 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:16:24.442 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:16:24.442 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:16:24.442 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:16:24.442 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:16:24.442 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:16:24.442 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:16:24.442 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:16:24.442 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:16:24.442 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:16:24.442 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:16:24.442 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:16:24.442 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:16:24.442 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:16:24.443 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:16:24.443 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:16:24.443 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:16:24.443 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:16:24.443 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:16:24.443 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:16:24.443 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:16:24.443 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:24.443 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:16:24.443 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:16:24.443 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:16:24.443 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:16:24.443 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:16:24.444 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:16:24.444 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:16:24.444 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:16:24.444 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:16:24.444 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:16:24.444 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:16:24.444 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:16:24.444 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:16:24.444 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:16:24.444 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:16:24.444 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:16:24.444 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:16:24.444 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:16:24.445 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:16:24.445 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:16:24.445 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:16:24.445 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:16:24.445 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:16:24.445 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:16:24.445 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:16:24.445 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:16:24.445 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:16:24.445 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:16:24.445 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:16:24.445 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:16:24.445 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:24.446 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:16:24.446 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:24.446 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:16:24.446 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:16:24.446 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:16:24.446 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:16:24.446 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:24.446 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:16:24.446 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:16:24.446 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:16:24.446 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:24.446 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:24.446 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:16:24.447 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:16:24.447 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:16:24.447 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:24.447 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:16:24.447 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:24.447 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:16:24.447 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:16:24.447 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:16:24.447 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:16:24.448 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:16:24.448 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:24.448 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:24.448 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:16:24.448 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:16:24.448 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:16:24.448 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:16:24.448 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:24.448 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:16:24.448 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:16:24.448 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:16:24.448 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:24.448 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:24.448 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:16:24.449 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:16:24.449 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:16:24.449 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:24.449 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:16:24.449 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:24.449 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:16:24.449 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:16:24.449 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:16:24.449 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:16:24.449 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:16:24.449 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:24.449 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:24.449 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:16:24.450 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:16:24.450 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:16:24.450 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:16:24.450 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:24.450 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:16:24.450 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:16:24.450 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:16:24.450 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:24.450 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:24.450 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:16:24.450 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:16:24.450 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:16:24.450 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:24.450 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:16:24.450 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:24.451 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:16:24.451 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:16:24.451 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:16:24.451 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:16:24.451 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:16:24.451 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:24.451 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:16:24.451 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:24.451 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:16:24.451 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:16:24.451 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:24.451 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:16:24.451 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:16:24.451 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:16:24.451 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:24.452 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:16:24.452 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:16:24.452 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:24.452 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:16:24.452 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:16:24.452 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:16:24.452 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:24.452 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:16:24.452 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:16:24.452 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:24.452 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:16:24.452 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:16:24.452 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:16:24.452 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:24.453 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:16:24.453 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:16:24.453 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:24.453 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:16:24.453 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:16:24.453 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:16:24.453 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:24.453 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:16:24.453 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:24.453 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:16:24.453 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:16:24.454 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.438+0000 7f5a103e4640 1 -- 172.21.15.19:0/3610358293 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f59ec077950 msgr2=0x7f59ec079e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:24.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.438+0000 7f5a103e4640 1 --2- 172.21.15.19:0/3610358293 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f59ec077950 0x7f59ec079e60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5a00035000 tx=0x7f5a0003c040 comp rx=0 tx=0).stop 2024-08-31T12:16:24.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.438+0000 7f5a103e4640 1 -- 172.21.15.19:0/3610358293 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5a0815bbd0 msgr2=0x7f5a08157ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:24.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.438+0000 7f5a103e4640 1 --2- 172.21.15.19:0/3610358293 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5a0815bbd0 0x7f5a08157ff0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f59f800b410 tx=0x7f59f800b8d0 comp rx=0 tx=0).stop 2024-08-31T12:16:24.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.438+0000 7f5a0e981640 1 -- 172.21.15.19:0/3610358293 reap_dead start 2024-08-31T12:16:24.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.438+0000 7f5a0e981640 1 -- 172.21.15.19:0/3610358293 reap_dead start 2024-08-31T12:16:24.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.438+0000 7f5a103e4640 1 -- 172.21.15.19:0/3610358293 shutdown_connections 2024-08-31T12:16:24.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.438+0000 7f5a103e4640 1 -- 172.21.15.19:0/3610358293 >> 172.21.15.19:0/3610358293 conn(0x7f5a0806dba0 msgr2=0x7f5a08151540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:24.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.439+0000 7f5a103e4640 1 -- 172.21.15.19:0/3610358293 shutdown_connections 2024-08-31T12:16:24.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.439+0000 7f5a103e4640 1 -- 172.21.15.19:0/3610358293 wait complete. 2024-08-31T12:16:24.455 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:16:24.456 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-08-31T12:16:24.456 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-08-31T12:16:24.456 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-08-31T12:16:24.456 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-08-31T12:16:24.456 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-08-31T12:16:24.456 INFO:teuthology.orchestra.run:waiting for 300 2024-08-31T12:16:24.470 INFO:tasks.ceph.mon.a:Stopped 2024-08-31T12:16:24.470 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-08-31T12:16:24.470 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:16:24.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.623+0000 7f71407c6640 1 -- 172.21.15.19:0/750898148 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f71380cc070 msgr2=0x7f71380cc480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:24.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.623+0000 7f71407c6640 1 --2- 172.21.15.19:0/750898148 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f71380cc070 0x7f71380cc480 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f712c005fd0 tx=0x7f712c02a5f0 comp rx=0 tx=0).stop 2024-08-31T12:16:24.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.623+0000 7f71407c6640 1 -- 172.21.15.19:0/750898148 shutdown_connections 2024-08-31T12:16:24.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.623+0000 7f71407c6640 1 --2- 172.21.15.19:0/750898148 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f71380cc070 0x7f71380cc480 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:24.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.623+0000 7f71407c6640 1 --2- 172.21.15.19:0/750898148 >> v2:172.21.15.19:3300/0 conn(0x7f71380ccbb0 0x7f71380cb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:24.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.623+0000 7f71407c6640 1 -- 172.21.15.19:0/750898148 >> 172.21.15.19:0/750898148 conn(0x7f7138074810 msgr2=0x7f7138068aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:24.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.623+0000 7f71407c6640 1 -- 172.21.15.19:0/750898148 shutdown_connections 2024-08-31T12:16:24.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.623+0000 7f71407c6640 1 -- 172.21.15.19:0/750898148 wait complete. 2024-08-31T12:16:24.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.625+0000 7f71407c6640 1 Processor -- start 2024-08-31T12:16:24.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.625+0000 7f71407c6640 1 -- start start 2024-08-31T12:16:24.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.626+0000 7f71407c6640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f71380cc070 0x7f7138155dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:24.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.626+0000 7f71407c6640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f71380ccbb0 0x7f71380d0500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:24.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.626+0000 7f71407c6640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f713815b120 0x7f7138157ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:24.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.626+0000 7f71407c6640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f71381557e0 con 0x7f71380cc070 2024-08-31T12:16:24.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.626+0000 7f71407c6640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7138155660 con 0x7f713815b120 2024-08-31T12:16:24.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.626+0000 7f71407c6640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f713806e1a0 con 0x7f71380ccbb0 2024-08-31T12:16:24.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.626+0000 7f713e562640 1 -- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f71380cc070 msgr2=0x7f7138155dd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:3300/0 2024-08-31T12:16:24.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.626+0000 7f713e562640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f71380cc070 0x7f7138155dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:16:24.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.626+0000 7f713dd61640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f71380ccbb0 0x7f71380d0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:24.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.626+0000 7f713dd61640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f71380ccbb0 0x7f71380d0500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.19:38166/0 (socket says 172.21.15.19:38166) 2024-08-31T12:16:24.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.626+0000 7f713dd61640 1 -- 172.21.15.19:0/643361248 learned_addr learned my addr 172.21.15.19:0/643361248 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:24.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.626+0000 7f713ed63640 1 --2- 172.21.15.19:0/643361248 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f713815b120 0x7f7138157ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:24.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.627+0000 7f713dd61640 1 -- 172.21.15.19:0/643361248 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f713815b120 msgr2=0x7f7138157ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:24.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.627+0000 7f713dd61640 1 --2- 172.21.15.19:0/643361248 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f713815b120 0x7f7138157ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:24.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.627+0000 7f713dd61640 1 -- 172.21.15.19:0/643361248 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f71380cc070 msgr2=0x7f7138155dd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:16:24.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.627+0000 7f713dd61640 1 --2- 172.21.15.19:0/643361248 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f71380cc070 0x7f7138155dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:24.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.627+0000 7f713dd61640 1 -- 172.21.15.19:0/643361248 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71380c6420 con 0x7f71380ccbb0 2024-08-31T12:16:24.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.627+0000 7f713dd61640 1 --2- 172.21.15.19:0/643361248 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f71380ccbb0 0x7f71380d0500 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f712c000c00 tx=0x7f712c003970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:24.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.627+0000 7f7136ffd640 1 -- 172.21.15.19:0/643361248 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f712c03e030 con 0x7f71380ccbb0 2024-08-31T12:16:24.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.627+0000 7f7136ffd640 1 -- 172.21.15.19:0/643361248 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f712c003e60 con 0x7f71380ccbb0 2024-08-31T12:16:24.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.627+0000 7f7136ffd640 1 -- 172.21.15.19:0/643361248 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f712c035040 con 0x7f71380ccbb0 2024-08-31T12:16:24.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.627+0000 7f71407c6640 1 -- 172.21.15.19:0/643361248 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71380cf6d0 con 0x7f71380ccbb0 2024-08-31T12:16:24.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.627+0000 7f71407c6640 1 -- 172.21.15.19:0/643361248 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f71381f58a0 con 0x7f71380ccbb0 2024-08-31T12:16:24.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.628+0000 7f71407c6640 1 -- 172.21.15.19:0/643361248 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71380cb280 con 0x7f71380ccbb0 2024-08-31T12:16:24.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.629+0000 7f7136ffd640 1 -- 172.21.15.19:0/643361248 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f712c041030 con 0x7f71380ccbb0 2024-08-31T12:16:24.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.630+0000 7f7136ffd640 1 --2- 172.21.15.19:0/643361248 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7124077a30 0x7f7124079f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:24.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.630+0000 7f7136ffd640 1 -- 172.21.15.19:0/643361248 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f712c0b9d80 con 0x7f71380ccbb0 2024-08-31T12:16:24.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.630+0000 7f713e562640 1 --2- 172.21.15.19:0/643361248 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7124077a30 0x7f7124079f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:24.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.630+0000 7f713e562640 1 --2- 172.21.15.19:0/643361248 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7124077a30 0x7f7124079f40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7128001350 tx=0x7f71280092b0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:24.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.634+0000 7f7136ffd640 1 -- 172.21.15.19:0/643361248 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f712c0824f0 con 0x7f71380ccbb0 2024-08-31T12:16:24.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.889+0000 7f71407c6640 1 -- 172.21.15.19:0/643361248 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7138158b40 con 0x7f71380ccbb0 2024-08-31T12:16:24.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.889+0000 7f7136ffd640 1 -- 172.21.15.19:0/643361248 <== mon.2 v2:172.21.15.43:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f712c086cb0 con 0x7f71380ccbb0 2024-08-31T12:16:24.890 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":52,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:16:24.894 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.892+0000 7f71407c6640 1 -- 172.21.15.19:0/643361248 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7124077a30 msgr2=0x7f7124079f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:24.894 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.892+0000 7f71407c6640 1 --2- 172.21.15.19:0/643361248 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7124077a30 0x7f7124079f40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7128001350 tx=0x7f71280092b0 comp rx=0 tx=0).stop 2024-08-31T12:16:24.894 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.892+0000 7f71407c6640 1 -- 172.21.15.19:0/643361248 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f71380ccbb0 msgr2=0x7f71380d0500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:24.894 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.892+0000 7f71407c6640 1 --2- 172.21.15.19:0/643361248 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f71380ccbb0 0x7f71380d0500 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f712c000c00 tx=0x7f712c003970 comp rx=0 tx=0).stop 2024-08-31T12:16:24.894 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.892+0000 7f71407c6640 1 -- 172.21.15.19:0/643361248 shutdown_connections 2024-08-31T12:16:24.894 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.892+0000 7f71407c6640 1 --2- 172.21.15.19:0/643361248 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7124077a30 0x7f7124079f40 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:24.894 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.892+0000 7f71407c6640 1 --2- 172.21.15.19:0/643361248 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f713815b120 0x7f7138157ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:24.894 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.892+0000 7f71407c6640 1 --2- 172.21.15.19:0/643361248 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f71380ccbb0 0x7f71380d0500 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:24.894 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.892+0000 7f71407c6640 1 --2- 172.21.15.19:0/643361248 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f71380cc070 0x7f7138155dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:24.894 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.892+0000 7f71407c6640 1 -- 172.21.15.19:0/643361248 >> 172.21.15.19:0/643361248 conn(0x7f7138074810 msgr2=0x7f71380c6680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:24.895 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.892+0000 7f71407c6640 1 -- 172.21.15.19:0/643361248 shutdown_connections 2024-08-31T12:16:24.895 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:24.892+0000 7f71407c6640 1 -- 172.21.15.19:0/643361248 wait complete. 2024-08-31T12:16:25.639 DEBUG:teuthology.orchestra.run.smithi019:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:16:25.642 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:16:27.904 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:16:28.033 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.031+0000 7f563fd0b640 1 -- 172.21.15.19:0/2870273321 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f563815f340 msgr2=0x7f563815bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:28.033 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.031+0000 7f563fd0b640 1 --2- 172.21.15.19:0/2870273321 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f563815f340 0x7f563815bbc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5634009860 tx=0x7f563402a4f0 comp rx=0 tx=0).stop 2024-08-31T12:16:28.033 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.031+0000 7f563fd0b640 1 -- 172.21.15.19:0/2870273321 shutdown_connections 2024-08-31T12:16:28.033 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.031+0000 7f563fd0b640 1 --2- 172.21.15.19:0/2870273321 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f563815f340 0x7f563815bbc0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:28.033 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.031+0000 7f563fd0b640 1 -- 172.21.15.19:0/2870273321 >> 172.21.15.19:0/2870273321 conn(0x7f563806f9d0 msgr2=0x7f563806e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:28.033 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.031+0000 7f563fd0b640 1 -- 172.21.15.19:0/2870273321 shutdown_connections 2024-08-31T12:16:28.033 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.031+0000 7f563fd0b640 1 -- 172.21.15.19:0/2870273321 wait complete. 2024-08-31T12:16:28.034 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.033+0000 7f563fd0b640 1 Processor -- start 2024-08-31T12:16:28.034 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.033+0000 7f563fd0b640 1 -- start start 2024-08-31T12:16:28.037 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.034+0000 7f563fd0b640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56380b4d00 0x7f56381f1c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:28.037 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.034+0000 7f563fd0b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5638151ba0 0x7f563814d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:28.037 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.034+0000 7f563fd0b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f563815f340 0x7f5638149050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:28.037 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.034+0000 7f563fd0b640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f563815ee20 con 0x7f5638151ba0 2024-08-31T12:16:28.037 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.034+0000 7f563fd0b640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f563815eb40 con 0x7f56380b4d00 2024-08-31T12:16:28.037 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.034+0000 7f563fd0b640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f56380b5110 con 0x7f563815f340 2024-08-31T12:16:28.037 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.034+0000 7f563d2a6640 1 -- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5638151ba0 msgr2=0x7f563814d0c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:3300/0 2024-08-31T12:16:28.038 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.034+0000 7f563d2a6640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5638151ba0 0x7f563814d0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:16:28.038 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.034+0000 7f563daa7640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56380b4d00 0x7f56381f1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:28.038 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.034+0000 7f563e2a8640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f563815f340 0x7f5638149050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:28.038 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.034+0000 7f563daa7640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56380b4d00 0x7f56381f1c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:33834/0 (socket says 172.21.15.19:33834) 2024-08-31T12:16:28.038 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.034+0000 7f563e2a8640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f563815f340 0x7f5638149050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.19:38182/0 (socket says 172.21.15.19:38182) 2024-08-31T12:16:28.038 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.034+0000 7f563daa7640 1 -- 172.21.15.19:0/408783149 learned_addr learned my addr 172.21.15.19:0/408783149 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:28.038 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.035+0000 7f563e2a8640 1 -- 172.21.15.19:0/408783149 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56380b4d00 msgr2=0x7f56381f1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:28.038 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.035+0000 7f563e2a8640 1 --2- 172.21.15.19:0/408783149 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56380b4d00 0x7f56381f1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:28.038 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.035+0000 7f563e2a8640 1 -- 172.21.15.19:0/408783149 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5638151ba0 msgr2=0x7f563814d0c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:16:28.038 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.035+0000 7f563e2a8640 1 --2- 172.21.15.19:0/408783149 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5638151ba0 0x7f563814d0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:28.038 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.035+0000 7f563e2a8640 1 -- 172.21.15.19:0/408783149 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56381f60a0 con 0x7f563815f340 2024-08-31T12:16:28.038 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.035+0000 7f563daa7640 1 --2- 172.21.15.19:0/408783149 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56380b4d00 0x7f56381f1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:16:28.039 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.035+0000 7f563e2a8640 1 --2- 172.21.15.19:0/408783149 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f563815f340 0x7f5638149050 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f563000ca70 tx=0x7f563000cf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:28.039 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.035+0000 7f562e7fc640 1 -- 172.21.15.19:0/408783149 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5630016020 con 0x7f563815f340 2024-08-31T12:16:28.039 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.035+0000 7f562e7fc640 1 -- 172.21.15.19:0/408783149 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f563000ac80 con 0x7f563815f340 2024-08-31T12:16:28.039 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.035+0000 7f563fd0b640 1 -- 172.21.15.19:0/408783149 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56381f5b40 con 0x7f563815f340 2024-08-31T12:16:28.039 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.035+0000 7f562e7fc640 1 -- 172.21.15.19:0/408783149 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5630012680 con 0x7f563815f340 2024-08-31T12:16:28.039 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.035+0000 7f563fd0b640 1 -- 172.21.15.19:0/408783149 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f563814aac0 con 0x7f563815f340 2024-08-31T12:16:28.040 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.036+0000 7f563fd0b640 1 -- 172.21.15.19:0/408783149 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56381f5e10 con 0x7f563815f340 2024-08-31T12:16:28.040 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.037+0000 7f562e7fc640 1 -- 172.21.15.19:0/408783149 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f5630019020 con 0x7f563815f340 2024-08-31T12:16:28.040 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.038+0000 7f562e7fc640 1 --2- 172.21.15.19:0/408783149 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f561c077a00 0x7f561c079f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:28.040 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.038+0000 7f562e7fc640 1 -- 172.21.15.19:0/408783149 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f563009a100 con 0x7f563815f340 2024-08-31T12:16:28.040 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.038+0000 7f563daa7640 1 --2- 172.21.15.19:0/408783149 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f561c077a00 0x7f561c079f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:28.040 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.039+0000 7f563daa7640 1 --2- 172.21.15.19:0/408783149 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f561c077a00 0x7f561c079f10 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f56340095f0 tx=0x7f5634004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:28.043 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.042+0000 7f562e7fc640 1 -- 172.21.15.19:0/408783149 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f5630020020 con 0x7f563815f340 2024-08-31T12:16:28.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.299+0000 7f563fd0b640 1 -- 172.21.15.19:0/408783149 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5638155700 con 0x7f563815f340 2024-08-31T12:16:28.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.299+0000 7f562e7fc640 1 -- 172.21.15.19:0/408783149 <== mon.2 v2:172.21.15.43:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f56381f5b40 con 0x7f563815f340 2024-08-31T12:16:28.301 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":56,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:16:28.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.302+0000 7f563fd0b640 1 -- 172.21.15.19:0/408783149 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f561c077a00 msgr2=0x7f561c079f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:28.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.302+0000 7f563fd0b640 1 --2- 172.21.15.19:0/408783149 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f561c077a00 0x7f561c079f10 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f56340095f0 tx=0x7f5634004730 comp rx=0 tx=0).stop 2024-08-31T12:16:28.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.302+0000 7f563fd0b640 1 -- 172.21.15.19:0/408783149 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f563815f340 msgr2=0x7f5638149050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:28.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.302+0000 7f563fd0b640 1 --2- 172.21.15.19:0/408783149 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f563815f340 0x7f5638149050 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f563000ca70 tx=0x7f563000cf30 comp rx=0 tx=0).stop 2024-08-31T12:16:28.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.302+0000 7f563fd0b640 1 -- 172.21.15.19:0/408783149 shutdown_connections 2024-08-31T12:16:28.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.302+0000 7f563fd0b640 1 --2- 172.21.15.19:0/408783149 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f561c077a00 0x7f561c079f10 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:28.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.302+0000 7f563fd0b640 1 --2- 172.21.15.19:0/408783149 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f563815f340 0x7f5638149050 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:28.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.302+0000 7f563fd0b640 1 --2- 172.21.15.19:0/408783149 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5638151ba0 0x7f563814d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:28.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.302+0000 7f563fd0b640 1 --2- 172.21.15.19:0/408783149 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56380b4d00 0x7f56381f1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:28.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.302+0000 7f563fd0b640 1 -- 172.21.15.19:0/408783149 >> 172.21.15.19:0/408783149 conn(0x7f563806f9d0 msgr2=0x7f56380b5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:28.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.302+0000 7f563fd0b640 1 -- 172.21.15.19:0/408783149 shutdown_connections 2024-08-31T12:16:28.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:28.302+0000 7f563fd0b640 1 -- 172.21.15.19:0/408783149 wait complete. 2024-08-31T12:16:31.316 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:16:31.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.442+0000 7f4d99377640 1 -- 172.21.15.19:0/2834713495 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4d9415c4b0 msgr2=0x7f4d9415c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:31.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.442+0000 7f4d99377640 1 --2- 172.21.15.19:0/2834713495 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4d9415c4b0 0x7f4d9415c8c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c005fd0 tx=0x7f4d8c02aa10 comp rx=0 tx=0).stop 2024-08-31T12:16:31.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.442+0000 7f4d99377640 1 -- 172.21.15.19:0/2834713495 shutdown_connections 2024-08-31T12:16:31.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.442+0000 7f4d99377640 1 --2- 172.21.15.19:0/2834713495 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4d9415c4b0 0x7f4d9415c8c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:31.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.442+0000 7f4d99377640 1 --2- 172.21.15.19:0/2834713495 >> v2:172.21.15.43:3300/0 conn(0x7f4d940ca7f0 0x7f4d941530f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:31.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.442+0000 7f4d99377640 1 -- 172.21.15.19:0/2834713495 >> 172.21.15.19:0/2834713495 conn(0x7f4d940cef20 msgr2=0x7f4d940cb340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:31.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.442+0000 7f4d99377640 1 -- 172.21.15.19:0/2834713495 shutdown_connections 2024-08-31T12:16:31.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.442+0000 7f4d99377640 1 -- 172.21.15.19:0/2834713495 wait complete. 2024-08-31T12:16:31.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.444+0000 7f4d99377640 1 Processor -- start 2024-08-31T12:16:31.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.444+0000 7f4d99377640 1 -- start start 2024-08-31T12:16:31.449 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.445+0000 7f4d99377640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4d940ca7f0 0x7f4d941f1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:31.449 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.445+0000 7f4d99377640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4d940cacb0 0x7f4d940cdc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:31.449 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.445+0000 7f4d99377640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4d9415c4b0 0x7f4d94143340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:31.449 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.445+0000 7f4d99377640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f4d940c58d0 con 0x7f4d940ca7f0 2024-08-31T12:16:31.449 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.445+0000 7f4d99377640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4d940c98b0 con 0x7f4d940cacb0 2024-08-31T12:16:31.449 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.445+0000 7f4d99377640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f4d940c5a50 con 0x7f4d9415c4b0 2024-08-31T12:16:31.449 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.445+0000 7f4d92ffd640 1 -- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4d940ca7f0 msgr2=0x7f4d941f1820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:3300/0 2024-08-31T12:16:31.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.445+0000 7f4d92ffd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4d940ca7f0 0x7f4d941f1820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:16:31.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.446+0000 7f4d937fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4d9415c4b0 0x7f4d94143340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:31.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.446+0000 7f4d927fc640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4d940cacb0 0x7f4d940cdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:31.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.446+0000 7f4d937fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4d9415c4b0 0x7f4d94143340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.19:41720/0 (socket says 172.21.15.19:41720) 2024-08-31T12:16:31.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.446+0000 7f4d937fe640 1 -- 172.21.15.19:0/2547885798 learned_addr learned my addr 172.21.15.19:0/2547885798 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:31.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.446+0000 7f4d937fe640 1 -- 172.21.15.19:0/2547885798 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4d940cacb0 msgr2=0x7f4d940cdc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:31.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.446+0000 7f4d937fe640 1 --2- 172.21.15.19:0/2547885798 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4d940cacb0 0x7f4d940cdc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:31.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.446+0000 7f4d937fe640 1 -- 172.21.15.19:0/2547885798 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4d940ca7f0 msgr2=0x7f4d941f1820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:16:31.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.446+0000 7f4d937fe640 1 --2- 172.21.15.19:0/2547885798 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4d940ca7f0 0x7f4d941f1820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:31.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.446+0000 7f4d937fe640 1 -- 172.21.15.19:0/2547885798 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d940b40b0 con 0x7f4d9415c4b0 2024-08-31T12:16:31.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.446+0000 7f4d927fc640 1 --2- 172.21.15.19:0/2547885798 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4d940cacb0 0x7f4d940cdc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:16:31.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.446+0000 7f4d937fe640 1 --2- 172.21.15.19:0/2547885798 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4d9415c4b0 0x7f4d94143340 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c02afd0 tx=0x7f4d8c003f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:31.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.446+0000 7f4d7b7fe640 1 -- 172.21.15.19:0/2547885798 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d8c033050 con 0x7f4d9415c4b0 2024-08-31T12:16:31.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.446+0000 7f4d7b7fe640 1 -- 172.21.15.19:0/2547885798 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4d8c004250 con 0x7f4d9415c4b0 2024-08-31T12:16:31.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.447+0000 7f4d99377640 1 -- 172.21.15.19:0/2547885798 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d940b4390 con 0x7f4d9415c4b0 2024-08-31T12:16:31.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.447+0000 7f4d7b7fe640 1 -- 172.21.15.19:0/2547885798 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d8c004680 con 0x7f4d9415c4b0 2024-08-31T12:16:31.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.447+0000 7f4d99377640 1 -- 172.21.15.19:0/2547885798 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d940ccb30 con 0x7f4d9415c4b0 2024-08-31T12:16:31.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.447+0000 7f4d99377640 1 -- 172.21.15.19:0/2547885798 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d940cc640 con 0x7f4d9415c4b0 2024-08-31T12:16:31.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.448+0000 7f4d7b7fe640 1 -- 172.21.15.19:0/2547885798 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f4d8c036070 con 0x7f4d9415c4b0 2024-08-31T12:16:31.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.449+0000 7f4d7b7fe640 1 --2- 172.21.15.19:0/2547885798 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4d80077a00 0x7f4d80079f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:31.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.449+0000 7f4d7b7fe640 1 -- 172.21.15.19:0/2547885798 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f4d8c0baaf0 con 0x7f4d9415c4b0 2024-08-31T12:16:31.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.449+0000 7f4d92ffd640 1 --2- 172.21.15.19:0/2547885798 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4d80077a00 0x7f4d80079f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:31.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.450+0000 7f4d92ffd640 1 --2- 172.21.15.19:0/2547885798 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4d80077a00 0x7f4d80079f10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4d84009670 tx=0x7f4d84005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:31.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.453+0000 7f4d7b7fe640 1 -- 172.21.15.19:0/2547885798 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f4d8c083850 con 0x7f4d9415c4b0 2024-08-31T12:16:31.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.715+0000 7f4d99377640 1 -- 172.21.15.19:0/2547885798 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4d9414d200 con 0x7f4d9415c4b0 2024-08-31T12:16:31.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.716+0000 7f4d7b7fe640 1 -- 172.21.15.19:0/2547885798 <== mon.2 v2:172.21.15.43:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f4d8c004820 con 0x7f4d9415c4b0 2024-08-31T12:16:31.718 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":59,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:16:31.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.719+0000 7f4d99377640 1 -- 172.21.15.19:0/2547885798 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4d80077a00 msgr2=0x7f4d80079f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:31.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.719+0000 7f4d99377640 1 --2- 172.21.15.19:0/2547885798 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4d80077a00 0x7f4d80079f10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4d84009670 tx=0x7f4d84005f90 comp rx=0 tx=0).stop 2024-08-31T12:16:31.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.719+0000 7f4d99377640 1 -- 172.21.15.19:0/2547885798 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4d9415c4b0 msgr2=0x7f4d94143340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:31.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.719+0000 7f4d99377640 1 --2- 172.21.15.19:0/2547885798 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4d9415c4b0 0x7f4d94143340 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c02afd0 tx=0x7f4d8c003f10 comp rx=0 tx=0).stop 2024-08-31T12:16:31.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.719+0000 7f4d99377640 1 -- 172.21.15.19:0/2547885798 shutdown_connections 2024-08-31T12:16:31.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.719+0000 7f4d99377640 1 --2- 172.21.15.19:0/2547885798 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4d80077a00 0x7f4d80079f10 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:31.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.719+0000 7f4d99377640 1 --2- 172.21.15.19:0/2547885798 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4d9415c4b0 0x7f4d94143340 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:31.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.719+0000 7f4d99377640 1 --2- 172.21.15.19:0/2547885798 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4d940cacb0 0x7f4d940cdc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:31.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.719+0000 7f4d99377640 1 --2- 172.21.15.19:0/2547885798 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4d940ca7f0 0x7f4d941f1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:31.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.719+0000 7f4d99377640 1 -- 172.21.15.19:0/2547885798 >> 172.21.15.19:0/2547885798 conn(0x7f4d940cef20 msgr2=0x7f4d940718b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:31.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.719+0000 7f4d99377640 1 -- 172.21.15.19:0/2547885798 shutdown_connections 2024-08-31T12:16:31.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:31.719+0000 7f4d99377640 1 -- 172.21.15.19:0/2547885798 wait complete. 2024-08-31T12:16:34.734 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:16:39.270 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.268+0000 7f6ab1e27640 1 -- 172.21.15.19:0/369289653 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a980036e0 msgr2=0x7f6a98005b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:39.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.268+0000 7f6ab1e27640 1 --2- 172.21.15.19:0/369289653 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a980036e0 0x7f6a98005b90 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c0010a0 tx=0x7f6a9c02acb0 comp rx=0 tx=0).stop 2024-08-31T12:16:39.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.268+0000 7f6aabfff640 1 -- 172.21.15.19:0/369289653 reap_dead start 2024-08-31T12:16:39.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.268+0000 7f6ab1e27640 1 -- 172.21.15.19:0/369289653 shutdown_connections 2024-08-31T12:16:39.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.268+0000 7f6ab1e27640 1 --2- 172.21.15.19:0/369289653 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a980036e0 0x7f6a98005b90 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:39.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.268+0000 7f6ab1e27640 1 -- 172.21.15.19:0/369289653 >> 172.21.15.19:0/369289653 conn(0x7f6aac0cc770 msgr2=0x7f6aac0ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:39.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.268+0000 7f6ab1e27640 1 -- 172.21.15.19:0/369289653 shutdown_connections 2024-08-31T12:16:39.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.268+0000 7f6ab1e27640 1 -- 172.21.15.19:0/369289653 wait complete. 2024-08-31T12:16:39.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.270+0000 7f6ab1e27640 1 Processor -- start 2024-08-31T12:16:39.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.271+0000 7f6ab1e27640 1 -- start start 2024-08-31T12:16:39.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.272+0000 7f6ab1e27640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6a980036e0 0x7f6aac153800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:39.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.272+0000 7f6ab1e27640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6aac13c190 0x7f6aac1552e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:39.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.272+0000 7f6ab1e27640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6aac139070 0x7f6aac1e40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:39.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.272+0000 7f6ab1e27640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f6aac071490 con 0x7f6a980036e0 2024-08-31T12:16:39.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.272+0000 7f6ab1e27640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6aac073eb0 con 0x7f6aac139070 2024-08-31T12:16:39.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.272+0000 7f6ab1e27640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f6aac06ffa0 con 0x7f6aac13c190 2024-08-31T12:16:39.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.272+0000 7f6aab7fe640 1 -- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6a980036e0 msgr2=0x7f6aac153800 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:3300/0 2024-08-31T12:16:39.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.273+0000 7f6aab7fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6a980036e0 0x7f6aac153800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:16:39.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.273+0000 7f6aabfff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6aac139070 0x7f6aac1e40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:39.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.273+0000 7f6aabfff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6aac139070 0x7f6aac1e40f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:50950/0 (socket says 172.21.15.19:50950) 2024-08-31T12:16:39.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.273+0000 7f6aabfff640 1 -- 172.21.15.19:0/1945697959 learned_addr learned my addr 172.21.15.19:0/1945697959 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:39.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.273+0000 7f6aaaffd640 1 --2- 172.21.15.19:0/1945697959 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6aac13c190 0x7f6aac1552e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:39.277 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.273+0000 7f6aabfff640 1 -- 172.21.15.19:0/1945697959 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6aac13c190 msgr2=0x7f6aac1552e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:39.277 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.273+0000 7f6aabfff640 1 --2- 172.21.15.19:0/1945697959 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6aac13c190 0x7f6aac1552e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:39.277 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.273+0000 7f6aabfff640 1 -- 172.21.15.19:0/1945697959 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6a980036e0 msgr2=0x7f6aac153800 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:16:39.277 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.273+0000 7f6aabfff640 1 --2- 172.21.15.19:0/1945697959 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6a980036e0 0x7f6aac153800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:39.277 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.273+0000 7f6aabfff640 1 -- 172.21.15.19:0/1945697959 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6aac155820 con 0x7f6aac139070 2024-08-31T12:16:39.277 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.273+0000 7f6aabfff640 1 --2- 172.21.15.19:0/1945697959 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6aac139070 0x7f6aac1e40f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6aa4013710 tx=0x7f6aa4013bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:39.277 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.274+0000 7f6a93fff640 1 -- 172.21.15.19:0/1945697959 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa4011dd0 con 0x7f6aac139070 2024-08-31T12:16:39.277 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.274+0000 7f6a93fff640 1 -- 172.21.15.19:0/1945697959 <== mon.1 v2:172.21.15.43:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6aa4004b20 con 0x7f6aac139070 2024-08-31T12:16:39.277 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.274+0000 7f6ab1e27640 1 -- 172.21.15.19:0/1945697959 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6aac153e20 con 0x7f6aac139070 2024-08-31T12:16:39.277 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.274+0000 7f6a93fff640 1 -- 172.21.15.19:0/1945697959 <== mon.1 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa4005020 con 0x7f6aac139070 2024-08-31T12:16:39.277 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.274+0000 7f6ab1e27640 1 -- 172.21.15.19:0/1945697959 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6aac154800 con 0x7f6aac139070 2024-08-31T12:16:39.278 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.275+0000 7f6ab1e27640 1 -- 172.21.15.19:0/1945697959 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6aac072b80 con 0x7f6aac139070 2024-08-31T12:16:39.278 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.276+0000 7f6a93fff640 1 -- 172.21.15.19:0/1945697959 <== mon.1 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f6aa400c040 con 0x7f6aac139070 2024-08-31T12:16:39.278 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.276+0000 7f6a93fff640 1 --2- 172.21.15.19:0/1945697959 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6a98078910 0x7f6a9807ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:39.279 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.277+0000 7f6aab7fe640 1 --2- 172.21.15.19:0/1945697959 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6a98078910 0x7f6a9807ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:39.279 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.277+0000 7f6a93fff640 1 -- 172.21.15.19:0/1945697959 <== mon.1 v2:172.21.15.43:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f6aa409a730 con 0x7f6aac139070 2024-08-31T12:16:39.279 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.277+0000 7f6aab7fe640 1 --2- 172.21.15.19:0/1945697959 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6a98078910 0x7f6a9807ae20 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c036000 tx=0x7f6a9c02d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:39.283 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.282+0000 7f6a93fff640 1 -- 172.21.15.19:0/1945697959 <== mon.1 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f6aa4063490 con 0x7f6aac139070 2024-08-31T12:16:39.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.541+0000 7f6ab1e27640 1 -- 172.21.15.19:0/1945697959 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6aac0c5ef0 con 0x7f6aac139070 2024-08-31T12:16:39.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.542+0000 7f6a93fff640 1 -- 172.21.15.19:0/1945697959 <== mon.1 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1549 (secure 0 0 0) 0x7f6aa4062d90 con 0x7f6aac139070 2024-08-31T12:16:39.544 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":6,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:16:39.546 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.544+0000 7f6ab1e27640 1 -- 172.21.15.19:0/1945697959 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6a98078910 msgr2=0x7f6a9807ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:39.546 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.544+0000 7f6ab1e27640 1 --2- 172.21.15.19:0/1945697959 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6a98078910 0x7f6a9807ae20 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c036000 tx=0x7f6a9c02d040 comp rx=0 tx=0).stop 2024-08-31T12:16:39.546 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.544+0000 7f6ab1e27640 1 -- 172.21.15.19:0/1945697959 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6aac139070 msgr2=0x7f6aac1e40f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:39.546 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.544+0000 7f6ab1e27640 1 --2- 172.21.15.19:0/1945697959 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6aac139070 0x7f6aac1e40f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6aa4013710 tx=0x7f6aa4013bd0 comp rx=0 tx=0).stop 2024-08-31T12:16:39.546 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.544+0000 7f6ab1e27640 1 -- 172.21.15.19:0/1945697959 shutdown_connections 2024-08-31T12:16:39.546 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.544+0000 7f6ab1e27640 1 --2- 172.21.15.19:0/1945697959 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6a98078910 0x7f6a9807ae20 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:39.546 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.544+0000 7f6ab1e27640 1 --2- 172.21.15.19:0/1945697959 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6aac139070 0x7f6aac1e40f0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:39.546 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.544+0000 7f6ab1e27640 1 --2- 172.21.15.19:0/1945697959 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6aac13c190 0x7f6aac1552e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:39.546 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.544+0000 7f6ab1e27640 1 --2- 172.21.15.19:0/1945697959 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6a980036e0 0x7f6aac153800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:39.547 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.544+0000 7f6ab1e27640 1 -- 172.21.15.19:0/1945697959 >> 172.21.15.19:0/1945697959 conn(0x7f6aac0cc770 msgr2=0x7f6aac141290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:39.547 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.544+0000 7f6ab1e27640 1 -- 172.21.15.19:0/1945697959 shutdown_connections 2024-08-31T12:16:39.547 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.544+0000 7f6ab1e27640 1 -- 172.21.15.19:0/1945697959 wait complete. 2024-08-31T12:16:39.557 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-08-31T12:16:39.557 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-08-31T12:16:39.557 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-31T12:16:39.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.677+0000 7fda343e6640 1 -- 172.21.15.19:0/2912758524 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fda2c0cb580 msgr2=0x7fda2c070bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:39.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.677+0000 7fda343e6640 1 --2- 172.21.15.19:0/2912758524 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fda2c0cb580 0x7fda2c070bf0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fda24009be0 tx=0x7fda2402a990 comp rx=0 tx=0).stop 2024-08-31T12:16:39.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.677+0000 7fda343e6640 1 -- 172.21.15.19:0/2912758524 shutdown_connections 2024-08-31T12:16:39.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.677+0000 7fda343e6640 1 --2- 172.21.15.19:0/2912758524 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fda2c0cb580 0x7fda2c070bf0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:39.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.677+0000 7fda343e6640 1 --2- 172.21.15.19:0/2912758524 >> v2:172.21.15.19:3300/0 conn(0x7fda2c159820 0x7fda2c0cb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:39.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.677+0000 7fda343e6640 1 -- 172.21.15.19:0/2912758524 >> 172.21.15.19:0/2912758524 conn(0x7fda2c0728b0 msgr2=0x7fda2c072cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:39.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.677+0000 7fda343e6640 1 -- 172.21.15.19:0/2912758524 shutdown_connections 2024-08-31T12:16:39.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.677+0000 7fda343e6640 1 -- 172.21.15.19:0/2912758524 wait complete. 2024-08-31T12:16:39.680 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.678+0000 7fda343e6640 1 Processor -- start 2024-08-31T12:16:39.680 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.679+0000 7fda343e6640 1 -- start start 2024-08-31T12:16:39.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.680+0000 7fda343e6640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fda2c0cb580 0x7fda2c155d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:39.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.680+0000 7fda343e6640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda2c153190 0x7fda2c0ce990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:39.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.680+0000 7fda343e6640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fda2c159820 0x7fda2c158670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:39.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.680+0000 7fda343e6640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fda2c15cdc0 con 0x7fda2c0cb580 2024-08-31T12:16:39.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.680+0000 7fda343e6640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fda2c1535a0 con 0x7fda2c153190 2024-08-31T12:16:39.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.680+0000 7fda343e6640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fda2c159c30 con 0x7fda2c159820 2024-08-31T12:16:39.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.680+0000 7fda32182640 1 -- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fda2c0cb580 msgr2=0x7fda2c155d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:3300/0 2024-08-31T12:16:39.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.680+0000 7fda32182640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fda2c0cb580 0x7fda2c155d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:16:39.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.680+0000 7fda31981640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda2c153190 0x7fda2c0ce990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:39.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.680+0000 7fda31981640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda2c153190 0x7fda2c0ce990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:50962/0 (socket says 172.21.15.19:50962) 2024-08-31T12:16:39.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.680+0000 7fda31981640 1 -- 172.21.15.19:0/2038302687 learned_addr learned my addr 172.21.15.19:0/2038302687 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:39.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.680+0000 7fda32983640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fda2c159820 0x7fda2c158670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:39.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.680+0000 7fda31981640 1 -- 172.21.15.19:0/2038302687 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fda2c159820 msgr2=0x7fda2c158670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:39.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.680+0000 7fda31981640 1 --2- 172.21.15.19:0/2038302687 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fda2c159820 0x7fda2c158670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:39.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.680+0000 7fda31981640 1 -- 172.21.15.19:0/2038302687 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fda2c0cb580 msgr2=0x7fda2c155d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:16:39.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.680+0000 7fda31981640 1 --2- 172.21.15.19:0/2038302687 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fda2c0cb580 0x7fda2c155d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:39.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.680+0000 7fda31981640 1 -- 172.21.15.19:0/2038302687 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda2c0c7850 con 0x7fda2c153190 2024-08-31T12:16:39.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.681+0000 7fda31981640 1 --2- 172.21.15.19:0/2038302687 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda2c153190 0x7fda2c0ce990 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fda1c00b4d0 tx=0x7fda1c00b990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:39.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.681+0000 7fda22ffd640 1 -- 172.21.15.19:0/2038302687 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda1c014070 con 0x7fda2c153190 2024-08-31T12:16:39.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.681+0000 7fda22ffd640 1 -- 172.21.15.19:0/2038302687 <== mon.1 v2:172.21.15.43:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fda1c010980 con 0x7fda2c153190 2024-08-31T12:16:39.685 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.681+0000 7fda343e6640 1 -- 172.21.15.19:0/2038302687 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda2c1f00c0 con 0x7fda2c153190 2024-08-31T12:16:39.685 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.681+0000 7fda22ffd640 1 -- 172.21.15.19:0/2038302687 <== mon.1 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda1c010dd0 con 0x7fda2c153190 2024-08-31T12:16:39.685 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.681+0000 7fda343e6640 1 -- 172.21.15.19:0/2038302687 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda2c0c82a0 con 0x7fda2c153190 2024-08-31T12:16:39.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.682+0000 7fda343e6640 1 --2- 172.21.15.19:0/2038302687 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda2c0c7230 0x7fda2c158bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:39.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.682+0000 7fda343e6640 1 -- 172.21.15.19:0/2038302687 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fda2c0c6020 con 0x7fda2c0c7230 2024-08-31T12:16:39.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.682+0000 7fda32182640 1 --2- 172.21.15.19:0/2038302687 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda2c0c7230 0x7fda2c158bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:39.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.683+0000 7fda32182640 1 -- 172.21.15.19:0/2038302687 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fda2c0b3e30 con 0x7fda2c0c7230 2024-08-31T12:16:39.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.683+0000 7fda32182640 1 --2- 172.21.15.19:0/2038302687 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda2c0c7230 0x7fda2c158bb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fda28003610 tx=0x7fda2800c810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:39.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.683+0000 7fda22ffd640 1 -- 172.21.15.19:0/2038302687 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda28017070 con 0x7fda2c0c7230 2024-08-31T12:16:39.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.683+0000 7fda22ffd640 1 -- 172.21.15.19:0/2038302687 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fda28005d50 con 0x7fda2c0c7230 2024-08-31T12:16:39.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.683+0000 7fda22ffd640 1 -- 172.21.15.19:0/2038302687 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda2c0c7230 msgr2=0x7fda2c158bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:39.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.683+0000 7fda22ffd640 1 --2- 172.21.15.19:0/2038302687 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda2c0c7230 0x7fda2c158bb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fda28003610 tx=0x7fda2800c810 comp rx=0 tx=0).stop 2024-08-31T12:16:39.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.683+0000 7fda22ffd640 1 -- 172.21.15.19:0/2038302687 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda2c0c7230 msgr2=0x7fda2c158bb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:16:39.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.683+0000 7fda22ffd640 1 --2- 172.21.15.19:0/2038302687 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda2c0c7230 0x7fda2c158bb0 secure :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fda28003610 tx=0x7fda2800c810 comp rx=0 tx=0).stop 2024-08-31T12:16:39.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.683+0000 7fda22ffd640 1 -- 172.21.15.19:0/2038302687 <== mon.1 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fda1c017070 con 0x7fda2c153190 2024-08-31T12:16:39.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.684+0000 7fda22ffd640 1 --2- 172.21.15.19:0/2038302687 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fda10077560 0x7fda10079a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:39.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.684+0000 7fda22ffd640 1 -- 172.21.15.19:0/2038302687 <== mon.1 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7fda1c0992e0 con 0x7fda2c153190 2024-08-31T12:16:39.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.684+0000 7fda32182640 1 --2- 172.21.15.19:0/2038302687 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fda10077560 0x7fda10079a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:39.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.685+0000 7fda32182640 1 --2- 172.21.15.19:0/2038302687 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fda10077560 0x7fda10079a70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fda28003610 tx=0x7fda2800a000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:39.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.690+0000 7fda343e6640 1 --2- 172.21.15.19:0/2038302687 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda2c0ceed0 0x7fda2c1f2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:39.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.690+0000 7fda343e6640 1 -- 172.21.15.19:0/2038302687 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fda2c0c7af0 con 0x7fda2c0ceed0 2024-08-31T12:16:39.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.690+0000 7fda32983640 1 --2- 172.21.15.19:0/2038302687 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda2c0ceed0 0x7fda2c1f2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:39.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.690+0000 7fda32983640 1 -- 172.21.15.19:0/2038302687 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fda2c072e10 con 0x7fda2c0ceed0 2024-08-31T12:16:39.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.691+0000 7fda32983640 1 --2- 172.21.15.19:0/2038302687 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda2c0ceed0 0x7fda2c1f2aa0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fda2c074850 tx=0x7fda24030960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:39.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.691+0000 7fda22ffd640 1 -- 172.21.15.19:0/2038302687 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda24004240 con 0x7fda2c0ceed0 2024-08-31T12:16:39.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.691+0000 7fda22ffd640 1 -- 172.21.15.19:0/2038302687 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5031 (secure 0 0 0) 0x7fda24004240 con 0x7fda2c0ceed0 2024-08-31T12:16:39.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.691+0000 7fda22ffd640 1 -- 172.21.15.19:0/2038302687 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda2c0ceed0 msgr2=0x7fda2c1f2aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:39.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.691+0000 7fda22ffd640 1 --2- 172.21.15.19:0/2038302687 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda2c0ceed0 0x7fda2c1f2aa0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fda2c074850 tx=0x7fda24030960 comp rx=0 tx=0).stop 2024-08-31T12:16:39.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.691+0000 7fda22ffd640 1 -- 172.21.15.19:0/2038302687 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda2c0ceed0 msgr2=0x7fda2c1f2aa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:16:39.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.691+0000 7fda22ffd640 1 --2- 172.21.15.19:0/2038302687 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda2c0ceed0 0x7fda2c1f2aa0 secure :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fda2c074850 tx=0x7fda24030960 comp rx=0 tx=0).stop 2024-08-31T12:16:39.694 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:16:39.694 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:16:39.694 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:16:39.694 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "leader", 2024-08-31T12:16:39.694 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 6, 2024-08-31T12:16:39.694 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 67788, 2024-08-31T12:16:39.694 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:16:39.694 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:16:39.694 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:16:39.694 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:16:39.694 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 1, 2024-08-31T12:16:39.694 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:16:39.695 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:16:39.695 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:16:39.695 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:16:39.695 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:16:39.695 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:16:39.695 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:16:39.695 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:16:39.695 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:16:39.695 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:16:39.695 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:16:39.695 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:16:39.695 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:16:39.695 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:16:39.695 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:16:39.695 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:16:39.696 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:16:39.696 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:16:39.696 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:16:39.696 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:16:39.696 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:16:39.696 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:16:39.696 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:16:39.696 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:16:39.696 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:16:39.696 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:16:39.696 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:16:39.696 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:16:39.696 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:16:39.696 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:39.697 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:16:39.697 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:16:39.697 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:16:39.697 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:16:39.697 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:16:39.697 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:16:39.697 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:16:39.697 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:16:39.697 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:16:39.697 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:16:39.697 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:16:39.697 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:16:39.697 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:16:39.698 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:16:39.698 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:16:39.698 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:16:39.698 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:16:39.698 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:16:39.698 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:16:39.698 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:16:39.698 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:16:39.698 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:16:39.698 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:16:39.698 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:16:39.698 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:16:39.698 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:16:39.698 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:16:39.698 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:16:39.699 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:16:39.699 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:16:39.699 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:39.699 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:16:39.699 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:39.699 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:16:39.699 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:16:39.699 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:16:39.699 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:16:39.699 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:39.699 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:16:39.699 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:16:39.699 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:16:39.700 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:39.700 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:39.700 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:16:39.700 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:16:39.700 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:16:39.700 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:39.700 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:16:39.700 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:39.700 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:16:39.700 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:16:39.700 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:16:39.700 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:16:39.700 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:16:39.700 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:39.701 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:39.701 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:16:39.701 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:16:39.701 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:16:39.701 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:16:39.701 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:39.701 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:16:39.701 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:16:39.701 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:16:39.701 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:39.701 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:39.701 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:16:39.701 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:16:39.702 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:16:39.702 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:39.702 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:16:39.702 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:39.702 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:16:39.702 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:16:39.702 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:16:39.702 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:16:39.702 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:16:39.703 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:39.703 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:39.703 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:16:39.703 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:16:39.703 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:16:39.703 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:16:39.703 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:39.703 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:16:39.704 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:16:39.704 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:16:39.704 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:39.704 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:39.704 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:16:39.704 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:16:39.704 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:16:39.704 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:39.704 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:16:39.704 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:39.704 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:16:39.704 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:16:39.704 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:16:39.705 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:16:39.705 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:16:39.705 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:39.705 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:16:39.705 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:39.705 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:16:39.705 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:16:39.705 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:39.705 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:16:39.705 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:16:39.705 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:16:39.705 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:39.705 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:16:39.706 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:16:39.706 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:39.706 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:16:39.706 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:16:39.706 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 3 2024-08-31T12:16:39.706 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:39.706 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:16:39.706 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:16:39.706 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:39.706 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:16:39.706 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:16:39.706 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 3 2024-08-31T12:16:39.706 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:39.707 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:16:39.707 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:16:39.707 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:39.707 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:16:39.707 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:16:39.707 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 5 2024-08-31T12:16:39.707 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:39.707 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:16:39.707 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:16:39.707 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:39.707 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:16:39.707 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:16:39.708 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:16:39.708 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:39.708 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:16:39.708 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:39.708 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:16:39.708 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:16:39.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.691+0000 7fda343e6640 1 -- 172.21.15.19:0/2038302687 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fda10077560 msgr2=0x7fda10079a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:39.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.691+0000 7fda343e6640 1 --2- 172.21.15.19:0/2038302687 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fda10077560 0x7fda10079a70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fda28003610 tx=0x7fda2800a000 comp rx=0 tx=0).stop 2024-08-31T12:16:39.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.691+0000 7fda343e6640 1 -- 172.21.15.19:0/2038302687 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda2c153190 msgr2=0x7fda2c0ce990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:39.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.691+0000 7fda343e6640 1 --2- 172.21.15.19:0/2038302687 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda2c153190 0x7fda2c0ce990 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fda1c00b4d0 tx=0x7fda1c00b990 comp rx=0 tx=0).stop 2024-08-31T12:16:39.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.691+0000 7fda32983640 1 -- 172.21.15.19:0/2038302687 reap_dead start 2024-08-31T12:16:39.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.691+0000 7fda32983640 1 -- 172.21.15.19:0/2038302687 reap_dead start 2024-08-31T12:16:39.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.691+0000 7fda343e6640 1 -- 172.21.15.19:0/2038302687 shutdown_connections 2024-08-31T12:16:39.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.691+0000 7fda343e6640 1 -- 172.21.15.19:0/2038302687 >> 172.21.15.19:0/2038302687 conn(0x7fda2c0728b0 msgr2=0x7fda2c15a450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:39.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.691+0000 7fda343e6640 1 -- 172.21.15.19:0/2038302687 shutdown_connections 2024-08-31T12:16:39.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.691+0000 7fda343e6640 1 -- 172.21.15.19:0/2038302687 wait complete. 2024-08-31T12:16:39.710 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-08-31T12:16:39.710 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-08-31T12:16:39.710 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-31T12:16:39.831 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.829+0000 7f8dc79b6640 1 -- 172.21.15.19:0/1708198837 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8dc00719b0 msgr2=0x7f8dc0071dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:39.831 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.829+0000 7f8dc79b6640 1 --2- 172.21.15.19:0/1708198837 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8dc00719b0 0x7f8dc0071dc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8dc00750c0 tx=0x7f8db802aa30 comp rx=0 tx=0).stop 2024-08-31T12:16:39.831 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.829+0000 7f8dc79b6640 1 -- 172.21.15.19:0/1708198837 shutdown_connections 2024-08-31T12:16:39.832 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.829+0000 7f8dc79b6640 1 --2- 172.21.15.19:0/1708198837 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8dc00719b0 0x7f8dc0071dc0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:39.832 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.829+0000 7f8dc79b6640 1 --2- 172.21.15.19:0/1708198837 >> v2:172.21.15.19:3300/0 conn(0x7f8dc015bd10 0x7f8dc00c7140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:39.832 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.829+0000 7f8dc79b6640 1 --2- 172.21.15.19:0/1708198837 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8dc0140c10 0x7f8dc00c7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:39.832 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.829+0000 7f8dc79b6640 1 -- 172.21.15.19:0/1708198837 >> 172.21.15.19:0/1708198837 conn(0x7f8dc00cd7d0 msgr2=0x7f8dc00cdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:39.832 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.829+0000 7f8dc79b6640 1 -- 172.21.15.19:0/1708198837 shutdown_connections 2024-08-31T12:16:39.832 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.829+0000 7f8dc79b6640 1 -- 172.21.15.19:0/1708198837 wait complete. 2024-08-31T12:16:39.833 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.831+0000 7f8dc79b6640 1 Processor -- start 2024-08-31T12:16:39.833 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.831+0000 7f8dc79b6640 1 -- start start 2024-08-31T12:16:39.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.832+0000 7f8dc79b6640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8dc00719b0 0x7f8dc01f1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:39.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.832+0000 7f8dc79b6640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8dc0140c10 0x7f8dc0153a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:39.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.832+0000 7f8dc79b6640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8dc015bd10 0x7f8dc0154c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:39.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.832+0000 7f8dc79b6640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f8dc015b2b0 con 0x7f8dc015bd10 2024-08-31T12:16:39.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.832+0000 7f8dc79b6640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8dc00c6920 con 0x7f8dc00719b0 2024-08-31T12:16:39.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.832+0000 7f8dc79b6640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f8dc01f2760 con 0x7f8dc0140c10 2024-08-31T12:16:39.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.832+0000 7f8dc5f53640 1 -- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8dc015bd10 msgr2=0x7f8dc0154c00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:3300/0 2024-08-31T12:16:39.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.832+0000 7f8dc5f53640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8dc015bd10 0x7f8dc0154c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:16:39.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.832+0000 7f8dc5752640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8dc00719b0 0x7f8dc01f1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:39.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.832+0000 7f8dc5752640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8dc00719b0 0x7f8dc01f1270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:50998/0 (socket says 172.21.15.19:50998) 2024-08-31T12:16:39.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.832+0000 7f8dc4f51640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8dc0140c10 0x7f8dc0153a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:39.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.832+0000 7f8dc5752640 1 -- 172.21.15.19:0/56031158 learned_addr learned my addr 172.21.15.19:0/56031158 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:16:39.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.833+0000 7f8dc5752640 1 -- 172.21.15.19:0/56031158 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8dc0140c10 msgr2=0x7f8dc0153a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:39.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.833+0000 7f8dc5752640 1 --2- 172.21.15.19:0/56031158 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8dc0140c10 0x7f8dc0153a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:39.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.833+0000 7f8dc5752640 1 -- 172.21.15.19:0/56031158 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8dc015bd10 msgr2=0x7f8dc0154c00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:16:39.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.833+0000 7f8dc5752640 1 --2- 172.21.15.19:0/56031158 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8dc015bd10 0x7f8dc0154c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:39.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.833+0000 7f8dc5752640 1 -- 172.21.15.19:0/56031158 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dc0153fd0 con 0x7f8dc00719b0 2024-08-31T12:16:39.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.833+0000 7f8dc5752640 1 --2- 172.21.15.19:0/56031158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8dc00719b0 0x7f8dc01f1270 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc00e7c0 tx=0x7f8dbc00ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:39.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.833+0000 7f8db5ffb640 1 -- 172.21.15.19:0/56031158 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dbc00ccc0 con 0x7f8dc00719b0 2024-08-31T12:16:39.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.833+0000 7f8db5ffb640 1 -- 172.21.15.19:0/56031158 <== mon.1 v2:172.21.15.43:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8dbc00ce80 con 0x7f8dc00719b0 2024-08-31T12:16:39.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.833+0000 7f8dc79b6640 1 -- 172.21.15.19:0/56031158 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dc00c3340 con 0x7f8dc00719b0 2024-08-31T12:16:39.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.833+0000 7f8db5ffb640 1 -- 172.21.15.19:0/56031158 <== mon.1 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dbc005280 con 0x7f8dc00719b0 2024-08-31T12:16:39.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.833+0000 7f8dc79b6640 1 -- 172.21.15.19:0/56031158 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dc0152cd0 con 0x7f8dc00719b0 2024-08-31T12:16:39.839 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.834+0000 7f8dc79b6640 1 --2- 172.21.15.19:0/56031158 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8dc00bd8d0 0x7f8dc0155140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:39.839 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.834+0000 7f8dc79b6640 1 -- 172.21.15.19:0/56031158 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f8dc015b430 con 0x7f8dc00bd8d0 2024-08-31T12:16:39.839 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.835+0000 7f8dc4f51640 1 --2- 172.21.15.19:0/56031158 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8dc00bd8d0 0x7f8dc0155140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:39.839 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.835+0000 7f8dc4f51640 1 -- 172.21.15.19:0/56031158 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8dc015c120 con 0x7f8dc00bd8d0 2024-08-31T12:16:39.839 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.835+0000 7f8dc4f51640 1 --2- 172.21.15.19:0/56031158 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8dc00bd8d0 0x7f8dc0155140 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8dc00c4060 tx=0x7f8db000ec00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:39.839 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.835+0000 7f8db5ffb640 1 -- 172.21.15.19:0/56031158 <== mon.1 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f8dbc005420 con 0x7f8dc00719b0 2024-08-31T12:16:39.839 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.836+0000 7f8db5ffb640 1 --2- 172.21.15.19:0/56031158 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8da4077a30 0x7f8da4079f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:39.839 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.836+0000 7f8db5ffb640 1 -- 172.21.15.19:0/56031158 <== mon.1 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f8dbc099ee0 con 0x7f8dc00719b0 2024-08-31T12:16:39.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.836+0000 7f8dc5f53640 1 --2- 172.21.15.19:0/56031158 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8da4077a30 0x7f8da4079f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:39.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.836+0000 7f8db5ffb640 1 -- 172.21.15.19:0/56031158 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db0017070 con 0x7f8dc00bd8d0 2024-08-31T12:16:39.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.836+0000 7f8db5ffb640 1 -- 172.21.15.19:0/56031158 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8db000c770 con 0x7f8dc00bd8d0 2024-08-31T12:16:39.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.836+0000 7f8db5ffb640 1 -- 172.21.15.19:0/56031158 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8dc00bd8d0 msgr2=0x7f8dc0155140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:39.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.836+0000 7f8db5ffb640 1 --2- 172.21.15.19:0/56031158 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8dc00bd8d0 0x7f8dc0155140 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8dc00c4060 tx=0x7f8db000ec00 comp rx=0 tx=0).stop 2024-08-31T12:16:39.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.836+0000 7f8db5ffb640 1 -- 172.21.15.19:0/56031158 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8dc00bd8d0 msgr2=0x7f8dc0155140 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:16:39.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.836+0000 7f8db5ffb640 1 --2- 172.21.15.19:0/56031158 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8dc00bd8d0 0x7f8dc0155140 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:39.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.836+0000 7f8dc5f53640 1 --2- 172.21.15.19:0/56031158 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8da4077a30 0x7f8da4079f40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8db8035000 tx=0x7f8db8004ab0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:39.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.843+0000 7f8dc79b6640 1 --2- 172.21.15.19:0/56031158 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8dc0152ea0 0x7f8dc01532b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:16:39.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.843+0000 7f8dc79b6640 1 -- 172.21.15.19:0/56031158 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f8dc0155730 con 0x7f8dc0152ea0 2024-08-31T12:16:39.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.843+0000 7f8dc4f51640 1 --2- 172.21.15.19:0/56031158 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8dc0152ea0 0x7f8dc01532b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:16:39.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.844+0000 7f8dc4f51640 1 -- 172.21.15.19:0/56031158 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8dc0151ad0 con 0x7f8dc0152ea0 2024-08-31T12:16:39.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.844+0000 7f8dc4f51640 1 --2- 172.21.15.19:0/56031158 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8dc0152ea0 0x7f8dc01532b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8dc00c8530 tx=0x7f8db0004800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:16:39.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.844+0000 7f8db5ffb640 1 -- 172.21.15.19:0/56031158 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db0017070 con 0x7f8dc0152ea0 2024-08-31T12:16:39.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.845+0000 7f8db5ffb640 1 -- 172.21.15.19:0/56031158 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5029 (secure 0 0 0) 0x7f8db0006540 con 0x7f8dc0152ea0 2024-08-31T12:16:39.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.845+0000 7f8db5ffb640 1 -- 172.21.15.19:0/56031158 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8dc0152ea0 msgr2=0x7f8dc01532b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:39.847 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.845+0000 7f8db5ffb640 1 --2- 172.21.15.19:0/56031158 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8dc0152ea0 0x7f8dc01532b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8dc00c8530 tx=0x7f8db0004800 comp rx=0 tx=0).stop 2024-08-31T12:16:39.847 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.845+0000 7f8db5ffb640 1 -- 172.21.15.19:0/56031158 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8dc0152ea0 msgr2=0x7f8dc01532b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:16:39.847 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.845+0000 7f8db5ffb640 1 --2- 172.21.15.19:0/56031158 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8dc0152ea0 0x7f8dc01532b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:16:39.847 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:16:39.847 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:16:39.847 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:16:39.847 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:16:39.847 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 6, 2024-08-31T12:16:39.848 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 67941, 2024-08-31T12:16:39.848 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:16:39.848 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:16:39.848 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:16:39.848 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:16:39.848 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 1, 2024-08-31T12:16:39.848 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:16:39.848 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:16:39.848 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:16:39.848 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:16:39.848 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:16:39.848 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:16:39.849 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:16:39.849 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:16:39.849 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:16:39.849 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:16:39.849 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:16:39.849 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:16:39.849 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:16:39.849 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:16:39.849 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:16:39.849 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:16:39.850 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:16:39.850 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:16:39.850 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:16:39.850 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:16:39.850 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:16:39.850 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:16:39.850 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:16:39.850 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:16:39.850 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:16:39.850 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:16:39.850 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:16:39.850 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:16:39.850 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:16:39.851 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:39.851 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:16:39.851 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:16:39.851 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:16:39.851 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:16:39.851 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:16:39.851 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:16:39.851 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:16:39.851 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:16:39.851 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:16:39.851 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:16:39.851 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:16:39.851 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:16:39.852 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:16:39.852 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:16:39.852 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:16:39.852 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:16:39.852 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:16:39.852 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:16:39.852 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:16:39.852 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:16:39.852 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:16:39.852 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:16:39.852 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:16:39.853 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:16:39.853 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:16:39.853 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:16:39.853 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:16:39.853 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:16:39.853 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:16:39.853 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:16:39.853 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:39.853 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:16:39.853 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:39.853 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:16:39.853 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:16:39.853 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:16:39.854 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:16:39.854 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:39.854 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:16:39.854 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:16:39.854 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:16:39.854 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:39.854 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:39.854 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:16:39.854 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:16:39.854 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:16:39.854 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:39.854 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:16:39.854 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:39.855 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:16:39.855 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:16:39.855 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:16:39.855 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:16:39.855 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:16:39.855 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:39.855 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:39.855 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:16:39.855 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:16:39.855 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:16:39.855 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:16:39.855 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:39.856 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:16:39.856 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:16:39.856 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:16:39.856 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:39.856 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:39.856 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:16:39.856 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:16:39.856 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:16:39.856 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:39.856 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:16:39.856 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:39.856 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:16:39.857 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:16:39.857 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:16:39.857 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:16:39.857 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:16:39.857 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:39.857 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:39.857 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:16:39.857 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:16:39.857 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:16:39.857 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:16:39.857 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:39.857 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:16:39.857 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:16:39.857 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:16:39.858 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:39.858 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:39.858 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:16:39.858 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:16:39.858 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:16:39.858 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:39.858 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:16:39.858 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:39.858 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:16:39.858 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:16:39.858 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:16:39.858 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:16:39.858 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:16:39.858 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:39.858 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:16:39.859 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:39.859 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:16:39.859 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:16:39.859 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:39.859 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:16:39.859 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:16:39.859 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:16:39.859 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:39.859 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:16:39.859 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:16:39.859 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:39.859 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:16:39.859 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:16:39.860 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 3 2024-08-31T12:16:39.860 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:39.860 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:16:39.860 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:16:39.860 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:39.860 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:16:39.860 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:16:39.860 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 5 2024-08-31T12:16:39.860 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:39.860 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:16:39.860 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:16:39.860 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:39.860 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:16:39.861 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:16:39.861 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 3 2024-08-31T12:16:39.861 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:39.861 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:16:39.861 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:16:39.861 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:16:39.861 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:16:39.861 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:16:39.861 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:16:39.861 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:16:39.861 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:16:39.861 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:16:39.862 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:16:39.862 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:16:39.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.845+0000 7f8dc79b6640 1 -- 172.21.15.19:0/56031158 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8da4077a30 msgr2=0x7f8da4079f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:39.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.845+0000 7f8dc79b6640 1 --2- 172.21.15.19:0/56031158 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8da4077a30 0x7f8da4079f40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8db8035000 tx=0x7f8db8004ab0 comp rx=0 tx=0).stop 2024-08-31T12:16:39.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.845+0000 7f8dc79b6640 1 -- 172.21.15.19:0/56031158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8dc00719b0 msgr2=0x7f8dc01f1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:16:39.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.845+0000 7f8dc79b6640 1 --2- 172.21.15.19:0/56031158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8dc00719b0 0x7f8dc01f1270 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc00e7c0 tx=0x7f8dbc00ec80 comp rx=0 tx=0).stop 2024-08-31T12:16:39.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.845+0000 7f8dc5f53640 1 -- 172.21.15.19:0/56031158 reap_dead start 2024-08-31T12:16:39.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.845+0000 7f8dc5f53640 1 -- 172.21.15.19:0/56031158 reap_dead start 2024-08-31T12:16:39.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.845+0000 7f8dc79b6640 1 -- 172.21.15.19:0/56031158 shutdown_connections 2024-08-31T12:16:39.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.845+0000 7f8dc79b6640 1 -- 172.21.15.19:0/56031158 >> 172.21.15.19:0/56031158 conn(0x7f8dc00cd7d0 msgr2=0x7f8dc00c7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:16:39.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.845+0000 7f8dc79b6640 1 -- 172.21.15.19:0/56031158 shutdown_connections 2024-08-31T12:16:39.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:16:39.845+0000 7f8dc79b6640 1 -- 172.21.15.19:0/56031158 wait complete. 2024-08-31T12:16:39.864 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-08-31T12:16:39.864 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-08-31T12:16:55.701 DEBUG:teuthology.orchestra.run.smithi019:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:16:55.704 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:16:59.865 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-08-31T12:16:59.865 INFO:tasks.ceph.mon.a:Restarting daemon 2024-08-31T12:16:59.866 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-08-31T12:16:59.868 INFO:tasks.ceph.mon.a:Started 2024-08-31T12:16:59.869 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-08-31T12:16:59.869 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:17:00.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.008+0000 7f7b68100640 1 -- 172.21.15.19:0/1940065648 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b60151c50 msgr2=0x7f7b600c8000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:00.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.008+0000 7f7b68100640 1 --2- 172.21.15.19:0/1940065648 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b60151c50 0x7f7b600c8000 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7b60072b10 tx=0x7f7b5802b8b0 comp rx=0 tx=0).stop 2024-08-31T12:17:00.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.008+0000 7f7b68100640 1 -- 172.21.15.19:0/1940065648 shutdown_connections 2024-08-31T12:17:00.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.008+0000 7f7b68100640 1 --2- 172.21.15.19:0/1940065648 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b60151c50 0x7f7b600c8000 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:00.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.008+0000 7f7b68100640 1 --2- 172.21.15.19:0/1940065648 >> v2:172.21.15.19:3300/0 conn(0x7f7b60151840 0x7f7b600c7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:00.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.008+0000 7f7b68100640 1 --2- 172.21.15.19:0/1940065648 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7b600c3a60 0x7f7b600c6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:00.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.008+0000 7f7b68100640 1 -- 172.21.15.19:0/1940065648 >> 172.21.15.19:0/1940065648 conn(0x7f7b600cc770 msgr2=0x7f7b600ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:17:00.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.008+0000 7f7b68100640 1 -- 172.21.15.19:0/1940065648 shutdown_connections 2024-08-31T12:17:00.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.008+0000 7f7b68100640 1 -- 172.21.15.19:0/1940065648 wait complete. 2024-08-31T12:17:00.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.010+0000 7f7b68100640 1 Processor -- start 2024-08-31T12:17:00.012 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.010+0000 7f7b68100640 1 -- start start 2024-08-31T12:17:00.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.011+0000 7f7b68100640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7b600c3a60 0x7f7b6014b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:00.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.011+0000 7f7b65e9c640 1 -- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7b600c3a60 msgr2=0x7f7b6014b460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:3300/0 2024-08-31T12:17:00.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.011+0000 7f7b68100640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b60151840 0x7f7b60152340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:00.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.011+0000 7f7b65e9c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7b600c3a60 0x7f7b6014b460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:17:00.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.011+0000 7f7b68100640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7b60151c50 0x7f7b60146e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:00.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.011+0000 7f7b68100640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f7b60070b20 con 0x7f7b600c3a60 2024-08-31T12:17:00.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.011+0000 7f7b68100640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7b60070840 con 0x7f7b60151840 2024-08-31T12:17:00.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.011+0000 7f7b68100640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f7b60071400 con 0x7f7b60151c50 2024-08-31T12:17:00.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.012+0000 7f7b6569b640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b60151840 0x7f7b60152340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:00.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.012+0000 7f7b6569b640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b60151840 0x7f7b60152340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:54284/0 (socket says 172.21.15.19:54284) 2024-08-31T12:17:00.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.012+0000 7f7b6569b640 1 -- 172.21.15.19:0/1031871698 learned_addr learned my addr 172.21.15.19:0/1031871698 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:17:00.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.012+0000 7f7b6669d640 1 --2- 172.21.15.19:0/1031871698 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7b60151c50 0x7f7b60146e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:00.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.012+0000 7f7b6669d640 1 -- 172.21.15.19:0/1031871698 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b60151840 msgr2=0x7f7b60152340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:00.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.012+0000 7f7b6669d640 1 --2- 172.21.15.19:0/1031871698 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b60151840 0x7f7b60152340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:00.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.012+0000 7f7b6669d640 1 -- 172.21.15.19:0/1031871698 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7b600c3a60 msgr2=0x7f7b6014b460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:17:00.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.012+0000 7f7b6669d640 1 --2- 172.21.15.19:0/1031871698 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7b600c3a60 0x7f7b6014b460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:00.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.012+0000 7f7b6669d640 1 -- 172.21.15.19:0/1031871698 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b600c0b80 con 0x7f7b60151c50 2024-08-31T12:17:00.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.012+0000 7f7b6569b640 1 --2- 172.21.15.19:0/1031871698 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b60151840 0x7f7b60152340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:17:00.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.012+0000 7f7b6669d640 1 --2- 172.21.15.19:0/1031871698 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7b60151c50 0x7f7b60146e90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7b58033000 tx=0x7f7b58032b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:00.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.013+0000 7f7b567fc640 1 -- 172.21.15.19:0/1031871698 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b58004240 con 0x7f7b60151c50 2024-08-31T12:17:00.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.013+0000 7f7b567fc640 1 -- 172.21.15.19:0/1031871698 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7b58040070 con 0x7f7b60151c50 2024-08-31T12:17:00.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.013+0000 7f7b567fc640 1 -- 172.21.15.19:0/1031871698 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b58004750 con 0x7f7b60151c50 2024-08-31T12:17:00.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.013+0000 7f7b68100640 1 -- 172.21.15.19:0/1031871698 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b6014cdc0 con 0x7f7b60151c50 2024-08-31T12:17:00.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.013+0000 7f7b68100640 1 -- 172.21.15.19:0/1031871698 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b600c08a0 con 0x7f7b60151c50 2024-08-31T12:17:00.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.015+0000 7f7b567fc640 1 -- 172.21.15.19:0/1031871698 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f7b5803c400 con 0x7f7b60151c50 2024-08-31T12:17:00.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.015+0000 7f7b68100640 1 -- 172.21.15.19:0/1031871698 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b6014fcc0 con 0x7f7b60151c50 2024-08-31T12:17:00.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.015+0000 7f7b567fc640 1 --2- 172.21.15.19:0/1031871698 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7b44077a00 0x7f7b44079f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:00.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.016+0000 7f7b567fc640 1 -- 172.21.15.19:0/1031871698 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f7b580bac80 con 0x7f7b60151c50 2024-08-31T12:17:00.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.016+0000 7f7b65e9c640 1 --2- 172.21.15.19:0/1031871698 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7b44077a00 0x7f7b44079f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:00.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.016+0000 7f7b65e9c640 1 --2- 172.21.15.19:0/1031871698 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7b44077a00 0x7f7b44079f10 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c0094e0 tx=0x7f7b5c005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:00.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.021+0000 7f7b567fc640 1 -- 172.21.15.19:0/1031871698 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f7b6014fcc0 con 0x7f7b60151c50 2024-08-31T12:17:00.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.280+0000 7f7b68100640 1 -- 172.21.15.19:0/1031871698 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7b600c6390 con 0x7f7b60151c50 2024-08-31T12:17:00.282 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.280+0000 7f7b567fc640 1 -- 172.21.15.19:0/1031871698 <== mon.2 v2:172.21.15.43:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7f7b58083840 con 0x7f7b60151c50 2024-08-31T12:17:00.282 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":6,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:17:00.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.283+0000 7f7b68100640 1 -- 172.21.15.19:0/1031871698 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7b44077a00 msgr2=0x7f7b44079f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:00.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.283+0000 7f7b68100640 1 --2- 172.21.15.19:0/1031871698 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7b44077a00 0x7f7b44079f10 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c0094e0 tx=0x7f7b5c005f90 comp rx=0 tx=0).stop 2024-08-31T12:17:00.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.283+0000 7f7b68100640 1 -- 172.21.15.19:0/1031871698 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7b60151c50 msgr2=0x7f7b60146e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:00.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.283+0000 7f7b68100640 1 --2- 172.21.15.19:0/1031871698 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7b60151c50 0x7f7b60146e90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7b58033000 tx=0x7f7b58032b00 comp rx=0 tx=0).stop 2024-08-31T12:17:00.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.283+0000 7f7b68100640 1 -- 172.21.15.19:0/1031871698 shutdown_connections 2024-08-31T12:17:00.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.283+0000 7f7b68100640 1 --2- 172.21.15.19:0/1031871698 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7b44077a00 0x7f7b44079f10 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:00.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.283+0000 7f7b68100640 1 --2- 172.21.15.19:0/1031871698 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7b60151c50 0x7f7b60146e90 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:00.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.283+0000 7f7b68100640 1 --2- 172.21.15.19:0/1031871698 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b60151840 0x7f7b60152340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:00.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.283+0000 7f7b68100640 1 --2- 172.21.15.19:0/1031871698 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7b600c3a60 0x7f7b6014b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:00.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.283+0000 7f7b68100640 1 -- 172.21.15.19:0/1031871698 >> 172.21.15.19:0/1031871698 conn(0x7f7b600cc770 msgr2=0x7f7b601500e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:17:00.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.284+0000 7f7b68100640 1 -- 172.21.15.19:0/1031871698 shutdown_connections 2024-08-31T12:17:00.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:00.284+0000 7f7b68100640 1 -- 172.21.15.19:0/1031871698 wait complete. 2024-08-31T12:17:03.301 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:17:03.477 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:03.475+0000 7fb771f58640 1 -- 172.21.15.19:0/2533614722 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb76c15c4b0 msgr2=0x7fb76c15c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:03.477 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:03.475+0000 7fb771f58640 1 --2- 172.21.15.19:0/2533614722 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb76c15c4b0 0x7fb76c15c8c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb76c0c5bb0 tx=0x7fb76402aa10 comp rx=0 tx=0).stop 2024-08-31T12:17:03.477 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:03.475+0000 7fb771f58640 1 -- 172.21.15.19:0/2533614722 shutdown_connections 2024-08-31T12:17:03.477 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:03.475+0000 7fb771f58640 1 --2- 172.21.15.19:0/2533614722 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb76c15c4b0 0x7fb76c15c8c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:03.477 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:03.475+0000 7fb771f58640 1 --2- 172.21.15.19:0/2533614722 >> v2:172.21.15.43:3300/0 conn(0x7fb76c0ca7d0 0x7fb76c1531f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:03.477 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:03.475+0000 7fb771f58640 1 -- 172.21.15.19:0/2533614722 >> 172.21.15.19:0/2533614722 conn(0x7fb76c0cefb0 msgr2=0x7fb76c0cb340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:17:03.478 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:03.475+0000 7fb771f58640 1 -- 172.21.15.19:0/2533614722 shutdown_connections 2024-08-31T12:17:03.478 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:03.475+0000 7fb771f58640 1 -- 172.21.15.19:0/2533614722 wait complete. 2024-08-31T12:17:03.479 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:03.478+0000 7fb771f58640 1 Processor -- start 2024-08-31T12:17:03.480 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:03.478+0000 7fb771f58640 1 -- start start 2024-08-31T12:17:03.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:03.480+0000 7fb771f58640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb76c0ca7d0 0x7fb76c155400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:03.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:03.480+0000 7fb771f58640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb76c0cac90 0x7fb76c1ef9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:03.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:03.480+0000 7fb771f58640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb76c15c4b0 0x7fb76c147230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:03.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:03.480+0000 7fb771f58640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fb76c15d0c0 con 0x7fb76c15c4b0 2024-08-31T12:17:03.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:03.480+0000 7fb771f58640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb76c0714f0 con 0x7fb76c0ca7d0 2024-08-31T12:17:03.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:03.480+0000 7fb771f58640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fb76c0c98d0 con 0x7fb76c0cac90 2024-08-31T12:17:03.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:03.481+0000 7fb76b7fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb76c0ca7d0 0x7fb76c155400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:03.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:03.481+0000 7fb76affd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb76c0cac90 0x7fb76c1ef9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:03.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:03.481+0000 7fb76b7fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb76c0ca7d0 0x7fb76c155400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:54324/0 (socket says 172.21.15.19:54324) 2024-08-31T12:17:03.485 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:03.481+0000 7fb76affd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb76c0cac90 0x7fb76c1ef9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.19:38494/0 (socket says 172.21.15.19:38494) 2024-08-31T12:17:03.485 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:03.481+0000 7fb76b7fe640 1 -- 172.21.15.19:0/1672538266 learned_addr learned my addr 172.21.15.19:0/1672538266 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:17:03.485 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:03.481+0000 7fb76bfff640 1 --2- 172.21.15.19:0/1672538266 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb76c15c4b0 0x7fb76c147230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:03.485 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:03.481+0000 7fb76affd640 1 -- 172.21.15.19:0/1672538266 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb76c0ca7d0 msgr2=0x7fb76c155400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:03.485 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:03.481+0000 7fb76affd640 1 --2- 172.21.15.19:0/1672538266 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb76c0ca7d0 0x7fb76c155400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:03.485 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:03.481+0000 7fb76affd640 1 -- 172.21.15.19:0/1672538266 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb76c15c4b0 msgr2=0x7fb76c147230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:03.485 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:03.481+0000 7fb76affd640 1 --2- 172.21.15.19:0/1672538266 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb76c15c4b0 0x7fb76c147230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:03.485 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:03.481+0000 7fb76affd640 1 -- 172.21.15.19:0/1672538266 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb76c1efef0 con 0x7fb76c0cac90 2024-08-31T12:17:03.485 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:03.481+0000 7fb76bfff640 1 --2- 172.21.15.19:0/1672538266 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb76c15c4b0 0x7fb76c147230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:17:03.485 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:03.482+0000 7fb76affd640 1 --2- 172.21.15.19:0/1672538266 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb76c0cac90 0x7fb76c1ef9b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb76000c3f0 tx=0x7fb76000c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:03.485 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:03.482+0000 7fb753fff640 1 -- 172.21.15.19:0/1672538266 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb760017070 con 0x7fb76c0cac90 2024-08-31T12:17:03.486 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:03.482+0000 7fb753fff640 1 -- 172.21.15.19:0/1672538266 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb7600053c0 con 0x7fb76c0cac90 2024-08-31T12:17:03.486 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:03.482+0000 7fb753fff640 1 -- 172.21.15.19:0/1672538266 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb760003850 con 0x7fb76c0cac90 2024-08-31T12:17:03.486 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:03.482+0000 7fb771f58640 1 -- 172.21.15.19:0/1672538266 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb76c1ef1d0 con 0x7fb76c0cac90 2024-08-31T12:17:03.486 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:03.482+0000 7fb771f58640 1 -- 172.21.15.19:0/1672538266 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb76c0b2a10 con 0x7fb76c0cac90 2024-08-31T12:17:03.486 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:03.483+0000 7fb771f58640 1 -- 172.21.15.19:0/1672538266 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb76c15d3a0 con 0x7fb76c0cac90 2024-08-31T12:17:03.486 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:03.485+0000 7fb753fff640 1 -- 172.21.15.19:0/1672538266 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fb76001a070 con 0x7fb76c0cac90 2024-08-31T12:17:03.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:03.485+0000 7fb753fff640 1 --2- 172.21.15.19:0/1672538266 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb758077a80 0x7fb758079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:03.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:03.485+0000 7fb753fff640 1 -- 172.21.15.19:0/1672538266 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fb7600988d0 con 0x7fb76c0cac90 2024-08-31T12:17:03.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:03.486+0000 7fb76b7fe640 1 --2- 172.21.15.19:0/1672538266 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb758077a80 0x7fb758079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:03.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:03.486+0000 7fb76b7fe640 1 --2- 172.21.15.19:0/1672538266 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb758077a80 0x7fb758079f90 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb75c0058c0 tx=0x7fb75c00f820 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:03.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:03.490+0000 7fb753fff640 1 -- 172.21.15.19:0/1672538266 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fb760060ef0 con 0x7fb76c0cac90 2024-08-31T12:17:04.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.017+0000 7fb771f58640 1 -- 172.21.15.19:0/1672538266 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb76c0ca160 con 0x7fb76c0cac90 2024-08-31T12:17:04.020 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.018+0000 7fb753fff640 1 -- 172.21.15.19:0/1672538266 <== mon.2 v2:172.21.15.43:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fb7600656b0 con 0x7fb76c0cac90 2024-08-31T12:17:04.020 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:17:04.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.021+0000 7fb771f58640 1 -- 172.21.15.19:0/1672538266 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb758077a80 msgr2=0x7fb758079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:04.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.021+0000 7fb771f58640 1 --2- 172.21.15.19:0/1672538266 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb758077a80 0x7fb758079f90 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb75c0058c0 tx=0x7fb75c00f820 comp rx=0 tx=0).stop 2024-08-31T12:17:04.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.021+0000 7fb771f58640 1 -- 172.21.15.19:0/1672538266 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb76c0cac90 msgr2=0x7fb76c1ef9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:04.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.021+0000 7fb771f58640 1 --2- 172.21.15.19:0/1672538266 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb76c0cac90 0x7fb76c1ef9b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb76000c3f0 tx=0x7fb76000c8b0 comp rx=0 tx=0).stop 2024-08-31T12:17:04.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.021+0000 7fb771f58640 1 -- 172.21.15.19:0/1672538266 shutdown_connections 2024-08-31T12:17:04.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.021+0000 7fb771f58640 1 --2- 172.21.15.19:0/1672538266 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb758077a80 0x7fb758079f90 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:04.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.021+0000 7fb771f58640 1 --2- 172.21.15.19:0/1672538266 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb76c15c4b0 0x7fb76c147230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:04.024 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.021+0000 7fb771f58640 1 --2- 172.21.15.19:0/1672538266 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb76c0cac90 0x7fb76c1ef9b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:04.024 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.021+0000 7fb771f58640 1 --2- 172.21.15.19:0/1672538266 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb76c0ca7d0 0x7fb76c155400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:04.024 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.021+0000 7fb771f58640 1 -- 172.21.15.19:0/1672538266 >> 172.21.15.19:0/1672538266 conn(0x7fb76c0cefb0 msgr2=0x7fb76c150aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:17:04.024 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.021+0000 7fb771f58640 1 -- 172.21.15.19:0/1672538266 shutdown_connections 2024-08-31T12:17:04.024 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.021+0000 7fb771f58640 1 -- 172.21.15.19:0/1672538266 wait complete. 2024-08-31T12:17:04.038 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-08-31T12:17:04.039 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-08-31T12:17:04.039 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-31T12:17:04.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.213+0000 7f889a557640 1 -- 172.21.15.19:0/2238767301 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8880005350 msgr2=0x7f888000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:04.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.213+0000 7f889a557640 1 --2- 172.21.15.19:0/2238767301 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8880005350 0x7f888000b9c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f888c001dc0 tx=0x7f8880005830 comp rx=0 tx=0).stop 2024-08-31T12:17:04.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.213+0000 7f889a557640 1 -- 172.21.15.19:0/2238767301 shutdown_connections 2024-08-31T12:17:04.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.213+0000 7f889a557640 1 --2- 172.21.15.19:0/2238767301 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8880005350 0x7f888000b9c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:04.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.213+0000 7f889a557640 1 -- 172.21.15.19:0/2238767301 >> 172.21.15.19:0/2238767301 conn(0x7f889406cae0 msgr2=0x7f889406cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:17:04.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.213+0000 7f889a557640 1 -- 172.21.15.19:0/2238767301 shutdown_connections 2024-08-31T12:17:04.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.213+0000 7f889a557640 1 -- 172.21.15.19:0/2238767301 wait complete. 2024-08-31T12:17:04.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.216+0000 7f889a557640 1 Processor -- start 2024-08-31T12:17:04.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.216+0000 7f889a557640 1 -- start start 2024-08-31T12:17:04.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.218+0000 7f889a557640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8880005350 0x7f8894159070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:04.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.218+0000 7f889a557640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f889406f360 0x7f8894145cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:04.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.218+0000 7f889a557640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f889406f770 0x7f8894146200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:04.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.219+0000 7f8893fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8880005350 0x7f8894159070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:04.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.219+0000 7f889a557640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f8894069110 con 0x7f8880005350 2024-08-31T12:17:04.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.219+0000 7f889a557640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8894068f90 con 0x7f889406f770 2024-08-31T12:17:04.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.219+0000 7f8893fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8880005350 0x7f8894159070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:50080/0 (socket says 172.21.15.19:50080) 2024-08-31T12:17:04.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.219+0000 7f889a557640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f88940c39c0 con 0x7f889406f360 2024-08-31T12:17:04.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.219+0000 7f8893fff640 1 -- 172.21.15.19:0/3226750257 learned_addr learned my addr 172.21.15.19:0/3226750257 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:17:04.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.219+0000 7f888bfff640 1 --2- 172.21.15.19:0/3226750257 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f889406f360 0x7f8894145cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:04.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.219+0000 7f8898af4640 1 --2- 172.21.15.19:0/3226750257 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f889406f770 0x7f8894146200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:04.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.219+0000 7f888bfff640 1 -- 172.21.15.19:0/3226750257 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f889406f770 msgr2=0x7f8894146200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:04.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.219+0000 7f888bfff640 1 --2- 172.21.15.19:0/3226750257 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f889406f770 0x7f8894146200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:04.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.219+0000 7f888bfff640 1 -- 172.21.15.19:0/3226750257 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8880005350 msgr2=0x7f8894159070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:04.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.219+0000 7f888bfff640 1 --2- 172.21.15.19:0/3226750257 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8880005350 0x7f8894159070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:04.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.219+0000 7f888bfff640 1 -- 172.21.15.19:0/3226750257 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88941595b0 con 0x7f889406f360 2024-08-31T12:17:04.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.219+0000 7f8893fff640 1 --2- 172.21.15.19:0/3226750257 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8880005350 0x7f8894159070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:17:04.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.219+0000 7f888bfff640 1 --2- 172.21.15.19:0/3226750257 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f889406f360 0x7f8894145cc0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f887c006680 tx=0x7f887c0039c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:04.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.220+0000 7f88917fa640 1 -- 172.21.15.19:0/3226750257 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f887c015070 con 0x7f889406f360 2024-08-31T12:17:04.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.220+0000 7f88917fa640 1 -- 172.21.15.19:0/3226750257 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f887c011920 con 0x7f889406f360 2024-08-31T12:17:04.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.220+0000 7f88917fa640 1 -- 172.21.15.19:0/3226750257 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f887c011d70 con 0x7f889406f360 2024-08-31T12:17:04.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.220+0000 7f889a557640 1 -- 172.21.15.19:0/3226750257 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88941eecf0 con 0x7f889406f360 2024-08-31T12:17:04.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.220+0000 7f889a557640 1 -- 172.21.15.19:0/3226750257 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f88941ef2b0 con 0x7f889406f360 2024-08-31T12:17:04.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.221+0000 7f889a557640 1 --2- 172.21.15.19:0/3226750257 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f88941ef4b0 0x7f8894149850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:04.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.221+0000 7f889a557640 1 -- 172.21.15.19:0/3226750257 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f8894159a80 con 0x7f88941ef4b0 2024-08-31T12:17:04.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.222+0000 7f8893fff640 1 --2- 172.21.15.19:0/3226750257 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f88941ef4b0 0x7f8894149850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:04.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.222+0000 7f8893fff640 1 -- 172.21.15.19:0/3226750257 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f88940c7a30 con 0x7f88941ef4b0 2024-08-31T12:17:04.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.222+0000 7f8893fff640 1 --2- 172.21.15.19:0/3226750257 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f88941ef4b0 0x7f8894149850 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8884005f60 tx=0x7f8884011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:04.225 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.223+0000 7f88917fa640 1 -- 172.21.15.19:0/3226750257 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8884011e90 con 0x7f88941ef4b0 2024-08-31T12:17:04.225 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.223+0000 7f88917fa640 1 -- 172.21.15.19:0/3226750257 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f888400f670 con 0x7f88941ef4b0 2024-08-31T12:17:04.226 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.223+0000 7f88917fa640 1 -- 172.21.15.19:0/3226750257 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f88941ef4b0 msgr2=0x7f8894149850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:04.226 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.223+0000 7f88917fa640 1 --2- 172.21.15.19:0/3226750257 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f88941ef4b0 0x7f8894149850 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8884005f60 tx=0x7f8884011810 comp rx=0 tx=0).stop 2024-08-31T12:17:04.226 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.223+0000 7f88917fa640 1 -- 172.21.15.19:0/3226750257 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f88941ef4b0 msgr2=0x7f8894149850 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:17:04.226 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.223+0000 7f88917fa640 1 --2- 172.21.15.19:0/3226750257 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f88941ef4b0 0x7f8894149850 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:04.226 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.223+0000 7f88917fa640 1 -- 172.21.15.19:0/3226750257 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f887c018070 con 0x7f889406f360 2024-08-31T12:17:04.227 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.224+0000 7f88917fa640 1 --2- 172.21.15.19:0/3226750257 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8874077660 0x7f8874079b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:04.227 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.224+0000 7f88917fa640 1 -- 172.21.15.19:0/3226750257 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f887c09a2c0 con 0x7f889406f360 2024-08-31T12:17:04.227 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.224+0000 7f8893fff640 1 --2- 172.21.15.19:0/3226750257 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8874077660 0x7f8874079b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:04.227 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.225+0000 7f8893fff640 1 --2- 172.21.15.19:0/3226750257 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8874077660 0x7f8874079b70 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8884011db0 tx=0x7f8884014000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:04.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.235+0000 7f889a557640 1 --2- 172.21.15.19:0/3226750257 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8894149d90 0x7f88941f20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:04.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.235+0000 7f889a557640 1 -- 172.21.15.19:0/3226750257 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f8894146bf0 con 0x7f8894149d90 2024-08-31T12:17:04.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.235+0000 7f8898af4640 1 --2- 172.21.15.19:0/3226750257 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8894149d90 0x7f88941f20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:04.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.236+0000 7f8898af4640 1 -- 172.21.15.19:0/3226750257 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f889406df60 con 0x7f8894149d90 2024-08-31T12:17:04.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.236+0000 7f8898af4640 1 --2- 172.21.15.19:0/3226750257 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8894149d90 0x7f88941f20c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f88940c98b0 tx=0x7f888c006d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:04.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.237+0000 7f88917fa640 1 -- 172.21.15.19:0/3226750257 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f888c003d80 con 0x7f8894149d90 2024-08-31T12:17:04.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.237+0000 7f88917fa640 1 -- 172.21.15.19:0/3226750257 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4541 (secure 0 0 0) 0x7f888c005070 con 0x7f8894149d90 2024-08-31T12:17:04.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.237+0000 7f88917fa640 1 -- 172.21.15.19:0/3226750257 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8894149d90 msgr2=0x7f88941f20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:04.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.237+0000 7f88917fa640 1 --2- 172.21.15.19:0/3226750257 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8894149d90 0x7f88941f20c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f88940c98b0 tx=0x7f888c006d10 comp rx=0 tx=0).stop 2024-08-31T12:17:04.239 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:17:04.239 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:17:04.239 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:17:04.240 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "leader", 2024-08-31T12:17:04.240 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 10, 2024-08-31T12:17:04.240 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 4227, 2024-08-31T12:17:04.240 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:17:04.240 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:17:04.240 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:17:04.240 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:17:04.240 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:04.240 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 3, 2024-08-31T12:17:04.240 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:17:04.240 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:17:04.241 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:17:04.241 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:17:04.241 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:17:04.241 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:17:04.241 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:17:04.241 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:17:04.241 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:17:04.241 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:17:04.241 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:17:04.241 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:17:04.241 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:17:04.241 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:17:04.241 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:04.242 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:17:04.242 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:17:04.242 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:17:04.242 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:17:04.242 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:17:04.242 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:17:04.242 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:17:04.242 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:17:04.242 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:17:04.242 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:17:04.242 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:17:04.242 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:17:04.242 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:17:04.243 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:04.243 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:04.243 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:17:04.243 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:17:04.243 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:17:04.243 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:17:04.243 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:17:04.243 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:17:04.243 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:17:04.243 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:17:04.243 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:17:04.243 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:17:04.243 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:17:04.243 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:17:04.244 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:17:04.244 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:17:04.244 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:17:04.244 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:17:04.244 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:17:04.244 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:17:04.244 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:17:04.244 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:17:04.244 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:17:04.244 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:17:04.244 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:17:04.244 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:17:04.244 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:17:04.244 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:17:04.245 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:17:04.245 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:17:04.245 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:04.245 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:17:04.245 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:04.245 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:17:04.245 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:04.245 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:17:04.245 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:17:04.245 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:17:04.245 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:17:04.245 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:04.245 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:17:04.245 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:17:04.246 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:04.246 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:04.246 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:04.246 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:17:04.246 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:17:04.246 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:04.246 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:04.246 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:04.246 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:04.246 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:17:04.246 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:17:04.246 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:17:04.246 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:17:04.246 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:17:04.247 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:04.247 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:04.247 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:17:04.247 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:17:04.247 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:17:04.247 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:17:04.247 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:04.247 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:17:04.247 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:17:04.247 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:04.247 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:04.247 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:04.247 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:17:04.247 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:17:04.248 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:04.248 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:04.248 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:04.248 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:04.248 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:17:04.248 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:17:04.248 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:17:04.248 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:17:04.248 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:17:04.248 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:04.248 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:04.248 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:17:04.248 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:17:04.248 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:17:04.249 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:17:04.249 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:04.249 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:17:04.249 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:17:04.249 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:04.249 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:04.249 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:04.249 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:17:04.249 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:17:04.249 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:04.249 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:04.249 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:04.249 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:04.249 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:17:04.250 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:17:04.250 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:17:04.250 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:17:04.250 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:17:04.250 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:04.250 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:04.250 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:04.250 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:17:04.250 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:17:04.250 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:04.250 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:17:04.250 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:17:04.250 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:17:04.250 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:04.250 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:04.251 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:17:04.251 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:04.251 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:17:04.251 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:17:04.251 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:17:04.251 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:04.251 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:04.251 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:04.251 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:17:04.251 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:17:04.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.237+0000 7f88917fa640 1 -- 172.21.15.19:0/3226750257 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8894149d90 msgr2=0x7f88941f20c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:17:04.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.237+0000 7f88917fa640 1 --2- 172.21.15.19:0/3226750257 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8894149d90 0x7f88941f20c0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:04.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.238+0000 7f889a557640 1 -- 172.21.15.19:0/3226750257 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8874077660 msgr2=0x7f8874079b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:04.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.238+0000 7f889a557640 1 --2- 172.21.15.19:0/3226750257 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8874077660 0x7f8874079b70 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8884011db0 tx=0x7f8884014000 comp rx=0 tx=0).stop 2024-08-31T12:17:04.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.238+0000 7f8898af4640 1 -- 172.21.15.19:0/3226750257 reap_dead start 2024-08-31T12:17:04.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.238+0000 7f889a557640 1 -- 172.21.15.19:0/3226750257 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f889406f360 msgr2=0x7f8894145cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:04.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.238+0000 7f889a557640 1 --2- 172.21.15.19:0/3226750257 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f889406f360 0x7f8894145cc0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f887c006680 tx=0x7f887c0039c0 comp rx=0 tx=0).stop 2024-08-31T12:17:04.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.238+0000 7f889a557640 1 -- 172.21.15.19:0/3226750257 shutdown_connections 2024-08-31T12:17:04.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.238+0000 7f889a557640 1 --2- 172.21.15.19:0/3226750257 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f889406f360 0x7f8894145cc0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:04.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.238+0000 7f889a557640 1 -- 172.21.15.19:0/3226750257 >> 172.21.15.19:0/3226750257 conn(0x7f889406cae0 msgr2=0x7f8894157690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:17:04.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.238+0000 7f889a557640 1 -- 172.21.15.19:0/3226750257 shutdown_connections 2024-08-31T12:17:04.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.238+0000 7f889a557640 1 -- 172.21.15.19:0/3226750257 wait complete. 2024-08-31T12:17:04.254 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:17:04.254 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-08-31T12:17:04.255 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-31T12:17:04.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.429+0000 7ff4389fe640 1 -- 172.21.15.19:0/1866406369 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff43414e530 msgr2=0x7ff434157b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:04.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.429+0000 7ff4389fe640 1 --2- 172.21.15.19:0/1866406369 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff43414e530 0x7ff434157b30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff4340ce2d0 tx=0x7ff42802a5d0 comp rx=0 tx=0).stop 2024-08-31T12:17:04.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.430+0000 7ff4389fe640 1 -- 172.21.15.19:0/1866406369 shutdown_connections 2024-08-31T12:17:04.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.430+0000 7ff4389fe640 1 --2- 172.21.15.19:0/1866406369 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff43414e530 0x7ff434157b30 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:04.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.430+0000 7ff4389fe640 1 --2- 172.21.15.19:0/1866406369 >> v2:172.21.15.43:3300/0 conn(0x7ff4340cb070 0x7ff4340c4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:04.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.430+0000 7ff4389fe640 1 -- 172.21.15.19:0/1866406369 >> 172.21.15.19:0/1866406369 conn(0x7ff4340718e0 msgr2=0x7ff434071d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:17:04.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.430+0000 7ff4389fe640 1 -- 172.21.15.19:0/1866406369 shutdown_connections 2024-08-31T12:17:04.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.430+0000 7ff4389fe640 1 -- 172.21.15.19:0/1866406369 wait complete. 2024-08-31T12:17:04.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.433+0000 7ff4389fe640 1 Processor -- start 2024-08-31T12:17:04.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.433+0000 7ff4389fe640 1 -- start start 2024-08-31T12:17:04.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.435+0000 7ff4389fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff4340cb070 0x7ff434158d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:04.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.435+0000 7ff4389fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff43414e530 0x7ff434152630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:04.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.435+0000 7ff4389fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff434158070 0x7ff43406ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:04.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.435+0000 7ff4389fe640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7ff4340c6d00 con 0x7ff4340cb070 2024-08-31T12:17:04.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.435+0000 7ff4389fe640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff4340c6a20 con 0x7ff434158070 2024-08-31T12:17:04.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.435+0000 7ff4389fe640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7ff43414e940 con 0x7ff43414e530 2024-08-31T12:17:04.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.435+0000 7ff432d9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff4340cb070 0x7ff434158d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:04.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.435+0000 7ff432d9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff4340cb070 0x7ff434158d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:50110/0 (socket says 172.21.15.19:50110) 2024-08-31T12:17:04.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.435+0000 7ff432d9d640 1 -- 172.21.15.19:0/465214891 learned_addr learned my addr 172.21.15.19:0/465214891 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:17:04.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.435+0000 7ff43259c640 1 --2- 172.21.15.19:0/465214891 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff43414e530 0x7ff434152630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:04.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.435+0000 7ff43359e640 1 --2- 172.21.15.19:0/465214891 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff434158070 0x7ff43406ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:04.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.436+0000 7ff43259c640 1 -- 172.21.15.19:0/465214891 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff434158070 msgr2=0x7ff43406ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:04.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.436+0000 7ff43259c640 1 --2- 172.21.15.19:0/465214891 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff434158070 0x7ff43406ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:04.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.436+0000 7ff43259c640 1 -- 172.21.15.19:0/465214891 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff4340cb070 msgr2=0x7ff434158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:04.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.436+0000 7ff43259c640 1 --2- 172.21.15.19:0/465214891 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff4340cb070 0x7ff434158d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:04.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.436+0000 7ff43259c640 1 -- 172.21.15.19:0/465214891 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff434152b70 con 0x7ff43414e530 2024-08-31T12:17:04.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.436+0000 7ff432d9d640 1 --2- 172.21.15.19:0/465214891 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff4340cb070 0x7ff434158d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:17:04.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.436+0000 7ff43259c640 1 --2- 172.21.15.19:0/465214891 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff43414e530 0x7ff434152630 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff434070e50 tx=0x7ff428033ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:04.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.436+0000 7ff41b7fe640 1 -- 172.21.15.19:0/465214891 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff428031d60 con 0x7ff43414e530 2024-08-31T12:17:04.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.436+0000 7ff41b7fe640 1 -- 172.21.15.19:0/465214891 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff428033e50 con 0x7ff43414e530 2024-08-31T12:17:04.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.436+0000 7ff41b7fe640 1 -- 172.21.15.19:0/465214891 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4280332d0 con 0x7ff43414e530 2024-08-31T12:17:04.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.438+0000 7ff4389fe640 1 -- 172.21.15.19:0/465214891 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4341544c0 con 0x7ff43414e530 2024-08-31T12:17:04.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.438+0000 7ff4389fe640 1 -- 172.21.15.19:0/465214891 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4340cbf90 con 0x7ff43414e530 2024-08-31T12:17:04.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.439+0000 7ff4389fe640 1 --2- 172.21.15.19:0/465214891 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff434154fd0 0x7ff434149b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:04.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.439+0000 7ff4389fe640 1 -- 172.21.15.19:0/465214891 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff434153040 con 0x7ff434154fd0 2024-08-31T12:17:04.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.439+0000 7ff432d9d640 1 --2- 172.21.15.19:0/465214891 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff434154fd0 0x7ff434149b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:04.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.439+0000 7ff432d9d640 1 -- 172.21.15.19:0/465214891 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff4340cb9c0 con 0x7ff434154fd0 2024-08-31T12:17:04.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.439+0000 7ff432d9d640 1 --2- 172.21.15.19:0/465214891 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff434154fd0 0x7ff434149b60 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff424011cf0 tx=0x7ff42400f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:04.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.440+0000 7ff41b7fe640 1 -- 172.21.15.19:0/465214891 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff424004230 con 0x7ff434154fd0 2024-08-31T12:17:04.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.440+0000 7ff41b7fe640 1 -- 172.21.15.19:0/465214891 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff4240043d0 con 0x7ff434154fd0 2024-08-31T12:17:04.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.440+0000 7ff41b7fe640 1 -- 172.21.15.19:0/465214891 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff434154fd0 msgr2=0x7ff434149b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:04.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.440+0000 7ff41b7fe640 1 --2- 172.21.15.19:0/465214891 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff434154fd0 0x7ff434149b60 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff424011cf0 tx=0x7ff42400f790 comp rx=0 tx=0).stop 2024-08-31T12:17:04.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.440+0000 7ff41b7fe640 1 -- 172.21.15.19:0/465214891 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff434154fd0 msgr2=0x7ff434149b60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:17:04.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.440+0000 7ff41b7fe640 1 --2- 172.21.15.19:0/465214891 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff434154fd0 0x7ff434149b60 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:04.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.440+0000 7ff41b7fe640 1 -- 172.21.15.19:0/465214891 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7ff428033530 con 0x7ff43414e530 2024-08-31T12:17:04.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.442+0000 7ff41b7fe640 1 --2- 172.21.15.19:0/465214891 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff4200774c0 0x7ff4200799d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:04.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.442+0000 7ff432d9d640 1 --2- 172.21.15.19:0/465214891 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff4200774c0 0x7ff4200799d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:04.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.442+0000 7ff41b7fe640 1 -- 172.21.15.19:0/465214891 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7ff4280ba690 con 0x7ff43414e530 2024-08-31T12:17:04.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.443+0000 7ff432d9d640 1 --2- 172.21.15.19:0/465214891 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff4200774c0 0x7ff4200799d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff424000c00 tx=0x7ff424015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:04.454 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.452+0000 7ff4389fe640 1 --2- 172.21.15.19:0/465214891 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff434157280 0x7ff43414a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:04.454 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.452+0000 7ff4389fe640 1 -- 172.21.15.19:0/465214891 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff4341e19c0 con 0x7ff434157280 2024-08-31T12:17:04.454 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.453+0000 7ff43359e640 1 --2- 172.21.15.19:0/465214891 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff434157280 0x7ff43414a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:04.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.453+0000 7ff43359e640 1 -- 172.21.15.19:0/465214891 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff434156fa0 con 0x7ff434157280 2024-08-31T12:17:04.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.453+0000 7ff43359e640 1 --2- 172.21.15.19:0/465214891 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff434157280 0x7ff43414a0a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff4340cadc0 tx=0x7ff42c00fd40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:04.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.453+0000 7ff41b7fe640 1 -- 172.21.15.19:0/465214891 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff42c003b80 con 0x7ff434157280 2024-08-31T12:17:04.456 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:17:04.456 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:17:04.456 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:17:04.456 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:17:04.456 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 10, 2024-08-31T12:17:04.456 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 92550, 2024-08-31T12:17:04.456 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:17:04.456 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:17:04.456 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:17:04.456 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:17:04.456 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:04.457 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 3, 2024-08-31T12:17:04.457 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:17:04.457 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:17:04.457 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:17:04.457 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:17:04.457 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:17:04.457 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:17:04.457 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:17:04.457 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:17:04.457 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:17:04.457 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:17:04.457 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:17:04.457 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:17:04.458 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:17:04.458 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:17:04.458 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:04.458 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:17:04.458 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:17:04.458 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:17:04.458 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:17:04.458 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:17:04.458 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:17:04.458 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:17:04.458 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:17:04.458 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:17:04.459 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:17:04.459 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:17:04.459 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:17:04.459 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:17:04.459 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:04.459 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:04.459 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:17:04.459 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:17:04.459 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:17:04.459 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:17:04.459 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:17:04.459 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:17:04.459 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:17:04.459 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:17:04.460 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:17:04.460 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:17:04.460 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:17:04.460 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:17:04.460 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:17:04.460 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:17:04.460 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:17:04.460 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:17:04.460 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:17:04.460 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:17:04.460 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:17:04.460 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:17:04.460 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:17:04.461 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:17:04.461 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:17:04.461 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:17:04.461 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:17:04.461 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:17:04.461 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:17:04.461 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:17:04.461 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:04.461 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:17:04.461 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:04.461 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:17:04.461 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:04.461 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:17:04.462 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:17:04.462 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:17:04.462 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:17:04.462 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:04.462 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:17:04.462 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:17:04.462 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:04.462 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:04.462 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:04.462 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:17:04.462 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:17:04.462 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:04.462 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:04.463 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:04.463 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:04.463 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:17:04.463 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:17:04.463 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:17:04.463 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:17:04.463 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:17:04.463 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:04.463 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:04.463 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:17:04.463 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:17:04.463 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:17:04.463 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:17:04.464 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:04.464 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:17:04.464 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:17:04.464 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:04.464 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:04.464 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:04.464 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:17:04.464 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:17:04.464 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:04.464 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:04.464 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:04.464 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:04.464 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:17:04.465 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:17:04.465 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:17:04.465 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:17:04.465 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:17:04.465 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:04.465 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:04.465 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:17:04.465 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:17:04.465 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:17:04.465 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:17:04.465 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:04.465 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:17:04.466 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:17:04.466 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:04.466 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:04.466 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:04.466 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:17:04.466 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:17:04.466 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:04.466 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:04.466 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:04.466 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:04.466 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:17:04.466 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:17:04.466 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:17:04.466 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:17:04.467 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:17:04.467 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:04.467 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:04.467 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:04.467 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:17:04.467 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:17:04.467 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:04.467 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:17:04.467 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:17:04.467 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:17:04.467 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:04.467 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:04.467 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:17:04.467 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:04.468 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:17:04.468 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:17:04.468 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 3 2024-08-31T12:17:04.468 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:04.468 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:04.468 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:17:04.468 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:04.468 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:17:04.468 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:17:04.468 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 3 2024-08-31T12:17:04.468 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:04.468 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:04.468 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:17:04.468 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:04.469 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:17:04.469 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:17:04.469 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 4 2024-08-31T12:17:04.469 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:04.469 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:04.469 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:17:04.469 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:04.469 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:17:04.469 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:17:04.469 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:17:04.469 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:04.469 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:04.469 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:04.470 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:17:04.470 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:17:04.471 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.454+0000 7ff41b7fe640 1 -- 172.21.15.19:0/465214891 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5041 (secure 0 0 0) 0x7ff42c003b80 con 0x7ff434157280 2024-08-31T12:17:04.471 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.454+0000 7ff41b7fe640 1 -- 172.21.15.19:0/465214891 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff434157280 msgr2=0x7ff43414a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:04.471 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.454+0000 7ff41b7fe640 1 --2- 172.21.15.19:0/465214891 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff434157280 0x7ff43414a0a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff4340cadc0 tx=0x7ff42c00fd40 comp rx=0 tx=0).stop 2024-08-31T12:17:04.471 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.454+0000 7ff41b7fe640 1 -- 172.21.15.19:0/465214891 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff434157280 msgr2=0x7ff43414a0a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:17:04.471 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.454+0000 7ff41b7fe640 1 --2- 172.21.15.19:0/465214891 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff434157280 0x7ff43414a0a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:04.471 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.454+0000 7ff4389fe640 1 -- 172.21.15.19:0/465214891 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff4200774c0 msgr2=0x7ff4200799d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:04.471 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.454+0000 7ff4389fe640 1 --2- 172.21.15.19:0/465214891 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff4200774c0 0x7ff4200799d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff424000c00 tx=0x7ff424015000 comp rx=0 tx=0).stop 2024-08-31T12:17:04.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.454+0000 7ff4389fe640 1 -- 172.21.15.19:0/465214891 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff43414e530 msgr2=0x7ff434152630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:04.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.454+0000 7ff43359e640 1 -- 172.21.15.19:0/465214891 reap_dead start 2024-08-31T12:17:04.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.454+0000 7ff4389fe640 1 --2- 172.21.15.19:0/465214891 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff43414e530 0x7ff434152630 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff434070e50 tx=0x7ff428033ba0 comp rx=0 tx=0).stop 2024-08-31T12:17:04.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.454+0000 7ff4389fe640 1 -- 172.21.15.19:0/465214891 shutdown_connections 2024-08-31T12:17:04.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.454+0000 7ff4389fe640 1 --2- 172.21.15.19:0/465214891 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff43414e530 0x7ff434152630 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:04.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.454+0000 7ff4389fe640 1 -- 172.21.15.19:0/465214891 >> 172.21.15.19:0/465214891 conn(0x7ff4340718e0 msgr2=0x7ff4340cc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:17:04.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.454+0000 7ff4389fe640 1 -- 172.21.15.19:0/465214891 shutdown_connections 2024-08-31T12:17:04.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.454+0000 7ff4389fe640 1 -- 172.21.15.19:0/465214891 wait complete. 2024-08-31T12:17:04.473 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:17:04.473 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-08-31T12:17:04.473 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-31T12:17:04.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.645+0000 7f54abfff640 1 -- 172.21.15.19:0/2320103640 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54ac036400 con 0x7f54b414e9a0 2024-08-31T12:17:04.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.646+0000 7f54bb332640 1 -- 172.21.15.19:0/2320103640 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f54b414e9a0 msgr2=0x7f54b40cc400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:04.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.646+0000 7f54bb332640 1 --2- 172.21.15.19:0/2320103640 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f54b414e9a0 0x7f54b40cc400 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f54b4073680 tx=0x7f54ac02aac0 comp rx=0 tx=0).stop 2024-08-31T12:17:04.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.646+0000 7f54bb332640 1 -- 172.21.15.19:0/2320103640 shutdown_connections 2024-08-31T12:17:04.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.646+0000 7f54bb332640 1 --2- 172.21.15.19:0/2320103640 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f54b414e9a0 0x7f54b40cc400 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:04.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.646+0000 7f54bb332640 1 --2- 172.21.15.19:0/2320103640 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f54b40cb2b0 0x7f54b414e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:04.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.646+0000 7f54bb332640 1 --2- 172.21.15.19:0/2320103640 >> v2:172.21.15.43:3300/0 conn(0x7f54b40c5370 0x7f54b406ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:04.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.646+0000 7f54bb332640 1 -- 172.21.15.19:0/2320103640 >> 172.21.15.19:0/2320103640 conn(0x7f54b40ca670 msgr2=0x7f54b40cd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:17:04.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.647+0000 7f54bb332640 1 -- 172.21.15.19:0/2320103640 shutdown_connections 2024-08-31T12:17:04.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.647+0000 7f54bb332640 1 -- 172.21.15.19:0/2320103640 wait complete. 2024-08-31T12:17:04.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.649+0000 7f54bb332640 1 Processor -- start 2024-08-31T12:17:04.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.650+0000 7f54bb332640 1 -- start start 2024-08-31T12:17:04.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.652+0000 7f54bb332640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54b40c5370 0x7f54b414ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:04.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.652+0000 7f54bb332640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f54b40cb2b0 0x7f54b40c8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:04.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.652+0000 7f54bb332640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f54b414e9a0 0x7f54b4147c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:04.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.652+0000 7f54bb332640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f54b40c72e0 con 0x7f54b40cb2b0 2024-08-31T12:17:04.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.652+0000 7f54bb332640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f54b40c7000 con 0x7f54b40c5370 2024-08-31T12:17:04.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.652+0000 7f54bb332640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f54b40cb8c0 con 0x7f54b414e9a0 2024-08-31T12:17:04.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.652+0000 7f54b90ce640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54b40c5370 0x7f54b414ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:04.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.652+0000 7f54b88cd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f54b40cb2b0 0x7f54b40c8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:04.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.652+0000 7f54b90ce640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54b40c5370 0x7f54b414ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:54402/0 (socket says 172.21.15.19:54402) 2024-08-31T12:17:04.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.652+0000 7f54b90ce640 1 -- 172.21.15.19:0/2445623379 learned_addr learned my addr 172.21.15.19:0/2445623379 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:17:04.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.652+0000 7f54b88cd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f54b40cb2b0 0x7f54b40c8e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:50132/0 (socket says 172.21.15.19:50132) 2024-08-31T12:17:04.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.652+0000 7f54b98cf640 1 --2- 172.21.15.19:0/2445623379 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f54b414e9a0 0x7f54b4147c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:04.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.652+0000 7f54b98cf640 1 -- 172.21.15.19:0/2445623379 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54b40c5370 msgr2=0x7f54b414ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:04.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.652+0000 7f54b98cf640 1 --2- 172.21.15.19:0/2445623379 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54b40c5370 0x7f54b414ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:04.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.652+0000 7f54b98cf640 1 -- 172.21.15.19:0/2445623379 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f54b40cb2b0 msgr2=0x7f54b40c8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:04.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.652+0000 7f54b98cf640 1 --2- 172.21.15.19:0/2445623379 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f54b40cb2b0 0x7f54b40c8e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:04.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.652+0000 7f54b98cf640 1 -- 172.21.15.19:0/2445623379 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54b41f1700 con 0x7f54b414e9a0 2024-08-31T12:17:04.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.653+0000 7f54b98cf640 1 --2- 172.21.15.19:0/2445623379 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f54b414e9a0 0x7f54b4147c80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f54b40cb890 tx=0x7f54ac003ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:04.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.653+0000 7f54a9ffb640 1 -- 172.21.15.19:0/2445623379 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54ac004040 con 0x7f54b414e9a0 2024-08-31T12:17:04.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.653+0000 7f54a9ffb640 1 -- 172.21.15.19:0/2445623379 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f54ac0041e0 con 0x7f54b414e9a0 2024-08-31T12:17:04.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.654+0000 7f54a9ffb640 1 -- 172.21.15.19:0/2445623379 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54ac004630 con 0x7f54b414e9a0 2024-08-31T12:17:04.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.654+0000 7f54bb332640 1 -- 172.21.15.19:0/2445623379 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54b41e06e0 con 0x7f54b414e9a0 2024-08-31T12:17:04.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.654+0000 7f54bb332640 1 -- 172.21.15.19:0/2445623379 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f54b4155db0 con 0x7f54b414e9a0 2024-08-31T12:17:04.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.655+0000 7f54bb332640 1 --2- 172.21.15.19:0/2445623379 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f54b4156890 0x7f54b41481c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:04.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.655+0000 7f54bb332640 1 -- 172.21.15.19:0/2445623379 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f54b4148700 con 0x7f54b4156890 2024-08-31T12:17:04.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.655+0000 7f54b90ce640 1 --2- 172.21.15.19:0/2445623379 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f54b4156890 0x7f54b41481c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:04.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.655+0000 7f54b90ce640 1 -- 172.21.15.19:0/2445623379 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f54b41e0c80 con 0x7f54b4156890 2024-08-31T12:17:04.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.656+0000 7f54b90ce640 1 --2- 172.21.15.19:0/2445623379 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f54b4156890 0x7f54b41481c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f54b0011a10 tx=0x7f54b0011ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:04.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.656+0000 7f54a9ffb640 1 -- 172.21.15.19:0/2445623379 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b000fce0 con 0x7f54b4156890 2024-08-31T12:17:04.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.656+0000 7f54a9ffb640 1 -- 172.21.15.19:0/2445623379 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f54b000fe80 con 0x7f54b4156890 2024-08-31T12:17:04.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.656+0000 7f54a9ffb640 1 -- 172.21.15.19:0/2445623379 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f54b4156890 msgr2=0x7f54b41481c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:04.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.656+0000 7f54a9ffb640 1 --2- 172.21.15.19:0/2445623379 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f54b4156890 0x7f54b41481c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f54b0011a10 tx=0x7f54b0011ed0 comp rx=0 tx=0).stop 2024-08-31T12:17:04.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.656+0000 7f54a9ffb640 1 -- 172.21.15.19:0/2445623379 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f54b4156890 msgr2=0x7f54b41481c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:17:04.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.656+0000 7f54a9ffb640 1 --2- 172.21.15.19:0/2445623379 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f54b4156890 0x7f54b41481c0 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f54b0011a10 tx=0x7f54b0011ed0 comp rx=0 tx=0).stop 2024-08-31T12:17:04.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.657+0000 7f54a9ffb640 1 -- 172.21.15.19:0/2445623379 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f54ac036400 con 0x7f54b414e9a0 2024-08-31T12:17:04.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.657+0000 7f54a9ffb640 1 --2- 172.21.15.19:0/2445623379 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5498077560 0x7f5498079a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:04.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.657+0000 7f54a9ffb640 1 -- 172.21.15.19:0/2445623379 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f54ac0ba250 con 0x7f54b414e9a0 2024-08-31T12:17:04.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.658+0000 7f54b90ce640 1 --2- 172.21.15.19:0/2445623379 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5498077560 0x7f5498079a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:04.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.658+0000 7f54b90ce640 1 --2- 172.21.15.19:0/2445623379 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5498077560 0x7f5498079a70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f54b000f830 tx=0x7f54b0016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:04.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.669+0000 7f54bb332640 1 --2- 172.21.15.19:0/2445623379 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f54b41ed350 0x7f54b41e15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:04.671 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.669+0000 7f54bb332640 1 -- 172.21.15.19:0/2445623379 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f54b406e760 con 0x7f54b41ed350 2024-08-31T12:17:04.671 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.669+0000 7f54b88cd640 1 --2- 172.21.15.19:0/2445623379 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f54b41ed350 0x7f54b41e15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:04.671 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.670+0000 7f54b88cd640 1 -- 172.21.15.19:0/2445623379 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f54b40cbfd0 con 0x7f54b41ed350 2024-08-31T12:17:04.672 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.670+0000 7f54b88cd640 1 --2- 172.21.15.19:0/2445623379 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f54b41ed350 0x7f54b41e15b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f54b414dc60 tx=0x7f54a400bd90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:04.672 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.670+0000 7f54a9ffb640 1 -- 172.21.15.19:0/2445623379 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a4004320 con 0x7f54b41ed350 2024-08-31T12:17:04.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.670+0000 7f54a9ffb640 1 -- 172.21.15.19:0/2445623379 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5041 (secure 0 0 0) 0x7f54a4004320 con 0x7f54b41ed350 2024-08-31T12:17:04.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.670+0000 7f54a9ffb640 1 -- 172.21.15.19:0/2445623379 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f54b41ed350 msgr2=0x7f54b41e15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:04.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.671+0000 7f54a9ffb640 1 --2- 172.21.15.19:0/2445623379 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f54b41ed350 0x7f54b41e15b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f54b414dc60 tx=0x7f54a400bd90 comp rx=0 tx=0).stop 2024-08-31T12:17:04.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.671+0000 7f54a9ffb640 1 -- 172.21.15.19:0/2445623379 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f54b41ed350 msgr2=0x7f54b41e15b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:17:04.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.671+0000 7f54a9ffb640 1 --2- 172.21.15.19:0/2445623379 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f54b41ed350 0x7f54b41e15b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:04.673 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:17:04.673 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:17:04.673 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:17:04.673 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:17:04.674 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 10, 2024-08-31T12:17:04.674 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 92765, 2024-08-31T12:17:04.674 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:17:04.674 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:17:04.674 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:17:04.674 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:17:04.674 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:04.674 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 4, 2024-08-31T12:17:04.674 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:17:04.674 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:17:04.674 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:17:04.674 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:17:04.674 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:17:04.675 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:17:04.675 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:17:04.675 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:17:04.675 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:17:04.675 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:17:04.675 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:17:04.675 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:17:04.675 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:17:04.675 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:17:04.675 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:04.675 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:17:04.676 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:17:04.676 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:17:04.676 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:17:04.676 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:17:04.676 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:17:04.676 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:17:04.676 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:17:04.676 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:17:04.676 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:17:04.676 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:17:04.676 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:17:04.676 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:17:04.677 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:04.677 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:04.677 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:17:04.677 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:17:04.677 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:17:04.677 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:17:04.677 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:17:04.677 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:17:04.677 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:17:04.677 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:17:04.677 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:17:04.677 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:17:04.677 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:17:04.678 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:17:04.678 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:17:04.678 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:17:04.678 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:17:04.678 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:17:04.678 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:17:04.678 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:17:04.678 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:17:04.678 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:17:04.678 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:17:04.678 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:17:04.678 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:17:04.678 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:17:04.678 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:17:04.678 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:17:04.679 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:17:04.679 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:17:04.679 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:04.679 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:17:04.679 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:04.679 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:17:04.679 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:04.679 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:17:04.679 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:17:04.679 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:17:04.679 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:17:04.679 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:04.679 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:17:04.679 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:17:04.679 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:04.680 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:04.680 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:04.680 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:17:04.680 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:17:04.680 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:04.680 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:04.680 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:04.680 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:04.680 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:17:04.680 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:17:04.680 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:17:04.680 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:17:04.680 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:17:04.680 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:04.681 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:04.681 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:17:04.681 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:17:04.681 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:17:04.681 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:17:04.681 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:04.681 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:17:04.681 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:17:04.681 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:04.681 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:04.681 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:04.681 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:17:04.681 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:17:04.681 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:04.682 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:04.682 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:04.682 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:04.682 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:17:04.682 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:17:04.682 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:17:04.682 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:17:04.682 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:17:04.682 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:04.682 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:04.682 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:17:04.682 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:17:04.682 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:17:04.683 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:17:04.683 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:04.683 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:17:04.683 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:17:04.683 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:04.683 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:04.683 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:04.683 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:17:04.683 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:17:04.683 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:04.683 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:04.683 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:04.683 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:04.683 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:17:04.684 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:17:04.684 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:17:04.684 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:17:04.684 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:17:04.684 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:04.684 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:04.684 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:04.684 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:17:04.684 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:17:04.684 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:04.684 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:17:04.684 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:17:04.684 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:17:04.685 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:04.685 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:04.685 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:17:04.685 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:04.685 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:17:04.685 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:17:04.685 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 3 2024-08-31T12:17:04.685 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:04.685 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:04.685 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:17:04.685 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:04.685 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:17:04.685 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:17:04.685 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 5 2024-08-31T12:17:04.685 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:04.686 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:04.686 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:17:04.686 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:04.686 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:17:04.686 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:17:04.686 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 4 2024-08-31T12:17:04.686 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:04.686 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:04.686 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:17:04.686 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:04.686 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:17:04.686 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:17:04.686 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:17:04.686 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:04.687 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:04.687 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:04.687 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:17:04.687 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:17:04.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.671+0000 7f54bb332640 1 -- 172.21.15.19:0/2445623379 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5498077560 msgr2=0x7f5498079a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:04.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.671+0000 7f54bb332640 1 --2- 172.21.15.19:0/2445623379 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5498077560 0x7f5498079a70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f54b000f830 tx=0x7f54b0016000 comp rx=0 tx=0).stop 2024-08-31T12:17:04.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.671+0000 7f54b98cf640 1 -- 172.21.15.19:0/2445623379 reap_dead start 2024-08-31T12:17:04.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.671+0000 7f54bb332640 1 -- 172.21.15.19:0/2445623379 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f54b414e9a0 msgr2=0x7f54b4147c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:04.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.671+0000 7f54bb332640 1 --2- 172.21.15.19:0/2445623379 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f54b414e9a0 0x7f54b4147c80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f54b40cb890 tx=0x7f54ac003ba0 comp rx=0 tx=0).stop 2024-08-31T12:17:04.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.672+0000 7f54bb332640 1 -- 172.21.15.19:0/2445623379 shutdown_connections 2024-08-31T12:17:04.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.672+0000 7f54bb332640 1 --2- 172.21.15.19:0/2445623379 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f54b414e9a0 0x7f54b4147c80 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:04.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.672+0000 7f54bb332640 1 -- 172.21.15.19:0/2445623379 >> 172.21.15.19:0/2445623379 conn(0x7f54b40ca670 msgr2=0x7f54b414ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:17:04.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.672+0000 7f54bb332640 1 -- 172.21.15.19:0/2445623379 shutdown_connections 2024-08-31T12:17:04.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.672+0000 7f54bb332640 1 -- 172.21.15.19:0/2445623379 wait complete. 2024-08-31T12:17:04.689 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:17:04.689 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-08-31T12:17:04.689 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-08-31T12:17:04.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.863+0000 7f12dd263640 1 -- 172.21.15.19:0/2897823598 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f12d814e700 msgr2=0x7f12d806dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:04.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.863+0000 7f12dd263640 1 --2- 172.21.15.19:0/2897823598 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f12d814e700 0x7f12d806dbf0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f12c8009830 tx=0x7f12c802a480 comp rx=0 tx=0).stop 2024-08-31T12:17:04.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.864+0000 7f12dd263640 1 -- 172.21.15.19:0/2897823598 shutdown_connections 2024-08-31T12:17:04.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.864+0000 7f12dd263640 1 --2- 172.21.15.19:0/2897823598 >> v2:172.21.15.43:3300/0 conn(0x7f12d8160610 0x7f12d8162a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:04.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.864+0000 7f12dd263640 1 --2- 172.21.15.19:0/2897823598 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f12d814e700 0x7f12d806dbf0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:04.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.864+0000 7f12dd263640 1 -- 172.21.15.19:0/2897823598 >> 172.21.15.19:0/2897823598 conn(0x7f12d80cda10 msgr2=0x7f12d80c9e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:17:04.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.864+0000 7f12dd263640 1 -- 172.21.15.19:0/2897823598 shutdown_connections 2024-08-31T12:17:04.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.864+0000 7f12dd263640 1 -- 172.21.15.19:0/2897823598 wait complete. 2024-08-31T12:17:04.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.866+0000 7f12dd263640 1 Processor -- start 2024-08-31T12:17:04.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.867+0000 7f12dd263640 1 -- start start 2024-08-31T12:17:04.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.869+0000 7f12dd263640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12d806e130 0x7f12d8155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:04.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.869+0000 7f12dd263640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f12d814e700 0x7f12d81484d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:04.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.869+0000 7f12dd263640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f12d8160610 0x7f12d8141bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:04.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.869+0000 7f12cffff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f12d814e700 0x7f12d81484d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:04.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.869+0000 7f12d6ffd640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12d806e130 0x7f12d8155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:04.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.869+0000 7f12cffff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f12d814e700 0x7f12d81484d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:50148/0 (socket says 172.21.15.19:50148) 2024-08-31T12:17:04.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.869+0000 7f12cffff640 1 -- 172.21.15.19:0/2611234262 learned_addr learned my addr 172.21.15.19:0/2611234262 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:17:04.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.869+0000 7f12d6ffd640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12d806e130 0x7f12d8155f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:54422/0 (socket says 172.21.15.19:54422) 2024-08-31T12:17:04.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.869+0000 7f12d77fe640 1 --2- 172.21.15.19:0/2611234262 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f12d8160610 0x7f12d8141bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:04.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.869+0000 7f12dd263640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f12d80c49c0 con 0x7f12d814e700 2024-08-31T12:17:04.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.869+0000 7f12dd263640 1 -- 172.21.15.19:0/2611234262 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f12d8158970 con 0x7f12d806e130 2024-08-31T12:17:04.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.870+0000 7f12dd263640 1 -- 172.21.15.19:0/2611234262 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f12d80c4ca0 con 0x7f12d8160610 2024-08-31T12:17:04.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.870+0000 7f12cffff640 1 -- 172.21.15.19:0/2611234262 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f12d8160610 msgr2=0x7f12d8141bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:04.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.870+0000 7f12cffff640 1 --2- 172.21.15.19:0/2611234262 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f12d8160610 0x7f12d8141bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:04.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.870+0000 7f12cffff640 1 -- 172.21.15.19:0/2611234262 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12d806e130 msgr2=0x7f12d8155f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:04.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.870+0000 7f12cffff640 1 --2- 172.21.15.19:0/2611234262 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12d806e130 0x7f12d8155f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:04.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.870+0000 7f12cffff640 1 -- 172.21.15.19:0/2611234262 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12d81f1680 con 0x7f12d814e700 2024-08-31T12:17:04.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.870+0000 7f12cffff640 1 --2- 172.21.15.19:0/2611234262 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f12d814e700 0x7f12d81484d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f12c000c370 tx=0x7f12c000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:04.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.871+0000 7f12cf7fe640 1 -- 172.21.15.19:0/2611234262 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12c00052d0 con 0x7f12d814e700 2024-08-31T12:17:04.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.871+0000 7f12cf7fe640 1 -- 172.21.15.19:0/2611234262 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f12c0007020 con 0x7f12d814e700 2024-08-31T12:17:04.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.872+0000 7f12cf7fe640 1 -- 172.21.15.19:0/2611234262 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12c0007470 con 0x7f12d814e700 2024-08-31T12:17:04.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.872+0000 7f12dd263640 1 -- 172.21.15.19:0/2611234262 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12d81f1960 con 0x7f12d814e700 2024-08-31T12:17:04.875 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.873+0000 7f12dd263640 1 -- 172.21.15.19:0/2611234262 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12d81f1f20 con 0x7f12d814e700 2024-08-31T12:17:04.876 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.874+0000 7f12dd263640 1 -- 172.21.15.19:0/2611234262 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12d8157f10 con 0x7f12d814e700 2024-08-31T12:17:04.876 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.874+0000 7f12cf7fe640 1 -- 172.21.15.19:0/2611234262 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f12c00067d0 con 0x7f12d814e700 2024-08-31T12:17:04.877 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.875+0000 7f12cf7fe640 1 --2- 172.21.15.19:0/2611234262 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f12b8077a80 0x7f12b8079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:04.877 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.875+0000 7f12cf7fe640 1 -- 172.21.15.19:0/2611234262 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f12c0098b10 con 0x7f12d814e700 2024-08-31T12:17:04.877 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.875+0000 7f12d6ffd640 1 --2- 172.21.15.19:0/2611234262 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f12b8077a80 0x7f12b8079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:04.878 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.876+0000 7f12d6ffd640 1 --2- 172.21.15.19:0/2611234262 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f12b8077a80 0x7f12b8079f90 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f12c8034000 tx=0x7f12c8004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:04.886 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:04.884+0000 7f12cf7fe640 1 -- 172.21.15.19:0/2611234262 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f12d8157f10 con 0x7f12d814e700 2024-08-31T12:17:05.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:05.408+0000 7f12dd263640 1 -- 172.21.15.19:0/2611234262 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f12d81440f0 con 0x7f12d814e700 2024-08-31T12:17:05.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:05.409+0000 7f12cf7fe640 1 -- 172.21.15.19:0/2611234262 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f12c0061120 con 0x7f12d814e700 2024-08-31T12:17:05.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:05.412+0000 7f12dd263640 1 -- 172.21.15.19:0/2611234262 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f12b8077a80 msgr2=0x7f12b8079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:05.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:05.412+0000 7f12dd263640 1 --2- 172.21.15.19:0/2611234262 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f12b8077a80 0x7f12b8079f90 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f12c8034000 tx=0x7f12c8004730 comp rx=0 tx=0).stop 2024-08-31T12:17:05.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:05.412+0000 7f12dd263640 1 -- 172.21.15.19:0/2611234262 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f12d814e700 msgr2=0x7f12d81484d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:05.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:05.412+0000 7f12dd263640 1 --2- 172.21.15.19:0/2611234262 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f12d814e700 0x7f12d81484d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f12c000c370 tx=0x7f12c000c830 comp rx=0 tx=0).stop 2024-08-31T12:17:05.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:05.412+0000 7f12dd263640 1 -- 172.21.15.19:0/2611234262 shutdown_connections 2024-08-31T12:17:05.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:05.412+0000 7f12dd263640 1 --2- 172.21.15.19:0/2611234262 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f12b8077a80 0x7f12b8079f90 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:05.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:05.412+0000 7f12dd263640 1 --2- 172.21.15.19:0/2611234262 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f12d8160610 0x7f12d8141bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:05.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:05.412+0000 7f12dd263640 1 --2- 172.21.15.19:0/2611234262 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f12d814e700 0x7f12d81484d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:05.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:05.412+0000 7f12dd263640 1 --2- 172.21.15.19:0/2611234262 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12d806e130 0x7f12d8155f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:05.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:05.412+0000 7f12dd263640 1 -- 172.21.15.19:0/2611234262 >> 172.21.15.19:0/2611234262 conn(0x7f12d80cda10 msgr2=0x7f12d80c68d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:17:05.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:05.412+0000 7f12dd263640 1 -- 172.21.15.19:0/2611234262 shutdown_connections 2024-08-31T12:17:05.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:05.412+0000 7f12dd263640 1 -- 172.21.15.19:0/2611234262 wait complete. 2024-08-31T12:17:05.430 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2024-08-31T12:17:15.431 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-08-31T12:17:15.431 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:17:15.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.555+0000 7f1d89d9b640 1 -- 172.21.15.19:0/728370404 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d7c032850 con 0x7f1d8c0ca7d0 2024-08-31T12:17:15.556 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.555+0000 7f1d91053640 1 -- 172.21.15.19:0/728370404 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d8c0ca7d0 msgr2=0x7f1d8c1531f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:15.556 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.555+0000 7f1d91053640 1 --2- 172.21.15.19:0/728370404 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d8c0ca7d0 0x7f1d8c1531f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c0c9800 tx=0x7f1d7c02a3b0 comp rx=0 tx=0).stop 2024-08-31T12:17:15.556 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.555+0000 7f1d91053640 1 -- 172.21.15.19:0/728370404 shutdown_connections 2024-08-31T12:17:15.556 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.555+0000 7f1d91053640 1 --2- 172.21.15.19:0/728370404 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d8c0ca7d0 0x7f1d8c1531f0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:15.556 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.555+0000 7f1d91053640 1 -- 172.21.15.19:0/728370404 >> 172.21.15.19:0/728370404 conn(0x7f1d8c0cefb0 msgr2=0x7f1d8c0cb340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:17:15.556 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.555+0000 7f1d91053640 1 -- 172.21.15.19:0/728370404 shutdown_connections 2024-08-31T12:17:15.557 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.556+0000 7f1d91053640 1 -- 172.21.15.19:0/728370404 wait complete. 2024-08-31T12:17:15.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.558+0000 7f1d91053640 1 Processor -- start 2024-08-31T12:17:15.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.558+0000 7f1d91053640 1 -- start start 2024-08-31T12:17:15.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.559+0000 7f1d91053640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d8c0ca7d0 0x7f1d8c0b4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:15.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.559+0000 7f1d91053640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d8c0cac90 0x7f1d8c0b28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:15.562 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.559+0000 7f1d91053640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d8c15c4b0 0x7f1d8c06e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:15.562 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.559+0000 7f1d91053640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f1d8c0714f0 con 0x7f1d8c0ca7d0 2024-08-31T12:17:15.562 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.559+0000 7f1d91053640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1d8c0c9bb0 con 0x7f1d8c0cac90 2024-08-31T12:17:15.562 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.559+0000 7f1d91053640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f1d8c15d3a0 con 0x7f1d8c15c4b0 2024-08-31T12:17:15.562 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.559+0000 7f1d8ad9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d8c0ca7d0 0x7f1d8c0b4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:15.562 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.559+0000 7f1d8ad9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d8c0ca7d0 0x7f1d8c0b4010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:44434/0 (socket says 172.21.15.19:44434) 2024-08-31T12:17:15.562 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.559+0000 7f1d8ad9d640 1 -- 172.21.15.19:0/457998158 learned_addr learned my addr 172.21.15.19:0/457998158 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:17:15.562 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.559+0000 7f1d8a59c640 1 --2- 172.21.15.19:0/457998158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d8c0cac90 0x7f1d8c0b28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:15.562 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.559+0000 7f1d8b59e640 1 --2- 172.21.15.19:0/457998158 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d8c15c4b0 0x7f1d8c06e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:15.562 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.560+0000 7f1d8ad9d640 1 -- 172.21.15.19:0/457998158 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d8c15c4b0 msgr2=0x7f1d8c06e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:15.562 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.560+0000 7f1d8ad9d640 1 --2- 172.21.15.19:0/457998158 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d8c15c4b0 0x7f1d8c06e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:15.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.560+0000 7f1d8ad9d640 1 -- 172.21.15.19:0/457998158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d8c0cac90 msgr2=0x7f1d8c0b28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:15.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.560+0000 7f1d8ad9d640 1 --2- 172.21.15.19:0/457998158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d8c0cac90 0x7f1d8c0b28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:15.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.560+0000 7f1d8ad9d640 1 -- 172.21.15.19:0/457998158 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d8c0b2de0 con 0x7f1d8c0ca7d0 2024-08-31T12:17:15.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.560+0000 7f1d8ad9d640 1 --2- 172.21.15.19:0/457998158 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d8c0ca7d0 0x7f1d8c0b4010 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c000c00 tx=0x7f1d7c033ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:15.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.560+0000 7f1d737fe640 1 -- 172.21.15.19:0/457998158 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d7c03f020 con 0x7f1d8c0ca7d0 2024-08-31T12:17:15.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.560+0000 7f1d91053640 1 -- 172.21.15.19:0/457998158 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d8c1f2010 con 0x7f1d8c0ca7d0 2024-08-31T12:17:15.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.560+0000 7f1d91053640 1 -- 172.21.15.19:0/457998158 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d8c06f7b0 con 0x7f1d8c0ca7d0 2024-08-31T12:17:15.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.560+0000 7f1d737fe640 1 -- 172.21.15.19:0/457998158 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1d7c033d90 con 0x7f1d8c0ca7d0 2024-08-31T12:17:15.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.560+0000 7f1d737fe640 1 -- 172.21.15.19:0/457998158 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d7c037040 con 0x7f1d8c0ca7d0 2024-08-31T12:17:15.564 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.562+0000 7f1d91053640 1 -- 172.21.15.19:0/457998158 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d8c072120 con 0x7f1d8c0ca7d0 2024-08-31T12:17:15.564 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.563+0000 7f1d737fe640 1 -- 172.21.15.19:0/457998158 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f1d7c042020 con 0x7f1d8c0ca7d0 2024-08-31T12:17:15.564 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.563+0000 7f1d737fe640 1 --2- 172.21.15.19:0/457998158 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1d78077c80 0x7f1d7807a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:15.564 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.563+0000 7f1d737fe640 1 -- 172.21.15.19:0/457998158 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f1d7c0035f0 con 0x7f1d8c0ca7d0 2024-08-31T12:17:15.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.564+0000 7f1d8a59c640 1 --2- 172.21.15.19:0/457998158 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1d78077c80 0x7f1d7807a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:15.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.564+0000 7f1d8a59c640 1 --2- 172.21.15.19:0/457998158 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1d78077c80 0x7f1d7807a190 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c0c4540 tx=0x7f1d80007cc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:15.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.568+0000 7f1d737fe640 1 -- 172.21.15.19:0/457998158 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f1d7c049070 con 0x7f1d8c0ca7d0 2024-08-31T12:17:15.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.826+0000 7f1d91053640 1 -- 172.21.15.19:0/457998158 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1d8c0ca160 con 0x7f1d8c0ca7d0 2024-08-31T12:17:15.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.826+0000 7f1d737fe640 1 -- 172.21.15.19:0/457998158 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f1d7c081910 con 0x7f1d8c0ca7d0 2024-08-31T12:17:15.827 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:17:15.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.828+0000 7f1d91053640 1 -- 172.21.15.19:0/457998158 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1d78077c80 msgr2=0x7f1d7807a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:15.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.828+0000 7f1d91053640 1 --2- 172.21.15.19:0/457998158 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1d78077c80 0x7f1d7807a190 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c0c4540 tx=0x7f1d80007cc0 comp rx=0 tx=0).stop 2024-08-31T12:17:15.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.828+0000 7f1d91053640 1 -- 172.21.15.19:0/457998158 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d8c0ca7d0 msgr2=0x7f1d8c0b4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:15.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.828+0000 7f1d91053640 1 --2- 172.21.15.19:0/457998158 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d8c0ca7d0 0x7f1d8c0b4010 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c000c00 tx=0x7f1d7c033ba0 comp rx=0 tx=0).stop 2024-08-31T12:17:15.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.829+0000 7f1d91053640 1 -- 172.21.15.19:0/457998158 shutdown_connections 2024-08-31T12:17:15.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.829+0000 7f1d91053640 1 --2- 172.21.15.19:0/457998158 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1d78077c80 0x7f1d7807a190 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:15.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.829+0000 7f1d91053640 1 --2- 172.21.15.19:0/457998158 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d8c15c4b0 0x7f1d8c06e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:15.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.829+0000 7f1d91053640 1 --2- 172.21.15.19:0/457998158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d8c0cac90 0x7f1d8c0b28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:15.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.829+0000 7f1d91053640 1 --2- 172.21.15.19:0/457998158 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d8c0ca7d0 0x7f1d8c0b4010 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:15.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.829+0000 7f1d91053640 1 -- 172.21.15.19:0/457998158 >> 172.21.15.19:0/457998158 conn(0x7f1d8c0cefb0 msgr2=0x7f1d8c149450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:17:15.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.829+0000 7f1d91053640 1 -- 172.21.15.19:0/457998158 shutdown_connections 2024-08-31T12:17:15.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.829+0000 7f1d91053640 1 -- 172.21.15.19:0/457998158 wait complete. 2024-08-31T12:17:15.841 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-08-31T12:17:15.842 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-08-31T12:17:15.842 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-08-31T12:17:15.842 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-31T12:17:15.959 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.957+0000 7fad36532640 1 -- 172.21.15.19:0/1353842047 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fad301532f0 msgr2=0x7fad3015c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:15.959 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.957+0000 7fad36532640 1 --2- 172.21.15.19:0/1353842047 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fad301532f0 0x7fad3015c4b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fad30072450 tx=0x7fad2402a560 comp rx=0 tx=0).stop 2024-08-31T12:17:15.959 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.958+0000 7fad36532640 1 -- 172.21.15.19:0/1353842047 shutdown_connections 2024-08-31T12:17:15.959 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.958+0000 7fad36532640 1 --2- 172.21.15.19:0/1353842047 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fad300c9880 0x7fad3015c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:15.959 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.958+0000 7fad36532640 1 --2- 172.21.15.19:0/1353842047 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fad301532f0 0x7fad3015c4b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:15.959 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.958+0000 7fad36532640 1 --2- 172.21.15.19:0/1353842047 >> v2:172.21.15.43:3300/0 conn(0x7fad300ca7f0 0x7fad30071bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:15.959 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.958+0000 7fad36532640 1 -- 172.21.15.19:0/1353842047 >> 172.21.15.19:0/1353842047 conn(0x7fad300cefb0 msgr2=0x7fad300cb340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:17:15.959 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.958+0000 7fad36532640 1 -- 172.21.15.19:0/1353842047 shutdown_connections 2024-08-31T12:17:15.959 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.958+0000 7fad36532640 1 -- 172.21.15.19:0/1353842047 wait complete. 2024-08-31T12:17:15.961 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.960+0000 7fad36532640 1 Processor -- start 2024-08-31T12:17:15.961 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.961+0000 7fad36532640 1 -- start start 2024-08-31T12:17:15.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.963+0000 7fad36532640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fad300c9880 0x7fad3014b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:15.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.963+0000 7fad36532640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fad300ca7f0 0x7fad300b36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:15.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.963+0000 7fad36532640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fad301532f0 0x7fad3014bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:15.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.963+0000 7fad2f7fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fad300ca7f0 0x7fad300b36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:15.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.963+0000 7fad2f7fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fad300ca7f0 0x7fad300b36a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:44442/0 (socket says 172.21.15.19:44442) 2024-08-31T12:17:15.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.963+0000 7fad2f7fe640 1 -- 172.21.15.19:0/2795484978 learned_addr learned my addr 172.21.15.19:0/2795484978 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:17:15.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.963+0000 7fad36532640 1 -- 172.21.15.19:0/2795484978 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fad3015d330 con 0x7fad300ca7f0 2024-08-31T12:17:15.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.963+0000 7fad36532640 1 -- 172.21.15.19:0/2795484978 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fad30072110 con 0x7fad300c9880 2024-08-31T12:17:15.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.963+0000 7fad36532640 1 -- 172.21.15.19:0/2795484978 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fad300c3860 con 0x7fad301532f0 2024-08-31T12:17:15.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.963+0000 7fad2ffff640 1 --2- 172.21.15.19:0/2795484978 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fad300c9880 0x7fad3014b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:15.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.963+0000 7fad34acf640 1 --2- 172.21.15.19:0/2795484978 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fad301532f0 0x7fad3014bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:15.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.964+0000 7fad2f7fe640 1 -- 172.21.15.19:0/2795484978 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fad301532f0 msgr2=0x7fad3014bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:15.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.964+0000 7fad2f7fe640 1 --2- 172.21.15.19:0/2795484978 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fad301532f0 0x7fad3014bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:15.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.964+0000 7fad2f7fe640 1 -- 172.21.15.19:0/2795484978 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fad300c9880 msgr2=0x7fad3014b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:15.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.964+0000 7fad2f7fe640 1 --2- 172.21.15.19:0/2795484978 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fad300c9880 0x7fad3014b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:15.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.964+0000 7fad2f7fe640 1 -- 172.21.15.19:0/2795484978 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad3014b5b0 con 0x7fad300ca7f0 2024-08-31T12:17:15.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.964+0000 7fad2f7fe640 1 --2- 172.21.15.19:0/2795484978 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fad300ca7f0 0x7fad300b36a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fad240099c0 tx=0x7fad240038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:15.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.964+0000 7fad2cff9640 1 -- 172.21.15.19:0/2795484978 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad2403e030 con 0x7fad300ca7f0 2024-08-31T12:17:15.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.964+0000 7fad36532640 1 -- 172.21.15.19:0/2795484978 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad300c8a00 con 0x7fad300ca7f0 2024-08-31T12:17:15.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.964+0000 7fad2cff9640 1 -- 172.21.15.19:0/2795484978 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fad24003ca0 con 0x7fad300ca7f0 2024-08-31T12:17:15.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.964+0000 7fad2cff9640 1 -- 172.21.15.19:0/2795484978 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad24035040 con 0x7fad300ca7f0 2024-08-31T12:17:15.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.964+0000 7fad36532640 1 -- 172.21.15.19:0/2795484978 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad3014cfc0 con 0x7fad300ca7f0 2024-08-31T12:17:15.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.967+0000 7fad36532640 1 --2- 172.21.15.19:0/2795484978 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fad3014c490 0x7fad30144100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:15.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.967+0000 7fad36532640 1 -- 172.21.15.19:0/2795484978 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fad301f0870 con 0x7fad3014c490 2024-08-31T12:17:15.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.967+0000 7fad2cff9640 1 -- 172.21.15.19:0/2795484978 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fad24041030 con 0x7fad300ca7f0 2024-08-31T12:17:15.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.967+0000 7fad2ffff640 1 --2- 172.21.15.19:0/2795484978 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fad3014c490 0x7fad30144100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:15.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.967+0000 7fad2ffff640 1 -- 172.21.15.19:0/2795484978 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fad301568e0 con 0x7fad3014c490 2024-08-31T12:17:15.969 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.968+0000 7fad2ffff640 1 --2- 172.21.15.19:0/2795484978 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fad3014c490 0x7fad30144100 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fad20011a10 tx=0x7fad20011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:15.969 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.968+0000 7fad2cff9640 1 --2- 172.21.15.19:0/2795484978 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fad1c077a50 0x7fad1c079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:15.969 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.968+0000 7fad2cff9640 1 -- 172.21.15.19:0/2795484978 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fad240b9f70 con 0x7fad300ca7f0 2024-08-31T12:17:15.969 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.968+0000 7fad2cff9640 1 -- 172.21.15.19:0/2795484978 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad2000fc20 con 0x7fad3014c490 2024-08-31T12:17:15.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.968+0000 7fad2cff9640 1 -- 172.21.15.19:0/2795484978 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fad2000fc20 con 0x7fad3014c490 2024-08-31T12:17:15.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.968+0000 7fad34acf640 1 --2- 172.21.15.19:0/2795484978 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fad1c077a50 0x7fad1c079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:15.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.968+0000 7fad2cff9640 1 -- 172.21.15.19:0/2795484978 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fad3014c490 msgr2=0x7fad30144100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:15.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.968+0000 7fad2cff9640 1 --2- 172.21.15.19:0/2795484978 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fad3014c490 0x7fad30144100 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fad20011a10 tx=0x7fad20011ed0 comp rx=0 tx=0).stop 2024-08-31T12:17:15.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.968+0000 7fad2cff9640 1 -- 172.21.15.19:0/2795484978 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fad3014c490 msgr2=0x7fad30144100 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:17:15.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.968+0000 7fad2cff9640 1 --2- 172.21.15.19:0/2795484978 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fad3014c490 0x7fad30144100 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fad20011a10 tx=0x7fad20011ed0 comp rx=0 tx=0).stop 2024-08-31T12:17:15.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.969+0000 7fad34acf640 1 --2- 172.21.15.19:0/2795484978 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fad1c077a50 0x7fad1c079f60 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fad300b4390 tx=0x7fad28009500 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:15.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.981+0000 7fad36532640 1 --2- 172.21.15.19:0/2795484978 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fad30144640 0x7fad301f2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:15.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.981+0000 7fad36532640 1 -- 172.21.15.19:0/2795484978 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fad300ca180 con 0x7fad30144640 2024-08-31T12:17:15.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.981+0000 7fad2ffff640 1 --2- 172.21.15.19:0/2795484978 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fad30144640 0x7fad301f2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:15.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.981+0000 7fad2ffff640 1 -- 172.21.15.19:0/2795484978 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fad300cb4c0 con 0x7fad30144640 2024-08-31T12:17:15.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.981+0000 7fad2ffff640 1 --2- 172.21.15.19:0/2795484978 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fad30144640 0x7fad301f2640 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fad20011a10 tx=0x7fad200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:15.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.982+0000 7fad2cff9640 1 -- 172.21.15.19:0/2795484978 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad200054c0 con 0x7fad30144640 2024-08-31T12:17:15.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.982+0000 7fad2cff9640 1 -- 172.21.15.19:0/2795484978 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4543 (secure 0 0 0) 0x7fad200054c0 con 0x7fad30144640 2024-08-31T12:17:15.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.982+0000 7fad2cff9640 1 -- 172.21.15.19:0/2795484978 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fad30144640 msgr2=0x7fad301f2640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:15.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.982+0000 7fad2cff9640 1 --2- 172.21.15.19:0/2795484978 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fad30144640 0x7fad301f2640 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fad20011a10 tx=0x7fad200046c0 comp rx=0 tx=0).stop 2024-08-31T12:17:15.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.982+0000 7fad2cff9640 1 -- 172.21.15.19:0/2795484978 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fad30144640 msgr2=0x7fad301f2640 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:17:15.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.982+0000 7fad2cff9640 1 --2- 172.21.15.19:0/2795484978 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fad30144640 0x7fad301f2640 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fad20011a10 tx=0x7fad200046c0 comp rx=0 tx=0).stop 2024-08-31T12:17:15.986 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:17:15.986 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:17:15.986 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:17:15.986 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "leader", 2024-08-31T12:17:15.986 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 10, 2024-08-31T12:17:15.987 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 15973, 2024-08-31T12:17:15.987 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:17:15.987 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:17:15.987 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:17:15.987 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:17:15.987 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:15.987 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 15, 2024-08-31T12:17:15.987 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:17:15.987 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:17:15.987 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:17:15.987 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:17:15.987 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:17:15.987 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:17:15.988 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:17:15.988 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:17:15.988 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:17:15.988 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:17:15.988 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:17:15.988 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:17:15.988 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:17:15.988 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:17:15.988 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:15.988 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:17:15.988 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:17:15.988 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:17:15.988 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:17:15.989 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:17:15.989 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:17:15.989 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:17:15.989 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:17:15.989 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:17:15.989 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:17:15.989 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:17:15.989 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:17:15.989 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:17:15.989 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:15.989 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:15.989 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:17:15.989 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:17:15.989 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:17:15.990 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:17:15.990 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:17:15.990 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:17:15.990 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:17:15.990 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:17:15.990 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:17:15.990 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:17:15.990 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:17:15.991 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:17:15.991 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:17:15.991 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:17:15.991 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:17:15.991 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:17:15.991 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:17:15.991 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:17:15.991 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:17:15.991 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:17:15.991 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:17:15.991 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:17:15.991 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:17:15.991 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:17:15.991 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:17:15.991 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:17:15.992 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:17:15.992 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:17:15.992 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:15.992 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:17:15.992 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:15.992 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:17:15.992 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:15.992 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:17:15.992 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:17:15.992 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:17:15.992 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:17:15.992 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:15.992 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:17:15.992 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:17:15.993 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:15.993 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:15.993 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:15.993 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:17:15.993 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:17:15.993 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:15.993 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:15.993 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:15.993 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:15.993 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:17:15.993 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:17:15.993 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:17:15.993 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:17:15.994 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:17:15.994 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:15.994 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:15.994 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:17:15.994 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:17:15.994 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:17:15.994 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:17:15.994 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:15.994 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:17:15.994 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:17:15.994 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:15.994 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:15.994 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:15.995 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:17:15.995 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:17:15.995 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:15.995 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:15.995 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:15.995 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:15.995 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:17:15.995 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:17:15.995 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:17:15.995 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:17:15.995 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:17:15.995 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:15.995 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:15.995 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:17:15.996 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:17:15.996 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:17:15.996 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:17:15.996 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:15.996 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:17:15.996 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:17:15.996 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:15.996 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:15.996 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:15.996 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:17:15.996 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:17:15.996 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:15.996 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:15.997 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:15.997 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:15.997 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:17:15.997 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:17:15.997 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:17:15.997 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:17:15.997 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:17:15.997 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:15.997 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:15.997 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:15.997 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:17:15.997 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:17:15.997 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:15.997 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:17:15.998 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:17:15.998 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:17:15.998 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:15.998 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:15.998 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:17:15.998 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:15.998 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:17:15.998 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:17:15.998 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:17:15.998 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:15.998 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:15.998 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:15.998 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:17:15.998 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:17:16.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.982+0000 7fad36532640 1 -- 172.21.15.19:0/2795484978 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fad1c077a50 msgr2=0x7fad1c079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:16.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.982+0000 7fad36532640 1 --2- 172.21.15.19:0/2795484978 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fad1c077a50 0x7fad1c079f60 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fad300b4390 tx=0x7fad28009500 comp rx=0 tx=0).stop 2024-08-31T12:17:16.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.982+0000 7fad36532640 1 -- 172.21.15.19:0/2795484978 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fad300ca7f0 msgr2=0x7fad300b36a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:16.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.982+0000 7fad36532640 1 --2- 172.21.15.19:0/2795484978 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fad300ca7f0 0x7fad300b36a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fad240099c0 tx=0x7fad240038a0 comp rx=0 tx=0).stop 2024-08-31T12:17:16.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.982+0000 7fad34acf640 1 -- 172.21.15.19:0/2795484978 reap_dead start 2024-08-31T12:17:16.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.982+0000 7fad34acf640 1 -- 172.21.15.19:0/2795484978 reap_dead start 2024-08-31T12:17:16.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.982+0000 7fad36532640 1 -- 172.21.15.19:0/2795484978 shutdown_connections 2024-08-31T12:17:16.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.983+0000 7fad36532640 1 -- 172.21.15.19:0/2795484978 >> 172.21.15.19:0/2795484978 conn(0x7fad300cefb0 msgr2=0x7fad30151640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:17:16.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.983+0000 7fad36532640 1 -- 172.21.15.19:0/2795484978 shutdown_connections 2024-08-31T12:17:16.001 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:15.983+0000 7fad36532640 1 -- 172.21.15.19:0/2795484978 wait complete. 2024-08-31T12:17:16.001 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:17:16.001 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-08-31T12:17:16.001 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-31T12:17:16.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.117+0000 7fd4c88dc640 1 -- 172.21.15.19:0/3883021349 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4c40ce050 msgr2=0x7fd4c4074290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:16.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.117+0000 7fd4c88dc640 1 --2- 172.21.15.19:0/3883021349 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4c40ce050 0x7fd4c4074290 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd4b4009830 tx=0x7fd4b402a4a0 comp rx=0 tx=0).stop 2024-08-31T12:17:16.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.117+0000 7fd4c88dc640 1 -- 172.21.15.19:0/3883021349 shutdown_connections 2024-08-31T12:17:16.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.117+0000 7fd4c88dc640 1 --2- 172.21.15.19:0/3883021349 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd4c40747d0 0x7fd4c40c6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:16.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.117+0000 7fd4c88dc640 1 --2- 172.21.15.19:0/3883021349 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4c40ce050 0x7fd4c4074290 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:16.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.117+0000 7fd4c88dc640 1 -- 172.21.15.19:0/3883021349 >> 172.21.15.19:0/3883021349 conn(0x7fd4c40ca670 msgr2=0x7fd4c40cd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:17:16.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.117+0000 7fd4c88dc640 1 -- 172.21.15.19:0/3883021349 shutdown_connections 2024-08-31T12:17:16.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.117+0000 7fd4c88dc640 1 -- 172.21.15.19:0/3883021349 wait complete. 2024-08-31T12:17:16.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.119+0000 7fd4c88dc640 1 Processor -- start 2024-08-31T12:17:16.120 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.119+0000 7fd4c88dc640 1 -- start start 2024-08-31T12:17:16.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.120+0000 7fd4c88dc640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd4c40747d0 0x7fd4c41e1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:16.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.120+0000 7fd4c88dc640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4c40c70e0 0x7fd4c4148ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:16.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.120+0000 7fd4c88dc640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd4c40ce050 0x7fd4c4149400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:16.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.120+0000 7fd4c88dc640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c40cd130 con 0x7fd4c40ce050 2024-08-31T12:17:16.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.120+0000 7fd4c88dc640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c4073d70 con 0x7fd4c40c70e0 2024-08-31T12:17:16.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.120+0000 7fd4c88dc640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fd4c406eb40 con 0x7fd4c40747d0 2024-08-31T12:17:16.122 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.120+0000 7fd4c2d9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd4c40ce050 0x7fd4c4149400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:16.122 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.120+0000 7fd4c2d9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd4c40ce050 0x7fd4c4149400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:44470/0 (socket says 172.21.15.19:44470) 2024-08-31T12:17:16.122 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.120+0000 7fd4c259c640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd4c40747d0 0x7fd4c41e1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:16.122 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.120+0000 7fd4c2d9d640 1 -- 172.21.15.19:0/4225615677 learned_addr learned my addr 172.21.15.19:0/4225615677 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:17:16.122 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.120+0000 7fd4c1d9b640 1 --2- 172.21.15.19:0/4225615677 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4c40c70e0 0x7fd4c4148ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:16.123 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.121+0000 7fd4c1d9b640 1 -- 172.21.15.19:0/4225615677 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd4c40747d0 msgr2=0x7fd4c41e1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:16.123 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.121+0000 7fd4c1d9b640 1 --2- 172.21.15.19:0/4225615677 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd4c40747d0 0x7fd4c41e1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:16.123 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.121+0000 7fd4c1d9b640 1 -- 172.21.15.19:0/4225615677 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd4c40ce050 msgr2=0x7fd4c4149400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:16.123 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.121+0000 7fd4c1d9b640 1 --2- 172.21.15.19:0/4225615677 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd4c40ce050 0x7fd4c4149400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:16.123 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.121+0000 7fd4c1d9b640 1 -- 172.21.15.19:0/4225615677 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4c414a670 con 0x7fd4c40c70e0 2024-08-31T12:17:16.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.121+0000 7fd4c259c640 1 --2- 172.21.15.19:0/4225615677 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd4c40747d0 0x7fd4c41e1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:17:16.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.121+0000 7fd4c2d9d640 1 --2- 172.21.15.19:0/4225615677 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd4c40ce050 0x7fd4c4149400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:17:16.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.121+0000 7fd4c1d9b640 1 --2- 172.21.15.19:0/4225615677 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4c40c70e0 0x7fd4c4148ec0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd4b800e6d0 tx=0x7fd4b800eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:16.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.121+0000 7fd4b2ffd640 1 -- 172.21.15.19:0/4225615677 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b8004220 con 0x7fd4c40c70e0 2024-08-31T12:17:16.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.121+0000 7fd4b2ffd640 1 -- 172.21.15.19:0/4225615677 <== mon.1 v2:172.21.15.43:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd4b80043c0 con 0x7fd4c40c70e0 2024-08-31T12:17:16.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.121+0000 7fd4c88dc640 1 -- 172.21.15.19:0/4225615677 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4c41dfaf0 con 0x7fd4c40c70e0 2024-08-31T12:17:16.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.121+0000 7fd4b2ffd640 1 -- 172.21.15.19:0/4225615677 <== mon.1 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b800cb10 con 0x7fd4c40c70e0 2024-08-31T12:17:16.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.122+0000 7fd4c88dc640 1 -- 172.21.15.19:0/4225615677 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4c41e00b0 con 0x7fd4c40c70e0 2024-08-31T12:17:16.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.123+0000 7fd4c88dc640 1 --2- 172.21.15.19:0/4225615677 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4c41e08b0 0x7fd4c41f2710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:16.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.123+0000 7fd4c88dc640 1 -- 172.21.15.19:0/4225615677 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c41e0280 con 0x7fd4c41e08b0 2024-08-31T12:17:16.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.123+0000 7fd4c259c640 1 --2- 172.21.15.19:0/4225615677 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4c41e08b0 0x7fd4c41f2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:16.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.124+0000 7fd4c259c640 1 -- 172.21.15.19:0/4225615677 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd4c40c69a0 con 0x7fd4c41e08b0 2024-08-31T12:17:16.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.124+0000 7fd4b2ffd640 1 -- 172.21.15.19:0/4225615677 <== mon.1 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fd4b800ccb0 con 0x7fd4c40c70e0 2024-08-31T12:17:16.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.124+0000 7fd4c259c640 1 --2- 172.21.15.19:0/4225615677 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4c41e08b0 0x7fd4c41f2710 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd4c406dcb0 tx=0x7fd4b4003f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:16.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.125+0000 7fd4b2ffd640 1 --2- 172.21.15.19:0/4225615677 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd4a4077a50 0x7fd4a4079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:16.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.125+0000 7fd4c2d9d640 1 --2- 172.21.15.19:0/4225615677 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd4a4077a50 0x7fd4a4079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:16.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.125+0000 7fd4b2ffd640 1 -- 172.21.15.19:0/4225615677 <== mon.1 v2:172.21.15.43:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fd4b8099900 con 0x7fd4c40c70e0 2024-08-31T12:17:16.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.125+0000 7fd4b2ffd640 1 -- 172.21.15.19:0/4225615677 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b4045640 con 0x7fd4c41e08b0 2024-08-31T12:17:16.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.125+0000 7fd4b2ffd640 1 -- 172.21.15.19:0/4225615677 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd4b40457e0 con 0x7fd4c41e08b0 2024-08-31T12:17:16.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.125+0000 7fd4b2ffd640 1 -- 172.21.15.19:0/4225615677 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4c41e08b0 msgr2=0x7fd4c41f2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:16.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.125+0000 7fd4b2ffd640 1 --2- 172.21.15.19:0/4225615677 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4c41e08b0 0x7fd4c41f2710 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd4c406dcb0 tx=0x7fd4b4003f20 comp rx=0 tx=0).stop 2024-08-31T12:17:16.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.125+0000 7fd4b2ffd640 1 -- 172.21.15.19:0/4225615677 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4c41e08b0 msgr2=0x7fd4c41f2710 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:17:16.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.125+0000 7fd4b2ffd640 1 --2- 172.21.15.19:0/4225615677 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4c41e08b0 0x7fd4c41f2710 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:16.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.125+0000 7fd4c2d9d640 1 --2- 172.21.15.19:0/4225615677 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd4a4077a50 0x7fd4a4079f60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc0015a0 tx=0x7fd4bc0020f0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:16.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.132+0000 7fd4c88dc640 1 --2- 172.21.15.19:0/4225615677 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4c41e1780 0x7fd4c41f2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:16.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.132+0000 7fd4c88dc640 1 -- 172.21.15.19:0/4225615677 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c40c88d0 con 0x7fd4c41e1780 2024-08-31T12:17:16.133 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.132+0000 7fd4c259c640 1 --2- 172.21.15.19:0/4225615677 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4c41e1780 0x7fd4c41f2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:16.133 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.132+0000 7fd4c259c640 1 -- 172.21.15.19:0/4225615677 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd4c4158f70 con 0x7fd4c41e1780 2024-08-31T12:17:16.133 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.133+0000 7fd4c259c640 1 --2- 172.21.15.19:0/4225615677 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4c41e1780 0x7fd4c41f2c50 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd4b4000c00 tx=0x7fd4b4005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:16.133 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.133+0000 7fd4b2ffd640 1 -- 172.21.15.19:0/4225615677 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b4045640 con 0x7fd4c41e1780 2024-08-31T12:17:16.134 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.133+0000 7fd4b2ffd640 1 -- 172.21.15.19:0/4225615677 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5043 (secure 0 0 0) 0x7fd4b4005220 con 0x7fd4c41e1780 2024-08-31T12:17:16.134 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.133+0000 7fd4b2ffd640 1 -- 172.21.15.19:0/4225615677 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4c41e1780 msgr2=0x7fd4c41f2c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:16.134 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.133+0000 7fd4b2ffd640 1 --2- 172.21.15.19:0/4225615677 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4c41e1780 0x7fd4c41f2c50 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd4b4000c00 tx=0x7fd4b4005f90 comp rx=0 tx=0).stop 2024-08-31T12:17:16.134 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.133+0000 7fd4b2ffd640 1 -- 172.21.15.19:0/4225615677 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4c41e1780 msgr2=0x7fd4c41f2c50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:17:16.135 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:17:16.135 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:17:16.135 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:17:16.135 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:17:16.135 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 10, 2024-08-31T12:17:16.135 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 104228, 2024-08-31T12:17:16.135 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:17:16.135 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:17:16.135 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:17:16.135 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:17:16.136 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:16.136 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 15, 2024-08-31T12:17:16.136 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:17:16.136 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:17:16.136 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:17:16.136 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:17:16.136 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:17:16.136 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:17:16.136 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:17:16.136 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:17:16.136 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:17:16.136 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:17:16.136 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:17:16.137 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:17:16.137 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:17:16.137 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:17:16.137 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:16.137 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:17:16.137 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:17:16.137 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:17:16.137 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:17:16.137 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:17:16.137 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:17:16.137 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:17:16.137 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:17:16.137 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:17:16.137 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:17:16.138 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:17:16.138 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:17:16.138 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:17:16.138 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:16.138 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:16.138 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:17:16.138 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:17:16.138 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:17:16.138 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:17:16.138 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:17:16.138 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:17:16.138 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:17:16.138 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:17:16.139 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:17:16.139 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:17:16.139 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:17:16.139 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:17:16.139 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:17:16.139 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:17:16.139 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:17:16.139 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:17:16.139 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:17:16.139 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:17:16.139 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:17:16.139 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:17:16.139 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:17:16.139 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:17:16.139 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:17:16.140 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:17:16.140 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:17:16.140 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:17:16.140 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:17:16.140 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:17:16.140 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:16.140 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:17:16.140 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:16.140 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:17:16.140 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:16.140 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:17:16.140 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:17:16.140 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:17:16.141 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:17:16.141 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:16.141 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:17:16.141 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:17:16.141 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:16.141 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:16.141 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:16.141 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:17:16.141 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:17:16.141 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:16.141 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:16.141 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:16.141 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:16.142 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:17:16.142 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:17:16.142 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:17:16.142 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:17:16.142 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:17:16.142 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:16.142 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:16.142 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:17:16.142 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:17:16.142 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:17:16.142 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:17:16.142 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:16.142 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:17:16.142 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:17:16.143 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:16.143 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:16.143 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:16.143 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:17:16.143 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:17:16.143 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:16.143 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:16.143 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:16.143 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:16.143 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:17:16.143 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:17:16.143 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:17:16.143 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:17:16.144 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:17:16.144 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:16.144 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:16.144 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:17:16.144 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:17:16.144 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:17:16.144 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:17:16.144 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:16.144 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:17:16.144 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:17:16.144 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:16.144 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:16.144 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:16.144 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:17:16.144 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:17:16.145 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:16.145 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:16.145 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:16.145 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:16.145 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:17:16.145 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:17:16.145 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:17:16.145 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:17:16.145 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:17:16.145 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:16.145 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:16.145 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:16.145 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:17:16.146 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:17:16.146 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:16.146 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:17:16.146 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:17:16.146 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:17:16.146 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:16.146 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:16.146 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:17:16.146 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:16.146 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:17:16.146 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:17:16.146 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 3 2024-08-31T12:17:16.146 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:16.146 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:16.146 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:17:16.147 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:16.147 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:17:16.147 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:17:16.147 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 3 2024-08-31T12:17:16.147 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:16.147 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:16.147 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:17:16.147 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:16.147 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:17:16.147 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:17:16.147 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 5 2024-08-31T12:17:16.147 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:16.147 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:16.147 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:17:16.148 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:16.148 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:17:16.148 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:17:16.148 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:17:16.148 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:16.148 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:16.148 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:16.148 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:17:16.148 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:17:16.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.133+0000 7fd4b2ffd640 1 --2- 172.21.15.19:0/4225615677 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4c41e1780 0x7fd4c41f2c50 secure :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd4b4000c00 tx=0x7fd4b4005f90 comp rx=0 tx=0).stop 2024-08-31T12:17:16.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.134+0000 7fd4c88dc640 1 -- 172.21.15.19:0/4225615677 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd4a4077a50 msgr2=0x7fd4a4079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:16.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.134+0000 7fd4c88dc640 1 --2- 172.21.15.19:0/4225615677 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd4a4077a50 0x7fd4a4079f60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc0015a0 tx=0x7fd4bc0020f0 comp rx=0 tx=0).stop 2024-08-31T12:17:16.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.134+0000 7fd4c88dc640 1 -- 172.21.15.19:0/4225615677 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4c40c70e0 msgr2=0x7fd4c4148ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:16.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.134+0000 7fd4c88dc640 1 --2- 172.21.15.19:0/4225615677 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4c40c70e0 0x7fd4c4148ec0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd4b800e6d0 tx=0x7fd4b800eb90 comp rx=0 tx=0).stop 2024-08-31T12:17:16.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.134+0000 7fd4c2d9d640 1 -- 172.21.15.19:0/4225615677 reap_dead start 2024-08-31T12:17:16.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.134+0000 7fd4c2d9d640 1 -- 172.21.15.19:0/4225615677 reap_dead start 2024-08-31T12:17:16.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.134+0000 7fd4c88dc640 1 -- 172.21.15.19:0/4225615677 shutdown_connections 2024-08-31T12:17:16.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.134+0000 7fd4c88dc640 1 -- 172.21.15.19:0/4225615677 >> 172.21.15.19:0/4225615677 conn(0x7fd4c40ca670 msgr2=0x7fd4c4152950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:17:16.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.134+0000 7fd4c88dc640 1 -- 172.21.15.19:0/4225615677 shutdown_connections 2024-08-31T12:17:16.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.134+0000 7fd4c88dc640 1 -- 172.21.15.19:0/4225615677 wait complete. 2024-08-31T12:17:16.151 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:17:16.151 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-08-31T12:17:16.151 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-31T12:17:16.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.300+0000 7f5428820640 1 -- 172.21.15.19:0/887511738 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5424153190 msgr2=0x7f5424159820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:16.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.300+0000 7f5428820640 1 --2- 172.21.15.19:0/887511738 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5424153190 0x7f5424159820 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f542415d2b0 tx=0x7f541402a690 comp rx=0 tx=0).stop 2024-08-31T12:17:16.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.300+0000 7f5428820640 1 -- 172.21.15.19:0/887511738 shutdown_connections 2024-08-31T12:17:16.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.300+0000 7f5428820640 1 --2- 172.21.15.19:0/887511738 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5424153190 0x7f5424159820 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:16.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.300+0000 7f5428820640 1 -- 172.21.15.19:0/887511738 >> 172.21.15.19:0/887511738 conn(0x7f54240728b0 msgr2=0x7f5424072cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:17:16.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.300+0000 7f5428820640 1 -- 172.21.15.19:0/887511738 shutdown_connections 2024-08-31T12:17:16.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.300+0000 7f5428820640 1 -- 172.21.15.19:0/887511738 wait complete. 2024-08-31T12:17:16.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.302+0000 7f5428820640 1 Processor -- start 2024-08-31T12:17:16.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.302+0000 7f5428820640 1 -- start start 2024-08-31T12:17:16.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.303+0000 7f5428820640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54240c9030 0x7f54241515e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:16.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.303+0000 7f5428820640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f54240c9440 0x7f54240c3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:16.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.303+0000 7f5428820640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5424153190 0x7f542406f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:16.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.303+0000 7f5428820640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f54240c4db0 con 0x7f54240c9440 2024-08-31T12:17:16.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.303+0000 7f5428820640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5424159d60 con 0x7f54240c9030 2024-08-31T12:17:16.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.303+0000 7f5428820640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f54241535a0 con 0x7f5424153190 2024-08-31T12:17:16.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.304+0000 7f5419d9b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f54240c9440 0x7f54240c3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:16.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.304+0000 7f5419d9b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f54240c9440 0x7f54240c3ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:44490/0 (socket says 172.21.15.19:44490) 2024-08-31T12:17:16.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.304+0000 7f5419d9b640 1 -- 172.21.15.19:0/3170862578 learned_addr learned my addr 172.21.15.19:0/3170862578 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:17:16.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.304+0000 7f542259c640 1 --2- 172.21.15.19:0/3170862578 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54240c9030 0x7f54241515e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:16.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.304+0000 7f5422d9d640 1 --2- 172.21.15.19:0/3170862578 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5424153190 0x7f542406f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:16.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.304+0000 7f5419d9b640 1 -- 172.21.15.19:0/3170862578 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5424153190 msgr2=0x7f542406f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:16.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.304+0000 7f5419d9b640 1 --2- 172.21.15.19:0/3170862578 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5424153190 0x7f542406f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:16.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.304+0000 7f5419d9b640 1 -- 172.21.15.19:0/3170862578 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54240c9030 msgr2=0x7f54241515e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:16.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.304+0000 7f5419d9b640 1 --2- 172.21.15.19:0/3170862578 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54240c9030 0x7f54241515e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:16.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.304+0000 7f5419d9b640 1 -- 172.21.15.19:0/3170862578 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f542414e260 con 0x7f54240c9440 2024-08-31T12:17:16.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.304+0000 7f5419d9b640 1 --2- 172.21.15.19:0/3170862578 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f54240c9440 0x7f54240c3ca0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f540c00c3c0 tx=0x7f540c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:16.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.304+0000 7f541b7fe640 1 -- 172.21.15.19:0/3170862578 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f540c017070 con 0x7f54240c9440 2024-08-31T12:17:16.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.305+0000 7f541b7fe640 1 -- 172.21.15.19:0/3170862578 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f540c0035f0 con 0x7f54240c9440 2024-08-31T12:17:16.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.305+0000 7f541b7fe640 1 -- 172.21.15.19:0/3170862578 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f540c003a80 con 0x7f54240c9440 2024-08-31T12:17:16.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.305+0000 7f5428820640 1 -- 172.21.15.19:0/3170862578 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f542414e520 con 0x7f54240c9440 2024-08-31T12:17:16.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.305+0000 7f5428820640 1 -- 172.21.15.19:0/3170862578 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5424150820 con 0x7f54240c9440 2024-08-31T12:17:16.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.306+0000 7f5428820640 1 --2- 172.21.15.19:0/3170862578 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f542406e980 0x7f542406f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:16.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.306+0000 7f5428820640 1 -- 172.21.15.19:0/3170862578 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f542406ee30 con 0x7f542406e980 2024-08-31T12:17:16.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.306+0000 7f542259c640 1 --2- 172.21.15.19:0/3170862578 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f542406e980 0x7f542406f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:16.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.307+0000 7f542259c640 1 -- 172.21.15.19:0/3170862578 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5424159820 con 0x7f542406e980 2024-08-31T12:17:16.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.307+0000 7f542259c640 1 --2- 172.21.15.19:0/3170862578 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f542406e980 0x7f542406f7c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f54240c5700 tx=0x7f5414003820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:16.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.307+0000 7f541b7fe640 1 -- 172.21.15.19:0/3170862578 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f540c01a070 con 0x7f54240c9440 2024-08-31T12:17:16.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.307+0000 7f541b7fe640 1 --2- 172.21.15.19:0/3170862578 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5404077ad0 0x7f5404079fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:16.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.308+0000 7f541b7fe640 1 -- 172.21.15.19:0/3170862578 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f540c098810 con 0x7f54240c9440 2024-08-31T12:17:16.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.308+0000 7f5422d9d640 1 --2- 172.21.15.19:0/3170862578 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5404077ad0 0x7f5404079fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:16.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.308+0000 7f541b7fe640 1 -- 172.21.15.19:0/3170862578 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5414040070 con 0x7f542406e980 2024-08-31T12:17:16.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.308+0000 7f541b7fe640 1 -- 172.21.15.19:0/3170862578 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5414003dc0 con 0x7f542406e980 2024-08-31T12:17:16.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.308+0000 7f541b7fe640 1 -- 172.21.15.19:0/3170862578 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f542406e980 msgr2=0x7f542406f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:16.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.308+0000 7f541b7fe640 1 --2- 172.21.15.19:0/3170862578 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f542406e980 0x7f542406f7c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f54240c5700 tx=0x7f5414003820 comp rx=0 tx=0).stop 2024-08-31T12:17:16.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.308+0000 7f541b7fe640 1 -- 172.21.15.19:0/3170862578 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f542406e980 msgr2=0x7f542406f7c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:17:16.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.308+0000 7f541b7fe640 1 --2- 172.21.15.19:0/3170862578 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f542406e980 0x7f542406f7c0 secure :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f54240c5700 tx=0x7f5414003820 comp rx=0 tx=0).stop 2024-08-31T12:17:16.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.308+0000 7f5422d9d640 1 --2- 172.21.15.19:0/3170862578 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5404077ad0 0x7f5404079fe0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f542414d390 tx=0x7f541c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:16.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.314+0000 7f5428820640 1 --2- 172.21.15.19:0/3170862578 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f542406df40 0x7f5424150a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:16.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.314+0000 7f5428820640 1 -- 172.21.15.19:0/3170862578 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f5424150f60 con 0x7f542406df40 2024-08-31T12:17:16.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.314+0000 7f542259c640 1 --2- 172.21.15.19:0/3170862578 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f542406df40 0x7f5424150a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:16.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.315+0000 7f542259c640 1 -- 172.21.15.19:0/3170862578 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f54240c9a20 con 0x7f542406df40 2024-08-31T12:17:16.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.315+0000 7f542259c640 1 --2- 172.21.15.19:0/3170862578 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f542406df40 0x7f5424150a20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5414006010 tx=0x7f5414009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:16.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.315+0000 7f541b7fe640 1 -- 172.21.15.19:0/3170862578 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5414035040 con 0x7f542406df40 2024-08-31T12:17:16.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.315+0000 7f541b7fe640 1 -- 172.21.15.19:0/3170862578 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5043 (secure 0 0 0) 0x7f541403c440 con 0x7f542406df40 2024-08-31T12:17:16.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.315+0000 7f541b7fe640 1 -- 172.21.15.19:0/3170862578 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f542406df40 msgr2=0x7f5424150a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:16.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.315+0000 7f541b7fe640 1 --2- 172.21.15.19:0/3170862578 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f542406df40 0x7f5424150a20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5414006010 tx=0x7f5414009260 comp rx=0 tx=0).stop 2024-08-31T12:17:16.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.315+0000 7f541b7fe640 1 -- 172.21.15.19:0/3170862578 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f542406df40 msgr2=0x7f5424150a20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:17:16.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.315+0000 7f541b7fe640 1 --2- 172.21.15.19:0/3170862578 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f542406df40 0x7f5424150a20 secure :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5414006010 tx=0x7f5414009260 comp rx=0 tx=0).stop 2024-08-31T12:17:16.317 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:17:16.317 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:17:16.317 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:17:16.317 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:17:16.317 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 10, 2024-08-31T12:17:16.317 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 104409, 2024-08-31T12:17:16.317 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:17:16.317 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:17:16.318 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:17:16.318 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:17:16.318 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:16.318 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 15, 2024-08-31T12:17:16.318 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:17:16.318 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:17:16.318 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:17:16.318 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:17:16.318 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:17:16.318 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:17:16.318 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:17:16.318 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:17:16.318 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:17:16.319 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:17:16.319 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:17:16.319 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:17:16.319 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:17:16.319 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:17:16.319 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:16.319 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:17:16.319 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:17:16.319 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:17:16.319 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:17:16.319 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:17:16.319 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:17:16.319 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:17:16.319 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:17:16.320 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:17:16.320 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:17:16.320 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:17:16.320 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:17:16.320 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:17:16.320 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:16.320 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:16.320 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:17:16.320 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:17:16.320 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:17:16.320 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:17:16.320 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:17:16.320 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:17:16.320 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:17:16.321 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:17:16.321 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:17:16.321 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:17:16.321 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:17:16.321 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:17:16.321 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:17:16.321 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:17:16.321 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:17:16.321 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:17:16.321 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:17:16.321 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:17:16.321 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:17:16.321 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:17:16.322 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:17:16.322 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:17:16.322 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:17:16.322 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:17:16.322 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:17:16.322 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:17:16.322 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:17:16.322 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:17:16.322 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:16.322 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:17:16.322 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:16.322 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:17:16.322 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:16.322 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:17:16.323 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:17:16.323 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:17:16.323 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:17:16.323 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:16.323 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:17:16.323 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:17:16.323 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:16.323 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:16.323 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:16.323 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:17:16.323 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:17:16.323 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:16.323 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:16.324 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:16.324 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:16.324 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:17:16.324 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:17:16.324 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:17:16.324 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:17:16.324 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:17:16.324 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:16.324 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:16.324 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:17:16.324 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:17:16.324 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:17:16.325 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:17:16.325 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:16.325 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:17:16.325 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:17:16.325 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:16.325 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:16.325 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:16.325 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:17:16.325 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:17:16.325 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:16.325 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:16.325 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:16.325 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:16.325 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:17:16.326 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:17:16.326 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:17:16.326 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:17:16.326 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:17:16.326 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:16.326 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:16.326 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:17:16.326 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:17:16.326 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:17:16.326 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:17:16.326 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:16.326 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:17:16.326 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:17:16.326 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:16.327 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:16.327 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:16.327 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:17:16.327 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:17:16.327 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:16.327 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:16.327 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:16.327 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:16.327 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:17:16.327 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:17:16.327 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:17:16.327 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:17:16.327 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:17:16.327 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:16.328 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:16.328 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:16.328 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:17:16.328 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:17:16.328 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:16.328 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:17:16.328 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:17:16.328 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:17:16.328 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:16.328 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:16.328 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:17:16.328 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:16.328 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:17:16.328 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:17:16.329 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 3 2024-08-31T12:17:16.329 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:16.329 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:16.329 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:17:16.329 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:16.329 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:17:16.329 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:17:16.329 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 5 2024-08-31T12:17:16.329 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:16.329 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:16.329 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:17:16.329 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:16.329 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:17:16.329 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:17:16.330 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 3 2024-08-31T12:17:16.330 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:16.330 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:16.330 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:17:16.330 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:16.330 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:17:16.330 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:17:16.330 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:17:16.330 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:16.330 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:16.330 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:16.330 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:17:16.330 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:17:16.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.316+0000 7f5428820640 1 -- 172.21.15.19:0/3170862578 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5404077ad0 msgr2=0x7f5404079fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:16.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.316+0000 7f5428820640 1 --2- 172.21.15.19:0/3170862578 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5404077ad0 0x7f5404079fe0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f542414d390 tx=0x7f541c009040 comp rx=0 tx=0).stop 2024-08-31T12:17:16.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.316+0000 7f5428820640 1 -- 172.21.15.19:0/3170862578 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f54240c9440 msgr2=0x7f54240c3ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:16.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.316+0000 7f5428820640 1 --2- 172.21.15.19:0/3170862578 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f54240c9440 0x7f54240c3ca0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f540c00c3c0 tx=0x7f540c00c880 comp rx=0 tx=0).stop 2024-08-31T12:17:16.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.316+0000 7f5422d9d640 1 -- 172.21.15.19:0/3170862578 reap_dead start 2024-08-31T12:17:16.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.316+0000 7f5422d9d640 1 -- 172.21.15.19:0/3170862578 reap_dead start 2024-08-31T12:17:16.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.316+0000 7f5428820640 1 -- 172.21.15.19:0/3170862578 shutdown_connections 2024-08-31T12:17:16.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.316+0000 7f5428820640 1 -- 172.21.15.19:0/3170862578 >> 172.21.15.19:0/3170862578 conn(0x7f54240728b0 msgr2=0x7f54241493d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:17:16.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.316+0000 7f5428820640 1 -- 172.21.15.19:0/3170862578 shutdown_connections 2024-08-31T12:17:16.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.316+0000 7f5428820640 1 -- 172.21.15.19:0/3170862578 wait complete. 2024-08-31T12:17:16.333 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:17:16.333 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-08-31T12:17:16.333 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-08-31T12:17:16.333 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-08-31T12:17:16.333 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-08-31T12:17:16.334 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-08-31T12:17:16.334 INFO:teuthology.orchestra.run:waiting for 300 2024-08-31T12:17:16.351 INFO:tasks.ceph.mon.c:Stopped 2024-08-31T12:17:16.351 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-08-31T12:17:16.351 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:17:16.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.465+0000 7f7df51f8640 1 -- 172.21.15.19:0/917487762 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7df00c72a0 msgr2=0x7f7df015c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:16.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.465+0000 7f7df51f8640 1 --2- 172.21.15.19:0/917487762 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7df00c72a0 0x7f7df015c570 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7df00cb210 tx=0x7f7de802aaf0 comp rx=0 tx=0).stop 2024-08-31T12:17:16.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.465+0000 7f7df51f8640 1 -- 172.21.15.19:0/917487762 shutdown_connections 2024-08-31T12:17:16.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.465+0000 7f7df51f8640 1 --2- 172.21.15.19:0/917487762 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7df00c72a0 0x7f7df015c570 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:16.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.465+0000 7f7df51f8640 1 --2- 172.21.15.19:0/917487762 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7df0153130 0x7f7df00c6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:16.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.465+0000 7f7df51f8640 1 --2- 172.21.15.19:0/917487762 >> v2:172.21.15.43:3300/0 conn(0x7f7df0158fd0 0x7f7df006d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:16.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.465+0000 7f7df51f8640 1 -- 172.21.15.19:0/917487762 >> 172.21.15.19:0/917487762 conn(0x7f7df00cdcb0 msgr2=0x7f7df00ca150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:17:16.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.466+0000 7f7df51f8640 1 -- 172.21.15.19:0/917487762 shutdown_connections 2024-08-31T12:17:16.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.466+0000 7f7df51f8640 1 -- 172.21.15.19:0/917487762 wait complete. 2024-08-31T12:17:16.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.468+0000 7f7df51f8640 1 Processor -- start 2024-08-31T12:17:16.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.468+0000 7f7df51f8640 1 -- start start 2024-08-31T12:17:16.471 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.469+0000 7f7df51f8640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7df00c72a0 0x7f7df01ef560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:16.471 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.469+0000 7f7df51f8640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7df0153130 0x7f7df0146f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:16.471 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.469+0000 7f7df51f8640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7df0158fd0 0x7f7df0147440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:16.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.469+0000 7f7df51f8640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f7df00c8780 con 0x7f7df00c72a0 2024-08-31T12:17:16.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.469+0000 7f7df51f8640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7df00c84a0 con 0x7f7df0158fd0 2024-08-31T12:17:16.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.469+0000 7f7df51f8640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f7df006cae0 con 0x7f7df0153130 2024-08-31T12:17:16.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.469+0000 7f7deed9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7df00c72a0 0x7f7df01ef560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:16.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.469+0000 7f7dee59c640 1 -- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7df0153130 msgr2=0x7f7df0146f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3301/0 2024-08-31T12:17:16.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.469+0000 7f7dee59c640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7df0153130 0x7f7df0146f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:17:16.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.469+0000 7f7deed9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7df00c72a0 0x7f7df01ef560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:44546/0 (socket says 172.21.15.19:44546) 2024-08-31T12:17:16.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.469+0000 7f7deed9d640 1 -- 172.21.15.19:0/1560732506 learned_addr learned my addr 172.21.15.19:0/1560732506 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:17:16.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.470+0000 7f7def59e640 1 --2- 172.21.15.19:0/1560732506 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7df0158fd0 0x7f7df0147440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:16.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.470+0000 7f7deed9d640 1 -- 172.21.15.19:0/1560732506 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7df0153130 msgr2=0x7f7df0146f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:17:16.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.470+0000 7f7deed9d640 1 --2- 172.21.15.19:0/1560732506 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7df0153130 0x7f7df0146f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:16.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.470+0000 7f7deed9d640 1 -- 172.21.15.19:0/1560732506 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7df0158fd0 msgr2=0x7f7df0147440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:16.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.470+0000 7f7deed9d640 1 --2- 172.21.15.19:0/1560732506 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7df0158fd0 0x7f7df0147440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:16.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.470+0000 7f7deed9d640 1 -- 172.21.15.19:0/1560732506 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7df01efaa0 con 0x7f7df00c72a0 2024-08-31T12:17:16.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.470+0000 7f7deed9d640 1 --2- 172.21.15.19:0/1560732506 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7df00c72a0 0x7f7df01ef560 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7de00099e0 tx=0x7f7de0009ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:16.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.470+0000 7f7dd77fe640 1 -- 172.21.15.19:0/1560732506 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7de0004020 con 0x7f7df00c72a0 2024-08-31T12:17:16.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.470+0000 7f7dd77fe640 1 -- 172.21.15.19:0/1560732506 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7de00041c0 con 0x7f7df00c72a0 2024-08-31T12:17:16.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.470+0000 7f7dd77fe640 1 -- 172.21.15.19:0/1560732506 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7de0007890 con 0x7f7df00c72a0 2024-08-31T12:17:16.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.470+0000 7f7df51f8640 1 -- 172.21.15.19:0/1560732506 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7df0155e50 con 0x7f7df00c72a0 2024-08-31T12:17:16.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.471+0000 7f7df51f8640 1 -- 172.21.15.19:0/1560732506 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7df0156410 con 0x7f7df00c72a0 2024-08-31T12:17:16.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.472+0000 7f7df51f8640 1 -- 172.21.15.19:0/1560732506 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7df00c8050 con 0x7f7df00c72a0 2024-08-31T12:17:16.474 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.473+0000 7f7dd77fe640 1 -- 172.21.15.19:0/1560732506 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f7de0021020 con 0x7f7df00c72a0 2024-08-31T12:17:16.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.474+0000 7f7dd77fe640 1 --2- 172.21.15.19:0/1560732506 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7ddc077a50 0x7f7ddc079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:16.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.474+0000 7f7dd77fe640 1 -- 172.21.15.19:0/1560732506 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f7de009d0f0 con 0x7f7df00c72a0 2024-08-31T12:17:16.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.474+0000 7f7dee59c640 1 --2- 172.21.15.19:0/1560732506 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7ddc077a50 0x7f7ddc079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:16.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.475+0000 7f7dee59c640 1 --2- 172.21.15.19:0/1560732506 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7ddc077a50 0x7f7ddc079f60 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7df00c3da0 tx=0x7f7de4006d50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:16.479 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.479+0000 7f7dd77fe640 1 -- 172.21.15.19:0/1560732506 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f7de0065dc0 con 0x7f7df00c72a0 2024-08-31T12:17:16.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.738+0000 7f7df51f8640 1 -- 172.21.15.19:0/1560732506 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7df00c8da0 con 0x7f7df00c72a0 2024-08-31T12:17:16.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.739+0000 7f7dd77fe640 1 -- 172.21.15.19:0/1560732506 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f7de006e270 con 0x7f7df00c72a0 2024-08-31T12:17:16.740 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:17:16.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.741+0000 7f7df51f8640 1 -- 172.21.15.19:0/1560732506 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7ddc077a50 msgr2=0x7f7ddc079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:16.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.741+0000 7f7df51f8640 1 --2- 172.21.15.19:0/1560732506 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7ddc077a50 0x7f7ddc079f60 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7df00c3da0 tx=0x7f7de4006d50 comp rx=0 tx=0).stop 2024-08-31T12:17:16.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.741+0000 7f7df51f8640 1 -- 172.21.15.19:0/1560732506 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7df00c72a0 msgr2=0x7f7df01ef560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:16.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.741+0000 7f7df51f8640 1 --2- 172.21.15.19:0/1560732506 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7df00c72a0 0x7f7df01ef560 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7de00099e0 tx=0x7f7de0009ea0 comp rx=0 tx=0).stop 2024-08-31T12:17:16.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.741+0000 7f7df51f8640 1 -- 172.21.15.19:0/1560732506 shutdown_connections 2024-08-31T12:17:16.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.741+0000 7f7df51f8640 1 --2- 172.21.15.19:0/1560732506 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7ddc077a50 0x7f7ddc079f60 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:16.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.741+0000 7f7df51f8640 1 --2- 172.21.15.19:0/1560732506 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7df0158fd0 0x7f7df0147440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:16.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.741+0000 7f7df51f8640 1 --2- 172.21.15.19:0/1560732506 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7df0153130 0x7f7df0146f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:16.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.741+0000 7f7df51f8640 1 --2- 172.21.15.19:0/1560732506 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7df00c72a0 0x7f7df01ef560 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:16.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.741+0000 7f7df51f8640 1 -- 172.21.15.19:0/1560732506 >> 172.21.15.19:0/1560732506 conn(0x7f7df00cdcb0 msgr2=0x7f7df0151380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:17:16.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.741+0000 7f7df51f8640 1 -- 172.21.15.19:0/1560732506 shutdown_connections 2024-08-31T12:17:16.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:16.741+0000 7f7df51f8640 1 -- 172.21.15.19:0/1560732506 wait complete. 2024-08-31T12:17:19.755 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:17:19.892 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:19.891+0000 7fccf472b640 1 -- 172.21.15.19:0/196508664 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fccec071b40 msgr2=0x7fccec0caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:19.892 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:19.891+0000 7fccf472b640 1 --2- 172.21.15.19:0/196508664 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fccec071b40 0x7fccec0caa20 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fccdc009830 tx=0x7fccdc02a5d0 comp rx=0 tx=0).stop 2024-08-31T12:17:19.892 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:19.891+0000 7fccf472b640 1 -- 172.21.15.19:0/196508664 shutdown_connections 2024-08-31T12:17:19.892 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:19.891+0000 7fccf472b640 1 --2- 172.21.15.19:0/196508664 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fccec159470 0x7fccec159880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:19.892 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:19.891+0000 7fccf472b640 1 --2- 172.21.15.19:0/196508664 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fccec071b40 0x7fccec0caa20 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:19.893 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:19.891+0000 7fccf472b640 1 -- 172.21.15.19:0/196508664 >> 172.21.15.19:0/196508664 conn(0x7fccec072c80 msgr2=0x7fccec073090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:17:19.893 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:19.891+0000 7fccf472b640 1 -- 172.21.15.19:0/196508664 shutdown_connections 2024-08-31T12:17:19.893 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:19.891+0000 7fccf472b640 1 -- 172.21.15.19:0/196508664 wait complete. 2024-08-31T12:17:19.895 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:19.894+0000 7fccf472b640 1 Processor -- start 2024-08-31T12:17:19.895 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:19.894+0000 7fccf472b640 1 -- start start 2024-08-31T12:17:19.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:19.896+0000 7fccf472b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fccec071b40 0x7fccec1f2680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:19.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:19.896+0000 7fccf472b640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fccec153230 0x7fccec14c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:19.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:19.896+0000 7fccf472b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fccec159470 0x7fccec14b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:19.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:19.896+0000 7fccf472b640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fccec072360 con 0x7fccec159470 2024-08-31T12:17:19.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:19.896+0000 7fccf472b640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fccec0721e0 con 0x7fccec153230 2024-08-31T12:17:19.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:19.896+0000 7fccf472b640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fccec0711e0 con 0x7fccec071b40 2024-08-31T12:17:19.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:19.896+0000 7fccf2cc8640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fccec159470 0x7fccec14b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:19.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:19.896+0000 7fccf2cc8640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fccec159470 0x7fccec14b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:57584/0 (socket says 172.21.15.19:57584) 2024-08-31T12:17:19.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:19.896+0000 7fccf24c7640 1 -- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fccec071b40 msgr2=0x7fccec1f2680 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3301/0 2024-08-31T12:17:19.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:19.896+0000 7fccf2cc8640 1 -- 172.21.15.19:0/3479417884 learned_addr learned my addr 172.21.15.19:0/3479417884 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:17:19.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:19.896+0000 7fccf1cc6640 1 --2- 172.21.15.19:0/3479417884 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fccec153230 0x7fccec14c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:19.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:19.896+0000 7fccf24c7640 1 --2- 172.21.15.19:0/3479417884 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fccec071b40 0x7fccec1f2680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:17:19.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:19.897+0000 7fccf2cc8640 1 -- 172.21.15.19:0/3479417884 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fccec071b40 msgr2=0x7fccec1f2680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:17:19.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:19.897+0000 7fccf2cc8640 1 --2- 172.21.15.19:0/3479417884 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fccec071b40 0x7fccec1f2680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:19.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:19.897+0000 7fccf2cc8640 1 -- 172.21.15.19:0/3479417884 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fccec153230 msgr2=0x7fccec14c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:19.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:19.897+0000 7fccf2cc8640 1 --2- 172.21.15.19:0/3479417884 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fccec153230 0x7fccec14c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:19.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:19.897+0000 7fccf2cc8640 1 -- 172.21.15.19:0/3479417884 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccec1f2bc0 con 0x7fccec159470 2024-08-31T12:17:19.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:19.897+0000 7fccf2cc8640 1 --2- 172.21.15.19:0/3479417884 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fccec159470 0x7fccec14b370 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcce400be50 tx=0x7fcce400bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:19.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:19.897+0000 7fcce2ffd640 1 -- 172.21.15.19:0/3479417884 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce4019070 con 0x7fccec159470 2024-08-31T12:17:19.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:19.897+0000 7fccf472b640 1 -- 172.21.15.19:0/3479417884 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccec14be10 con 0x7fccec159470 2024-08-31T12:17:19.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:19.897+0000 7fcce2ffd640 1 -- 172.21.15.19:0/3479417884 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcce4014440 con 0x7fccec159470 2024-08-31T12:17:19.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:19.897+0000 7fcce2ffd640 1 -- 172.21.15.19:0/3479417884 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce4014890 con 0x7fccec159470 2024-08-31T12:17:19.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:19.897+0000 7fccf472b640 1 -- 172.21.15.19:0/3479417884 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccec14acd0 con 0x7fccec159470 2024-08-31T12:17:19.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:19.899+0000 7fcce0ff9640 1 -- 172.21.15.19:0/3479417884 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccec0caa20 con 0x7fccec159470 2024-08-31T12:17:19.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:19.899+0000 7fcce2ffd640 1 -- 172.21.15.19:0/3479417884 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fcce401c070 con 0x7fccec159470 2024-08-31T12:17:19.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:19.900+0000 7fcce2ffd640 1 --2- 172.21.15.19:0/3479417884 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fccd0077a00 0x7fccd0079f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:19.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:19.900+0000 7fcce2ffd640 1 -- 172.21.15.19:0/3479417884 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fcce4099700 con 0x7fccec159470 2024-08-31T12:17:19.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:19.900+0000 7fccf24c7640 1 --2- 172.21.15.19:0/3479417884 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fccd0077a00 0x7fccd0079f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:19.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:19.901+0000 7fccf24c7640 1 --2- 172.21.15.19:0/3479417884 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fccd0077a00 0x7fccd0079f10 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fcce8007fd0 tx=0x7fcce8007980 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:19.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:19.905+0000 7fcce2ffd640 1 -- 172.21.15.19:0/3479417884 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fcce4061db0 con 0x7fccec159470 2024-08-31T12:17:20.200 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:20.199+0000 7fcce0ff9640 1 -- 172.21.15.19:0/3479417884 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fccec0ca110 con 0x7fccec159470 2024-08-31T12:17:20.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:20.200+0000 7fcce2ffd640 1 -- 172.21.15.19:0/3479417884 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fcce4066570 con 0x7fccec159470 2024-08-31T12:17:20.201 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":19,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:17:20.203 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:20.202+0000 7fcce0ff9640 1 -- 172.21.15.19:0/3479417884 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fccd0077a00 msgr2=0x7fccd0079f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:20.203 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:20.202+0000 7fcce0ff9640 1 --2- 172.21.15.19:0/3479417884 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fccd0077a00 0x7fccd0079f10 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fcce8007fd0 tx=0x7fcce8007980 comp rx=0 tx=0).stop 2024-08-31T12:17:20.203 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:20.202+0000 7fcce0ff9640 1 -- 172.21.15.19:0/3479417884 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fccec159470 msgr2=0x7fccec14b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:20.203 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:20.202+0000 7fcce0ff9640 1 --2- 172.21.15.19:0/3479417884 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fccec159470 0x7fccec14b370 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcce400be50 tx=0x7fcce400bf40 comp rx=0 tx=0).stop 2024-08-31T12:17:20.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:20.202+0000 7fcce0ff9640 1 -- 172.21.15.19:0/3479417884 shutdown_connections 2024-08-31T12:17:20.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:20.202+0000 7fcce0ff9640 1 --2- 172.21.15.19:0/3479417884 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fccd0077a00 0x7fccd0079f10 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:20.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:20.202+0000 7fcce0ff9640 1 --2- 172.21.15.19:0/3479417884 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fccec159470 0x7fccec14b370 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:20.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:20.202+0000 7fcce0ff9640 1 --2- 172.21.15.19:0/3479417884 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fccec153230 0x7fccec14c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:20.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:20.202+0000 7fcce0ff9640 1 --2- 172.21.15.19:0/3479417884 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fccec071b40 0x7fccec1f2680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:20.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:20.202+0000 7fcce0ff9640 1 -- 172.21.15.19:0/3479417884 >> 172.21.15.19:0/3479417884 conn(0x7fccec072c80 msgr2=0x7fccec14dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:17:20.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:20.202+0000 7fcce0ff9640 1 -- 172.21.15.19:0/3479417884 shutdown_connections 2024-08-31T12:17:20.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:20.202+0000 7fcce0ff9640 1 -- 172.21.15.19:0/3479417884 wait complete. 2024-08-31T12:17:23.080 INFO:tasks.workunit.client.0.smithi019.stderr:Updating files: 94% (12025/12697) Updating files: 95% (12063/12697) Updating files: 96% (12190/12697) Updating files: 97% (12317/12697) Updating files: 98% (12444/12697) Updating files: 99% (12571/12697) Updating files: 100% (12697/12697) Updating files: 100% (12697/12697), done. 2024-08-31T12:17:23.218 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:17:23.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.331+0000 7f7e77f2b640 1 -- 172.21.15.19:0/4269644637 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7e70151ff0 msgr2=0x7f7e7006dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:23.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.331+0000 7f7e77f2b640 1 --2- 172.21.15.19:0/4269644637 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7e70151ff0 0x7f7e7006dbf0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7e700ce3c0 tx=0x7f7e6402a4a0 comp rx=0 tx=0).stop 2024-08-31T12:17:23.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.331+0000 7f7e77f2b640 1 -- 172.21.15.19:0/4269644637 shutdown_connections 2024-08-31T12:17:23.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.331+0000 7f7e77f2b640 1 --2- 172.21.15.19:0/4269644637 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7e70151ff0 0x7f7e7006dbf0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:23.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.331+0000 7f7e77f2b640 1 --2- 172.21.15.19:0/4269644637 >> v2:172.21.15.43:3300/0 conn(0x7f7e700c7f00 0x7f7e700c53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:23.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.331+0000 7f7e77f2b640 1 -- 172.21.15.19:0/4269644637 >> 172.21.15.19:0/4269644637 conn(0x7f7e700ca630 msgr2=0x7f7e700cd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:17:23.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.331+0000 7f7e77f2b640 1 -- 172.21.15.19:0/4269644637 shutdown_connections 2024-08-31T12:17:23.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.331+0000 7f7e77f2b640 1 -- 172.21.15.19:0/4269644637 wait complete. 2024-08-31T12:17:23.334 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.333+0000 7f7e77f2b640 1 Processor -- start 2024-08-31T12:17:23.334 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.333+0000 7f7e77f2b640 1 -- start start 2024-08-31T12:17:23.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.334+0000 7f7e77f2b640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e7006e130 0x7f7e701f2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:23.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.334+0000 7f7e77f2b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7e700c7f00 0x7f7e701e4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:23.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.334+0000 7f7e77f2b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7e70151ff0 0x7f7e701e4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:23.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.334+0000 7f7e77f2b640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f7e70152400 con 0x7f7e70151ff0 2024-08-31T12:17:23.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.334+0000 7f7e77f2b640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7e700cded0 con 0x7f7e7006e130 2024-08-31T12:17:23.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.334+0000 7f7e77f2b640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f7e70152580 con 0x7f7e700c7f00 2024-08-31T12:17:23.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.335+0000 7f7e754c6640 1 -- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7e700c7f00 msgr2=0x7f7e701e4450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3301/0 2024-08-31T12:17:23.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.335+0000 7f7e754c6640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7e700c7f00 0x7f7e701e4450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:17:23.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.335+0000 7f7e764c8640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7e70151ff0 0x7f7e701e4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:23.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.335+0000 7f7e764c8640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7e70151ff0 0x7f7e701e4990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:57602/0 (socket says 172.21.15.19:57602) 2024-08-31T12:17:23.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.335+0000 7f7e764c8640 1 -- 172.21.15.19:0/712674066 learned_addr learned my addr 172.21.15.19:0/712674066 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:17:23.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.335+0000 7f7e75cc7640 1 --2- 172.21.15.19:0/712674066 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e7006e130 0x7f7e701f2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:23.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.335+0000 7f7e764c8640 1 -- 172.21.15.19:0/712674066 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7e700c7f00 msgr2=0x7f7e701e4450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:17:23.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.335+0000 7f7e764c8640 1 --2- 172.21.15.19:0/712674066 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7e700c7f00 0x7f7e701e4450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:23.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.335+0000 7f7e764c8640 1 -- 172.21.15.19:0/712674066 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e7006e130 msgr2=0x7f7e701f2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:23.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.335+0000 7f7e764c8640 1 --2- 172.21.15.19:0/712674066 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e7006e130 0x7f7e701f2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:23.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.335+0000 7f7e764c8640 1 -- 172.21.15.19:0/712674066 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e70150d30 con 0x7f7e70151ff0 2024-08-31T12:17:23.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.335+0000 7f7e764c8640 1 --2- 172.21.15.19:0/712674066 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7e70151ff0 0x7f7e701e4990 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7e6800cab0 tx=0x7f7e6800cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:23.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.335+0000 7f7e627fc640 1 -- 172.21.15.19:0/712674066 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e68016020 con 0x7f7e70151ff0 2024-08-31T12:17:23.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.335+0000 7f7e627fc640 1 -- 172.21.15.19:0/712674066 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7e6800ad20 con 0x7f7e70151ff0 2024-08-31T12:17:23.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.335+0000 7f7e77f2b640 1 -- 172.21.15.19:0/712674066 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e70151660 con 0x7f7e70151ff0 2024-08-31T12:17:23.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.335+0000 7f7e627fc640 1 -- 172.21.15.19:0/712674066 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e68012680 con 0x7f7e70151ff0 2024-08-31T12:17:23.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.335+0000 7f7e77f2b640 1 -- 172.21.15.19:0/712674066 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e70150a50 con 0x7f7e70151ff0 2024-08-31T12:17:23.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.336+0000 7f7e77f2b640 1 -- 172.21.15.19:0/712674066 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e70159020 con 0x7f7e70151ff0 2024-08-31T12:17:23.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.337+0000 7f7e627fc640 1 -- 172.21.15.19:0/712674066 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f7e68019020 con 0x7f7e70151ff0 2024-08-31T12:17:23.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.338+0000 7f7e627fc640 1 --2- 172.21.15.19:0/712674066 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7e54077a00 0x7f7e54079f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:23.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.338+0000 7f7e627fc640 1 -- 172.21.15.19:0/712674066 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f7e6809a020 con 0x7f7e70151ff0 2024-08-31T12:17:23.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.338+0000 7f7e75cc7640 1 --2- 172.21.15.19:0/712674066 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7e54077a00 0x7f7e54079f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:23.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.339+0000 7f7e75cc7640 1 --2- 172.21.15.19:0/712674066 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7e54077a00 0x7f7e54079f10 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c005fd0 tx=0x7f7e6c005dc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:23.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.342+0000 7f7e627fc640 1 -- 172.21.15.19:0/712674066 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f7e70159020 con 0x7f7e70151ff0 2024-08-31T12:17:23.414 INFO:tasks.workunit.client.0.smithi019.stderr:Note: switching to 'd19e6118f7015f0047264006cb0c873528eb3ae6'. 2024-08-31T12:17:23.414 INFO:tasks.workunit.client.0.smithi019.stderr: 2024-08-31T12:17:23.414 INFO:tasks.workunit.client.0.smithi019.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-08-31T12:17:23.414 INFO:tasks.workunit.client.0.smithi019.stderr:changes and commit them, and you can discard any commits you make in this 2024-08-31T12:17:23.414 INFO:tasks.workunit.client.0.smithi019.stderr:state without impacting any branches by switching back to a branch. 2024-08-31T12:17:23.415 INFO:tasks.workunit.client.0.smithi019.stderr: 2024-08-31T12:17:23.415 INFO:tasks.workunit.client.0.smithi019.stderr:If you want to create a new branch to retain commits you create, you may 2024-08-31T12:17:23.415 INFO:tasks.workunit.client.0.smithi019.stderr:do so (now or later) by using -c with the switch command. Example: 2024-08-31T12:17:23.415 INFO:tasks.workunit.client.0.smithi019.stderr: 2024-08-31T12:17:23.415 INFO:tasks.workunit.client.0.smithi019.stderr: git switch -c 2024-08-31T12:17:23.415 INFO:tasks.workunit.client.0.smithi019.stderr: 2024-08-31T12:17:23.415 INFO:tasks.workunit.client.0.smithi019.stderr:Or undo this operation with: 2024-08-31T12:17:23.415 INFO:tasks.workunit.client.0.smithi019.stderr: 2024-08-31T12:17:23.415 INFO:tasks.workunit.client.0.smithi019.stderr: git switch - 2024-08-31T12:17:23.415 INFO:tasks.workunit.client.0.smithi019.stderr: 2024-08-31T12:17:23.415 INFO:tasks.workunit.client.0.smithi019.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-08-31T12:17:23.415 INFO:tasks.workunit.client.0.smithi019.stderr: 2024-08-31T12:17:23.415 INFO:tasks.workunit.client.0.smithi019.stderr:HEAD is now at d19e6118f70 Merge PR #44359 into wip-mchangir-testing-20240828.085847-main-debug 2024-08-31T12:17:23.420 DEBUG:teuthology.orchestra.run.smithi019:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-08-31T12:17:23.454 INFO:tasks.workunit.client.0.smithi019.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-08-31T12:17:23.456 INFO:tasks.workunit.client.0.smithi019.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-08-31T12:17:23.456 INFO:tasks.workunit.client.0.smithi019.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-08-31T12:17:23.604 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.603+0000 7f7e77f2b640 1 -- 172.21.15.19:0/712674066 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7e700c9ce0 con 0x7f7e70151ff0 2024-08-31T12:17:23.604 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.603+0000 7f7e627fc640 1 -- 172.21.15.19:0/712674066 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f7e68020030 con 0x7f7e70151ff0 2024-08-31T12:17:23.604 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:17:23.607 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.606+0000 7f7e77f2b640 1 -- 172.21.15.19:0/712674066 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7e54077a00 msgr2=0x7f7e54079f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:23.607 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.606+0000 7f7e77f2b640 1 --2- 172.21.15.19:0/712674066 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7e54077a00 0x7f7e54079f10 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c005fd0 tx=0x7f7e6c005dc0 comp rx=0 tx=0).stop 2024-08-31T12:17:23.607 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.606+0000 7f7e77f2b640 1 -- 172.21.15.19:0/712674066 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7e70151ff0 msgr2=0x7f7e701e4990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:23.607 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.606+0000 7f7e77f2b640 1 --2- 172.21.15.19:0/712674066 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7e70151ff0 0x7f7e701e4990 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7e6800cab0 tx=0x7f7e6800cf70 comp rx=0 tx=0).stop 2024-08-31T12:17:23.607 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.606+0000 7f7e77f2b640 1 -- 172.21.15.19:0/712674066 shutdown_connections 2024-08-31T12:17:23.607 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.606+0000 7f7e77f2b640 1 --2- 172.21.15.19:0/712674066 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7e54077a00 0x7f7e54079f10 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:23.607 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.606+0000 7f7e77f2b640 1 --2- 172.21.15.19:0/712674066 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7e70151ff0 0x7f7e701e4990 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:23.608 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.606+0000 7f7e77f2b640 1 --2- 172.21.15.19:0/712674066 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7e700c7f00 0x7f7e701e4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:23.608 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.606+0000 7f7e77f2b640 1 --2- 172.21.15.19:0/712674066 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e7006e130 0x7f7e701f2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:23.608 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.606+0000 7f7e77f2b640 1 -- 172.21.15.19:0/712674066 >> 172.21.15.19:0/712674066 conn(0x7f7e700ca630 msgr2=0x7f7e700cd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:17:23.608 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.606+0000 7f7e77f2b640 1 -- 172.21.15.19:0/712674066 shutdown_connections 2024-08-31T12:17:23.608 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:23.606+0000 7f7e77f2b640 1 -- 172.21.15.19:0/712674066 wait complete. 2024-08-31T12:17:24.025 INFO:tasks.workunit.client.0.smithi019.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-08-31T12:17:24.091 INFO:tasks.workunit.client.0.smithi019.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-08-31T12:17:24.130 INFO:tasks.workunit.client.0.smithi019.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-08-31T12:17:24.132 INFO:tasks.workunit.client.0.smithi019.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-08-31T12:17:24.132 INFO:tasks.workunit.client.0.smithi019.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-08-31T12:17:24.170 INFO:tasks.workunit.client.0.smithi019.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-08-31T12:17:24.174 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-08-31T12:17:24.174 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-08-31T12:17:24.206 INFO:tasks.workunit:Running workunits matching suites/ffsb.sh on client.0... 2024-08-31T12:17:24.208 INFO:tasks.workunit:Running workunit suites/ffsb.sh... 2024-08-31T12:17:24.208 DEBUG:teuthology.orchestra.run.smithi019:workunit test suites/ffsb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d19e6118f7015f0047264006cb0c873528eb3ae6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-08-31T12:17:24.273 INFO:tasks.workunit.client.0.smithi019.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-08-31T12:17:24.273 INFO:tasks.workunit.client.0.smithi019.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.0/qa/workunits/suites 2024-08-31T12:17:24.274 INFO:tasks.workunit.client.0.smithi019.stderr:+ trap -- retry EXIT 2024-08-31T12:17:24.274 INFO:tasks.workunit.client.0.smithi019.stderr:+ rm -rf ffsb 2024-08-31T12:17:24.274 INFO:tasks.workunit.client.0.smithi019.stderr:+ timeout 1800 git clone https://git.ceph.com/ffsb.git --depth 1 2024-08-31T12:17:24.279 INFO:tasks.workunit.client.0.smithi019.stderr:Cloning into 'ffsb'... 2024-08-31T12:17:25.766 DEBUG:teuthology.orchestra.run.smithi019:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:17:25.769 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:17:25.933 INFO:tasks.workunit.client.0.smithi019.stderr:+ trap - EXIT 2024-08-31T12:17:25.933 INFO:tasks.workunit.client.0.smithi019.stderr:+ cd ffsb 2024-08-31T12:17:25.934 INFO:tasks.workunit.client.0.smithi019.stderr:+ ./configure 2024-08-31T12:17:26.063 INFO:tasks.workunit.client.0.smithi019.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-08-31T12:17:26.618 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:17:27.072 INFO:tasks.workunit.client.0.smithi019.stdout:checking whether build environment is sane... yes 2024-08-31T12:17:27.087 INFO:tasks.workunit.client.0.smithi019.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-08-31T12:17:27.088 INFO:tasks.workunit.client.0.smithi019.stdout:checking for gawk... gawk 2024-08-31T12:17:27.111 INFO:tasks.workunit.client.0.smithi019.stdout:checking whether make sets $(MAKE)... yes 2024-08-31T12:17:27.153 INFO:tasks.workunit.client.0.smithi019.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-08-31T12:17:27.153 INFO:tasks.workunit.client.0.smithi019.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-08-31T12:17:27.173 INFO:tasks.workunit.client.0.smithi019.stdout:checking for style of include used by make... GNU 2024-08-31T12:17:27.177 INFO:tasks.workunit.client.0.smithi019.stdout:checking for gcc... gcc 2024-08-31T12:17:27.259 INFO:tasks.workunit.client.0.smithi019.stdout:checking for C compiler default output file name... a.out 2024-08-31T12:17:27.262 INFO:tasks.workunit.client.0.smithi019.stdout:checking whether the C compiler works... yes 2024-08-31T12:17:27.265 INFO:tasks.workunit.client.0.smithi019.stdout:checking whether we are cross compiling... no 2024-08-31T12:17:27.314 INFO:tasks.workunit.client.0.smithi019.stdout:checking for suffix of executables... 2024-08-31T12:17:27.357 INFO:tasks.workunit.client.0.smithi019.stdout:checking for suffix of object files... o 2024-08-31T12:17:27.398 INFO:tasks.workunit.client.0.smithi019.stdout:checking whether we are using the GNU C compiler... yes 2024-08-31T12:17:27.439 INFO:tasks.workunit.client.0.smithi019.stdout:checking whether gcc accepts -g... yes 2024-08-31T12:17:27.503 INFO:tasks.workunit.client.0.smithi019.stdout:checking for gcc option to accept ISO C89... none needed 2024-08-31T12:17:30.867 INFO:tasks.workunit.client.0.smithi019.stdout:checking dependency style of gcc... gcc3 2024-08-31T12:17:30.940 INFO:tasks.workunit.client.0.smithi019.stdout:checking how to run the C preprocessor... gcc -E 2024-08-31T12:17:30.999 INFO:tasks.workunit.client.0.smithi019.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-08-31T12:17:31.000 INFO:tasks.workunit.client.0.smithi019.stdout:checking for egrep... /usr/bin/grep -E 2024-08-31T12:17:31.013 INFO:tasks.workunit.client.0.smithi019.stdout:checking for AIX... no 2024-08-31T12:17:31.038 INFO:tasks.workunit.client.0.smithi019.stdout:checking for gcc... (cached) gcc 2024-08-31T12:17:31.050 INFO:tasks.workunit.client.0.smithi019.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-08-31T12:17:31.052 INFO:tasks.workunit.client.0.smithi019.stdout:checking whether gcc accepts -g... (cached) yes 2024-08-31T12:17:31.052 INFO:tasks.workunit.client.0.smithi019.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-08-31T12:17:31.053 INFO:tasks.workunit.client.0.smithi019.stdout:checking dependency style of gcc... (cached) gcc3 2024-08-31T12:17:31.120 INFO:tasks.workunit.client.0.smithi019.stdout:checking for main in -lm... yes 2024-08-31T12:17:31.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.140+0000 7fa2b1194640 1 -- 172.21.15.19:0/1152545274 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa298006230 msgr2=0x7fa298008700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:31.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.140+0000 7fa2b1194640 1 --2- 172.21.15.19:0/1152545274 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa298006230 0x7fa298008700 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa298006200 tx=0x7fa2a002b520 comp rx=0 tx=0).stop 2024-08-31T12:17:31.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.140+0000 7fa2ab59e640 1 -- 172.21.15.19:0/1152545274 reap_dead start 2024-08-31T12:17:31.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.140+0000 7fa2b1194640 1 -- 172.21.15.19:0/1152545274 shutdown_connections 2024-08-31T12:17:31.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.140+0000 7fa2b1194640 1 --2- 172.21.15.19:0/1152545274 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa298006230 0x7fa298008700 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:31.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.140+0000 7fa2b1194640 1 -- 172.21.15.19:0/1152545274 >> 172.21.15.19:0/1152545274 conn(0x7fa2ac06f9d0 msgr2=0x7fa2ac06e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:17:31.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.140+0000 7fa2b1194640 1 -- 172.21.15.19:0/1152545274 shutdown_connections 2024-08-31T12:17:31.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.140+0000 7fa2b1194640 1 -- 172.21.15.19:0/1152545274 wait complete. 2024-08-31T12:17:31.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.142+0000 7fa2b1194640 1 Processor -- start 2024-08-31T12:17:31.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.142+0000 7fa2b1194640 1 -- start start 2024-08-31T12:17:31.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.144+0000 7fa2b1194640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa298006230 0x7fa2ac14c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:31.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.144+0000 7fa2b1194640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa2ac159450 0x7fa2ac15d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:31.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.144+0000 7fa2b1194640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa2ac14da80 0x7fa2ac15c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:31.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.144+0000 7fa2b1194640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ac15ee20 con 0x7fa2ac14da80 2024-08-31T12:17:31.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.144+0000 7fa2b1194640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ac15eb40 con 0x7fa2ac159450 2024-08-31T12:17:31.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.144+0000 7fa2b1194640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fa2ac0b5110 con 0x7fa298006230 2024-08-31T12:17:31.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.144+0000 7fa2aad9d640 1 -- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa298006230 msgr2=0x7fa2ac14c310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3301/0 2024-08-31T12:17:31.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.144+0000 7fa2aad9d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa298006230 0x7fa2ac14c310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:17:31.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.144+0000 7fa2ab59e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa2ac14da80 0x7fa2ac15c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:31.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.144+0000 7fa2ab59e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa2ac14da80 0x7fa2ac15c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:46494/0 (socket says 172.21.15.19:46494) 2024-08-31T12:17:31.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.144+0000 7fa2ab59e640 1 -- 172.21.15.19:0/1275438242 learned_addr learned my addr 172.21.15.19:0/1275438242 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:17:31.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.144+0000 7fa2aa59c640 1 --2- 172.21.15.19:0/1275438242 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa2ac159450 0x7fa2ac15d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:31.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.144+0000 7fa2ab59e640 1 -- 172.21.15.19:0/1275438242 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa298006230 msgr2=0x7fa2ac14c310 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:17:31.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.144+0000 7fa2ab59e640 1 --2- 172.21.15.19:0/1275438242 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa298006230 0x7fa2ac14c310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:31.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.144+0000 7fa2ab59e640 1 -- 172.21.15.19:0/1275438242 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa2ac159450 msgr2=0x7fa2ac15d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:31.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.144+0000 7fa2ab59e640 1 --2- 172.21.15.19:0/1275438242 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa2ac159450 0x7fa2ac15d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:31.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.144+0000 7fa2ab59e640 1 -- 172.21.15.19:0/1275438242 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2ac15d6d0 con 0x7fa2ac14da80 2024-08-31T12:17:31.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.144+0000 7fa2ab59e640 1 --2- 172.21.15.19:0/1275438242 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa2ac14da80 0x7fa2ac15c4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac0c7920 tx=0x7fa2a400cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:31.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.145+0000 7fa2937fe640 1 -- 172.21.15.19:0/1275438242 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a4017070 con 0x7fa2ac14da80 2024-08-31T12:17:31.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.145+0000 7fa2937fe640 1 -- 172.21.15.19:0/1275438242 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa2a400acc0 con 0x7fa2ac14da80 2024-08-31T12:17:31.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.145+0000 7fa2b1194640 1 -- 172.21.15.19:0/1275438242 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2ac14cab0 con 0x7fa2ac14da80 2024-08-31T12:17:31.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.145+0000 7fa2937fe640 1 -- 172.21.15.19:0/1275438242 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a40072d0 con 0x7fa2ac14da80 2024-08-31T12:17:31.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.145+0000 7fa2b1194640 1 -- 172.21.15.19:0/1275438242 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2ac14d050 con 0x7fa2ac14da80 2024-08-31T12:17:31.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.146+0000 7fa2b1194640 1 -- 172.21.15.19:0/1275438242 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2ac0b3390 con 0x7fa2ac14da80 2024-08-31T12:17:31.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.147+0000 7fa2937fe640 1 -- 172.21.15.19:0/1275438242 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fa2a401a070 con 0x7fa2ac14da80 2024-08-31T12:17:31.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.147+0000 7fa2937fe640 1 --2- 172.21.15.19:0/1275438242 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa2980787b0 0x7fa29807acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:31.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.147+0000 7fa2937fe640 1 -- 172.21.15.19:0/1275438242 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fa2a4099150 con 0x7fa2ac14da80 2024-08-31T12:17:31.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.147+0000 7fa2aad9d640 1 --2- 172.21.15.19:0/1275438242 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa2980787b0 0x7fa29807acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:31.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.148+0000 7fa2aad9d640 1 --2- 172.21.15.19:0/1275438242 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa2980787b0 0x7fa29807acc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa29c0094b0 tx=0x7fa29c00b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:31.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.152+0000 7fa2937fe640 1 -- 172.21.15.19:0/1275438242 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fa2a4061800 con 0x7fa2ac14da80 2024-08-31T12:17:31.199 INFO:tasks.workunit.client.0.smithi019.stdout:checking for main in -lpthread... yes 2024-08-31T12:17:31.407 INFO:tasks.workunit.client.0.smithi019.stdout:checking for ANSI C header files... yes 2024-08-31T12:17:31.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.435+0000 7fa2b1194640 1 -- 172.21.15.19:0/1275438242 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa2ac0cc860 con 0x7fa2ac14da80 2024-08-31T12:17:31.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.435+0000 7fa2937fe640 1 -- 172.21.15.19:0/1275438242 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7fa2a406a300 con 0x7fa2ac14da80 2024-08-31T12:17:31.436 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":14,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:17:31.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.437+0000 7fa2b1194640 1 -- 172.21.15.19:0/1275438242 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa2980787b0 msgr2=0x7fa29807acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:31.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.437+0000 7fa2b1194640 1 --2- 172.21.15.19:0/1275438242 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa2980787b0 0x7fa29807acc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa29c0094b0 tx=0x7fa29c00b040 comp rx=0 tx=0).stop 2024-08-31T12:17:31.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.437+0000 7fa2b1194640 1 -- 172.21.15.19:0/1275438242 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa2ac14da80 msgr2=0x7fa2ac15c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:31.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.437+0000 7fa2b1194640 1 --2- 172.21.15.19:0/1275438242 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa2ac14da80 0x7fa2ac15c4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac0c7920 tx=0x7fa2a400cf00 comp rx=0 tx=0).stop 2024-08-31T12:17:31.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.437+0000 7fa2b1194640 1 -- 172.21.15.19:0/1275438242 shutdown_connections 2024-08-31T12:17:31.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.437+0000 7fa2b1194640 1 --2- 172.21.15.19:0/1275438242 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa2980787b0 0x7fa29807acc0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:31.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.437+0000 7fa2b1194640 1 --2- 172.21.15.19:0/1275438242 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa2ac14da80 0x7fa2ac15c4e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:31.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.437+0000 7fa2b1194640 1 --2- 172.21.15.19:0/1275438242 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa2ac159450 0x7fa2ac15d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:31.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.437+0000 7fa2b1194640 1 --2- 172.21.15.19:0/1275438242 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa298006230 0x7fa2ac14c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:31.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.437+0000 7fa2b1194640 1 -- 172.21.15.19:0/1275438242 >> 172.21.15.19:0/1275438242 conn(0x7fa2ac06f9d0 msgr2=0x7fa2ac0c6bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:17:31.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.437+0000 7fa2b1194640 1 -- 172.21.15.19:0/1275438242 shutdown_connections 2024-08-31T12:17:31.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.437+0000 7fa2b1194640 1 -- 172.21.15.19:0/1275438242 wait complete. 2024-08-31T12:17:31.449 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-08-31T12:17:31.449 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-08-31T12:17:31.449 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-31T12:17:31.545 INFO:tasks.workunit.client.0.smithi019.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-08-31T12:17:31.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.568+0000 7fd309053640 1 -- 172.21.15.19:0/2091638829 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd2ec003ee0 msgr2=0x7fd2ec00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:31.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.568+0000 7fd309053640 1 --2- 172.21.15.19:0/2091638829 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd2ec003ee0 0x7fd2ec00b9c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc001fa0 tx=0x7fd2ec005830 comp rx=0 tx=0).stop 2024-08-31T12:17:31.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.568+0000 7fd30359e640 1 -- 172.21.15.19:0/2091638829 reap_dead start 2024-08-31T12:17:31.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.568+0000 7fd309053640 1 -- 172.21.15.19:0/2091638829 shutdown_connections 2024-08-31T12:17:31.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.568+0000 7fd309053640 1 -- 172.21.15.19:0/2091638829 >> 172.21.15.19:0/2091638829 conn(0x7fd3040cd7d0 msgr2=0x7fd3040cdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:17:31.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.568+0000 7fd309053640 1 -- 172.21.15.19:0/2091638829 shutdown_connections 2024-08-31T12:17:31.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.569+0000 7fd309053640 1 -- 172.21.15.19:0/2091638829 wait complete. 2024-08-31T12:17:31.571 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.571+0000 7fd309053640 1 Processor -- start 2024-08-31T12:17:31.571 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.571+0000 7fd309053640 1 -- start start 2024-08-31T12:17:31.574 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.572+0000 7fd309053640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd2ec003ee0 0x7fd304154b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:31.574 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.572+0000 7fd309053640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd304153800 0x7fd3041e5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:31.574 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.572+0000 7fd309053640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd30413cd10 0x7fd30413d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:31.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.572+0000 7fd309053640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fd3040c8170 con 0x7fd30413cd10 2024-08-31T12:17:31.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.572+0000 7fd309053640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd3040c6920 con 0x7fd304153800 2024-08-31T12:17:31.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.572+0000 7fd309053640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fd30415b2b0 con 0x7fd2ec003ee0 2024-08-31T12:17:31.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.572+0000 7fd302d9d640 1 -- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd2ec003ee0 msgr2=0x7fd304154b40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3301/0 2024-08-31T12:17:31.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.572+0000 7fd302d9d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd2ec003ee0 0x7fd304154b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:17:31.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.572+0000 7fd30359e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd30413cd10 0x7fd30413d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:31.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.572+0000 7fd30359e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd30413cd10 0x7fd30413d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:46504/0 (socket says 172.21.15.19:46504) 2024-08-31T12:17:31.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.572+0000 7fd30359e640 1 -- 172.21.15.19:0/3846092470 learned_addr learned my addr 172.21.15.19:0/3846092470 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:17:31.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.572+0000 7fd30259c640 1 --2- 172.21.15.19:0/3846092470 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd304153800 0x7fd3041e5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:31.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.572+0000 7fd30359e640 1 -- 172.21.15.19:0/3846092470 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd2ec003ee0 msgr2=0x7fd304154b40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:17:31.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.572+0000 7fd30359e640 1 --2- 172.21.15.19:0/3846092470 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd2ec003ee0 0x7fd304154b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:31.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.572+0000 7fd30359e640 1 -- 172.21.15.19:0/3846092470 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd304153800 msgr2=0x7fd3041e5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:31.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.572+0000 7fd30359e640 1 --2- 172.21.15.19:0/3846092470 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd304153800 0x7fd3041e5010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:31.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.572+0000 7fd30359e640 1 -- 172.21.15.19:0/3846092470 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd30415a130 con 0x7fd30413cd10 2024-08-31T12:17:31.576 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.573+0000 7fd30259c640 1 --2- 172.21.15.19:0/3846092470 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd304153800 0x7fd3041e5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:17:31.576 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.573+0000 7fd30359e640 1 --2- 172.21.15.19:0/3846092470 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd30413cd10 0x7fd30413d120 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc0020c0 tx=0x7fd2fc006810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:31.576 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.573+0000 7fd2eb7fe640 1 -- 172.21.15.19:0/3846092470 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc042070 con 0x7fd30413cd10 2024-08-31T12:17:31.576 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.573+0000 7fd2eb7fe640 1 -- 172.21.15.19:0/3846092470 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd2fc003e20 con 0x7fd30413cd10 2024-08-31T12:17:31.576 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.573+0000 7fd309053640 1 -- 172.21.15.19:0/3846092470 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd304154080 con 0x7fd30413cd10 2024-08-31T12:17:31.576 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.573+0000 7fd2eb7fe640 1 -- 172.21.15.19:0/3846092470 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc02e070 con 0x7fd30413cd10 2024-08-31T12:17:31.576 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.573+0000 7fd309053640 1 -- 172.21.15.19:0/3846092470 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3041534e0 con 0x7fd30413cd10 2024-08-31T12:17:31.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.574+0000 7fd309053640 1 --2- 172.21.15.19:0/3846092470 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd3041f1190 0x7fd30406cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:31.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.574+0000 7fd309053640 1 -- 172.21.15.19:0/3846092470 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fd304152f70 con 0x7fd3041f1190 2024-08-31T12:17:31.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.574+0000 7fd302d9d640 1 --2- 172.21.15.19:0/3846092470 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd3041f1190 0x7fd30406cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:31.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.575+0000 7fd302d9d640 1 -- 172.21.15.19:0/3846092470 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd3041f15a0 con 0x7fd3041f1190 2024-08-31T12:17:31.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.575+0000 7fd302d9d640 1 --2- 172.21.15.19:0/3846092470 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd3041f1190 0x7fd30406cae0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd2f400c680 tx=0x7fd2f400cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:31.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.575+0000 7fd2eb7fe640 1 -- 172.21.15.19:0/3846092470 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f4017070 con 0x7fd3041f1190 2024-08-31T12:17:31.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.575+0000 7fd2eb7fe640 1 -- 172.21.15.19:0/3846092470 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fd2fc045070 con 0x7fd30413cd10 2024-08-31T12:17:31.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.576+0000 7fd2eb7fe640 1 --2- 172.21.15.19:0/3846092470 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd2f0077930 0x7fd2f0079e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:31.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.576+0000 7fd2eb7fe640 1 -- 172.21.15.19:0/3846092470 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fd2fc0b8d70 con 0x7fd30413cd10 2024-08-31T12:17:31.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.576+0000 7fd2eb7fe640 1 -- 172.21.15.19:0/3846092470 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd2f400ce80 con 0x7fd3041f1190 2024-08-31T12:17:31.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.576+0000 7fd2eb7fe640 1 -- 172.21.15.19:0/3846092470 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd3041f1190 msgr2=0x7fd30406cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:31.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.576+0000 7fd2eb7fe640 1 --2- 172.21.15.19:0/3846092470 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd3041f1190 0x7fd30406cae0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd2f400c680 tx=0x7fd2f400cb40 comp rx=0 tx=0).stop 2024-08-31T12:17:31.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.576+0000 7fd2eb7fe640 1 -- 172.21.15.19:0/3846092470 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd3041f1190 msgr2=0x7fd30406cae0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:17:31.579 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.576+0000 7fd2eb7fe640 1 --2- 172.21.15.19:0/3846092470 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd3041f1190 0x7fd30406cae0 secure :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd2f400c680 tx=0x7fd2f400cb40 comp rx=0 tx=0).stop 2024-08-31T12:17:31.579 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.576+0000 7fd30259c640 1 --2- 172.21.15.19:0/3846092470 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd2f0077930 0x7fd2f0079e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:31.579 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.576+0000 7fd30259c640 1 --2- 172.21.15.19:0/3846092470 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd2f0077930 0x7fd2f0079e40 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8001020 tx=0x7fd2f8005ea0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:31.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.587+0000 7fd309053640 1 --2- 172.21.15.19:0/3846092470 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd304154250 0x7fd30406d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:31.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.587+0000 7fd309053640 1 -- 172.21.15.19:0/3846092470 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fd304141240 con 0x7fd304154250 2024-08-31T12:17:31.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.587+0000 7fd302d9d640 1 --2- 172.21.15.19:0/3846092470 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd304154250 0x7fd30406d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:31.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.587+0000 7fd302d9d640 1 -- 172.21.15.19:0/3846092470 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd3040c89e0 con 0x7fd304154250 2024-08-31T12:17:31.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.587+0000 7fd302d9d640 1 --2- 172.21.15.19:0/3846092470 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd304154250 0x7fd30406d020 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd2f400c680 tx=0x7fd3040c5410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:31.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.588+0000 7fd2eb7fe640 1 -- 172.21.15.19:0/3846092470 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f401a6c0 con 0x7fd304154250 2024-08-31T12:17:31.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.588+0000 7fd2eb7fe640 1 -- 172.21.15.19:0/3846092470 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4865 (secure 0 0 0) 0x7fd2f400a660 con 0x7fd304154250 2024-08-31T12:17:31.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.588+0000 7fd2eb7fe640 1 -- 172.21.15.19:0/3846092470 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd304154250 msgr2=0x7fd30406d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:31.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.588+0000 7fd2eb7fe640 1 --2- 172.21.15.19:0/3846092470 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd304154250 0x7fd30406d020 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd2f400c680 tx=0x7fd3040c5410 comp rx=0 tx=0).stop 2024-08-31T12:17:31.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.588+0000 7fd2eb7fe640 1 -- 172.21.15.19:0/3846092470 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd304154250 msgr2=0x7fd30406d020 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:17:31.590 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.588+0000 7fd2eb7fe640 1 --2- 172.21.15.19:0/3846092470 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd304154250 0x7fd30406d020 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:31.590 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:17:31.590 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:17:31.590 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:17:31.590 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "leader", 2024-08-31T12:17:31.590 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 14, 2024-08-31T12:17:31.590 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 31579, 2024-08-31T12:17:31.590 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:17:31.590 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:17:31.591 INFO:teuthology.orchestra.run.smithi019.stdout: 1 2024-08-31T12:17:31.591 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:31.591 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 0, 2024-08-31T12:17:31.591 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:17:31.591 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:17:31.591 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:17:31.591 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:17:31.591 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:17:31.591 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:17:31.591 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:17:31.591 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:17:31.591 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:17:31.592 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:17:31.592 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:17:31.592 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:17:31.592 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:17:31.592 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:17:31.592 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:31.592 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:17:31.592 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:17:31.592 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:17:31.592 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:17:31.592 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:17:31.592 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:17:31.593 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:17:31.593 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:17:31.593 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:17:31.593 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:17:31.593 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:17:31.593 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:17:31.593 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:17:31.593 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:31.593 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:31.593 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:17:31.593 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:17:31.593 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:17:31.594 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:17:31.594 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:17:31.594 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:17:31.594 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:17:31.594 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:17:31.594 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:17:31.594 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:17:31.594 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:17:31.594 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:17:31.594 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:17:31.594 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:17:31.594 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:17:31.595 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:17:31.595 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:17:31.595 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:17:31.595 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:17:31.595 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:17:31.595 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:17:31.595 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:17:31.595 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:17:31.595 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:17:31.595 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:17:31.595 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:17:31.595 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:17:31.595 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:17:31.596 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:31.596 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:17:31.596 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:31.596 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:17:31.596 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:31.596 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:17:31.596 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:17:31.596 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:17:31.596 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:17:31.596 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:31.596 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:17:31.596 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:17:31.597 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:31.597 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:31.597 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:31.597 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:17:31.597 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:17:31.597 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:31.597 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:31.597 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:31.597 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:31.597 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:17:31.597 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:17:31.597 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:17:31.597 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:17:31.598 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:17:31.598 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:31.598 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:31.598 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:17:31.598 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:17:31.598 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:17:31.598 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:17:31.598 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:31.598 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:17:31.598 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:17:31.598 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:31.598 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:31.598 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:31.598 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:17:31.599 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:17:31.599 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:31.599 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:31.599 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:31.599 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:31.599 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:17:31.599 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:17:31.599 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:17:31.599 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:17:31.599 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:17:31.599 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:31.599 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:31.599 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:17:31.600 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:17:31.600 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:17:31.600 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:17:31.600 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:31.600 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:17:31.600 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:17:31.600 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:31.600 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:31.600 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:31.600 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:17:31.600 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:17:31.600 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:31.600 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:31.601 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:31.601 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:31.601 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:17:31.601 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:17:31.601 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:17:31.601 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:17:31.601 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:17:31.601 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:31.601 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:31.601 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:31.601 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:17:31.601 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:17:31.602 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:31.602 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:17:31.602 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:17:31.602 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:17:31.602 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:31.602 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:31.602 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:17:31.602 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:31.602 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:17:31.602 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:17:31.602 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:17:31.602 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:31.602 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:31.602 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:17:31.603 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:31.603 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:17:31.603 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:17:31.603 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:17:31.603 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:31.603 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:31.603 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:17:31.603 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:31.603 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:17:31.603 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:17:31.603 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:17:31.603 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:31.603 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:31.604 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:31.604 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:17:31.604 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:17:31.605 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.588+0000 7fd309053640 1 -- 172.21.15.19:0/3846092470 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd2f0077930 msgr2=0x7fd2f0079e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:31.605 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.588+0000 7fd309053640 1 --2- 172.21.15.19:0/3846092470 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd2f0077930 0x7fd2f0079e40 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8001020 tx=0x7fd2f8005ea0 comp rx=0 tx=0).stop 2024-08-31T12:17:31.605 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.588+0000 7fd309053640 1 -- 172.21.15.19:0/3846092470 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd30413cd10 msgr2=0x7fd30413d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:31.605 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.588+0000 7fd309053640 1 --2- 172.21.15.19:0/3846092470 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd30413cd10 0x7fd30413d120 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc0020c0 tx=0x7fd2fc006810 comp rx=0 tx=0).stop 2024-08-31T12:17:31.605 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.588+0000 7fd30359e640 1 -- 172.21.15.19:0/3846092470 reap_dead start 2024-08-31T12:17:31.605 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.588+0000 7fd30359e640 1 -- 172.21.15.19:0/3846092470 reap_dead start 2024-08-31T12:17:31.605 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.588+0000 7fd309053640 1 -- 172.21.15.19:0/3846092470 shutdown_connections 2024-08-31T12:17:31.605 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.588+0000 7fd309053640 1 -- 172.21.15.19:0/3846092470 >> 172.21.15.19:0/3846092470 conn(0x7fd3040cd7d0 msgr2=0x7fd3040cbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:17:31.606 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.588+0000 7fd309053640 1 -- 172.21.15.19:0/3846092470 shutdown_connections 2024-08-31T12:17:31.606 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.588+0000 7fd309053640 1 -- 172.21.15.19:0/3846092470 wait complete. 2024-08-31T12:17:31.606 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-08-31T12:17:31.606 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-08-31T12:17:31.606 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-31T12:17:31.609 INFO:tasks.workunit.client.0.smithi019.stdout:checking for sys/types.h... yes 2024-08-31T12:17:31.680 INFO:tasks.workunit.client.0.smithi019.stdout:checking for sys/stat.h... yes 2024-08-31T12:17:31.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.728+0000 7fc267e63640 1 -- 172.21.15.19:0/71418413 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc248003bc0 msgr2=0x7fc248004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:31.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.728+0000 7fc267e63640 1 --2- 172.21.15.19:0/71418413 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc248003bc0 0x7fc248004010 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc250001f20 tx=0x7fc248005830 comp rx=0 tx=0).stop 2024-08-31T12:17:31.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.728+0000 7fc266400640 1 -- 172.21.15.19:0/71418413 reap_dead start 2024-08-31T12:17:31.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.728+0000 7fc267e63640 1 -- 172.21.15.19:0/71418413 shutdown_connections 2024-08-31T12:17:31.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.728+0000 7fc267e63640 1 -- 172.21.15.19:0/71418413 >> 172.21.15.19:0/71418413 conn(0x7fc26006e220 msgr2=0x7fc2600ca000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:17:31.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.728+0000 7fc267e63640 1 -- 172.21.15.19:0/71418413 shutdown_connections 2024-08-31T12:17:31.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.728+0000 7fc267e63640 1 -- 172.21.15.19:0/71418413 wait complete. 2024-08-31T12:17:31.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.730+0000 7fc267e63640 1 Processor -- start 2024-08-31T12:17:31.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.730+0000 7fc267e63640 1 -- start start 2024-08-31T12:17:31.734 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.732+0000 7fc267e63640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc248003bc0 0x7fc2600cba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:31.735 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.732+0000 7fc267e63640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc260158c40 0x7fc260146130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:31.735 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.732+0000 7fc267e63640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc260144eb0 0x7fc2601452c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:31.735 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.732+0000 7fc267e63640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fc2600c74e0 con 0x7fc260158c40 2024-08-31T12:17:31.735 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.732+0000 7fc267e63640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc2600c85a0 con 0x7fc260144eb0 2024-08-31T12:17:31.735 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.732+0000 7fc267e63640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fc2600c7660 con 0x7fc248003bc0 2024-08-31T12:17:31.735 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.732+0000 7fc265bff640 1 -- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc248003bc0 msgr2=0x7fc2600cba00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3301/0 2024-08-31T12:17:31.735 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.732+0000 7fc265bff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc248003bc0 0x7fc2600cba00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:17:31.735 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.732+0000 7fc2653fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc260158c40 0x7fc260146130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:31.735 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.732+0000 7fc2653fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc260158c40 0x7fc260146130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:46544/0 (socket says 172.21.15.19:46544) 2024-08-31T12:17:31.735 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.732+0000 7fc2653fe640 1 -- 172.21.15.19:0/831180207 learned_addr learned my addr 172.21.15.19:0/831180207 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:17:31.735 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.733+0000 7fc266400640 1 --2- 172.21.15.19:0/831180207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc260144eb0 0x7fc2601452c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:31.735 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.733+0000 7fc2653fe640 1 -- 172.21.15.19:0/831180207 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc248003bc0 msgr2=0x7fc2600cba00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:17:31.735 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.733+0000 7fc2653fe640 1 --2- 172.21.15.19:0/831180207 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc248003bc0 0x7fc2600cba00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:31.735 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.733+0000 7fc2653fe640 1 -- 172.21.15.19:0/831180207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc260144eb0 msgr2=0x7fc2601452c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:31.736 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.733+0000 7fc2653fe640 1 --2- 172.21.15.19:0/831180207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc260144eb0 0x7fc2601452c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:31.736 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.733+0000 7fc2653fe640 1 -- 172.21.15.19:0/831180207 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc260150500 con 0x7fc260158c40 2024-08-31T12:17:31.736 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.733+0000 7fc266400640 1 --2- 172.21.15.19:0/831180207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc260144eb0 0x7fc2601452c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:17:31.736 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.733+0000 7fc2653fe640 1 --2- 172.21.15.19:0/831180207 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc260158c40 0x7fc260146130 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc25400e6b0 tx=0x7fc254009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:31.736 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.733+0000 7fc25e7fc640 1 -- 172.21.15.19:0/831180207 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc254004b70 con 0x7fc260158c40 2024-08-31T12:17:31.736 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.733+0000 7fc267e63640 1 -- 172.21.15.19:0/831180207 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc260159bb0 con 0x7fc260158c40 2024-08-31T12:17:31.736 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.733+0000 7fc25e7fc640 1 -- 172.21.15.19:0/831180207 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc254004d10 con 0x7fc260158c40 2024-08-31T12:17:31.736 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.733+0000 7fc25e7fc640 1 -- 172.21.15.19:0/831180207 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc25400ca70 con 0x7fc260158c40 2024-08-31T12:17:31.736 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.733+0000 7fc267e63640 1 -- 172.21.15.19:0/831180207 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc26015a110 con 0x7fc260158c40 2024-08-31T12:17:31.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.734+0000 7fc267e63640 1 --2- 172.21.15.19:0/831180207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc26014d250 0x7fc26014d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:31.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.734+0000 7fc267e63640 1 -- 172.21.15.19:0/831180207 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc26006a020 con 0x7fc26014d250 2024-08-31T12:17:31.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.735+0000 7fc265bff640 1 --2- 172.21.15.19:0/831180207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc26014d250 0x7fc26014d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:31.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.735+0000 7fc265bff640 1 -- 172.21.15.19:0/831180207 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc260145800 con 0x7fc26014d250 2024-08-31T12:17:31.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.735+0000 7fc265bff640 1 --2- 172.21.15.19:0/831180207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc26014d250 0x7fc26014d660 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc250001ef0 tx=0x7fc250003ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:31.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.735+0000 7fc25e7fc640 1 -- 172.21.15.19:0/831180207 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc250006050 con 0x7fc26014d250 2024-08-31T12:17:31.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.735+0000 7fc25e7fc640 1 -- 172.21.15.19:0/831180207 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fc25400cc10 con 0x7fc260158c40 2024-08-31T12:17:31.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.736+0000 7fc25e7fc640 1 --2- 172.21.15.19:0/831180207 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc24c077950 0x7fc24c079e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:31.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.736+0000 7fc25e7fc640 1 -- 172.21.15.19:0/831180207 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fc254099fc0 con 0x7fc260158c40 2024-08-31T12:17:31.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.736+0000 7fc266400640 1 --2- 172.21.15.19:0/831180207 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc24c077950 0x7fc24c079e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:31.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.736+0000 7fc25e7fc640 1 -- 172.21.15.19:0/831180207 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc2500061f0 con 0x7fc26014d250 2024-08-31T12:17:31.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.736+0000 7fc25e7fc640 1 -- 172.21.15.19:0/831180207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc26014d250 msgr2=0x7fc26014d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:31.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.736+0000 7fc25e7fc640 1 --2- 172.21.15.19:0/831180207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc26014d250 0x7fc26014d660 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc250001ef0 tx=0x7fc250003ab0 comp rx=0 tx=0).stop 2024-08-31T12:17:31.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.736+0000 7fc25e7fc640 1 -- 172.21.15.19:0/831180207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc26014d250 msgr2=0x7fc26014d660 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:17:31.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.736+0000 7fc25e7fc640 1 --2- 172.21.15.19:0/831180207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc26014d250 0x7fc26014d660 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:31.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.737+0000 7fc266400640 1 --2- 172.21.15.19:0/831180207 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc24c077950 0x7fc24c079e60 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc260070cc0 tx=0x7fc258009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:31.745 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.744+0000 7fc267e63640 1 --2- 172.21.15.19:0/831180207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc26014dba0 0x7fc260156940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:31.745 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.744+0000 7fc267e63640 1 -- 172.21.15.19:0/831180207 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc260151f10 con 0x7fc26014dba0 2024-08-31T12:17:31.745 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.744+0000 7fc265bff640 1 --2- 172.21.15.19:0/831180207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc26014dba0 0x7fc260156940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:31.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.745+0000 7fc265bff640 1 -- 172.21.15.19:0/831180207 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc2601488c0 con 0x7fc26014dba0 2024-08-31T12:17:31.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.745+0000 7fc265bff640 1 --2- 172.21.15.19:0/831180207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc26014dba0 0x7fc260156940 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc260051d70 tx=0x7fc248005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:31.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.745+0000 7fc25e7fc640 1 -- 172.21.15.19:0/831180207 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc250045df0 con 0x7fc26014dba0 2024-08-31T12:17:31.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.745+0000 7fc25e7fc640 1 -- 172.21.15.19:0/831180207 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5031 (secure 0 0 0) 0x7fc250045df0 con 0x7fc26014dba0 2024-08-31T12:17:31.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.745+0000 7fc25e7fc640 1 -- 172.21.15.19:0/831180207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc26014dba0 msgr2=0x7fc260156940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:31.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.745+0000 7fc25e7fc640 1 --2- 172.21.15.19:0/831180207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc26014dba0 0x7fc260156940 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc260051d70 tx=0x7fc248005830 comp rx=0 tx=0).stop 2024-08-31T12:17:31.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.745+0000 7fc25e7fc640 1 -- 172.21.15.19:0/831180207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc26014dba0 msgr2=0x7fc260156940 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:17:31.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.745+0000 7fc25e7fc640 1 --2- 172.21.15.19:0/831180207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc26014dba0 0x7fc260156940 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:31.747 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:17:31.747 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:17:31.747 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:17:31.747 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:17:31.747 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 14, 2024-08-31T12:17:31.747 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 119840, 2024-08-31T12:17:31.747 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:17:31.747 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:17:31.748 INFO:teuthology.orchestra.run.smithi019.stdout: 1 2024-08-31T12:17:31.748 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:31.748 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 1, 2024-08-31T12:17:31.748 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:17:31.748 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:17:31.748 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:17:31.748 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:17:31.748 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:17:31.748 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:17:31.748 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:17:31.748 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:17:31.748 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:17:31.749 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:17:31.749 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:17:31.749 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:17:31.749 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:17:31.749 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:17:31.749 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:31.749 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:17:31.749 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:17:31.749 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:17:31.749 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:17:31.749 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:17:31.749 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:17:31.749 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:17:31.749 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:17:31.750 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:17:31.750 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:17:31.750 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:17:31.750 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:17:31.750 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:17:31.750 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:31.750 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:31.750 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:17:31.750 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:17:31.750 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:17:31.750 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:17:31.750 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:17:31.750 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:17:31.750 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:17:31.750 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:17:31.751 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:17:31.751 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:17:31.751 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:17:31.751 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:17:31.751 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:17:31.751 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:17:31.751 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:17:31.751 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:17:31.751 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:17:31.751 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:17:31.751 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:17:31.751 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:17:31.751 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:17:31.751 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:17:31.752 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:17:31.752 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:17:31.752 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:17:31.752 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:17:31.752 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:17:31.752 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:17:31.752 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:31.752 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:17:31.752 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:31.752 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:17:31.752 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:31.752 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:17:31.752 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:17:31.752 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:17:31.753 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:17:31.753 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:31.753 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:17:31.753 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:17:31.753 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:31.753 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:31.753 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:31.753 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:17:31.753 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:17:31.753 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:31.753 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:31.753 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:31.753 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:31.753 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:17:31.754 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:17:31.754 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:17:31.754 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:17:31.754 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:17:31.754 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:31.754 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:31.754 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:17:31.754 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:17:31.754 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:17:31.754 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:17:31.754 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:31.754 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:17:31.754 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:17:31.754 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:31.755 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:31.755 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:31.755 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:17:31.755 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:17:31.755 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:31.755 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:31.755 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:31.755 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:31.755 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:17:31.755 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:17:31.755 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:17:31.755 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:17:31.755 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:17:31.755 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:31.756 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:31.756 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:17:31.756 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:17:31.756 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:17:31.756 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:17:31.756 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:31.756 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:17:31.756 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:17:31.756 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:31.756 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:31.756 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:31.756 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:17:31.756 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:17:31.756 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:31.757 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:31.757 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:31.757 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:31.757 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:17:31.757 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:17:31.757 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:17:31.757 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:17:31.757 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:17:31.757 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:31.757 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:31.757 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:31.757 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:17:31.757 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:17:31.758 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:31.758 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:17:31.758 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:17:31.758 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:17:31.758 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:31.758 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:31.758 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:17:31.758 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:31.758 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:17:31.758 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:17:31.758 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 5 2024-08-31T12:17:31.758 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:31.758 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:31.759 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:17:31.759 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:31.759 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:17:31.759 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:17:31.759 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 7 2024-08-31T12:17:31.759 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:31.759 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:31.759 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:17:31.759 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:31.759 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:17:31.759 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:17:31.759 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 6 2024-08-31T12:17:31.759 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:31.760 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:31.760 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:17:31.760 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:31.760 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:17:31.760 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:17:31.760 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:17:31.760 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:31.760 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:31.760 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:31.760 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:17:31.760 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:17:31.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.746+0000 7fc267e63640 1 -- 172.21.15.19:0/831180207 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc24c077950 msgr2=0x7fc24c079e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:31.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.746+0000 7fc267e63640 1 --2- 172.21.15.19:0/831180207 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc24c077950 0x7fc24c079e60 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc260070cc0 tx=0x7fc258009040 comp rx=0 tx=0).stop 2024-08-31T12:17:31.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.746+0000 7fc267e63640 1 -- 172.21.15.19:0/831180207 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc260158c40 msgr2=0x7fc260146130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:31.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.746+0000 7fc267e63640 1 --2- 172.21.15.19:0/831180207 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc260158c40 0x7fc260146130 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc25400e6b0 tx=0x7fc254009f90 comp rx=0 tx=0).stop 2024-08-31T12:17:31.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.746+0000 7fc266400640 1 -- 172.21.15.19:0/831180207 reap_dead start 2024-08-31T12:17:31.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.746+0000 7fc266400640 1 -- 172.21.15.19:0/831180207 reap_dead start 2024-08-31T12:17:31.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.746+0000 7fc267e63640 1 -- 172.21.15.19:0/831180207 shutdown_connections 2024-08-31T12:17:31.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.746+0000 7fc267e63640 1 -- 172.21.15.19:0/831180207 >> 172.21.15.19:0/831180207 conn(0x7fc26006e220 msgr2=0x7fc26014ef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:17:31.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.746+0000 7fc267e63640 1 -- 172.21.15.19:0/831180207 shutdown_connections 2024-08-31T12:17:31.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:31.746+0000 7fc267e63640 1 -- 172.21.15.19:0/831180207 wait complete. 2024-08-31T12:17:31.762 INFO:tasks.workunit.client.0.smithi019.stdout:checking for stdlib.h... yes 2024-08-31T12:17:31.763 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-08-31T12:17:31.763 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-08-31T12:17:31.824 INFO:tasks.workunit.client.0.smithi019.stdout:checking for string.h... yes 2024-08-31T12:17:31.894 INFO:tasks.workunit.client.0.smithi019.stdout:checking for memory.h... yes 2024-08-31T12:17:31.957 INFO:tasks.workunit.client.0.smithi019.stdout:checking for strings.h... yes 2024-08-31T12:17:32.019 INFO:tasks.workunit.client.0.smithi019.stdout:checking for inttypes.h... yes 2024-08-31T12:17:32.085 INFO:tasks.workunit.client.0.smithi019.stdout:checking for stdint.h... yes 2024-08-31T12:17:32.154 INFO:tasks.workunit.client.0.smithi019.stdout:checking for unistd.h... yes 2024-08-31T12:17:32.223 INFO:tasks.workunit.client.0.smithi019.stdout:checking pthread.h usability... yes 2024-08-31T12:17:32.252 INFO:tasks.workunit.client.0.smithi019.stdout:checking pthread.h presence... yes 2024-08-31T12:17:32.253 INFO:tasks.workunit.client.0.smithi019.stdout:checking for pthread.h... yes 2024-08-31T12:17:32.320 INFO:tasks.workunit.client.0.smithi019.stdout:checking fcntl.h usability... yes 2024-08-31T12:17:32.363 INFO:tasks.workunit.client.0.smithi019.stdout:checking fcntl.h presence... yes 2024-08-31T12:17:32.364 INFO:tasks.workunit.client.0.smithi019.stdout:checking for fcntl.h... yes 2024-08-31T12:17:32.427 INFO:tasks.workunit.client.0.smithi019.stdout:checking limits.h usability... yes 2024-08-31T12:17:32.454 INFO:tasks.workunit.client.0.smithi019.stdout:checking limits.h presence... yes 2024-08-31T12:17:32.455 INFO:tasks.workunit.client.0.smithi019.stdout:checking for limits.h... yes 2024-08-31T12:17:32.481 INFO:tasks.workunit.client.0.smithi019.stdout:checking for stdint.h... (cached) yes 2024-08-31T12:17:32.547 INFO:tasks.workunit.client.0.smithi019.stdout:checking sys/time.h usability... yes 2024-08-31T12:17:32.574 INFO:tasks.workunit.client.0.smithi019.stdout:checking sys/time.h presence... yes 2024-08-31T12:17:32.575 INFO:tasks.workunit.client.0.smithi019.stdout:checking for sys/time.h... yes 2024-08-31T12:17:32.598 INFO:tasks.workunit.client.0.smithi019.stdout:checking for unistd.h... (cached) yes 2024-08-31T12:17:32.667 INFO:tasks.workunit.client.0.smithi019.stdout:checking sys/vfs.h usability... yes 2024-08-31T12:17:32.698 INFO:tasks.workunit.client.0.smithi019.stdout:checking sys/vfs.h presence... yes 2024-08-31T12:17:32.698 INFO:tasks.workunit.client.0.smithi019.stdout:checking for sys/vfs.h... yes 2024-08-31T12:17:32.765 INFO:tasks.workunit.client.0.smithi019.stdout:checking sys/limits.h usability... no 2024-08-31T12:17:32.798 INFO:tasks.workunit.client.0.smithi019.stdout:checking sys/limits.h presence... no 2024-08-31T12:17:32.798 INFO:tasks.workunit.client.0.smithi019.stdout:checking for sys/limits.h... no 2024-08-31T12:17:32.842 INFO:tasks.workunit.client.0.smithi019.stdout:checking for an ANSI C-conforming const... yes 2024-08-31T12:17:32.893 INFO:tasks.workunit.client.0.smithi019.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-08-31T12:17:32.896 INFO:tasks.workunit.client.0.smithi019.stdout:checking for function prototypes... yes 2024-08-31T12:17:32.938 INFO:tasks.workunit.client.0.smithi019.stdout:checking whether setvbuf arguments are reversed... no 2024-08-31T12:17:33.021 INFO:tasks.workunit.client.0.smithi019.stdout:checking for system... yes 2024-08-31T12:17:33.104 INFO:tasks.workunit.client.0.smithi019.stdout:checking for gettimeofday... yes 2024-08-31T12:17:33.186 INFO:tasks.workunit.client.0.smithi019.stdout:checking for mkdir... yes 2024-08-31T12:17:33.270 INFO:tasks.workunit.client.0.smithi019.stdout:checking for strerror... yes 2024-08-31T12:17:33.356 INFO:tasks.workunit.client.0.smithi019.stdout:checking for open64... yes 2024-08-31T12:17:33.447 INFO:tasks.workunit.client.0.smithi019.stdout:checking for stat64... yes 2024-08-31T12:17:33.539 INFO:tasks.workunit.client.0.smithi019.stdout:checking for fseeko64... yes 2024-08-31T12:17:33.626 INFO:tasks.workunit.client.0.smithi019.stdout:checking for lrand48_r... yes 2024-08-31T12:17:33.714 INFO:tasks.workunit.client.0.smithi019.stdout:checking for srand48_r... yes 2024-08-31T12:17:33.743 INFO:tasks.workunit.client.0.smithi019.stdout:configure: creating ./config.status 2024-08-31T12:17:33.878 INFO:tasks.workunit.client.0.smithi019.stdout:config.status: creating Makefile 2024-08-31T12:17:33.895 INFO:tasks.workunit.client.0.smithi019.stdout:config.status: creating config.h 2024-08-31T12:17:33.921 INFO:tasks.workunit.client.0.smithi019.stdout:config.status: executing depfiles commands 2024-08-31T12:17:33.992 INFO:tasks.workunit.client.0.smithi019.stdout:config.status: executing default commands 2024-08-31T12:17:34.056 INFO:tasks.workunit.client.0.smithi019.stderr:+ make 2024-08-31T12:17:34.083 INFO:tasks.workunit.client.0.smithi019.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run aclocal-1.10 2024-08-31T12:17:34.090 INFO:tasks.workunit.client.0.smithi019.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: aclocal-1.10: command not found 2024-08-31T12:17:34.091 INFO:tasks.workunit.client.0.smithi019.stderr:WARNING: `aclocal-1.10' is missing on your system. You should only need it if 2024-08-31T12:17:34.091 INFO:tasks.workunit.client.0.smithi019.stderr: you modified `acinclude.m4' or `configure.in'. You might want 2024-08-31T12:17:34.091 INFO:tasks.workunit.client.0.smithi019.stderr: to install the `Automake' and `Perl' packages. Grab them from 2024-08-31T12:17:34.091 INFO:tasks.workunit.client.0.smithi019.stderr: any GNU archive site. 2024-08-31T12:17:34.096 INFO:tasks.workunit.client.0.smithi019.stdout: cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run automake-1.10 --gnu 2024-08-31T12:17:34.101 INFO:tasks.workunit.client.0.smithi019.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: automake-1.10: command not found 2024-08-31T12:17:34.102 INFO:tasks.workunit.client.0.smithi019.stderr:WARNING: `automake-1.10' is missing on your system. You should only need it if 2024-08-31T12:17:34.102 INFO:tasks.workunit.client.0.smithi019.stderr: you modified `Makefile.am', `acinclude.m4' or `configure.in'. 2024-08-31T12:17:34.102 INFO:tasks.workunit.client.0.smithi019.stderr: You might want to install the `Automake' and `Perl' packages. 2024-08-31T12:17:34.102 INFO:tasks.workunit.client.0.smithi019.stderr: Grab them from any GNU archive site. 2024-08-31T12:17:34.275 INFO:tasks.workunit.client.0.smithi019.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoconf 2024-08-31T12:17:34.572 INFO:tasks.workunit.client.0.smithi019.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2024-08-31T12:17:34.572 INFO:tasks.workunit.client.0.smithi019.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2024-08-31T12:17:34.572 INFO:tasks.workunit.client.0.smithi019.stderr:If you have problems, you may need to regenerate the build system entirely. 2024-08-31T12:17:34.572 INFO:tasks.workunit.client.0.smithi019.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2024-08-31T12:17:34.711 INFO:tasks.workunit.client.0.smithi019.stdout:/bin/sh ./config.status --recheck 2024-08-31T12:17:34.772 INFO:tasks.workunit.client.0.smithi019.stdout:running CONFIG_SHELL=/bin/sh /bin/sh ./configure --no-create --no-recursion 2024-08-31T12:17:34.916 INFO:tasks.workunit.client.0.smithi019.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-08-31T12:17:35.927 INFO:tasks.workunit.client.0.smithi019.stdout:checking whether build environment is sane... yes 2024-08-31T12:17:35.937 INFO:tasks.workunit.client.0.smithi019.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-08-31T12:17:35.938 INFO:tasks.workunit.client.0.smithi019.stdout:checking for gawk... gawk 2024-08-31T12:17:35.963 INFO:tasks.workunit.client.0.smithi019.stdout:checking whether make sets $(MAKE)... yes 2024-08-31T12:17:36.010 INFO:tasks.workunit.client.0.smithi019.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-08-31T12:17:36.010 INFO:tasks.workunit.client.0.smithi019.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-08-31T12:17:36.029 INFO:tasks.workunit.client.0.smithi019.stdout:checking for style of include used by make... GNU 2024-08-31T12:17:36.035 INFO:tasks.workunit.client.0.smithi019.stdout:checking for gcc... gcc 2024-08-31T12:17:36.156 INFO:tasks.workunit.client.0.smithi019.stdout:checking whether the C compiler works... yes 2024-08-31T12:17:36.156 INFO:tasks.workunit.client.0.smithi019.stdout:checking for C compiler default output file name... a.out 2024-08-31T12:17:36.208 INFO:tasks.workunit.client.0.smithi019.stdout:checking for suffix of executables... 2024-08-31T12:17:36.261 INFO:tasks.workunit.client.0.smithi019.stdout:checking whether we are cross compiling... no 2024-08-31T12:17:36.307 INFO:tasks.workunit.client.0.smithi019.stdout:checking for suffix of object files... o 2024-08-31T12:17:36.339 INFO:tasks.workunit.client.0.smithi019.stdout:checking whether we are using the GNU C compiler... yes 2024-08-31T12:17:36.372 INFO:tasks.workunit.client.0.smithi019.stdout:checking whether gcc accepts -g... yes 2024-08-31T12:17:36.418 INFO:tasks.workunit.client.0.smithi019.stdout:checking for gcc option to accept ISO C89... none needed 2024-08-31T12:17:36.580 INFO:tasks.workunit.client.0.smithi019.stdout:checking dependency style of gcc... gcc3 2024-08-31T12:17:36.644 INFO:tasks.workunit.client.0.smithi019.stdout:checking how to run the C preprocessor... gcc -E 2024-08-31T12:17:36.707 INFO:tasks.workunit.client.0.smithi019.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-08-31T12:17:36.708 INFO:tasks.workunit.client.0.smithi019.stdout:checking for egrep... /usr/bin/grep -E 2024-08-31T12:17:36.875 INFO:tasks.workunit.client.0.smithi019.stdout:checking for ANSI C header files... yes 2024-08-31T12:17:36.921 INFO:tasks.workunit.client.0.smithi019.stdout:checking for sys/types.h... yes 2024-08-31T12:17:36.973 INFO:tasks.workunit.client.0.smithi019.stdout:checking for sys/stat.h... yes 2024-08-31T12:17:37.023 INFO:tasks.workunit.client.0.smithi019.stdout:checking for stdlib.h... yes 2024-08-31T12:17:37.076 INFO:tasks.workunit.client.0.smithi019.stdout:checking for string.h... yes 2024-08-31T12:17:37.129 INFO:tasks.workunit.client.0.smithi019.stdout:checking for memory.h... yes 2024-08-31T12:17:37.182 INFO:tasks.workunit.client.0.smithi019.stdout:checking for strings.h... yes 2024-08-31T12:17:37.236 INFO:tasks.workunit.client.0.smithi019.stdout:checking for inttypes.h... yes 2024-08-31T12:17:37.288 INFO:tasks.workunit.client.0.smithi019.stdout:checking for stdint.h... yes 2024-08-31T12:17:37.343 INFO:tasks.workunit.client.0.smithi019.stdout:checking for unistd.h... yes 2024-08-31T12:17:37.403 INFO:tasks.workunit.client.0.smithi019.stdout:checking minix/config.h usability... no 2024-08-31T12:17:37.440 INFO:tasks.workunit.client.0.smithi019.stdout:checking minix/config.h presence... no 2024-08-31T12:17:37.441 INFO:tasks.workunit.client.0.smithi019.stdout:checking for minix/config.h... no 2024-08-31T12:17:37.484 INFO:tasks.workunit.client.0.smithi019.stdout:checking whether it is safe to define __EXTENSIONS__... yes 2024-08-31T12:17:37.495 INFO:tasks.workunit.client.0.smithi019.stdout:checking for gcc... (cached) gcc 2024-08-31T12:17:37.546 INFO:tasks.workunit.client.0.smithi019.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-08-31T12:17:37.546 INFO:tasks.workunit.client.0.smithi019.stdout:checking whether gcc accepts -g... (cached) yes 2024-08-31T12:17:37.547 INFO:tasks.workunit.client.0.smithi019.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-08-31T12:17:37.547 INFO:tasks.workunit.client.0.smithi019.stdout:checking dependency style of gcc... (cached) gcc3 2024-08-31T12:17:37.608 INFO:tasks.workunit.client.0.smithi019.stdout:checking for main in -lm... yes 2024-08-31T12:17:37.671 INFO:tasks.workunit.client.0.smithi019.stdout:checking for main in -lpthread... yes 2024-08-31T12:17:37.676 INFO:tasks.workunit.client.0.smithi019.stdout:checking for ANSI C header files... (cached) yes 2024-08-31T12:17:37.718 INFO:tasks.workunit.client.0.smithi019.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-08-31T12:17:37.775 INFO:tasks.workunit.client.0.smithi019.stdout:checking pthread.h usability... yes 2024-08-31T12:17:37.801 INFO:tasks.workunit.client.0.smithi019.stdout:checking pthread.h presence... yes 2024-08-31T12:17:37.801 INFO:tasks.workunit.client.0.smithi019.stdout:checking for pthread.h... yes 2024-08-31T12:17:37.868 INFO:tasks.workunit.client.0.smithi019.stdout:checking fcntl.h usability... yes 2024-08-31T12:17:37.890 INFO:tasks.workunit.client.0.smithi019.stdout:checking fcntl.h presence... yes 2024-08-31T12:17:37.891 INFO:tasks.workunit.client.0.smithi019.stdout:checking for fcntl.h... yes 2024-08-31T12:17:37.979 INFO:tasks.workunit.client.0.smithi019.stdout:checking limits.h usability... yes 2024-08-31T12:17:38.001 INFO:tasks.workunit.client.0.smithi019.stdout:checking limits.h presence... yes 2024-08-31T12:17:38.001 INFO:tasks.workunit.client.0.smithi019.stdout:checking for limits.h... yes 2024-08-31T12:17:38.021 INFO:tasks.workunit.client.0.smithi019.stdout:checking for stdint.h... (cached) yes 2024-08-31T12:17:38.134 INFO:tasks.workunit.client.0.smithi019.stdout:checking sys/time.h usability... yes 2024-08-31T12:17:38.157 INFO:tasks.workunit.client.0.smithi019.stdout:checking sys/time.h presence... yes 2024-08-31T12:17:38.157 INFO:tasks.workunit.client.0.smithi019.stdout:checking for sys/time.h... yes 2024-08-31T12:17:38.177 INFO:tasks.workunit.client.0.smithi019.stdout:checking for unistd.h... (cached) yes 2024-08-31T12:17:38.281 INFO:tasks.workunit.client.0.smithi019.stdout:checking sys/vfs.h usability... yes 2024-08-31T12:17:38.303 INFO:tasks.workunit.client.0.smithi019.stdout:checking sys/vfs.h presence... yes 2024-08-31T12:17:38.303 INFO:tasks.workunit.client.0.smithi019.stdout:checking for sys/vfs.h... yes 2024-08-31T12:17:38.424 INFO:tasks.workunit.client.0.smithi019.stdout:checking sys/limits.h usability... no 2024-08-31T12:17:38.485 INFO:tasks.workunit.client.0.smithi019.stdout:checking sys/limits.h presence... no 2024-08-31T12:17:38.485 INFO:tasks.workunit.client.0.smithi019.stdout:checking for sys/limits.h... no 2024-08-31T12:17:38.520 INFO:tasks.workunit.client.0.smithi019.stdout:checking for an ANSI C-conforming const... yes 2024-08-31T12:17:38.556 INFO:tasks.workunit.client.0.smithi019.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-08-31T12:17:38.636 INFO:tasks.workunit.client.0.smithi019.stdout:checking for system... yes 2024-08-31T12:17:38.714 INFO:tasks.workunit.client.0.smithi019.stdout:checking for gettimeofday... yes 2024-08-31T12:17:38.791 INFO:tasks.workunit.client.0.smithi019.stdout:checking for mkdir... yes 2024-08-31T12:17:38.869 INFO:tasks.workunit.client.0.smithi019.stdout:checking for strerror... yes 2024-08-31T12:17:38.951 INFO:tasks.workunit.client.0.smithi019.stdout:checking for open64... yes 2024-08-31T12:17:39.034 INFO:tasks.workunit.client.0.smithi019.stdout:checking for stat64... yes 2024-08-31T12:17:39.110 INFO:tasks.workunit.client.0.smithi019.stdout:checking for fseeko64... yes 2024-08-31T12:17:39.188 INFO:tasks.workunit.client.0.smithi019.stdout:checking for lrand48_r... yes 2024-08-31T12:17:39.266 INFO:tasks.workunit.client.0.smithi019.stdout:checking for srand48_r... yes 2024-08-31T12:17:39.292 INFO:tasks.workunit.client.0.smithi019.stdout:configure: creating ./config.status 2024-08-31T12:17:39.409 INFO:tasks.workunit.client.0.smithi019.stdout: /bin/sh ./config.status 2024-08-31T12:17:39.476 INFO:tasks.workunit.client.0.smithi019.stdout:config.status: creating Makefile 2024-08-31T12:17:39.494 INFO:tasks.workunit.client.0.smithi019.stdout:config.status: creating config.h 2024-08-31T12:17:39.525 INFO:tasks.workunit.client.0.smithi019.stdout:config.status: executing depfiles commands 2024-08-31T12:17:39.544 INFO:tasks.workunit.client.0.smithi019.stdout:config.status: executing default commands 2024-08-31T12:17:39.605 INFO:tasks.workunit.client.0.smithi019.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoheader 2024-08-31T12:17:39.781 INFO:tasks.workunit.client.0.smithi019.stdout:rm -f stamp-h1 2024-08-31T12:17:39.784 INFO:tasks.workunit.client.0.smithi019.stdout:touch config.h.in 2024-08-31T12:17:39.786 INFO:tasks.workunit.client.0.smithi019.stdout:cd . && /bin/sh ./config.status config.h 2024-08-31T12:17:39.845 INFO:tasks.workunit.client.0.smithi019.stdout:config.status: creating config.h 2024-08-31T12:17:39.881 INFO:tasks.workunit.client.0.smithi019.stdout:make all-am 2024-08-31T12:17:39.907 INFO:tasks.workunit.client.0.smithi019.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-08-31T12:17:39.907 INFO:tasks.workunit.client.0.smithi019.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fileops.o -MD -MP -MF .deps/fileops.Tpo -c -o fileops.o fileops.c 2024-08-31T12:17:40.102 INFO:tasks.workunit.client.0.smithi019.stdout:mv -f .deps/fileops.Tpo .deps/fileops.Po 2024-08-31T12:17:40.107 INFO:tasks.workunit.client.0.smithi019.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rand.o -MD -MP -MF .deps/rand.Tpo -c -o rand.o rand.c 2024-08-31T12:17:40.216 INFO:tasks.workunit.client.0.smithi019.stdout:mv -f .deps/rand.Tpo .deps/rand.Po 2024-08-31T12:17:40.221 INFO:tasks.workunit.client.0.smithi019.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.c 2024-08-31T12:17:40.338 INFO:tasks.workunit.client.0.smithi019.stdout:mv -f .deps/main.Tpo .deps/main.Po 2024-08-31T12:17:40.343 INFO:tasks.workunit.client.0.smithi019.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fh.o -MD -MP -MF .deps/fh.Tpo -c -o fh.o fh.c 2024-08-31T12:17:40.466 INFO:tasks.workunit.client.0.smithi019.stdout:mv -f .deps/fh.Tpo .deps/fh.Po 2024-08-31T12:17:40.472 INFO:tasks.workunit.client.0.smithi019.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT filelist.o -MD -MP -MF .deps/filelist.Tpo -c -o filelist.o filelist.c 2024-08-31T12:17:40.616 INFO:tasks.workunit.client.0.smithi019.stdout:mv -f .deps/filelist.Tpo .deps/filelist.Po 2024-08-31T12:17:40.621 INFO:tasks.workunit.client.0.smithi019.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT metaops.o -MD -MP -MF .deps/metaops.Tpo -c -o metaops.o metaops.c 2024-08-31T12:17:40.693 INFO:tasks.workunit.client.0.smithi019.stdout:mv -f .deps/metaops.Tpo .deps/metaops.Po 2024-08-31T12:17:40.698 INFO:tasks.workunit.client.0.smithi019.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rwlock.o -MD -MP -MF .deps/rwlock.Tpo -c -o rwlock.o rwlock.c 2024-08-31T12:17:40.752 INFO:tasks.workunit.client.0.smithi019.stdout:mv -f .deps/rwlock.Tpo .deps/rwlock.Po 2024-08-31T12:17:40.757 INFO:tasks.workunit.client.0.smithi019.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT cirlist.o -MD -MP -MF .deps/cirlist.Tpo -c -o cirlist.o cirlist.c 2024-08-31T12:17:40.820 INFO:tasks.workunit.client.0.smithi019.stdout:mv -f .deps/cirlist.Tpo .deps/cirlist.Po 2024-08-31T12:17:40.824 INFO:tasks.workunit.client.0.smithi019.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rbt.o -MD -MP -MF .deps/rbt.Tpo -c -o rbt.o rbt.c 2024-08-31T12:17:41.046 INFO:tasks.workunit.client.0.smithi019.stdout:mv -f .deps/rbt.Tpo .deps/rbt.Po 2024-08-31T12:17:41.051 INFO:tasks.workunit.client.0.smithi019.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_tg.o -MD -MP -MF .deps/ffsb_tg.Tpo -c -o ffsb_tg.o ffsb_tg.c 2024-08-31T12:17:41.205 INFO:tasks.workunit.client.0.smithi019.stdout:mv -f .deps/ffsb_tg.Tpo .deps/ffsb_tg.Po 2024-08-31T12:17:41.210 INFO:tasks.workunit.client.0.smithi019.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fs.o -MD -MP -MF .deps/ffsb_fs.Tpo -c -o ffsb_fs.o ffsb_fs.c 2024-08-31T12:17:41.390 INFO:tasks.workunit.client.0.smithi019.stdout:mv -f .deps/ffsb_fs.Tpo .deps/ffsb_fs.Po 2024-08-31T12:17:41.395 INFO:tasks.workunit.client.0.smithi019.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_thread.o -MD -MP -MF .deps/ffsb_thread.Tpo -c -o ffsb_thread.o ffsb_thread.c 2024-08-31T12:17:41.485 INFO:tasks.workunit.client.0.smithi019.stdout:mv -f .deps/ffsb_thread.Tpo .deps/ffsb_thread.Po 2024-08-31T12:17:41.491 INFO:tasks.workunit.client.0.smithi019.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_op.o -MD -MP -MF .deps/ffsb_op.Tpo -c -o ffsb_op.o ffsb_op.c 2024-08-31T12:17:41.584 INFO:tasks.workunit.client.0.smithi019.stdout:mv -f .deps/ffsb_op.Tpo .deps/ffsb_op.Po 2024-08-31T12:17:41.589 INFO:tasks.workunit.client.0.smithi019.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT util.o -MD -MP -MF .deps/util.Tpo -c -o util.o util.c 2024-08-31T12:17:41.718 INFO:tasks.workunit.client.0.smithi019.stdout:mv -f .deps/util.Tpo .deps/util.Po 2024-08-31T12:17:41.723 INFO:tasks.workunit.client.0.smithi019.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT parser.o -MD -MP -MF .deps/parser.Tpo -c -o parser.o parser.c 2024-08-31T12:17:41.767 INFO:tasks.workunit.client.0.smithi019.stderr:In file included from parser.h:22, 2024-08-31T12:17:41.767 INFO:tasks.workunit.client.0.smithi019.stderr: from parser.c:25: 2024-08-31T12:17:41.767 INFO:tasks.workunit.client.0.smithi019.stderr:parser.c: In function ‘init_filesys’: 2024-08-31T12:17:41.769 INFO:tasks.workunit.client.0.smithi019.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-08-31T12:17:41.769 INFO:tasks.workunit.client.0.smithi019.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-08-31T12:17:41.769 INFO:tasks.workunit.client.0.smithi019.stderr: | ^ 2024-08-31T12:17:41.769 INFO:tasks.workunit.client.0.smithi019.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-08-31T12:17:41.769 INFO:tasks.workunit.client.0.smithi019.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-08-31T12:17:41.769 INFO:tasks.workunit.client.0.smithi019.stderr: | ^~~~~~~~ 2024-08-31T12:17:41.769 INFO:tasks.workunit.client.0.smithi019.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-08-31T12:17:41.769 INFO:tasks.workunit.client.0.smithi019.stderr: 36 | container_of(ptr, type, member) 2024-08-31T12:17:41.770 INFO:tasks.workunit.client.0.smithi019.stderr: | ^~~~~~~~~~~~ 2024-08-31T12:17:41.770 INFO:tasks.workunit.client.0.smithi019.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-08-31T12:17:41.770 INFO:tasks.workunit.client.0.smithi019.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-08-31T12:17:41.770 INFO:tasks.workunit.client.0.smithi019.stderr: | ^~~~~~~~~~ 2024-08-31T12:17:41.770 INFO:tasks.workunit.client.0.smithi019.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-08-31T12:17:41.770 INFO:tasks.workunit.client.0.smithi019.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-08-31T12:17:41.770 INFO:tasks.workunit.client.0.smithi019.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-08-31T12:17:41.770 INFO:tasks.workunit.client.0.smithi019.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-08-31T12:17:41.770 INFO:tasks.workunit.client.0.smithi019.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-08-31T12:17:41.770 INFO:tasks.workunit.client.0.smithi019.stderr: | ^ 2024-08-31T12:17:41.770 INFO:tasks.workunit.client.0.smithi019.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-08-31T12:17:41.770 INFO:tasks.workunit.client.0.smithi019.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-08-31T12:17:41.770 INFO:tasks.workunit.client.0.smithi019.stderr: | ^~~~~~~~ 2024-08-31T12:17:41.770 INFO:tasks.workunit.client.0.smithi019.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-08-31T12:17:41.771 INFO:tasks.workunit.client.0.smithi019.stderr: 36 | container_of(ptr, type, member) 2024-08-31T12:17:41.771 INFO:tasks.workunit.client.0.smithi019.stderr: | ^~~~~~~~~~~~ 2024-08-31T12:17:41.771 INFO:tasks.workunit.client.0.smithi019.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-08-31T12:17:41.771 INFO:tasks.workunit.client.0.smithi019.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-08-31T12:17:41.771 INFO:tasks.workunit.client.0.smithi019.stderr: | ^~~~~~~~~~ 2024-08-31T12:17:41.771 INFO:tasks.workunit.client.0.smithi019.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-08-31T12:17:41.771 INFO:tasks.workunit.client.0.smithi019.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-08-31T12:17:41.771 INFO:tasks.workunit.client.0.smithi019.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-08-31T12:17:41.771 INFO:tasks.workunit.client.0.smithi019.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-08-31T12:17:41.771 INFO:tasks.workunit.client.0.smithi019.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-08-31T12:17:41.771 INFO:tasks.workunit.client.0.smithi019.stderr: | ^ 2024-08-31T12:17:41.771 INFO:tasks.workunit.client.0.smithi019.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-08-31T12:17:41.771 INFO:tasks.workunit.client.0.smithi019.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-08-31T12:17:41.771 INFO:tasks.workunit.client.0.smithi019.stderr: | ^~~~~~~~ 2024-08-31T12:17:41.772 INFO:tasks.workunit.client.0.smithi019.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-08-31T12:17:41.772 INFO:tasks.workunit.client.0.smithi019.stderr: 36 | container_of(ptr, type, member) 2024-08-31T12:17:41.772 INFO:tasks.workunit.client.0.smithi019.stderr: | ^~~~~~~~~~~~ 2024-08-31T12:17:41.772 INFO:tasks.workunit.client.0.smithi019.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-08-31T12:17:41.772 INFO:tasks.workunit.client.0.smithi019.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-08-31T12:17:41.772 INFO:tasks.workunit.client.0.smithi019.stderr: | ^~~~~~~~~~ 2024-08-31T12:17:41.772 INFO:tasks.workunit.client.0.smithi019.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-08-31T12:17:41.772 INFO:tasks.workunit.client.0.smithi019.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-08-31T12:17:41.772 INFO:tasks.workunit.client.0.smithi019.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-08-31T12:17:41.772 INFO:tasks.workunit.client.0.smithi019.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-08-31T12:17:41.772 INFO:tasks.workunit.client.0.smithi019.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-08-31T12:17:41.772 INFO:tasks.workunit.client.0.smithi019.stderr: | ^ 2024-08-31T12:17:41.772 INFO:tasks.workunit.client.0.smithi019.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-08-31T12:17:41.772 INFO:tasks.workunit.client.0.smithi019.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-08-31T12:17:41.773 INFO:tasks.workunit.client.0.smithi019.stderr: | ^~~~~~~~ 2024-08-31T12:17:41.773 INFO:tasks.workunit.client.0.smithi019.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-08-31T12:17:41.773 INFO:tasks.workunit.client.0.smithi019.stderr: 36 | container_of(ptr, type, member) 2024-08-31T12:17:41.773 INFO:tasks.workunit.client.0.smithi019.stderr: | ^~~~~~~~~~~~ 2024-08-31T12:17:41.773 INFO:tasks.workunit.client.0.smithi019.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-08-31T12:17:41.773 INFO:tasks.workunit.client.0.smithi019.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-08-31T12:17:41.773 INFO:tasks.workunit.client.0.smithi019.stderr: | ^~~~~~~~~~ 2024-08-31T12:17:41.773 INFO:tasks.workunit.client.0.smithi019.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-08-31T12:17:41.773 INFO:tasks.workunit.client.0.smithi019.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-08-31T12:17:41.773 INFO:tasks.workunit.client.0.smithi019.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-08-31T12:17:41.773 INFO:tasks.workunit.client.0.smithi019.stderr:parser.c: In function ‘init_tg_stats’: 2024-08-31T12:17:41.773 INFO:tasks.workunit.client.0.smithi019.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-08-31T12:17:41.773 INFO:tasks.workunit.client.0.smithi019.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-08-31T12:17:41.773 INFO:tasks.workunit.client.0.smithi019.stderr: | ^ 2024-08-31T12:17:41.774 INFO:tasks.workunit.client.0.smithi019.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-08-31T12:17:41.774 INFO:tasks.workunit.client.0.smithi019.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-08-31T12:17:41.774 INFO:tasks.workunit.client.0.smithi019.stderr: | ^~~~~~~~ 2024-08-31T12:17:41.774 INFO:tasks.workunit.client.0.smithi019.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-08-31T12:17:41.774 INFO:tasks.workunit.client.0.smithi019.stderr: 36 | container_of(ptr, type, member) 2024-08-31T12:17:41.774 INFO:tasks.workunit.client.0.smithi019.stderr: | ^~~~~~~~~~~~ 2024-08-31T12:17:41.774 INFO:tasks.workunit.client.0.smithi019.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-08-31T12:17:41.774 INFO:tasks.workunit.client.0.smithi019.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-08-31T12:17:41.774 INFO:tasks.workunit.client.0.smithi019.stderr: | ^~~~~~~~~~ 2024-08-31T12:17:41.774 INFO:tasks.workunit.client.0.smithi019.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-08-31T12:17:41.774 INFO:tasks.workunit.client.0.smithi019.stderr: 971 | list_for_each_entry(tmp_list, 2024-08-31T12:17:41.774 INFO:tasks.workunit.client.0.smithi019.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-08-31T12:17:41.774 INFO:tasks.workunit.client.0.smithi019.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-08-31T12:17:41.774 INFO:tasks.workunit.client.0.smithi019.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-08-31T12:17:41.775 INFO:tasks.workunit.client.0.smithi019.stderr: | ^ 2024-08-31T12:17:41.775 INFO:tasks.workunit.client.0.smithi019.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-08-31T12:17:41.775 INFO:tasks.workunit.client.0.smithi019.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-08-31T12:17:41.775 INFO:tasks.workunit.client.0.smithi019.stderr: | ^~~~~~~~ 2024-08-31T12:17:41.775 INFO:tasks.workunit.client.0.smithi019.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-08-31T12:17:41.775 INFO:tasks.workunit.client.0.smithi019.stderr: 36 | container_of(ptr, type, member) 2024-08-31T12:17:41.775 INFO:tasks.workunit.client.0.smithi019.stderr: | ^~~~~~~~~~~~ 2024-08-31T12:17:41.775 INFO:tasks.workunit.client.0.smithi019.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-08-31T12:17:41.775 INFO:tasks.workunit.client.0.smithi019.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-08-31T12:17:41.775 INFO:tasks.workunit.client.0.smithi019.stderr: | ^~~~~~~~~~ 2024-08-31T12:17:41.775 INFO:tasks.workunit.client.0.smithi019.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-08-31T12:17:41.775 INFO:tasks.workunit.client.0.smithi019.stderr: 971 | list_for_each_entry(tmp_list, 2024-08-31T12:17:41.775 INFO:tasks.workunit.client.0.smithi019.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-08-31T12:17:41.775 INFO:tasks.workunit.client.0.smithi019.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-08-31T12:17:41.776 INFO:tasks.workunit.client.0.smithi019.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-08-31T12:17:41.776 INFO:tasks.workunit.client.0.smithi019.stderr: | ^ 2024-08-31T12:17:41.776 INFO:tasks.workunit.client.0.smithi019.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-08-31T12:17:41.776 INFO:tasks.workunit.client.0.smithi019.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-08-31T12:17:41.776 INFO:tasks.workunit.client.0.smithi019.stderr: | ^~~~~~~~ 2024-08-31T12:17:41.776 INFO:tasks.workunit.client.0.smithi019.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-08-31T12:17:41.776 INFO:tasks.workunit.client.0.smithi019.stderr: 36 | container_of(ptr, type, member) 2024-08-31T12:17:41.776 INFO:tasks.workunit.client.0.smithi019.stderr: | ^~~~~~~~~~~~ 2024-08-31T12:17:41.776 INFO:tasks.workunit.client.0.smithi019.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-08-31T12:17:41.776 INFO:tasks.workunit.client.0.smithi019.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-08-31T12:17:41.776 INFO:tasks.workunit.client.0.smithi019.stderr: | ^~~~~~~~~~ 2024-08-31T12:17:41.776 INFO:tasks.workunit.client.0.smithi019.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-08-31T12:17:41.776 INFO:tasks.workunit.client.0.smithi019.stderr: 980 | list_for_each_entry(tmp_list, 2024-08-31T12:17:41.777 INFO:tasks.workunit.client.0.smithi019.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-08-31T12:17:41.777 INFO:tasks.workunit.client.0.smithi019.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-08-31T12:17:41.777 INFO:tasks.workunit.client.0.smithi019.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-08-31T12:17:41.777 INFO:tasks.workunit.client.0.smithi019.stderr: | ^ 2024-08-31T12:17:41.777 INFO:tasks.workunit.client.0.smithi019.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-08-31T12:17:41.777 INFO:tasks.workunit.client.0.smithi019.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-08-31T12:17:41.777 INFO:tasks.workunit.client.0.smithi019.stderr: | ^~~~~~~~ 2024-08-31T12:17:41.777 INFO:tasks.workunit.client.0.smithi019.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-08-31T12:17:41.777 INFO:tasks.workunit.client.0.smithi019.stderr: 36 | container_of(ptr, type, member) 2024-08-31T12:17:41.777 INFO:tasks.workunit.client.0.smithi019.stderr: | ^~~~~~~~~~~~ 2024-08-31T12:17:41.777 INFO:tasks.workunit.client.0.smithi019.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-08-31T12:17:41.777 INFO:tasks.workunit.client.0.smithi019.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-08-31T12:17:41.777 INFO:tasks.workunit.client.0.smithi019.stderr: | ^~~~~~~~~~ 2024-08-31T12:17:41.777 INFO:tasks.workunit.client.0.smithi019.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-08-31T12:17:41.778 INFO:tasks.workunit.client.0.smithi019.stderr: 980 | list_for_each_entry(tmp_list, 2024-08-31T12:17:41.778 INFO:tasks.workunit.client.0.smithi019.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-08-31T12:17:41.821 INFO:tasks.workunit.client.0.smithi019.stderr:parser.c: In function ‘get_optstr’: 2024-08-31T12:17:41.821 INFO:tasks.workunit.client.0.smithi019.stderr:parser.c:206:23: warning: ‘strnlen’ specified bound 4095 exceeds source size 1024 [-Wstringop-overread] 2024-08-31T12:17:41.821 INFO:tasks.workunit.client.0.smithi019.stderr: 206 | len = strnlen(temp, 4095) + 1; 2024-08-31T12:17:41.821 INFO:tasks.workunit.client.0.smithi019.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-08-31T12:17:41.821 INFO:tasks.workunit.client.0.smithi019.stderr:parser.c:200:14: note: source object declared here 2024-08-31T12:17:41.821 INFO:tasks.workunit.client.0.smithi019.stderr: 200 | char temp[BUFSIZE]; 2024-08-31T12:17:41.821 INFO:tasks.workunit.client.0.smithi019.stderr: | ^~~~ 2024-08-31T12:17:41.827 INFO:tasks.workunit.client.0.smithi019.stderr:parser.c: In function ‘size64_convert’: 2024-08-31T12:17:41.828 INFO:tasks.workunit.client.0.smithi019.stderr:parser.c:119:9: warning: ‘memcpy’ reading 3 bytes from a region of size 2 [-Wstringop-overread] 2024-08-31T12:17:41.828 INFO:tasks.workunit.client.0.smithi019.stderr: 119 | memcpy(unit, "\0", 3); 2024-08-31T12:17:41.828 INFO:tasks.workunit.client.0.smithi019.stderr: | ^~~~~~~~~~~~~~~~~~~~~ 2024-08-31T12:17:42.120 INFO:tasks.workunit.client.0.smithi019.stdout:mv -f .deps/parser.Tpo .deps/parser.Po 2024-08-31T12:17:42.125 INFO:tasks.workunit.client.0.smithi019.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fc.o -MD -MP -MF .deps/ffsb_fc.Tpo -c -o ffsb_fc.o ffsb_fc.c 2024-08-31T12:17:42.202 INFO:tasks.workunit.client.0.smithi019.stdout:mv -f .deps/ffsb_fc.Tpo .deps/ffsb_fc.Po 2024-08-31T12:17:42.207 INFO:tasks.workunit.client.0.smithi019.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_stats.o -MD -MP -MF .deps/ffsb_stats.Tpo -c -o ffsb_stats.o ffsb_stats.c 2024-08-31T12:17:42.312 INFO:tasks.workunit.client.0.smithi019.stdout:mv -f .deps/ffsb_stats.Tpo .deps/ffsb_stats.Po 2024-08-31T12:17:42.317 INFO:tasks.workunit.client.0.smithi019.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT list.o -MD -MP -MF .deps/list.Tpo -c -o list.o list.c 2024-08-31T12:17:42.365 INFO:tasks.workunit.client.0.smithi019.stdout:mv -f .deps/list.Tpo .deps/list.Po 2024-08-31T12:17:42.372 INFO:tasks.workunit.client.0.smithi019.stdout:gcc -D_GNU_SOURCE -g -O2 -D_REENTRANT -o ffsb fileops.o rand.o main.o fh.o filelist.o metaops.o rwlock.o cirlist.o rbt.o ffsb_tg.o ffsb_fs.o ffsb_thread.o ffsb_op.o util.o parser.o ffsb_fc.o ffsb_stats.o list.o -lpthread -lm 2024-08-31T12:17:42.475 INFO:tasks.workunit.client.0.smithi019.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-08-31T12:17:42.476 INFO:tasks.workunit.client.0.smithi019.stderr:+ cd .. 2024-08-31T12:17:42.477 INFO:tasks.workunit.client.0.smithi019.stderr:+ mkdir tmp 2024-08-31T12:17:42.479 INFO:tasks.workunit.client.0.smithi019.stderr:+ cd tmp 2024-08-31T12:17:42.481 INFO:tasks.workunit.client.0.smithi019.stderr:+ for f in $mydir/*.ffsb 2024-08-31T12:17:42.481 INFO:tasks.workunit.client.0.smithi019.stderr:+ ../ffsb/ffsb /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/random_write.32.ffsb 2024-08-31T12:17:42.481 INFO:tasks.workunit.client.0.smithi019.stdout:FFSB version 6.0-RC2 started 2024-08-31T12:17:42.481 INFO:tasks.workunit.client.0.smithi019.stdout: 2024-08-31T12:17:42.484 INFO:tasks.workunit.client.0.smithi019.stdout:benchmark time = 300 2024-08-31T12:17:42.484 INFO:tasks.workunit.client.0.smithi019.stdout:ThreadGroup 0 2024-08-31T12:17:42.484 INFO:tasks.workunit.client.0.smithi019.stdout:================ 2024-08-31T12:17:42.484 INFO:tasks.workunit.client.0.smithi019.stdout: num_threads = 32 2024-08-31T12:17:42.484 INFO:tasks.workunit.client.0.smithi019.stdout: 2024-08-31T12:17:42.484 INFO:tasks.workunit.client.0.smithi019.stdout: read_random = off 2024-08-31T12:17:42.484 INFO:tasks.workunit.client.0.smithi019.stdout: read_size = 0 (0B) 2024-08-31T12:17:42.484 INFO:tasks.workunit.client.0.smithi019.stdout: read_blocksize = 0 (0B) 2024-08-31T12:17:42.485 INFO:tasks.workunit.client.0.smithi019.stdout: read_skip = off 2024-08-31T12:17:42.485 INFO:tasks.workunit.client.0.smithi019.stdout: read_skipsize = 0 (0B) 2024-08-31T12:17:42.485 INFO:tasks.workunit.client.0.smithi019.stdout: 2024-08-31T12:17:42.485 INFO:tasks.workunit.client.0.smithi019.stdout: write_random = on 2024-08-31T12:17:42.485 INFO:tasks.workunit.client.0.smithi019.stdout: write_size = 5242880 (5MB) 2024-08-31T12:17:42.485 INFO:tasks.workunit.client.0.smithi019.stdout: fsync_file = 0 2024-08-31T12:17:42.485 INFO:tasks.workunit.client.0.smithi019.stdout: write_blocksize = 4096 (4KB) 2024-08-31T12:17:42.485 INFO:tasks.workunit.client.0.smithi019.stdout: wait time = 0 2024-08-31T12:17:42.485 INFO:tasks.workunit.client.0.smithi019.stdout: 2024-08-31T12:17:42.485 INFO:tasks.workunit.client.0.smithi019.stdout: op weights 2024-08-31T12:17:42.485 INFO:tasks.workunit.client.0.smithi019.stdout: read = 0 (0.00%) 2024-08-31T12:17:42.485 INFO:tasks.workunit.client.0.smithi019.stdout: readall = 0 (0.00%) 2024-08-31T12:17:42.485 INFO:tasks.workunit.client.0.smithi019.stdout: write = 1 (100.00%) 2024-08-31T12:17:42.486 INFO:tasks.workunit.client.0.smithi019.stdout: create = 0 (0.00%) 2024-08-31T12:17:42.486 INFO:tasks.workunit.client.0.smithi019.stdout: append = 0 (0.00%) 2024-08-31T12:17:42.486 INFO:tasks.workunit.client.0.smithi019.stdout: delete = 0 (0.00%) 2024-08-31T12:17:42.486 INFO:tasks.workunit.client.0.smithi019.stdout: metaop = 0 (0.00%) 2024-08-31T12:17:42.486 INFO:tasks.workunit.client.0.smithi019.stdout: createdir = 0 (0.00%) 2024-08-31T12:17:42.486 INFO:tasks.workunit.client.0.smithi019.stdout: stat = 0 (0.00%) 2024-08-31T12:17:42.486 INFO:tasks.workunit.client.0.smithi019.stdout: writeall = 0 (0.00%) 2024-08-31T12:17:42.486 INFO:tasks.workunit.client.0.smithi019.stdout: writeall_fsync = 0 (0.00%) 2024-08-31T12:17:42.486 INFO:tasks.workunit.client.0.smithi019.stdout: open_close = 0 (0.00%) 2024-08-31T12:17:42.486 INFO:tasks.workunit.client.0.smithi019.stdout: write_fsync = 0 (0.00%) 2024-08-31T12:17:42.486 INFO:tasks.workunit.client.0.smithi019.stdout: create_fsync = 0 (0.00%) 2024-08-31T12:17:42.486 INFO:tasks.workunit.client.0.smithi019.stdout: append_fsync = 0 (0.00%) 2024-08-31T12:17:42.486 INFO:tasks.workunit.client.0.smithi019.stdout: 2024-08-31T12:17:42.487 INFO:tasks.workunit.client.0.smithi019.stdout:FileSystem . 2024-08-31T12:17:42.487 INFO:tasks.workunit.client.0.smithi019.stdout:========== 2024-08-31T12:17:42.487 INFO:tasks.workunit.client.0.smithi019.stdout: num_dirs = 0 2024-08-31T12:17:42.487 INFO:tasks.workunit.client.0.smithi019.stdout: starting files = 128 2024-08-31T12:17:42.487 INFO:tasks.workunit.client.0.smithi019.stdout: 2024-08-31T12:17:42.487 INFO:tasks.workunit.client.0.smithi019.stdout: min file size = 104857600 (100MB) 2024-08-31T12:17:42.487 INFO:tasks.workunit.client.0.smithi019.stdout: max file size = 104857600 (100MB) 2024-08-31T12:17:42.487 INFO:tasks.workunit.client.0.smithi019.stdout: directio = off 2024-08-31T12:17:42.487 INFO:tasks.workunit.client.0.smithi019.stdout: alignedio = on 2024-08-31T12:17:42.487 INFO:tasks.workunit.client.0.smithi019.stdout: bufferedio = off 2024-08-31T12:17:42.487 INFO:tasks.workunit.client.0.smithi019.stdout: 2024-08-31T12:17:42.487 INFO:tasks.workunit.client.0.smithi019.stdout: aging is off 2024-08-31T12:17:42.488 INFO:tasks.workunit.client.0.smithi019.stdout: current utilization = 0.00% 2024-08-31T12:17:42.488 INFO:tasks.workunit.client.0.smithi019.stdout: 2024-08-31T12:17:42.488 INFO:tasks.workunit.client.0.smithi019.stdout:checking existing fs: . 2024-08-31T12:17:42.488 INFO:tasks.workunit.client.0.smithi019.stdout:recreating new fileset 2024-08-31T12:17:42.488 INFO:tasks.workunit.client.0.smithi019.stderr:opendir: No such file or directory 2024-08-31T12:17:51.763 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-08-31T12:17:51.764 INFO:tasks.ceph.mon.c:Restarting daemon 2024-08-31T12:17:51.764 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-08-31T12:17:51.767 INFO:tasks.ceph.mon.c:Started 2024-08-31T12:17:51.767 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-08-31T12:17:51.767 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:17:51.896 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:51.895+0000 7f2662969640 1 -- 172.21.15.19:0/2925090499 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2644004040 msgr2=0x7f2644004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:51.897 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:51.895+0000 7f2662969640 1 --2- 172.21.15.19:0/2925090499 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2644004040 0x7f2644004800 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f265000c470 tx=0x7f2644005830 comp rx=0 tx=0).stop 2024-08-31T12:17:51.897 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:51.895+0000 7f2660f06640 1 -- 172.21.15.19:0/2925090499 reap_dead start 2024-08-31T12:17:51.897 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:51.895+0000 7f2662969640 1 -- 172.21.15.19:0/2925090499 shutdown_connections 2024-08-31T12:17:51.897 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:51.895+0000 7f2662969640 1 -- 172.21.15.19:0/2925090499 >> 172.21.15.19:0/2925090499 conn(0x7f265c0ca630 msgr2=0x7f265c0cd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:17:51.897 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:51.895+0000 7f2662969640 1 -- 172.21.15.19:0/2925090499 shutdown_connections 2024-08-31T12:17:51.897 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:51.895+0000 7f2662969640 1 -- 172.21.15.19:0/2925090499 wait complete. 2024-08-31T12:17:51.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:51.897+0000 7f2662969640 1 Processor -- start 2024-08-31T12:17:51.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:51.898+0000 7f2662969640 1 -- start start 2024-08-31T12:17:51.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:51.899+0000 7f2662969640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2644004040 0x7f265c14da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:51.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:51.899+0000 7f2662969640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f265c1ddf10 0x7f265c1482c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:51.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:51.899+0000 7f2662969640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f265c1defc0 0x7f265c14ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:51.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:51.899+0000 7f2662969640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f265c0cded0 con 0x7f265c1defc0 2024-08-31T12:17:51.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:51.899+0000 7f2662969640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f265c0c8ff0 con 0x7f265c1ddf10 2024-08-31T12:17:51.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:51.899+0000 7f2662969640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f265c152400 con 0x7f2644004040 2024-08-31T12:17:51.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:51.899+0000 7f2660f06640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f265c1defc0 0x7f265c14ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:51.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:51.899+0000 7f265bfff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2644004040 0x7f265c14da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:51.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:51.899+0000 7f265b7fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f265c1ddf10 0x7f265c1482c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:51.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:51.899+0000 7f2660f06640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f265c1defc0 0x7f265c14ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:44572/0 (socket says 172.21.15.19:44572) 2024-08-31T12:17:51.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:51.899+0000 7f2660f06640 1 -- 172.21.15.19:0/1984498727 learned_addr learned my addr 172.21.15.19:0/1984498727 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:17:51.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:51.899+0000 7f265bfff640 1 -- 172.21.15.19:0/1984498727 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2644004040 msgr2=0x7f265c14da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-08-31T12:17:51.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:51.899+0000 7f265bfff640 1 -- 172.21.15.19:0/1984498727 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2644004040 msgr2=0x7f265c14da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-31T12:17:51.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:51.899+0000 7f265bfff640 1 --2- 172.21.15.19:0/1984498727 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2644004040 0x7f265c14da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-31T12:17:51.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:51.899+0000 7f265bfff640 1 --2- 172.21.15.19:0/1984498727 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2644004040 0x7f265c14da90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:17:51.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:51.900+0000 7f2660f06640 1 -- 172.21.15.19:0/1984498727 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2644004040 msgr2=0x7f265c14da90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:17:51.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:51.900+0000 7f2660f06640 1 --2- 172.21.15.19:0/1984498727 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2644004040 0x7f265c14da90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:51.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:51.900+0000 7f2660f06640 1 -- 172.21.15.19:0/1984498727 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f265c1ddf10 msgr2=0x7f265c1482c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:51.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:51.900+0000 7f2660f06640 1 --2- 172.21.15.19:0/1984498727 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f265c1ddf10 0x7f265c1482c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:51.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:51.900+0000 7f2660f06640 1 -- 172.21.15.19:0/1984498727 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f265c1eef20 con 0x7f265c1defc0 2024-08-31T12:17:51.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:51.900+0000 7f265b7fe640 1 --2- 172.21.15.19:0/1984498727 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f265c1ddf10 0x7f265c1482c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:17:51.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:51.900+0000 7f2660f06640 1 --2- 172.21.15.19:0/1984498727 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f265c1defc0 0x7f265c14ea50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f265c073930 tx=0x7f265400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:51.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:51.900+0000 7f2658ff9640 1 -- 172.21.15.19:0/1984498727 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2654003d80 con 0x7f265c1defc0 2024-08-31T12:17:51.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:51.900+0000 7f2658ff9640 1 -- 172.21.15.19:0/1984498727 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f265400fe50 con 0x7f265c1defc0 2024-08-31T12:17:51.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:51.900+0000 7f2658ff9640 1 -- 172.21.15.19:0/1984498727 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2654004d80 con 0x7f265c1defc0 2024-08-31T12:17:51.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:51.900+0000 7f2662969640 1 -- 172.21.15.19:0/1984498727 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f265c14dfd0 con 0x7f265c1defc0 2024-08-31T12:17:51.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:51.900+0000 7f2662969640 1 -- 172.21.15.19:0/1984498727 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f265c0cb960 con 0x7f265c1defc0 2024-08-31T12:17:51.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:51.901+0000 7f2662969640 1 -- 172.21.15.19:0/1984498727 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f265c1ef0f0 con 0x7f265c1defc0 2024-08-31T12:17:51.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:51.902+0000 7f2658ff9640 1 -- 172.21.15.19:0/1984498727 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f2654004030 con 0x7f265c1defc0 2024-08-31T12:17:51.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:51.903+0000 7f2658ff9640 1 --2- 172.21.15.19:0/1984498727 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2648077a50 0x7f2648079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:51.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:51.903+0000 7f2658ff9640 1 -- 172.21.15.19:0/1984498727 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f265409a390 con 0x7f265c1defc0 2024-08-31T12:17:51.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:51.903+0000 7f265bfff640 1 --2- 172.21.15.19:0/1984498727 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2648077a50 0x7f2648079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:51.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:51.903+0000 7f265bfff640 1 --2- 172.21.15.19:0/1984498727 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2648077a50 0x7f2648079f60 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f264c005df0 tx=0x7f264c00b260 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:51.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:51.907+0000 7f2658ff9640 1 -- 172.21.15.19:0/1984498727 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f2654063110 con 0x7f265c1defc0 2024-08-31T12:17:52.263 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:52.262+0000 7f2662969640 1 -- 172.21.15.19:0/1984498727 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f265c06e3e0 con 0x7f265c1defc0 2024-08-31T12:17:52.263 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:52.262+0000 7f2658ff9640 1 -- 172.21.15.19:0/1984498727 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f2654062a10 con 0x7f265c1defc0 2024-08-31T12:17:52.264 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":14,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:17:52.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:52.265+0000 7f2662969640 1 -- 172.21.15.19:0/1984498727 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2648077a50 msgr2=0x7f2648079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:52.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:52.265+0000 7f2662969640 1 --2- 172.21.15.19:0/1984498727 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2648077a50 0x7f2648079f60 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f264c005df0 tx=0x7f264c00b260 comp rx=0 tx=0).stop 2024-08-31T12:17:52.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:52.265+0000 7f2662969640 1 -- 172.21.15.19:0/1984498727 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f265c1defc0 msgr2=0x7f265c14ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:52.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:52.265+0000 7f2662969640 1 --2- 172.21.15.19:0/1984498727 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f265c1defc0 0x7f265c14ea50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f265c073930 tx=0x7f265400f790 comp rx=0 tx=0).stop 2024-08-31T12:17:52.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:52.265+0000 7f2662969640 1 -- 172.21.15.19:0/1984498727 shutdown_connections 2024-08-31T12:17:52.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:52.265+0000 7f2662969640 1 --2- 172.21.15.19:0/1984498727 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2648077a50 0x7f2648079f60 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:52.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:52.265+0000 7f2662969640 1 --2- 172.21.15.19:0/1984498727 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f265c1defc0 0x7f265c14ea50 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:52.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:52.265+0000 7f2662969640 1 --2- 172.21.15.19:0/1984498727 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f265c1ddf10 0x7f265c1482c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:52.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:52.265+0000 7f2662969640 1 --2- 172.21.15.19:0/1984498727 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2644004040 0x7f265c14da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:52.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:52.265+0000 7f2662969640 1 -- 172.21.15.19:0/1984498727 >> 172.21.15.19:0/1984498727 conn(0x7f265c0ca630 msgr2=0x7f265c0cd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:17:52.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:52.266+0000 7f2662969640 1 -- 172.21.15.19:0/1984498727 shutdown_connections 2024-08-31T12:17:52.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:52.266+0000 7f2662969640 1 -- 172.21.15.19:0/1984498727 wait complete. 2024-08-31T12:17:55.281 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:17:55.821 DEBUG:teuthology.orchestra.run.smithi019:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:17:55.823 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:17:58.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.405+0000 7fb395cc5640 1 -- 172.21.15.19:0/1681807791 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb37c004800 msgr2=0x7fb37c004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:58.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.405+0000 7fb395cc5640 1 --2- 172.21.15.19:0/1681807791 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb37c004800 0x7fb37c004cd0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb3840058c0 tx=0x7fb37c005830 comp rx=0 tx=0).stop 2024-08-31T12:17:58.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.405+0000 7fb395cc5640 1 -- 172.21.15.19:0/1681807791 shutdown_connections 2024-08-31T12:17:58.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.405+0000 7fb395cc5640 1 --2- 172.21.15.19:0/1681807791 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb37c0099b0 0x7fb37c00bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:58.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.405+0000 7fb395cc5640 1 --2- 172.21.15.19:0/1681807791 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb37c004800 0x7fb37c004cd0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:58.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.405+0000 7fb395cc5640 1 --2- 172.21.15.19:0/1681807791 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb37c003ad0 0x7fb37c003f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:58.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.405+0000 7fb395cc5640 1 -- 172.21.15.19:0/1681807791 >> 172.21.15.19:0/1681807791 conn(0x7fb390068ec0 msgr2=0x7fb3900692d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:17:58.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.405+0000 7fb395cc5640 1 -- 172.21.15.19:0/1681807791 shutdown_connections 2024-08-31T12:17:58.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.405+0000 7fb395cc5640 1 -- 172.21.15.19:0/1681807791 wait complete. 2024-08-31T12:17:58.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.406+0000 7fb395cc5640 1 Processor -- start 2024-08-31T12:17:58.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.407+0000 7fb395cc5640 1 -- start start 2024-08-31T12:17:58.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.408+0000 7fb395cc5640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb37c003ad0 0x7fb39014e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:58.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.408+0000 7fb395cc5640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb37c004800 0x7fb3900cd8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:58.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.408+0000 7fb395cc5640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb37c0099b0 0x7fb390148fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:58.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.408+0000 7fb395cc5640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fb39015c690 con 0x7fb37c004800 2024-08-31T12:17:58.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.408+0000 7fb395cc5640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb3901599c0 con 0x7fb37c0099b0 2024-08-31T12:17:58.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.408+0000 7fb395cc5640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fb39015c970 con 0x7fb37c003ad0 2024-08-31T12:17:58.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.408+0000 7fb38effd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb37c004800 0x7fb3900cd8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:58.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.408+0000 7fb38effd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb37c004800 0x7fb3900cd8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:56770/0 (socket says 172.21.15.19:56770) 2024-08-31T12:17:58.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.408+0000 7fb38effd640 1 -- 172.21.15.19:0/152538263 learned_addr learned my addr 172.21.15.19:0/152538263 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:17:58.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.408+0000 7fb38f7fe640 1 --2- 172.21.15.19:0/152538263 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb37c003ad0 0x7fb39014e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:58.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.408+0000 7fb38ffff640 1 --2- 172.21.15.19:0/152538263 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb37c0099b0 0x7fb390148fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:58.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.408+0000 7fb38effd640 1 -- 172.21.15.19:0/152538263 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb37c003ad0 msgr2=0x7fb39014e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:58.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.408+0000 7fb38effd640 1 --2- 172.21.15.19:0/152538263 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb37c003ad0 0x7fb39014e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:58.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.408+0000 7fb38effd640 1 -- 172.21.15.19:0/152538263 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb37c0099b0 msgr2=0x7fb390148fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:58.412 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.408+0000 7fb38effd640 1 --2- 172.21.15.19:0/152538263 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb37c0099b0 0x7fb390148fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:58.412 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.408+0000 7fb38effd640 1 -- 172.21.15.19:0/152538263 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb39014e900 con 0x7fb37c004800 2024-08-31T12:17:58.412 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.408+0000 7fb38f7fe640 1 --2- 172.21.15.19:0/152538263 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb37c003ad0 0x7fb39014e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-31T12:17:58.412 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.408+0000 7fb38ffff640 1 --2- 172.21.15.19:0/152538263 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb37c0099b0 0x7fb390148fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-31T12:17:58.412 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.409+0000 7fb38effd640 1 --2- 172.21.15.19:0/152538263 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb37c004800 0x7fb3900cd8e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb384013970 tx=0x7fb38400d730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:58.412 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.409+0000 7fb377fff640 1 -- 172.21.15.19:0/152538263 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb384032070 con 0x7fb37c004800 2024-08-31T12:17:58.412 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.409+0000 7fb377fff640 1 -- 172.21.15.19:0/152538263 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb384005020 con 0x7fb37c004800 2024-08-31T12:17:58.412 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.409+0000 7fb377fff640 1 -- 172.21.15.19:0/152538263 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb384005430 con 0x7fb37c004800 2024-08-31T12:17:58.412 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.409+0000 7fb395cc5640 1 -- 172.21.15.19:0/152538263 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3900b3670 con 0x7fb37c004800 2024-08-31T12:17:58.412 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.409+0000 7fb395cc5640 1 -- 172.21.15.19:0/152538263 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3900b2b50 con 0x7fb37c004800 2024-08-31T12:17:58.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.410+0000 7fb395cc5640 1 -- 172.21.15.19:0/152538263 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3900b2980 con 0x7fb37c004800 2024-08-31T12:17:58.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.411+0000 7fb377fff640 1 -- 172.21.15.19:0/152538263 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fb3840047a0 con 0x7fb37c004800 2024-08-31T12:17:58.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.411+0000 7fb377fff640 1 --2- 172.21.15.19:0/152538263 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb378078390 0x7fb37807a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:58.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.411+0000 7fb377fff640 1 -- 172.21.15.19:0/152538263 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fb3840ba730 con 0x7fb37c004800 2024-08-31T12:17:58.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.412+0000 7fb38f7fe640 1 --2- 172.21.15.19:0/152538263 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb378078390 0x7fb37807a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:58.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.412+0000 7fb38f7fe640 1 --2- 172.21.15.19:0/152538263 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb378078390 0x7fb37807a8a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb380000c00 tx=0x7fb3800113e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:58.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.417+0000 7fb377fff640 1 -- 172.21.15.19:0/152538263 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fb3840834b0 con 0x7fb37c004800 2024-08-31T12:17:58.749 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.747+0000 7fb395cc5640 1 -- 172.21.15.19:0/152538263 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb3900c8f40 con 0x7fb37c004800 2024-08-31T12:17:58.749 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.748+0000 7fb377fff640 1 -- 172.21.15.19:0/152538263 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fb384082db0 con 0x7fb37c004800 2024-08-31T12:17:58.750 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:17:58.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.751+0000 7fb395cc5640 1 -- 172.21.15.19:0/152538263 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb378078390 msgr2=0x7fb37807a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:58.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.751+0000 7fb395cc5640 1 --2- 172.21.15.19:0/152538263 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb378078390 0x7fb37807a8a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb380000c00 tx=0x7fb3800113e0 comp rx=0 tx=0).stop 2024-08-31T12:17:58.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.751+0000 7fb395cc5640 1 -- 172.21.15.19:0/152538263 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb37c004800 msgr2=0x7fb3900cd8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:58.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.751+0000 7fb395cc5640 1 --2- 172.21.15.19:0/152538263 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb37c004800 0x7fb3900cd8e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb384013970 tx=0x7fb38400d730 comp rx=0 tx=0).stop 2024-08-31T12:17:58.754 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.751+0000 7fb395cc5640 1 -- 172.21.15.19:0/152538263 shutdown_connections 2024-08-31T12:17:58.754 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.751+0000 7fb395cc5640 1 --2- 172.21.15.19:0/152538263 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb378078390 0x7fb37807a8a0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:58.754 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.751+0000 7fb395cc5640 1 --2- 172.21.15.19:0/152538263 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb37c0099b0 0x7fb390148fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:58.754 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.751+0000 7fb395cc5640 1 --2- 172.21.15.19:0/152538263 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb37c004800 0x7fb3900cd8e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:58.754 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.751+0000 7fb395cc5640 1 --2- 172.21.15.19:0/152538263 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb37c003ad0 0x7fb39014e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:58.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.751+0000 7fb395cc5640 1 -- 172.21.15.19:0/152538263 >> 172.21.15.19:0/152538263 conn(0x7fb390068ec0 msgr2=0x7fb39014af00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:17:58.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.751+0000 7fb395cc5640 1 -- 172.21.15.19:0/152538263 shutdown_connections 2024-08-31T12:17:58.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.751+0000 7fb395cc5640 1 -- 172.21.15.19:0/152538263 wait complete. 2024-08-31T12:17:58.766 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-08-31T12:17:58.766 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-08-31T12:17:58.767 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-31T12:17:58.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.889+0000 7f453539e640 1 -- 172.21.15.19:0/3641604067 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f45300cd050 msgr2=0x7f45300cd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:58.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.889+0000 7f453539e640 1 --2- 172.21.15.19:0/3641604067 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f45300cd050 0x7f45300cd460 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4528005fd0 tx=0x7f452802aa90 comp rx=0 tx=0).stop 2024-08-31T12:17:58.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.889+0000 7f453539e640 1 -- 172.21.15.19:0/3641604067 shutdown_connections 2024-08-31T12:17:58.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.889+0000 7f453539e640 1 --2- 172.21.15.19:0/3641604067 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f45300cd050 0x7f45300cd460 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:58.891 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.889+0000 7f453539e640 1 --2- 172.21.15.19:0/3641604067 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f45300cbc90 0x7f45300cc0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:58.891 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.889+0000 7f453539e640 1 -- 172.21.15.19:0/3641604067 >> 172.21.15.19:0/3641604067 conn(0x7f453006d6c0 msgr2=0x7f453006dad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:17:58.891 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.889+0000 7f453539e640 1 -- 172.21.15.19:0/3641604067 shutdown_connections 2024-08-31T12:17:58.891 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.889+0000 7f453539e640 1 -- 172.21.15.19:0/3641604067 wait complete. 2024-08-31T12:17:58.892 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.891+0000 7f453539e640 1 Processor -- start 2024-08-31T12:17:58.892 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.891+0000 7f453539e640 1 -- start start 2024-08-31T12:17:58.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.893+0000 7f453539e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f45300ca500 0x7f45301f26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:58.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.893+0000 7f453539e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f45300cbc90 0x7f453015a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:58.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.893+0000 7f453539e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f45300cd050 0x7f4530152a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:58.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.893+0000 7f453539e640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f45300cd9a0 con 0x7f45300ca500 2024-08-31T12:17:58.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.893+0000 7f453539e640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f453015f080 con 0x7f45300cbc90 2024-08-31T12:17:58.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.893+0000 7f453539e640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f45300b4fb0 con 0x7f45300cd050 2024-08-31T12:17:58.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.893+0000 7f452ffff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f45300ca500 0x7f45301f26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:58.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.893+0000 7f452ffff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f45300ca500 0x7f45301f26d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:56800/0 (socket says 172.21.15.19:56800) 2024-08-31T12:17:58.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.893+0000 7f452ffff640 1 -- 172.21.15.19:0/2282863411 learned_addr learned my addr 172.21.15.19:0/2282863411 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:17:58.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.893+0000 7f452f7fe640 1 --2- 172.21.15.19:0/2282863411 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f45300cbc90 0x7f453015a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:58.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.893+0000 7f4534b9d640 1 --2- 172.21.15.19:0/2282863411 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f45300cd050 0x7f4530152a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:58.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.893+0000 7f452ffff640 1 -- 172.21.15.19:0/2282863411 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f45300cd050 msgr2=0x7f4530152a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:58.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.893+0000 7f452ffff640 1 --2- 172.21.15.19:0/2282863411 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f45300cd050 0x7f4530152a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:58.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.893+0000 7f452ffff640 1 -- 172.21.15.19:0/2282863411 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f45300cbc90 msgr2=0x7f453015a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:58.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.893+0000 7f452ffff640 1 --2- 172.21.15.19:0/2282863411 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f45300cbc90 0x7f453015a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:58.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.893+0000 7f452ffff640 1 -- 172.21.15.19:0/2282863411 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45301f2c10 con 0x7f45300ca500 2024-08-31T12:17:58.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.893+0000 7f4534b9d640 1 --2- 172.21.15.19:0/2282863411 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f45300cd050 0x7f4530152a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-31T12:17:58.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.893+0000 7f452f7fe640 1 --2- 172.21.15.19:0/2282863411 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f45300cbc90 0x7f453015a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-31T12:17:58.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.893+0000 7f452ffff640 1 --2- 172.21.15.19:0/2282863411 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f45300ca500 0x7f45301f26d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f452000b3f0 tx=0x7f452000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:58.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.894+0000 7f452cff9640 1 -- 172.21.15.19:0/2282863411 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4520015020 con 0x7f45300ca500 2024-08-31T12:17:58.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.894+0000 7f452cff9640 1 -- 172.21.15.19:0/2282863411 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4520011920 con 0x7f45300ca500 2024-08-31T12:17:58.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.894+0000 7f453539e640 1 -- 172.21.15.19:0/2282863411 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f453015b3a0 con 0x7f45300ca500 2024-08-31T12:17:58.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.894+0000 7f452cff9640 1 -- 172.21.15.19:0/2282863411 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4520011d70 con 0x7f45300ca500 2024-08-31T12:17:58.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.894+0000 7f453539e640 1 -- 172.21.15.19:0/2282863411 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f453015b960 con 0x7f45300ca500 2024-08-31T12:17:58.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.895+0000 7f453539e640 1 --2- 172.21.15.19:0/2282863411 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f45301f2de0 0x7f453015bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:58.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.895+0000 7f453539e640 1 -- 172.21.15.19:0/2282863411 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f45300ca910 con 0x7f45301f2de0 2024-08-31T12:17:58.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.895+0000 7f452f7fe640 1 --2- 172.21.15.19:0/2282863411 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f45301f2de0 0x7f453015bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:58.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.895+0000 7f452f7fe640 1 -- 172.21.15.19:0/2282863411 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f45301f2330 con 0x7f45301f2de0 2024-08-31T12:17:58.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.895+0000 7f452f7fe640 1 --2- 172.21.15.19:0/2282863411 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f45301f2de0 0x7f453015bb60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f45240058c0 tx=0x7f452400dd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:58.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.896+0000 7f452cff9640 1 -- 172.21.15.19:0/2282863411 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4524008be0 con 0x7f45301f2de0 2024-08-31T12:17:58.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.896+0000 7f452cff9640 1 -- 172.21.15.19:0/2282863411 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f4520018020 con 0x7f45300ca500 2024-08-31T12:17:58.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.896+0000 7f452cff9640 1 --2- 172.21.15.19:0/2282863411 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f451c077950 0x7f451c079e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:58.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.896+0000 7f452cff9640 1 -- 172.21.15.19:0/2282863411 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f45200996f0 con 0x7f45300ca500 2024-08-31T12:17:58.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.897+0000 7f4534b9d640 1 --2- 172.21.15.19:0/2282863411 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f451c077950 0x7f451c079e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:58.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.897+0000 7f452cff9640 1 -- 172.21.15.19:0/2282863411 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4524008d80 con 0x7f45301f2de0 2024-08-31T12:17:58.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.897+0000 7f452cff9640 1 -- 172.21.15.19:0/2282863411 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f45301f2de0 msgr2=0x7f453015bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:58.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.897+0000 7f452cff9640 1 --2- 172.21.15.19:0/2282863411 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f45301f2de0 0x7f453015bb60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f45240058c0 tx=0x7f452400dd70 comp rx=0 tx=0).stop 2024-08-31T12:17:58.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.897+0000 7f452cff9640 1 -- 172.21.15.19:0/2282863411 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f45301f2de0 msgr2=0x7f453015bb60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:17:58.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.897+0000 7f452cff9640 1 --2- 172.21.15.19:0/2282863411 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f45301f2de0 0x7f453015bb60 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:58.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.897+0000 7f4534b9d640 1 --2- 172.21.15.19:0/2282863411 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f451c077950 0x7f451c079e60 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4528035000 tx=0x7f4528004ab0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:58.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.905+0000 7f453539e640 1 --2- 172.21.15.19:0/2282863411 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f453015d920 0x7f4530152fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:58.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.905+0000 7f453539e640 1 -- 172.21.15.19:0/2282863411 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f45300c9b20 con 0x7f453015d920 2024-08-31T12:17:58.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.905+0000 7f452f7fe640 1 --2- 172.21.15.19:0/2282863411 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f453015d920 0x7f4530152fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:58.907 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.905+0000 7f452f7fe640 1 -- 172.21.15.19:0/2282863411 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f45300cc820 con 0x7f453015d920 2024-08-31T12:17:58.907 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.906+0000 7f452f7fe640 1 --2- 172.21.15.19:0/2282863411 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f453015d920 0x7f4530152fb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f45240058c0 tx=0x7f45240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:58.907 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.906+0000 7f452cff9640 1 -- 172.21.15.19:0/2282863411 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f452401e2b0 con 0x7f453015d920 2024-08-31T12:17:58.907 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.906+0000 7f452cff9640 1 -- 172.21.15.19:0/2282863411 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f452401e2b0 con 0x7f453015d920 2024-08-31T12:17:58.907 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.906+0000 7f452cff9640 1 -- 172.21.15.19:0/2282863411 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f453015d920 msgr2=0x7f4530152fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:58.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.906+0000 7f452cff9640 1 --2- 172.21.15.19:0/2282863411 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f453015d920 0x7f4530152fb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f45240058c0 tx=0x7f45240046c0 comp rx=0 tx=0).stop 2024-08-31T12:17:58.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.906+0000 7f452cff9640 1 -- 172.21.15.19:0/2282863411 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f453015d920 msgr2=0x7f4530152fb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:17:58.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.906+0000 7f452cff9640 1 --2- 172.21.15.19:0/2282863411 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f453015d920 0x7f4530152fb0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:58.908 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:17:58.908 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:17:58.908 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:17:58.908 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "leader", 2024-08-31T12:17:58.908 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 20, 2024-08-31T12:17:58.908 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 58897, 2024-08-31T12:17:58.909 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:17:58.909 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:17:58.909 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:17:58.909 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:17:58.909 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:58.909 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 0, 2024-08-31T12:17:58.909 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:17:58.909 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:17:58.909 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:17:58.909 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:17:58.909 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:17:58.909 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:17:58.909 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:17:58.910 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:17:58.910 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:17:58.910 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:17:58.910 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:17:58.910 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:17:58.910 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:17:58.910 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:17:58.910 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:58.910 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:17:58.910 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:17:58.910 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:17:58.910 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:17:58.911 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:17:58.911 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:17:58.911 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:17:58.911 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:17:58.911 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:17:58.911 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:17:58.911 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:17:58.911 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:17:58.911 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:17:58.911 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:58.911 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:58.911 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:17:58.911 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:17:58.912 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:17:58.912 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:17:58.912 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:17:58.912 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:17:58.912 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:17:58.912 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:17:58.912 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:17:58.912 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:17:58.912 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:17:58.912 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:17:58.912 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:17:58.912 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:17:58.912 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:17:58.913 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:17:58.913 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:17:58.913 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:17:58.913 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:17:58.913 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:17:58.913 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:17:58.913 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:17:58.913 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:17:58.913 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:17:58.913 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:17:58.913 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:17:58.913 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:17:58.914 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:17:58.914 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:58.914 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:17:58.914 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:58.914 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:17:58.914 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:58.914 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:17:58.914 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:17:58.914 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:17:58.914 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:17:58.914 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:58.914 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:17:58.915 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:17:58.915 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:58.915 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:58.915 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:58.915 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:17:58.915 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:17:58.915 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:58.915 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:58.915 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:58.915 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:58.915 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:17:58.915 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:17:58.915 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:17:58.915 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:17:58.916 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:17:58.916 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:58.916 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:58.916 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:17:58.916 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:17:58.916 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:17:58.916 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:17:58.916 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:58.916 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:17:58.916 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:17:58.916 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:58.916 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:58.916 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:58.917 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:17:58.917 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:17:58.917 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:58.917 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:58.917 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:58.917 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:58.917 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:17:58.917 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:17:58.917 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:17:58.917 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:17:58.917 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:17:58.917 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:58.918 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:58.918 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:17:58.918 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:17:58.918 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:17:58.918 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:17:58.918 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:58.918 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:17:58.918 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:17:58.918 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:58.918 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:58.918 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:58.918 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:17:58.918 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:17:58.919 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:58.919 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:58.919 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:58.919 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:58.919 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:17:58.919 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:17:58.919 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:17:58.919 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:17:58.919 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:17:58.919 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:58.920 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:58.920 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:58.920 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:17:58.920 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:17:58.920 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:58.920 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:17:58.920 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:17:58.920 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:17:58.920 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:58.920 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:58.920 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:17:58.920 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:58.920 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:17:58.921 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:17:58.921 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:17:58.921 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:58.921 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:58.921 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:17:58.921 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:58.921 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:17:58.921 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:17:58.921 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:17:58.921 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:58.921 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:58.921 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:17:58.921 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:58.921 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:17:58.922 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:17:58.922 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:17:58.922 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:58.922 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:58.922 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:58.922 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:17:58.922 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:17:58.923 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.907+0000 7f453539e640 1 -- 172.21.15.19:0/2282863411 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f451c077950 msgr2=0x7f451c079e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:58.923 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.907+0000 7f453539e640 1 --2- 172.21.15.19:0/2282863411 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f451c077950 0x7f451c079e60 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4528035000 tx=0x7f4528004ab0 comp rx=0 tx=0).stop 2024-08-31T12:17:58.923 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.907+0000 7f453539e640 1 -- 172.21.15.19:0/2282863411 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f45300ca500 msgr2=0x7f45301f26d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:58.923 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.907+0000 7f453539e640 1 --2- 172.21.15.19:0/2282863411 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f45300ca500 0x7f45301f26d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f452000b3f0 tx=0x7f452000b8b0 comp rx=0 tx=0).stop 2024-08-31T12:17:58.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.907+0000 7f4534b9d640 1 -- 172.21.15.19:0/2282863411 reap_dead start 2024-08-31T12:17:58.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.907+0000 7f453539e640 1 -- 172.21.15.19:0/2282863411 shutdown_connections 2024-08-31T12:17:58.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.907+0000 7f453539e640 1 --2- 172.21.15.19:0/2282863411 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f45300ca500 0x7f45301f26d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:58.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.907+0000 7f453539e640 1 -- 172.21.15.19:0/2282863411 >> 172.21.15.19:0/2282863411 conn(0x7f453006d6c0 msgr2=0x7f45300b4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:17:58.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.907+0000 7f453539e640 1 -- 172.21.15.19:0/2282863411 shutdown_connections 2024-08-31T12:17:58.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:58.907+0000 7f453539e640 1 -- 172.21.15.19:0/2282863411 wait complete. 2024-08-31T12:17:58.924 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:17:58.924 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-08-31T12:17:58.925 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-31T12:17:59.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.054+0000 7fe4861e6640 1 -- 172.21.15.19:0/4206103801 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe48014e9a0 msgr2=0x7fe4800cc400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:59.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.055+0000 7fe4861e6640 1 --2- 172.21.15.19:0/4206103801 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe48014e9a0 0x7fe4800cc400 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe478005fd0 tx=0x7fe47802a940 comp rx=0 tx=0).stop 2024-08-31T12:17:59.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.055+0000 7fe4861e6640 1 -- 172.21.15.19:0/4206103801 shutdown_connections 2024-08-31T12:17:59.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.055+0000 7fe4861e6640 1 --2- 172.21.15.19:0/4206103801 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe48014e9a0 0x7fe4800cc400 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:59.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.055+0000 7fe4861e6640 1 --2- 172.21.15.19:0/4206103801 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe4800cb2b0 0x7fe48014e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:59.057 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.055+0000 7fe4861e6640 1 -- 172.21.15.19:0/4206103801 >> 172.21.15.19:0/4206103801 conn(0x7fe4800ca670 msgr2=0x7fe4800cd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:17:59.057 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.055+0000 7fe4861e6640 1 -- 172.21.15.19:0/4206103801 shutdown_connections 2024-08-31T12:17:59.057 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.055+0000 7fe4861e6640 1 -- 172.21.15.19:0/4206103801 wait complete. 2024-08-31T12:17:59.057 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.057+0000 7fe4861e6640 1 Processor -- start 2024-08-31T12:17:59.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.057+0000 7fe4861e6640 1 -- start start 2024-08-31T12:17:59.061 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.059+0000 7fe4861e6640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4800c5370 0x7fe480155560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:59.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.059+0000 7fe4861e6640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe4800cb2b0 0x7fe48014f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:59.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.059+0000 7fe4861e6640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe48014e9a0 0x7fe480073db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:59.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.059+0000 7fe4861e6640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fe4800c7650 con 0x7fe4800cb2b0 2024-08-31T12:17:59.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.059+0000 7fe4861e6640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe4800ce140 con 0x7fe4800c5370 2024-08-31T12:17:59.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.059+0000 7fe4861e6640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fe4800c72e0 con 0x7fe48014e9a0 2024-08-31T12:17:59.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.059+0000 7fe47effd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe4800cb2b0 0x7fe48014f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:59.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.059+0000 7fe47effd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe4800cb2b0 0x7fe48014f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:56820/0 (socket says 172.21.15.19:56820) 2024-08-31T12:17:59.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.059+0000 7fe47effd640 1 -- 172.21.15.19:0/3068110701 learned_addr learned my addr 172.21.15.19:0/3068110701 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:17:59.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.059+0000 7fe47f7fe640 1 --2- 172.21.15.19:0/3068110701 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4800c5370 0x7fe480155560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:59.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.059+0000 7fe47ffff640 1 --2- 172.21.15.19:0/3068110701 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe48014e9a0 0x7fe480073db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:59.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.059+0000 7fe47ffff640 1 -- 172.21.15.19:0/3068110701 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe48014e9a0 msgr2=0x7fe480073db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-08-31T12:17:59.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.059+0000 7fe47ffff640 1 -- 172.21.15.19:0/3068110701 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe48014e9a0 msgr2=0x7fe480073db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-31T12:17:59.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.059+0000 7fe47ffff640 1 --2- 172.21.15.19:0/3068110701 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe48014e9a0 0x7fe480073db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-31T12:17:59.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.059+0000 7fe47ffff640 1 --2- 172.21.15.19:0/3068110701 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe48014e9a0 0x7fe480073db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:17:59.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.059+0000 7fe47effd640 1 -- 172.21.15.19:0/3068110701 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe48014e9a0 msgr2=0x7fe480073db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:17:59.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.059+0000 7fe47effd640 1 --2- 172.21.15.19:0/3068110701 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe48014e9a0 0x7fe480073db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:59.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.059+0000 7fe47effd640 1 -- 172.21.15.19:0/3068110701 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4800c5370 msgr2=0x7fe480155560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:59.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.059+0000 7fe47effd640 1 --2- 172.21.15.19:0/3068110701 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4800c5370 0x7fe480155560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:59.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.059+0000 7fe47effd640 1 -- 172.21.15.19:0/3068110701 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe48014f900 con 0x7fe4800cb2b0 2024-08-31T12:17:59.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.060+0000 7fe47effd640 1 --2- 172.21.15.19:0/3068110701 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe4800cb2b0 0x7fe48014f3c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe47400d980 tx=0x7fe47400de40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:59.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.060+0000 7fe467fff640 1 -- 172.21.15.19:0/3068110701 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe474017070 con 0x7fe4800cb2b0 2024-08-31T12:17:59.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.060+0000 7fe467fff640 1 -- 172.21.15.19:0/3068110701 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe474009d40 con 0x7fe4800cb2b0 2024-08-31T12:17:59.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.060+0000 7fe4861e6640 1 -- 172.21.15.19:0/3068110701 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe480150be0 con 0x7fe4800cb2b0 2024-08-31T12:17:59.064 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.060+0000 7fe467fff640 1 -- 172.21.15.19:0/3068110701 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4740106c0 con 0x7fe4800cb2b0 2024-08-31T12:17:59.064 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.060+0000 7fe4861e6640 1 -- 172.21.15.19:0/3068110701 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4801513c0 con 0x7fe4800cb2b0 2024-08-31T12:17:59.065 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.061+0000 7fe4861e6640 1 --2- 172.21.15.19:0/3068110701 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe480151c00 0x7fe480147d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:59.065 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.061+0000 7fe4861e6640 1 -- 172.21.15.19:0/3068110701 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe48006e760 con 0x7fe480151c00 2024-08-31T12:17:59.065 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.062+0000 7fe47f7fe640 1 --2- 172.21.15.19:0/3068110701 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe480151c00 0x7fe480147d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:59.065 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.062+0000 7fe467fff640 1 -- 172.21.15.19:0/3068110701 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fe47401a070 con 0x7fe4800cb2b0 2024-08-31T12:17:59.065 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.062+0000 7fe47f7fe640 1 -- 172.21.15.19:0/3068110701 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe4801eec90 con 0x7fe480151c00 2024-08-31T12:17:59.066 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.062+0000 7fe467fff640 1 --2- 172.21.15.19:0/3068110701 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe46c077a50 0x7fe46c079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:59.066 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.062+0000 7fe47f7fe640 1 --2- 172.21.15.19:0/3068110701 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe480151c00 0x7fe480147d60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe47000f770 tx=0x7fe47000fc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:59.066 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.062+0000 7fe47ffff640 1 --2- 172.21.15.19:0/3068110701 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe46c077a50 0x7fe46c079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:59.066 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.062+0000 7fe467fff640 1 -- 172.21.15.19:0/3068110701 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fe474099860 con 0x7fe4800cb2b0 2024-08-31T12:17:59.066 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.063+0000 7fe467fff640 1 -- 172.21.15.19:0/3068110701 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4700050d0 con 0x7fe480151c00 2024-08-31T12:17:59.066 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.063+0000 7fe467fff640 1 -- 172.21.15.19:0/3068110701 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe470018070 con 0x7fe480151c00 2024-08-31T12:17:59.066 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.063+0000 7fe467fff640 1 -- 172.21.15.19:0/3068110701 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe480151c00 msgr2=0x7fe480147d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:59.066 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.063+0000 7fe467fff640 1 --2- 172.21.15.19:0/3068110701 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe480151c00 0x7fe480147d60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe47000f770 tx=0x7fe47000fc30 comp rx=0 tx=0).stop 2024-08-31T12:17:59.066 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.063+0000 7fe467fff640 1 -- 172.21.15.19:0/3068110701 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe480151c00 msgr2=0x7fe480147d60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:17:59.066 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.063+0000 7fe467fff640 1 --2- 172.21.15.19:0/3068110701 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe480151c00 0x7fe480147d60 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:59.066 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.063+0000 7fe47ffff640 1 --2- 172.21.15.19:0/3068110701 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe46c077a50 0x7fe46c079f60 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe478034000 tx=0x7fe478004ab0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:59.071 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.070+0000 7fe4861e6640 1 --2- 172.21.15.19:0/3068110701 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4801482a0 0x7fe4801f1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:59.071 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.070+0000 7fe4861e6640 1 -- 172.21.15.19:0/3068110701 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe4801486b0 con 0x7fe4801482a0 2024-08-31T12:17:59.071 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.071+0000 7fe47f7fe640 1 --2- 172.21.15.19:0/3068110701 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4801482a0 0x7fe4801f1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:59.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.072+0000 7fe47f7fe640 1 -- 172.21.15.19:0/3068110701 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe48006e1e0 con 0x7fe4801482a0 2024-08-31T12:17:59.073 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.072+0000 7fe47f7fe640 1 --2- 172.21.15.19:0/3068110701 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4801482a0 0x7fe4801f1680 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe47000f770 tx=0x7fe470000fe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:59.073 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.072+0000 7fe467fff640 1 -- 172.21.15.19:0/3068110701 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe47000d840 con 0x7fe4801482a0 2024-08-31T12:17:59.073 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.073+0000 7fe467fff640 1 -- 172.21.15.19:0/3068110701 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5042 (secure 0 0 0) 0x7fe47000d840 con 0x7fe4801482a0 2024-08-31T12:17:59.074 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.073+0000 7fe467fff640 1 -- 172.21.15.19:0/3068110701 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4801482a0 msgr2=0x7fe4801f1680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:59.074 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.073+0000 7fe467fff640 1 --2- 172.21.15.19:0/3068110701 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4801482a0 0x7fe4801f1680 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe47000f770 tx=0x7fe470000fe0 comp rx=0 tx=0).stop 2024-08-31T12:17:59.074 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.073+0000 7fe467fff640 1 -- 172.21.15.19:0/3068110701 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4801482a0 msgr2=0x7fe4801f1680 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:17:59.074 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.073+0000 7fe467fff640 1 --2- 172.21.15.19:0/3068110701 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4801482a0 0x7fe4801f1680 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:59.074 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:17:59.074 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:17:59.074 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:17:59.074 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:17:59.074 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 20, 2024-08-31T12:17:59.075 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 147169, 2024-08-31T12:17:59.075 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:17:59.075 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:17:59.075 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:17:59.075 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:17:59.075 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:59.075 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 0, 2024-08-31T12:17:59.075 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:17:59.075 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:17:59.075 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:17:59.075 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:17:59.075 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:17:59.075 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:17:59.076 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:17:59.076 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:17:59.076 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:17:59.076 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:17:59.076 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:17:59.076 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:17:59.076 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:17:59.076 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:17:59.076 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:59.076 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:17:59.076 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:17:59.076 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:17:59.076 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:17:59.077 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:17:59.077 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:17:59.077 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:17:59.077 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:17:59.077 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:17:59.077 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:17:59.077 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:17:59.077 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:17:59.077 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:17:59.077 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:59.077 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:59.077 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:17:59.077 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:17:59.078 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:17:59.078 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:17:59.078 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:17:59.078 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:17:59.078 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:17:59.078 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:17:59.078 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:17:59.078 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:17:59.078 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:17:59.078 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:17:59.078 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:17:59.078 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:17:59.078 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:17:59.078 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:17:59.078 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:17:59.079 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:17:59.079 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:17:59.079 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:17:59.079 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:17:59.079 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:17:59.079 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:17:59.079 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:17:59.079 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:17:59.079 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:17:59.079 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:17:59.079 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:17:59.079 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:59.079 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:17:59.079 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:59.080 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:17:59.080 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:59.080 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:17:59.080 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:17:59.080 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:17:59.080 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:17:59.080 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:59.080 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:17:59.080 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:17:59.080 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:59.080 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:59.080 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:59.080 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:17:59.081 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:17:59.081 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:59.081 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:59.081 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:59.081 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:59.081 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:17:59.081 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:17:59.081 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:17:59.081 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:17:59.081 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:17:59.081 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:59.081 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:59.081 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:17:59.082 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:17:59.082 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:17:59.082 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:17:59.082 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:59.082 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:17:59.082 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:17:59.082 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:59.082 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:59.082 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:59.082 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:17:59.082 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:17:59.082 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:59.082 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:59.082 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:59.083 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:59.083 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:17:59.083 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:17:59.083 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:17:59.083 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:17:59.083 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:17:59.083 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:59.083 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:59.083 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:17:59.083 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:17:59.083 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:17:59.083 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:17:59.083 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:59.083 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:17:59.084 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:17:59.084 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:59.084 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:59.084 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:59.084 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:17:59.084 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:17:59.084 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:17:59.084 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:59.084 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:59.084 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:59.084 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:17:59.084 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:17:59.085 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:17:59.085 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:17:59.085 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:17:59.085 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:59.085 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:59.085 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:59.085 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:17:59.085 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:17:59.085 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:59.085 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:17:59.085 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:17:59.085 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:17:59.086 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:59.086 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:59.086 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:17:59.086 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:59.086 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:17:59.086 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:17:59.086 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 5 2024-08-31T12:17:59.086 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:59.086 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:59.086 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:17:59.086 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:59.086 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:17:59.087 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:17:59.087 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 7 2024-08-31T12:17:59.087 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:59.087 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:59.087 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:17:59.087 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:59.087 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:17:59.087 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:17:59.087 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 6 2024-08-31T12:17:59.087 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:59.087 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:17:59.087 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:17:59.087 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:17:59.088 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:17:59.088 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:17:59.088 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:17:59.088 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:17:59.088 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:17:59.088 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:17:59.088 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:17:59.088 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:17:59.090 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.073+0000 7fe4861e6640 1 -- 172.21.15.19:0/3068110701 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe46c077a50 msgr2=0x7fe46c079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:59.090 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.073+0000 7fe4861e6640 1 --2- 172.21.15.19:0/3068110701 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe46c077a50 0x7fe46c079f60 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe478034000 tx=0x7fe478004ab0 comp rx=0 tx=0).stop 2024-08-31T12:17:59.090 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.073+0000 7fe4861e6640 1 -- 172.21.15.19:0/3068110701 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe4800cb2b0 msgr2=0x7fe48014f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:59.090 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.073+0000 7fe4861e6640 1 --2- 172.21.15.19:0/3068110701 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe4800cb2b0 0x7fe48014f3c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe47400d980 tx=0x7fe47400de40 comp rx=0 tx=0).stop 2024-08-31T12:17:59.090 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.073+0000 7fe47ffff640 1 -- 172.21.15.19:0/3068110701 reap_dead start 2024-08-31T12:17:59.090 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.073+0000 7fe47ffff640 1 -- 172.21.15.19:0/3068110701 reap_dead start 2024-08-31T12:17:59.090 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.074+0000 7fe4861e6640 1 -- 172.21.15.19:0/3068110701 shutdown_connections 2024-08-31T12:17:59.090 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.074+0000 7fe4861e6640 1 -- 172.21.15.19:0/3068110701 >> 172.21.15.19:0/3068110701 conn(0x7fe4800ca670 msgr2=0x7fe480149eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:17:59.090 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.074+0000 7fe4861e6640 1 -- 172.21.15.19:0/3068110701 shutdown_connections 2024-08-31T12:17:59.091 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.074+0000 7fe4861e6640 1 -- 172.21.15.19:0/3068110701 wait complete. 2024-08-31T12:17:59.091 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:17:59.091 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-08-31T12:17:59.091 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-31T12:17:59.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.216+0000 7feb7619d640 1 -- 172.21.15.19:0/2369278033 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7feb700ce820 msgr2=0x7feb700cec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:59.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.216+0000 7feb7619d640 1 --2- 172.21.15.19:0/2369278033 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7feb700ce820 0x7feb700cec30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7feb68005fd0 tx=0x7feb6802a940 comp rx=0 tx=0).stop 2024-08-31T12:17:59.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.216+0000 7feb7619d640 1 -- 172.21.15.19:0/2369278033 shutdown_connections 2024-08-31T12:17:59.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.216+0000 7feb7619d640 1 --2- 172.21.15.19:0/2369278033 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7feb700ce820 0x7feb700cec30 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:59.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.216+0000 7feb7619d640 1 --2- 172.21.15.19:0/2369278033 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb700c4ca0 0x7feb70152ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:59.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.216+0000 7feb7619d640 1 -- 172.21.15.19:0/2369278033 >> 172.21.15.19:0/2369278033 conn(0x7feb70068ec0 msgr2=0x7feb700692d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:17:59.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.216+0000 7feb7619d640 1 -- 172.21.15.19:0/2369278033 shutdown_connections 2024-08-31T12:17:59.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.216+0000 7feb7619d640 1 -- 172.21.15.19:0/2369278033 wait complete. 2024-08-31T12:17:59.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.218+0000 7feb7619d640 1 Processor -- start 2024-08-31T12:17:59.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.218+0000 7feb7619d640 1 -- start start 2024-08-31T12:17:59.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.219+0000 7feb7619d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7feb700c4ca0 0x7feb7006f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:59.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.219+0000 7feb7619d640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb700c8c50 0x7feb70146d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:59.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.219+0000 7feb7619d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb700ce820 0x7feb701472d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:59.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.219+0000 7feb7619d640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7feb70153000 con 0x7feb700c4ca0 2024-08-31T12:17:59.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.219+0000 7feb7619d640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7feb70159490 con 0x7feb700c8c50 2024-08-31T12:17:59.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.219+0000 7feb7619d640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7feb700c9160 con 0x7feb700ce820 2024-08-31T12:17:59.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.220+0000 7feb7519b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7feb700c4ca0 0x7feb7006f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:59.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.220+0000 7feb7519b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7feb700c4ca0 0x7feb7006f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:56850/0 (socket says 172.21.15.19:56850) 2024-08-31T12:17:59.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.220+0000 7feb7519b640 1 -- 172.21.15.19:0/2015469984 learned_addr learned my addr 172.21.15.19:0/2015469984 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:17:59.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.220+0000 7feb7499a640 1 --2- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb700c8c50 0x7feb70146d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:59.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.220+0000 7feb7599c640 1 --2- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb700ce820 0x7feb701472d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:59.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.220+0000 7feb7599c640 1 -- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb700ce820 msgr2=0x7feb701472d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-08-31T12:17:59.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.220+0000 7feb7599c640 1 -- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb700ce820 msgr2=0x7feb701472d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-31T12:17:59.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.220+0000 7feb7599c640 1 --2- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb700ce820 0x7feb701472d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-31T12:17:59.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.220+0000 7feb7519b640 1 -- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb700ce820 msgr2=0x7feb701472d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:59.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.220+0000 7feb7599c640 1 --2- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb700ce820 0x7feb701472d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:17:59.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.220+0000 7feb7519b640 1 --2- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb700ce820 0x7feb701472d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:59.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.220+0000 7feb7519b640 1 -- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb700c8c50 msgr2=0x7feb70146d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:17:59.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.220+0000 7feb7519b640 1 --2- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb700c8c50 0x7feb70146d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:17:59.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.220+0000 7feb7519b640 1 -- 172.21.15.19:0/2015469984 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb7006fa00 con 0x7feb700c4ca0 2024-08-31T12:17:59.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.220+0000 7feb7499a640 1 --2- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb700c8c50 0x7feb70146d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:17:59.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.220+0000 7feb7519b640 1 --2- 172.21.15.19:0/2015469984 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7feb700c4ca0 0x7feb7006f4c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7feb6c00c430 tx=0x7feb6c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:59.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.220+0000 7feb65ffb640 1 -- 172.21.15.19:0/2015469984 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb6c011790 con 0x7feb700c4ca0 2024-08-31T12:17:59.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.220+0000 7feb65ffb640 1 -- 172.21.15.19:0/2015469984 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feb6c017070 con 0x7feb700c4ca0 2024-08-31T12:17:59.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.220+0000 7feb7619d640 1 -- 172.21.15.19:0/2015469984 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb7006e960 con 0x7feb700c4ca0 2024-08-31T12:17:59.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.220+0000 7feb65ffb640 1 -- 172.21.15.19:0/2015469984 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb6c011ce0 con 0x7feb700c4ca0 2024-08-31T12:17:59.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.221+0000 7feb7619d640 1 -- 172.21.15.19:0/2015469984 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb700c9e10 con 0x7feb700c4ca0 2024-08-31T12:17:59.225 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.222+0000 7feb7619d640 1 --2- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb700c9fe0 0x7feb70156850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:59.225 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.222+0000 7feb7619d640 1 -- 172.21.15.19:0/2015469984 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7feb7006f170 con 0x7feb700c9fe0 2024-08-31T12:17:59.225 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.222+0000 7feb7499a640 1 --2- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb700c9fe0 0x7feb70156850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:59.225 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.222+0000 7feb7499a640 1 -- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb700c9fe0 msgr2=0x7feb70156850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 20 2024-08-31T12:17:59.225 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.222+0000 7feb7499a640 1 -- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb700c9fe0 msgr2=0x7feb70156850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-31T12:17:59.225 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.222+0000 7feb7499a640 1 --2- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb700c9fe0 0x7feb70156850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-31T12:17:59.225 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.222+0000 7feb7499a640 1 --2- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb700c9fe0 0x7feb70156850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:17:59.225 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.223+0000 7feb65ffb640 1 -- 172.21.15.19:0/2015469984 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7feb6c01c460 con 0x7feb700c4ca0 2024-08-31T12:17:59.225 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.223+0000 7feb65ffb640 1 --2- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7feb54077a50 0x7feb54079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:17:59.225 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.223+0000 7feb65ffb640 1 -- 172.21.15.19:0/2015469984 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7feb6c099420 con 0x7feb700c4ca0 2024-08-31T12:17:59.225 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.223+0000 7feb7599c640 1 --2- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7feb54077a50 0x7feb54079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:59.225 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.224+0000 7feb7599c640 1 --2- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7feb54077a50 0x7feb54079f60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7feb68034000 tx=0x7feb68004ab0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:17:59.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.424+0000 7feb7499a640 1 --2- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb700c9fe0 0x7feb70156850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:59.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.424+0000 7feb7499a640 1 -- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb700c9fe0 msgr2=0x7feb70156850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 21 2024-08-31T12:17:59.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.424+0000 7feb7499a640 1 -- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb700c9fe0 msgr2=0x7feb70156850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-31T12:17:59.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.424+0000 7feb7499a640 1 --2- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb700c9fe0 0x7feb70156850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-31T12:17:59.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.424+0000 7feb7499a640 1 --2- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb700c9fe0 0x7feb70156850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-31T12:17:59.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.826+0000 7feb7499a640 1 --2- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb700c9fe0 0x7feb70156850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:17:59.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.826+0000 7feb7499a640 1 -- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb700c9fe0 msgr2=0x7feb70156850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 21 2024-08-31T12:17:59.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.826+0000 7feb7499a640 1 -- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb700c9fe0 msgr2=0x7feb70156850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-31T12:17:59.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.826+0000 7feb7499a640 1 --2- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb700c9fe0 0x7feb70156850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-31T12:17:59.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:17:59.826+0000 7feb7499a640 1 --2- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb700c9fe0 0x7feb70156850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-31T12:18:00.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:00.627+0000 7feb7499a640 1 --2- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb700c9fe0 0x7feb70156850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:00.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:00.627+0000 7feb7499a640 1 -- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb700c9fe0 msgr2=0x7feb70156850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 21 2024-08-31T12:18:00.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:00.627+0000 7feb7499a640 1 -- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb700c9fe0 msgr2=0x7feb70156850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-31T12:18:00.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:00.627+0000 7feb7499a640 1 --2- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb700c9fe0 0x7feb70156850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-31T12:18:00.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:00.627+0000 7feb7499a640 1 --2- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb700c9fe0 0x7feb70156850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-31T12:18:02.230 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.229+0000 7feb7499a640 1 --2- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb700c9fe0 0x7feb70156850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:02.230 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.229+0000 7feb7499a640 1 -- 172.21.15.19:0/2015469984 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feb7006fbd0 con 0x7feb700c9fe0 2024-08-31T12:18:02.231 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.230+0000 7feb7499a640 1 --2- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb700c9fe0 0x7feb70156850 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7feb7006e8f0 tx=0x7feb60014ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:02.231 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.230+0000 7feb65ffb640 1 -- 172.21.15.19:0/2015469984 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb60011b00 con 0x7feb700c9fe0 2024-08-31T12:18:02.231 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.230+0000 7feb65ffb640 1 -- 172.21.15.19:0/2015469984 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7feb60011b00 con 0x7feb700c9fe0 2024-08-31T12:18:02.231 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.230+0000 7feb65ffb640 1 -- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb700c9fe0 msgr2=0x7feb70156850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:02.231 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.230+0000 7feb65ffb640 1 --2- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb700c9fe0 0x7feb70156850 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7feb7006e8f0 tx=0x7feb60014ea0 comp rx=0 tx=0).stop 2024-08-31T12:18:02.231 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.230+0000 7feb65ffb640 1 -- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb700c9fe0 msgr2=0x7feb70156850 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:18:02.231 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.230+0000 7feb65ffb640 1 --2- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb700c9fe0 0x7feb70156850 secure :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7feb7006e8f0 tx=0x7feb60014ea0 comp rx=0 tx=0).stop 2024-08-31T12:18:02.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.237+0000 7feb7619d640 1 --2- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb70156d90 0x7feb701571a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:02.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.237+0000 7feb7619d640 1 -- 172.21.15.19:0/2015469984 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7feb700c3c20 con 0x7feb70156d90 2024-08-31T12:18:02.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.237+0000 7feb7499a640 1 --2- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb70156d90 0x7feb701571a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:02.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.238+0000 7feb7499a640 1 -- 172.21.15.19:0/2015469984 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7feb700c9790 con 0x7feb70156d90 2024-08-31T12:18:02.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.238+0000 7feb7499a640 1 --2- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb70156d90 0x7feb701571a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7feb6001e000 tx=0x7feb600046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:02.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.238+0000 7feb65ffb640 1 -- 172.21.15.19:0/2015469984 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb6002d450 con 0x7feb70156d90 2024-08-31T12:18:02.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.238+0000 7feb65ffb640 1 -- 172.21.15.19:0/2015469984 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4540 (secure 0 0 0) 0x7feb60012450 con 0x7feb70156d90 2024-08-31T12:18:02.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.239+0000 7feb65ffb640 1 -- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb70156d90 msgr2=0x7feb701571a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:02.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.239+0000 7feb65ffb640 1 --2- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb70156d90 0x7feb701571a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7feb6001e000 tx=0x7feb600046c0 comp rx=0 tx=0).stop 2024-08-31T12:18:02.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.239+0000 7feb65ffb640 1 -- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb70156d90 msgr2=0x7feb701571a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:18:02.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.239+0000 7feb65ffb640 1 --2- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb70156d90 0x7feb701571a0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:02.240 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:18:02.240 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:18:02.240 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:18:02.240 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:18:02.240 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 20, 2024-08-31T12:18:02.240 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 10360, 2024-08-31T12:18:02.241 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:18:02.241 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:18:02.241 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:18:02.241 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:18:02.241 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:02.241 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 3, 2024-08-31T12:18:02.241 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:18:02.241 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:18:02.241 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:18:02.241 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:18:02.241 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:18:02.241 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:18:02.241 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:18:02.242 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:18:02.242 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:18:02.242 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:18:02.242 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:18:02.242 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:18:02.242 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:18:02.242 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:18:02.242 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:02.242 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:18:02.242 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:18:02.242 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:18:02.242 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:18:02.242 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:18:02.242 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:18:02.243 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:18:02.243 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:18:02.243 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:18:02.243 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:18:02.243 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:18:02.243 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:18:02.243 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:18:02.243 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:02.243 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:02.243 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:18:02.243 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:18:02.243 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:18:02.243 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:18:02.243 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:18:02.244 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:18:02.244 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:18:02.244 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:18:02.244 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:18:02.244 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:18:02.244 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:18:02.244 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:18:02.244 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:18:02.244 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:18:02.244 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:18:02.244 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:18:02.244 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:18:02.244 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:18:02.245 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:18:02.245 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:18:02.245 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:18:02.245 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:18:02.245 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:18:02.245 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:18:02.245 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:18:02.245 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:18:02.245 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:18:02.245 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:18:02.245 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:02.245 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:18:02.245 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:02.246 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:18:02.246 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:02.246 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:18:02.246 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:18:02.246 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:18:02.246 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:18:02.246 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:02.246 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:18:02.246 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:18:02.246 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:02.246 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:02.246 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:02.246 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:18:02.246 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:18:02.247 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:02.247 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:02.247 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:02.247 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:02.247 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:18:02.247 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:18:02.247 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:18:02.247 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:18:02.247 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:18:02.247 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:02.247 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:02.247 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:18:02.247 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:18:02.248 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:18:02.248 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:18:02.248 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:02.248 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:18:02.248 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:18:02.248 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:02.248 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:02.248 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:02.248 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:18:02.248 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:18:02.248 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:02.248 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:02.248 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:02.249 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:02.249 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:18:02.249 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:18:02.249 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:18:02.249 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:18:02.249 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:18:02.249 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:02.249 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:02.249 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:18:02.249 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:18:02.249 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:18:02.249 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:18:02.249 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:02.250 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:18:02.250 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:18:02.250 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:02.250 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:02.250 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:02.250 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:18:02.250 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:18:02.250 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:02.250 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:02.250 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:02.250 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:02.250 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:18:02.250 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:18:02.250 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:18:02.251 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:18:02.251 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:18:02.251 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:02.251 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:02.251 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:02.251 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:18:02.251 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:18:02.251 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:02.251 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:18:02.251 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:18:02.251 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:18:02.251 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:02.251 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:02.252 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:18:02.252 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:02.252 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:18:02.252 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:18:02.252 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:18:02.252 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:02.252 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:02.252 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:02.252 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:18:02.252 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:18:02.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.239+0000 7feb7619d640 1 -- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7feb54077a50 msgr2=0x7feb54079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:02.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.239+0000 7feb7619d640 1 --2- 172.21.15.19:0/2015469984 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7feb54077a50 0x7feb54079f60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7feb68034000 tx=0x7feb68004ab0 comp rx=0 tx=0).stop 2024-08-31T12:18:02.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.239+0000 7feb7619d640 1 -- 172.21.15.19:0/2015469984 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7feb700c4ca0 msgr2=0x7feb7006f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:02.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.239+0000 7feb7619d640 1 --2- 172.21.15.19:0/2015469984 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7feb700c4ca0 0x7feb7006f4c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7feb6c00c430 tx=0x7feb6c00c8f0 comp rx=0 tx=0).stop 2024-08-31T12:18:02.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.239+0000 7feb7599c640 1 -- 172.21.15.19:0/2015469984 reap_dead start 2024-08-31T12:18:02.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.239+0000 7feb7599c640 1 -- 172.21.15.19:0/2015469984 reap_dead start 2024-08-31T12:18:02.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.239+0000 7feb7619d640 1 -- 172.21.15.19:0/2015469984 shutdown_connections 2024-08-31T12:18:02.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.239+0000 7feb7619d640 1 -- 172.21.15.19:0/2015469984 >> 172.21.15.19:0/2015469984 conn(0x7feb70068ec0 msgr2=0x7feb700c3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:18:02.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.239+0000 7feb7619d640 1 -- 172.21.15.19:0/2015469984 shutdown_connections 2024-08-31T12:18:02.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.239+0000 7feb7619d640 1 -- 172.21.15.19:0/2015469984 wait complete. 2024-08-31T12:18:02.255 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:18:02.255 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-08-31T12:18:02.255 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-08-31T12:18:02.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.376+0000 7f5308684640 1 -- 172.21.15.19:0/3005363251 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f53000c9440 msgr2=0x7f53000cb580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:02.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.376+0000 7f5308684640 1 --2- 172.21.15.19:0/3005363251 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f53000c9440 0x7f53000cb580 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5300074850 tx=0x7f52f802aac0 comp rx=0 tx=0).stop 2024-08-31T12:18:02.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.376+0000 7f5308684640 1 -- 172.21.15.19:0/3005363251 shutdown_connections 2024-08-31T12:18:02.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.376+0000 7f5308684640 1 --2- 172.21.15.19:0/3005363251 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f53000c9440 0x7f53000cb580 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:02.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.376+0000 7f5308684640 1 --2- 172.21.15.19:0/3005363251 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f53000c9030 0x7f53000cb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:02.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.376+0000 7f5308684640 1 --2- 172.21.15.19:0/3005363251 >> v2:172.21.15.43:3300/0 conn(0x7f5300153190 0x7f5300159820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:02.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.376+0000 7f5308684640 1 -- 172.21.15.19:0/3005363251 >> 172.21.15.19:0/3005363251 conn(0x7f53000728b0 msgr2=0x7f5300072cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:18:02.378 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.376+0000 7f5308684640 1 -- 172.21.15.19:0/3005363251 shutdown_connections 2024-08-31T12:18:02.378 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.376+0000 7f5308684640 1 -- 172.21.15.19:0/3005363251 wait complete. 2024-08-31T12:18:02.378 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.378+0000 7f5308684640 1 Processor -- start 2024-08-31T12:18:02.378 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.378+0000 7f5308684640 1 -- start start 2024-08-31T12:18:02.382 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.380+0000 7f5308684640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53000c9030 0x7f5300157970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:02.382 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.380+0000 7f5308684640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f53000c9440 0x7f5300144160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:02.382 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.380+0000 7f5308684640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5300153190 0x7f530015b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:02.382 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.380+0000 7f5308684640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f530015cae0 con 0x7f5300153190 2024-08-31T12:18:02.382 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.380+0000 7f5308684640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f53001535a0 con 0x7f53000c9030 2024-08-31T12:18:02.382 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.380+0000 7f5308684640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f530015cdc0 con 0x7f53000c9440 2024-08-31T12:18:02.382 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.380+0000 7f5306c21640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5300153190 0x7f530015b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:02.382 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.380+0000 7f5306c21640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5300153190 0x7f530015b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:56856/0 (socket says 172.21.15.19:56856) 2024-08-31T12:18:02.382 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.380+0000 7f5306c21640 1 -- 172.21.15.19:0/3355542480 learned_addr learned my addr 172.21.15.19:0/3355542480 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:18:02.382 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.380+0000 7f5305c1f640 1 --2- 172.21.15.19:0/3355542480 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f53000c9440 0x7f5300144160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:02.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.380+0000 7f5306420640 1 --2- 172.21.15.19:0/3355542480 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53000c9030 0x7f5300157970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:02.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.380+0000 7f5306c21640 1 -- 172.21.15.19:0/3355542480 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f53000c9440 msgr2=0x7f5300144160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:02.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.380+0000 7f5306c21640 1 --2- 172.21.15.19:0/3355542480 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f53000c9440 0x7f5300144160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:02.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.380+0000 7f5306c21640 1 -- 172.21.15.19:0/3355542480 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53000c9030 msgr2=0x7f5300157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:02.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.380+0000 7f5306c21640 1 --2- 172.21.15.19:0/3355542480 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53000c9030 0x7f5300157970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:02.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.380+0000 7f5306c21640 1 -- 172.21.15.19:0/3355542480 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5300157eb0 con 0x7f5300153190 2024-08-31T12:18:02.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.380+0000 7f5306420640 1 --2- 172.21.15.19:0/3355542480 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53000c9030 0x7f5300157970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:18:02.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.380+0000 7f5306c21640 1 --2- 172.21.15.19:0/3355542480 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5300153190 0x7f530015b990 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f52f8009d60 tx=0x7f52f8003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:02.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.381+0000 7f52f6ffd640 1 -- 172.21.15.19:0/3355542480 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f8044070 con 0x7f5300153190 2024-08-31T12:18:02.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.381+0000 7f52f6ffd640 1 -- 172.21.15.19:0/3355542480 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f52f8004040 con 0x7f5300153190 2024-08-31T12:18:02.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.381+0000 7f5308684640 1 -- 172.21.15.19:0/3355542480 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f530015a180 con 0x7f5300153190 2024-08-31T12:18:02.384 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.381+0000 7f52f6ffd640 1 -- 172.21.15.19:0/3355542480 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f8004470 con 0x7f5300153190 2024-08-31T12:18:02.384 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.381+0000 7f5308684640 1 -- 172.21.15.19:0/3355542480 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f530015a440 con 0x7f5300153190 2024-08-31T12:18:02.384 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.382+0000 7f5308684640 1 -- 172.21.15.19:0/3355542480 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5300159820 con 0x7f5300153190 2024-08-31T12:18:02.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.382+0000 7f52f6ffd640 1 -- 172.21.15.19:0/3355542480 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f52f8004610 con 0x7f5300153190 2024-08-31T12:18:02.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.383+0000 7f52f6ffd640 1 --2- 172.21.15.19:0/3355542480 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f52e4077a80 0x7f52e4079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:02.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.383+0000 7f52f6ffd640 1 -- 172.21.15.19:0/3355542480 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f52f80b9d00 con 0x7f5300153190 2024-08-31T12:18:02.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.383+0000 7f5306420640 1 --2- 172.21.15.19:0/3355542480 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f52e4077a80 0x7f52e4079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:02.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.384+0000 7f5306420640 1 --2- 172.21.15.19:0/3355542480 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f52e4077a80 0x7f52e4079f90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f52fc009b80 tx=0x7f52fc005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:02.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.388+0000 7f52f6ffd640 1 -- 172.21.15.19:0/3355542480 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f52f8032030 con 0x7f5300153190 2024-08-31T12:18:02.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.717+0000 7f5308684640 1 -- 172.21.15.19:0/3355542480 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f53000c9a20 con 0x7f5300153190 2024-08-31T12:18:02.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.718+0000 7f52f6ffd640 1 -- 172.21.15.19:0/3355542480 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f52f80823b0 con 0x7f5300153190 2024-08-31T12:18:02.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.721+0000 7f5308684640 1 -- 172.21.15.19:0/3355542480 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f52e4077a80 msgr2=0x7f52e4079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:02.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.721+0000 7f5308684640 1 --2- 172.21.15.19:0/3355542480 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f52e4077a80 0x7f52e4079f90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f52fc009b80 tx=0x7f52fc005f90 comp rx=0 tx=0).stop 2024-08-31T12:18:02.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.721+0000 7f5308684640 1 -- 172.21.15.19:0/3355542480 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5300153190 msgr2=0x7f530015b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:02.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.721+0000 7f5308684640 1 --2- 172.21.15.19:0/3355542480 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5300153190 0x7f530015b990 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f52f8009d60 tx=0x7f52f8003c10 comp rx=0 tx=0).stop 2024-08-31T12:18:02.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.721+0000 7f5308684640 1 -- 172.21.15.19:0/3355542480 shutdown_connections 2024-08-31T12:18:02.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.721+0000 7f5308684640 1 --2- 172.21.15.19:0/3355542480 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f52e4077a80 0x7f52e4079f90 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:02.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.721+0000 7f5308684640 1 --2- 172.21.15.19:0/3355542480 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5300153190 0x7f530015b990 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:02.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.721+0000 7f5308684640 1 --2- 172.21.15.19:0/3355542480 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f53000c9440 0x7f5300144160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:02.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.721+0000 7f5308684640 1 --2- 172.21.15.19:0/3355542480 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53000c9030 0x7f5300157970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:02.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.721+0000 7f5308684640 1 -- 172.21.15.19:0/3355542480 >> 172.21.15.19:0/3355542480 conn(0x7f53000728b0 msgr2=0x7f5300150990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:18:02.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.722+0000 7f5308684640 1 -- 172.21.15.19:0/3355542480 shutdown_connections 2024-08-31T12:18:02.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:02.722+0000 7f5308684640 1 -- 172.21.15.19:0/3355542480 wait complete. 2024-08-31T12:18:02.735 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2024-08-31T12:18:12.737 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-08-31T12:18:12.737 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:18:12.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:12.857+0000 7f5efe3e2640 1 -- 172.21.15.19:0/175102374 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5ef80c7f00 msgr2=0x7f5ef80c53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:12.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:12.857+0000 7f5efe3e2640 1 --2- 172.21.15.19:0/175102374 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5ef80c7f00 0x7f5ef80c53c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5ef80c6780 tx=0x7f5ee802a530 comp rx=0 tx=0).stop 2024-08-31T12:18:12.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:12.857+0000 7f5efe3e2640 1 -- 172.21.15.19:0/175102374 shutdown_connections 2024-08-31T12:18:12.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:12.858+0000 7f5efe3e2640 1 --2- 172.21.15.19:0/175102374 >> v2:172.21.15.43:3300/0 conn(0x7f5ef80c6dd0 0x7f5ef80c71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:12.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:12.858+0000 7f5efe3e2640 1 --2- 172.21.15.19:0/175102374 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5ef80c7f00 0x7f5ef80c53c0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:12.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:12.858+0000 7f5efe3e2640 1 -- 172.21.15.19:0/175102374 >> 172.21.15.19:0/175102374 conn(0x7f5ef80ca630 msgr2=0x7f5ef80cd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:18:12.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:12.858+0000 7f5efe3e2640 1 -- 172.21.15.19:0/175102374 shutdown_connections 2024-08-31T12:18:12.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:12.858+0000 7f5efe3e2640 1 -- 172.21.15.19:0/175102374 wait complete. 2024-08-31T12:18:12.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:12.859+0000 7f5efe3e2640 1 Processor -- start 2024-08-31T12:18:12.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:12.860+0000 7f5efe3e2640 1 -- start start 2024-08-31T12:18:12.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:12.861+0000 7f5efe3e2640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5ef80c6dd0 0x7f5ef814cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:12.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:12.861+0000 7f5efe3e2640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5ef80c7f00 0x7f5ef8147d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:12.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:12.861+0000 7f5efe3e2640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ef8151ff0 0x7f5ef81e3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:12.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:12.861+0000 7f5efe3e2640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef80cded0 con 0x7f5ef80c6dd0 2024-08-31T12:18:12.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:12.861+0000 7f5efe3e2640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef80c8ff0 con 0x7f5ef8151ff0 2024-08-31T12:18:12.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:12.861+0000 7f5efe3e2640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f5ef8152400 con 0x7f5ef80c7f00 2024-08-31T12:18:12.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:12.861+0000 7f5ef7fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5ef80c6dd0 0x7f5ef814cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:12.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:12.861+0000 7f5ef7fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5ef80c6dd0 0x7f5ef814cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:34598/0 (socket says 172.21.15.19:34598) 2024-08-31T12:18:12.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:12.861+0000 7f5ef7fff640 1 -- 172.21.15.19:0/2078291484 learned_addr learned my addr 172.21.15.19:0/2078291484 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:18:12.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:12.861+0000 7f5ef77fe640 1 --2- 172.21.15.19:0/2078291484 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5ef80c7f00 0x7f5ef8147d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:12.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:12.861+0000 7f5efc97f640 1 --2- 172.21.15.19:0/2078291484 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ef8151ff0 0x7f5ef81e3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:12.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:12.862+0000 7f5ef7fff640 1 -- 172.21.15.19:0/2078291484 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5ef80c7f00 msgr2=0x7f5ef8147d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:12.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:12.862+0000 7f5ef7fff640 1 --2- 172.21.15.19:0/2078291484 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5ef80c7f00 0x7f5ef8147d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:12.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:12.862+0000 7f5ef7fff640 1 -- 172.21.15.19:0/2078291484 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ef8151ff0 msgr2=0x7f5ef81e3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:12.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:12.862+0000 7f5ef7fff640 1 --2- 172.21.15.19:0/2078291484 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ef8151ff0 0x7f5ef81e3720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:12.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:12.862+0000 7f5ef7fff640 1 -- 172.21.15.19:0/2078291484 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ef814d2f0 con 0x7f5ef80c6dd0 2024-08-31T12:18:12.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:12.862+0000 7f5ef77fe640 1 --2- 172.21.15.19:0/2078291484 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5ef80c7f00 0x7f5ef8147d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:18:12.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:12.862+0000 7f5efc97f640 1 --2- 172.21.15.19:0/2078291484 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ef8151ff0 0x7f5ef81e3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:18:12.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:12.862+0000 7f5ef7fff640 1 --2- 172.21.15.19:0/2078291484 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5ef80c6dd0 0x7f5ef814cdb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5ee80099a0 tx=0x7f5ee8031720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:12.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:12.862+0000 7f5ef4ff9640 1 -- 172.21.15.19:0/2078291484 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee80055d0 con 0x7f5ef80c6dd0 2024-08-31T12:18:12.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:12.862+0000 7f5efe3e2640 1 -- 172.21.15.19:0/2078291484 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ef81efea0 con 0x7f5ef80c6dd0 2024-08-31T12:18:12.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:12.862+0000 7f5ef4ff9640 1 -- 172.21.15.19:0/2078291484 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5ee8004890 con 0x7f5ef80c6dd0 2024-08-31T12:18:12.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:12.862+0000 7f5ef4ff9640 1 -- 172.21.15.19:0/2078291484 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee8004ce0 con 0x7f5ef80c6dd0 2024-08-31T12:18:12.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:12.862+0000 7f5efe3e2640 1 -- 172.21.15.19:0/2078291484 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ef81f0160 con 0x7f5ef80c6dd0 2024-08-31T12:18:12.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:12.863+0000 7f5efe3e2640 1 -- 172.21.15.19:0/2078291484 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ef806e350 con 0x7f5ef80c6dd0 2024-08-31T12:18:12.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:12.864+0000 7f5ef4ff9640 1 -- 172.21.15.19:0/2078291484 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f5ee803b410 con 0x7f5ef80c6dd0 2024-08-31T12:18:12.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:12.864+0000 7f5ef4ff9640 1 --2- 172.21.15.19:0/2078291484 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5ee4077a80 0x7f5ee4079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:12.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:12.865+0000 7f5ef4ff9640 1 -- 172.21.15.19:0/2078291484 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f5ee80bacc0 con 0x7f5ef80c6dd0 2024-08-31T12:18:12.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:12.865+0000 7f5ef77fe640 1 --2- 172.21.15.19:0/2078291484 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5ee4077a80 0x7f5ee4079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:12.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:12.865+0000 7f5ef77fe640 1 --2- 172.21.15.19:0/2078291484 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5ee4077a80 0x7f5ee4079f90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5ef81e4760 tx=0x7f5eec009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:12.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:12.870+0000 7f5ef4ff9640 1 -- 172.21.15.19:0/2078291484 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f5ee803f030 con 0x7f5ef80c6dd0 2024-08-31T12:18:13.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.169+0000 7f5efe3e2640 1 -- 172.21.15.19:0/2078291484 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5ef814c320 con 0x7f5ef80c6dd0 2024-08-31T12:18:13.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.169+0000 7f5ef4ff9640 1 -- 172.21.15.19:0/2078291484 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f5ee8083a50 con 0x7f5ef80c6dd0 2024-08-31T12:18:13.172 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:18:13.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.172+0000 7f5efe3e2640 1 -- 172.21.15.19:0/2078291484 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5ee4077a80 msgr2=0x7f5ee4079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:13.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.172+0000 7f5efe3e2640 1 --2- 172.21.15.19:0/2078291484 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5ee4077a80 0x7f5ee4079f90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5ef81e4760 tx=0x7f5eec009040 comp rx=0 tx=0).stop 2024-08-31T12:18:13.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.172+0000 7f5efe3e2640 1 -- 172.21.15.19:0/2078291484 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5ef80c6dd0 msgr2=0x7f5ef814cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:13.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.172+0000 7f5efe3e2640 1 --2- 172.21.15.19:0/2078291484 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5ef80c6dd0 0x7f5ef814cdb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5ee80099a0 tx=0x7f5ee8031720 comp rx=0 tx=0).stop 2024-08-31T12:18:13.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.172+0000 7f5efe3e2640 1 -- 172.21.15.19:0/2078291484 shutdown_connections 2024-08-31T12:18:13.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.172+0000 7f5efe3e2640 1 --2- 172.21.15.19:0/2078291484 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5ee4077a80 0x7f5ee4079f90 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:13.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.172+0000 7f5efe3e2640 1 --2- 172.21.15.19:0/2078291484 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ef8151ff0 0x7f5ef81e3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:13.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.172+0000 7f5efe3e2640 1 --2- 172.21.15.19:0/2078291484 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5ef80c7f00 0x7f5ef8147d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:13.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.172+0000 7f5efe3e2640 1 --2- 172.21.15.19:0/2078291484 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5ef80c6dd0 0x7f5ef814cdb0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:13.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.172+0000 7f5efe3e2640 1 -- 172.21.15.19:0/2078291484 >> 172.21.15.19:0/2078291484 conn(0x7f5ef80ca630 msgr2=0x7f5ef81e4f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:18:13.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.172+0000 7f5efe3e2640 1 -- 172.21.15.19:0/2078291484 shutdown_connections 2024-08-31T12:18:13.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.172+0000 7f5efe3e2640 1 -- 172.21.15.19:0/2078291484 wait complete. 2024-08-31T12:18:13.185 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-08-31T12:18:13.185 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-08-31T12:18:13.185 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-08-31T12:18:13.186 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-31T12:18:13.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.311+0000 7f14bbfad640 1 -- 172.21.15.19:0/2389246260 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f14b40c9030 msgr2=0x7f14b40cb040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:13.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.311+0000 7f14bbfad640 1 --2- 172.21.15.19:0/2389246260 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f14b40c9030 0x7f14b40cb040 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f14a8009830 tx=0x7f14a802a5a0 comp rx=0 tx=0).stop 2024-08-31T12:18:13.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.312+0000 7f14bbfad640 1 -- 172.21.15.19:0/2389246260 shutdown_connections 2024-08-31T12:18:13.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.312+0000 7f14bbfad640 1 --2- 172.21.15.19:0/2389246260 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f14b40c9030 0x7f14b40cb040 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:13.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.312+0000 7f14bbfad640 1 --2- 172.21.15.19:0/2389246260 >> v2:172.21.15.43:3300/0 conn(0x7f14b4153190 0x7f14b4159820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:13.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.312+0000 7f14bbfad640 1 -- 172.21.15.19:0/2389246260 >> 172.21.15.19:0/2389246260 conn(0x7f14b40728b0 msgr2=0x7f14b4072cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:18:13.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.312+0000 7f14bbfad640 1 -- 172.21.15.19:0/2389246260 shutdown_connections 2024-08-31T12:18:13.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.312+0000 7f14bbfad640 1 -- 172.21.15.19:0/2389246260 wait complete. 2024-08-31T12:18:13.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.314+0000 7f14bbfad640 1 Processor -- start 2024-08-31T12:18:13.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.314+0000 7f14bbfad640 1 -- start start 2024-08-31T12:18:13.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.315+0000 7f14bbfad640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14b40c9030 0x7f14b41f2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:13.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.315+0000 7f14bbfad640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f14b40c9440 0x7f14b40ca680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:13.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.315+0000 7f14bbfad640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f14b4153190 0x7f14b415b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:13.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.315+0000 7f14bbfad640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f14b41535a0 con 0x7f14b4153190 2024-08-31T12:18:13.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.315+0000 7f14bbfad640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f14b40c4db0 con 0x7f14b40c9030 2024-08-31T12:18:13.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.315+0000 7f14bbfad640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f14b415cdc0 con 0x7f14b40c9440 2024-08-31T12:18:13.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.316+0000 7f14b9d49640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14b40c9030 0x7f14b41f2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:13.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.316+0000 7f14b9d49640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14b40c9030 0x7f14b41f2de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:53422/0 (socket says 172.21.15.19:53422) 2024-08-31T12:18:13.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.316+0000 7f14b9d49640 1 -- 172.21.15.19:0/1624567726 learned_addr learned my addr 172.21.15.19:0/1624567726 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:18:13.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.316+0000 7f14b9548640 1 --2- 172.21.15.19:0/1624567726 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f14b40c9440 0x7f14b40ca680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:13.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.316+0000 7f14ba54a640 1 --2- 172.21.15.19:0/1624567726 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f14b4153190 0x7f14b415b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:13.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.316+0000 7f14ba54a640 1 -- 172.21.15.19:0/1624567726 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f14b40c9440 msgr2=0x7f14b40ca680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:13.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.316+0000 7f14ba54a640 1 --2- 172.21.15.19:0/1624567726 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f14b40c9440 0x7f14b40ca680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:13.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.316+0000 7f14ba54a640 1 -- 172.21.15.19:0/1624567726 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14b40c9030 msgr2=0x7f14b41f2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:13.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.316+0000 7f14ba54a640 1 --2- 172.21.15.19:0/1624567726 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f14b40c9030 0x7f14b41f2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:13.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.316+0000 7f14ba54a640 1 -- 172.21.15.19:0/1624567726 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14b41f3320 con 0x7f14b4153190 2024-08-31T12:18:13.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.316+0000 7f14ba54a640 1 --2- 172.21.15.19:0/1624567726 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f14b4153190 0x7f14b415b8b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f14a400c3c0 tx=0x7f14a400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:13.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.317+0000 7f14b27fc640 1 -- 172.21.15.19:0/1624567726 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a4011790 con 0x7f14b4153190 2024-08-31T12:18:13.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.317+0000 7f14b27fc640 1 -- 172.21.15.19:0/1624567726 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f14a4017070 con 0x7f14b4153190 2024-08-31T12:18:13.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.317+0000 7f14bbfad640 1 -- 172.21.15.19:0/1624567726 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14b4063b40 con 0x7f14b4153190 2024-08-31T12:18:13.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.317+0000 7f14b27fc640 1 -- 172.21.15.19:0/1624567726 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a4011ca0 con 0x7f14b4153190 2024-08-31T12:18:13.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.317+0000 7f14bbfad640 1 -- 172.21.15.19:0/1624567726 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14b41efbe0 con 0x7f14b4153190 2024-08-31T12:18:13.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.318+0000 7f14bbfad640 1 --2- 172.21.15.19:0/1624567726 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f14b41f34f0 0x7f14b415bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:13.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.318+0000 7f14bbfad640 1 -- 172.21.15.19:0/1624567726 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f14b41f0510 con 0x7f14b41f34f0 2024-08-31T12:18:13.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.318+0000 7f14b9d49640 1 --2- 172.21.15.19:0/1624567726 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f14b41f34f0 0x7f14b415bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:13.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.318+0000 7f14b9d49640 1 -- 172.21.15.19:0/1624567726 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f14b4150a40 con 0x7f14b41f34f0 2024-08-31T12:18:13.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.319+0000 7f14b9d49640 1 --2- 172.21.15.19:0/1624567726 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f14b41f34f0 0x7f14b415bdf0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f14ac011cf0 tx=0x7f14ac00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:13.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.319+0000 7f14b27fc640 1 -- 172.21.15.19:0/1624567726 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14ac004d20 con 0x7f14b41f34f0 2024-08-31T12:18:13.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.319+0000 7f14b27fc640 1 -- 172.21.15.19:0/1624567726 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f14ac00e5c0 con 0x7f14b41f34f0 2024-08-31T12:18:13.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.319+0000 7f14b27fc640 1 -- 172.21.15.19:0/1624567726 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f14b41f34f0 msgr2=0x7f14b415bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:13.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.319+0000 7f14b27fc640 1 --2- 172.21.15.19:0/1624567726 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f14b41f34f0 0x7f14b415bdf0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f14ac011cf0 tx=0x7f14ac00f790 comp rx=0 tx=0).stop 2024-08-31T12:18:13.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.319+0000 7f14b27fc640 1 -- 172.21.15.19:0/1624567726 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f14b41f34f0 msgr2=0x7f14b415bdf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:18:13.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.319+0000 7f14b27fc640 1 --2- 172.21.15.19:0/1624567726 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f14b41f34f0 0x7f14b415bdf0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:13.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.319+0000 7f14b27fc640 1 -- 172.21.15.19:0/1624567726 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f14a401c460 con 0x7f14b4153190 2024-08-31T12:18:13.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.320+0000 7f14b27fc640 1 --2- 172.21.15.19:0/1624567726 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f14a0077430 0x7f14a0079940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:13.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.320+0000 7f14b27fc640 1 -- 172.21.15.19:0/1624567726 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f14a40992b0 con 0x7f14b4153190 2024-08-31T12:18:13.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.320+0000 7f14b9d49640 1 --2- 172.21.15.19:0/1624567726 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f14a0077430 0x7f14a0079940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:13.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.321+0000 7f14b9d49640 1 --2- 172.21.15.19:0/1624567726 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f14a0077430 0x7f14a0079940 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f14ac000f80 tx=0x7f14ac015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:13.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.326+0000 7f14bbfad640 1 --2- 172.21.15.19:0/1624567726 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f14b40c86f0 0x7f14b41efe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:13.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.326+0000 7f14bbfad640 1 -- 172.21.15.19:0/1624567726 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f14b41f0360 con 0x7f14b40c86f0 2024-08-31T12:18:13.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.326+0000 7f14b9548640 1 --2- 172.21.15.19:0/1624567726 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f14b40c86f0 0x7f14b41efe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:13.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.327+0000 7f14b9548640 1 -- 172.21.15.19:0/1624567726 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f14b4159b50 con 0x7f14b40c86f0 2024-08-31T12:18:13.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.327+0000 7f14b9548640 1 --2- 172.21.15.19:0/1624567726 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f14b40c86f0 0x7f14b41efe20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f14b4074d30 tx=0x7f14a8003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:13.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.327+0000 7f14b27fc640 1 -- 172.21.15.19:0/1624567726 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a803c620 con 0x7f14b40c86f0 2024-08-31T12:18:13.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.327+0000 7f14b27fc640 1 -- 172.21.15.19:0/1624567726 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4877 (secure 0 0 0) 0x7f14a8040070 con 0x7f14b40c86f0 2024-08-31T12:18:13.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.327+0000 7f14b27fc640 1 -- 172.21.15.19:0/1624567726 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f14b40c86f0 msgr2=0x7f14b41efe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:13.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.327+0000 7f14b27fc640 1 --2- 172.21.15.19:0/1624567726 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f14b40c86f0 0x7f14b41efe20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f14b4074d30 tx=0x7f14a8003b90 comp rx=0 tx=0).stop 2024-08-31T12:18:13.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.327+0000 7f14b27fc640 1 -- 172.21.15.19:0/1624567726 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f14b40c86f0 msgr2=0x7f14b41efe20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:18:13.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.327+0000 7f14b27fc640 1 --2- 172.21.15.19:0/1624567726 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f14b40c86f0 0x7f14b41efe20 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:13.330 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:18:13.330 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:18:13.330 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:18:13.330 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "leader", 2024-08-31T12:18:13.331 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 20, 2024-08-31T12:18:13.331 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 73319, 2024-08-31T12:18:13.331 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:18:13.331 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:18:13.331 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:18:13.331 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:18:13.331 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:13.331 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 14, 2024-08-31T12:18:13.331 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:18:13.331 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:18:13.331 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:18:13.331 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:18:13.331 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:18:13.332 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:18:13.332 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:18:13.332 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:18:13.332 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:18:13.332 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:18:13.332 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:18:13.332 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:18:13.332 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:18:13.332 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:18:13.332 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:13.332 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:18:13.333 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:18:13.333 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:18:13.333 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:18:13.333 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:18:13.333 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:18:13.333 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:18:13.333 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:18:13.334 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:18:13.334 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:18:13.334 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:18:13.334 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:18:13.334 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:18:13.334 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:13.335 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:13.335 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:18:13.335 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:18:13.335 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:18:13.335 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:18:13.335 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:18:13.335 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:18:13.335 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:18:13.335 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:18:13.335 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:18:13.336 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:18:13.336 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:18:13.336 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:18:13.336 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:18:13.336 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:18:13.336 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:18:13.336 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:18:13.336 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:18:13.336 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:18:13.336 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:18:13.336 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:18:13.336 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:18:13.337 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:18:13.337 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:18:13.337 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:18:13.337 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:18:13.337 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:18:13.337 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:18:13.337 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:18:13.337 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:13.337 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:18:13.338 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:13.338 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:18:13.338 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:13.338 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:18:13.338 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:18:13.338 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:18:13.338 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:18:13.338 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:13.338 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:18:13.338 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:18:13.339 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:13.339 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:13.339 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:13.339 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:18:13.339 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:18:13.339 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:13.339 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:13.339 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:13.339 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:13.339 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:18:13.340 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:18:13.340 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:18:13.340 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:18:13.340 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:18:13.340 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:13.340 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:13.340 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:18:13.340 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:18:13.340 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:18:13.340 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:18:13.340 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:13.340 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:18:13.341 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:18:13.341 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:13.341 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:13.341 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:13.341 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:18:13.341 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:18:13.341 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:13.341 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:13.341 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:13.341 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:13.341 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:18:13.341 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:18:13.341 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:18:13.342 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:18:13.342 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:18:13.342 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:13.342 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:13.342 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:18:13.342 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:18:13.342 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:18:13.342 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:18:13.342 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:13.342 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:18:13.342 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:18:13.342 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:13.342 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:13.342 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:13.343 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:18:13.343 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:18:13.343 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:13.343 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:13.343 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:13.343 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:13.343 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:18:13.343 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:18:13.343 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:18:13.343 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:18:13.343 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:18:13.343 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:13.343 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:13.344 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:13.344 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:18:13.344 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:18:13.344 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:13.344 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:18:13.344 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:18:13.344 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:18:13.344 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:13.345 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:13.345 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:18:13.345 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:13.345 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:18:13.345 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:18:13.345 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:18:13.345 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:13.345 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:13.345 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:18:13.345 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:13.345 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:18:13.345 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:18:13.345 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:18:13.346 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:13.346 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:13.346 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:18:13.346 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:13.346 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:18:13.346 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:18:13.346 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:18:13.346 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:13.346 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:13.346 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:13.346 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:18:13.346 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:18:13.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.328+0000 7f14bbfad640 1 -- 172.21.15.19:0/1624567726 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f14a0077430 msgr2=0x7f14a0079940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:13.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.328+0000 7f14bbfad640 1 --2- 172.21.15.19:0/1624567726 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f14a0077430 0x7f14a0079940 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f14ac000f80 tx=0x7f14ac015000 comp rx=0 tx=0).stop 2024-08-31T12:18:13.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.328+0000 7f14bbfad640 1 -- 172.21.15.19:0/1624567726 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f14b4153190 msgr2=0x7f14b415b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:13.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.328+0000 7f14ba54a640 1 -- 172.21.15.19:0/1624567726 reap_dead start 2024-08-31T12:18:13.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.328+0000 7f14bbfad640 1 --2- 172.21.15.19:0/1624567726 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f14b4153190 0x7f14b415b8b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f14a400c3c0 tx=0x7f14a400c880 comp rx=0 tx=0).stop 2024-08-31T12:18:13.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.328+0000 7f14bbfad640 1 -- 172.21.15.19:0/1624567726 shutdown_connections 2024-08-31T12:18:13.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.328+0000 7f14bbfad640 1 --2- 172.21.15.19:0/1624567726 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f14b4153190 0x7f14b415b8b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:13.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.328+0000 7f14bbfad640 1 -- 172.21.15.19:0/1624567726 >> 172.21.15.19:0/1624567726 conn(0x7f14b40728b0 msgr2=0x7f14b4149700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:18:13.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.328+0000 7f14bbfad640 1 -- 172.21.15.19:0/1624567726 shutdown_connections 2024-08-31T12:18:13.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.328+0000 7f14bbfad640 1 -- 172.21.15.19:0/1624567726 wait complete. 2024-08-31T12:18:13.349 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:18:13.349 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-08-31T12:18:13.349 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-31T12:18:13.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.478+0000 7ff12c426640 1 -- 172.21.15.19:0/37172892 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff1240c5370 msgr2=0x7ff12406ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:13.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.478+0000 7ff12c426640 1 --2- 172.21.15.19:0/37172892 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff1240c5370 0x7ff12406ddd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff120009830 tx=0x7ff12002a530 comp rx=0 tx=0).stop 2024-08-31T12:18:13.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.478+0000 7ff12c426640 1 -- 172.21.15.19:0/37172892 shutdown_connections 2024-08-31T12:18:13.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.478+0000 7ff12c426640 1 --2- 172.21.15.19:0/37172892 >> v2:172.21.15.43:3300/0 conn(0x7ff1240cc400 0x7ff1240cc810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:13.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.478+0000 7ff12c426640 1 --2- 172.21.15.19:0/37172892 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff1240c5370 0x7ff12406ddd0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:13.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.478+0000 7ff12c426640 1 -- 172.21.15.19:0/37172892 >> 172.21.15.19:0/37172892 conn(0x7ff1240ca670 msgr2=0x7ff1240cd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:18:13.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.478+0000 7ff12c426640 1 -- 172.21.15.19:0/37172892 shutdown_connections 2024-08-31T12:18:13.482 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.478+0000 7ff12c426640 1 -- 172.21.15.19:0/37172892 wait complete. 2024-08-31T12:18:13.482 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.480+0000 7ff12c426640 1 Processor -- start 2024-08-31T12:18:13.482 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.480+0000 7ff12c426640 1 -- start start 2024-08-31T12:18:13.486 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.482+0000 7ff12c426640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff1240c5370 0x7ff1241e0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:13.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.482+0000 7ff12c426640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff1240cb2b0 0x7ff1241f2700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:13.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.482+0000 7ff12c426640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1240cc400 0x7ff124158a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:13.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.482+0000 7ff12c426640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7ff1240c72e0 con 0x7ff1240cb2b0 2024-08-31T12:18:13.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.482+0000 7ff12c426640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff1240ccd50 con 0x7ff1240cc400 2024-08-31T12:18:13.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.482+0000 7ff12c426640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7ff1240cb8c0 con 0x7ff1240c5370 2024-08-31T12:18:13.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.482+0000 7ff1299c1640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff1240cb2b0 0x7ff1241f2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:13.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.482+0000 7ff1299c1640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff1240cb2b0 0x7ff1241f2700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:34650/0 (socket says 172.21.15.19:34650) 2024-08-31T12:18:13.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.482+0000 7ff12a1c2640 1 --2- 172.21.15.19:0/3179409568 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff1240c5370 0x7ff1241e0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:13.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.482+0000 7ff1299c1640 1 -- 172.21.15.19:0/3179409568 learned_addr learned my addr 172.21.15.19:0/3179409568 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:18:13.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.482+0000 7ff12a9c3640 1 --2- 172.21.15.19:0/3179409568 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1240cc400 0x7ff124158a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:13.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.482+0000 7ff1299c1640 1 -- 172.21.15.19:0/3179409568 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff1240c5370 msgr2=0x7ff1241e0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:13.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.482+0000 7ff1299c1640 1 --2- 172.21.15.19:0/3179409568 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff1240c5370 0x7ff1241e0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:13.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.482+0000 7ff1299c1640 1 -- 172.21.15.19:0/3179409568 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1240cc400 msgr2=0x7ff124158a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:13.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.482+0000 7ff1299c1640 1 --2- 172.21.15.19:0/3179409568 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1240cc400 0x7ff124158a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:13.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.482+0000 7ff1299c1640 1 -- 172.21.15.19:0/3179409568 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1241f2c40 con 0x7ff1240cb2b0 2024-08-31T12:18:13.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.482+0000 7ff12a9c3640 1 --2- 172.21.15.19:0/3179409568 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1240cc400 0x7ff124158a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:18:13.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.482+0000 7ff12a1c2640 1 --2- 172.21.15.19:0/3179409568 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff1240c5370 0x7ff1241e0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:18:13.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.483+0000 7ff1299c1640 1 --2- 172.21.15.19:0/3179409568 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff1240cb2b0 0x7ff1241f2700 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff11400c530 tx=0x7ff11400c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:13.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.483+0000 7ff11affd640 1 -- 172.21.15.19:0/3179409568 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff114011790 con 0x7ff1240cb2b0 2024-08-31T12:18:13.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.483+0000 7ff11affd640 1 -- 172.21.15.19:0/3179409568 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff114017070 con 0x7ff1240cb2b0 2024-08-31T12:18:13.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.483+0000 7ff12c426640 1 -- 172.21.15.19:0/3179409568 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1241ef240 con 0x7ff1240cb2b0 2024-08-31T12:18:13.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.483+0000 7ff11affd640 1 -- 172.21.15.19:0/3179409568 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff114011cc0 con 0x7ff1240cb2b0 2024-08-31T12:18:13.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.483+0000 7ff12c426640 1 -- 172.21.15.19:0/3179409568 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1241ef800 con 0x7ff1240cb2b0 2024-08-31T12:18:13.489 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.484+0000 7ff12c426640 1 --2- 172.21.15.19:0/3179409568 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1241e1590 0x7ff12415a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:13.489 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.484+0000 7ff12c426640 1 -- 172.21.15.19:0/3179409568 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff1241f3110 con 0x7ff1241e1590 2024-08-31T12:18:13.489 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.484+0000 7ff12a1c2640 1 --2- 172.21.15.19:0/3179409568 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1241e1590 0x7ff12415a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:13.489 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.484+0000 7ff12a1c2640 1 -- 172.21.15.19:0/3179409568 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff12406e760 con 0x7ff1241e1590 2024-08-31T12:18:13.489 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.485+0000 7ff12a1c2640 1 --2- 172.21.15.19:0/3179409568 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1241e1590 0x7ff12415a7e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff124074d70 tx=0x7ff120003f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:13.489 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.485+0000 7ff11affd640 1 -- 172.21.15.19:0/3179409568 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff120045640 con 0x7ff1241e1590 2024-08-31T12:18:13.489 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.485+0000 7ff11affd640 1 -- 172.21.15.19:0/3179409568 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7ff114004020 con 0x7ff1240cb2b0 2024-08-31T12:18:13.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.485+0000 7ff11affd640 1 --2- 172.21.15.19:0/3179409568 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff108077980 0x7ff108079e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:13.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.485+0000 7ff11affd640 1 -- 172.21.15.19:0/3179409568 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7ff1140992a0 con 0x7ff1240cb2b0 2024-08-31T12:18:13.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.486+0000 7ff12a9c3640 1 --2- 172.21.15.19:0/3179409568 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff108077980 0x7ff108079e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:13.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.486+0000 7ff11affd640 1 -- 172.21.15.19:0/3179409568 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff12003b410 con 0x7ff1241e1590 2024-08-31T12:18:13.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.486+0000 7ff11affd640 1 -- 172.21.15.19:0/3179409568 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1241e1590 msgr2=0x7ff12415a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:13.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.486+0000 7ff11affd640 1 --2- 172.21.15.19:0/3179409568 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1241e1590 0x7ff12415a7e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff124074d70 tx=0x7ff120003f20 comp rx=0 tx=0).stop 2024-08-31T12:18:13.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.486+0000 7ff11affd640 1 -- 172.21.15.19:0/3179409568 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1241e1590 msgr2=0x7ff12415a7e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:18:13.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.486+0000 7ff11affd640 1 --2- 172.21.15.19:0/3179409568 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1241e1590 0x7ff12415a7e0 secure :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff124074d70 tx=0x7ff120003f20 comp rx=0 tx=0).stop 2024-08-31T12:18:13.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.486+0000 7ff12a9c3640 1 --2- 172.21.15.19:0/3179409568 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff108077980 0x7ff108079e90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff11c0099c0 tx=0x7ff11c009500 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:13.493 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.492+0000 7ff12c426640 1 --2- 172.21.15.19:0/3179409568 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1241efa00 0x7ff12415ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:13.494 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.492+0000 7ff12c426640 1 -- 172.21.15.19:0/3179409568 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff12415b260 con 0x7ff1241efa00 2024-08-31T12:18:13.494 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.492+0000 7ff12a1c2640 1 --2- 172.21.15.19:0/3179409568 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1241efa00 0x7ff12415ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:13.494 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.493+0000 7ff12a1c2640 1 -- 172.21.15.19:0/3179409568 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff1240cbfd0 con 0x7ff1241efa00 2024-08-31T12:18:13.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.493+0000 7ff12a1c2640 1 --2- 172.21.15.19:0/3179409568 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1241efa00 0x7ff12415ad20 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff120000c00 tx=0x7ff120005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:13.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.493+0000 7ff11affd640 1 -- 172.21.15.19:0/3179409568 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff12003bcd0 con 0x7ff1241efa00 2024-08-31T12:18:13.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.493+0000 7ff11affd640 1 -- 172.21.15.19:0/3179409568 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5043 (secure 0 0 0) 0x7ff12003bcd0 con 0x7ff1241efa00 2024-08-31T12:18:13.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.493+0000 7ff11affd640 1 -- 172.21.15.19:0/3179409568 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1241efa00 msgr2=0x7ff12415ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:13.496 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.493+0000 7ff11affd640 1 --2- 172.21.15.19:0/3179409568 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1241efa00 0x7ff12415ad20 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff120000c00 tx=0x7ff120005f90 comp rx=0 tx=0).stop 2024-08-31T12:18:13.496 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.493+0000 7ff11affd640 1 -- 172.21.15.19:0/3179409568 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1241efa00 msgr2=0x7ff12415ad20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:18:13.496 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.493+0000 7ff11affd640 1 --2- 172.21.15.19:0/3179409568 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1241efa00 0x7ff12415ad20 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:13.496 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:18:13.496 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:18:13.496 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:18:13.496 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:18:13.496 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 20, 2024-08-31T12:18:13.496 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 161589, 2024-08-31T12:18:13.497 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:18:13.497 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:18:13.497 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:18:13.497 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:18:13.497 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:13.497 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 14, 2024-08-31T12:18:13.497 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:18:13.497 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:18:13.497 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:18:13.497 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:18:13.497 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:18:13.497 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:18:13.497 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:18:13.497 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:18:13.498 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:18:13.498 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:18:13.498 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:18:13.498 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:18:13.498 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:18:13.498 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:18:13.498 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:13.498 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:18:13.498 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:18:13.498 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:18:13.498 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:18:13.498 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:18:13.498 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:18:13.498 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:18:13.499 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:18:13.499 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:18:13.499 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:18:13.499 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:18:13.499 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:18:13.499 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:18:13.499 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:13.499 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:13.499 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:18:13.499 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:18:13.499 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:18:13.499 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:18:13.499 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:18:13.500 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:18:13.500 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:18:13.500 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:18:13.500 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:18:13.500 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:18:13.500 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:18:13.500 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:18:13.500 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:18:13.500 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:18:13.500 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:18:13.500 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:18:13.500 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:18:13.500 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:18:13.501 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:18:13.501 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:18:13.501 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:18:13.501 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:18:13.501 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:18:13.501 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:18:13.501 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:18:13.501 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:18:13.501 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:18:13.501 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:18:13.501 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:13.501 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:18:13.501 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:13.502 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:18:13.502 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:13.502 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:18:13.502 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:18:13.502 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:18:13.502 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:18:13.502 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:13.502 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:18:13.502 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:18:13.502 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:13.502 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:13.502 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:13.502 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:18:13.502 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:18:13.503 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:13.503 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:13.503 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:13.503 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:13.503 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:18:13.503 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:18:13.503 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:18:13.503 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:18:13.503 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:18:13.503 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:13.503 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:13.503 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:18:13.503 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:18:13.503 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:18:13.504 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:18:13.504 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:13.504 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:18:13.504 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:18:13.504 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:13.504 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:13.504 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:13.504 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:18:13.504 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:18:13.504 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:13.504 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:13.504 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:13.504 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:13.505 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:18:13.505 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:18:13.505 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:18:13.505 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:18:13.505 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:18:13.505 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:13.505 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:13.505 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:18:13.505 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:18:13.505 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:18:13.505 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:18:13.505 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:13.505 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:18:13.505 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:18:13.506 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:13.506 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:13.506 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:13.506 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:18:13.506 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:18:13.506 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:13.506 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:13.506 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:13.506 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:13.506 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:18:13.506 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:18:13.506 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:18:13.507 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:18:13.507 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:18:13.507 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:13.507 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:13.507 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:13.507 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:18:13.507 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:18:13.507 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:13.507 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:18:13.507 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:18:13.507 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:18:13.507 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:13.507 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:13.507 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:18:13.508 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:13.508 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:18:13.508 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:18:13.508 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 5 2024-08-31T12:18:13.508 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:13.508 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:13.508 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:18:13.508 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:13.508 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:18:13.508 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:18:13.508 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 7 2024-08-31T12:18:13.508 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:13.508 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:13.508 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:18:13.508 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:13.509 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:18:13.509 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:18:13.509 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 6 2024-08-31T12:18:13.509 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:13.509 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:13.509 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:18:13.509 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:13.509 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:18:13.509 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:18:13.509 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:18:13.509 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:13.509 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:13.509 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:13.510 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:18:13.510 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:18:13.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.494+0000 7ff12c426640 1 -- 172.21.15.19:0/3179409568 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff108077980 msgr2=0x7ff108079e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:13.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.494+0000 7ff12c426640 1 --2- 172.21.15.19:0/3179409568 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff108077980 0x7ff108079e90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff11c0099c0 tx=0x7ff11c009500 comp rx=0 tx=0).stop 2024-08-31T12:18:13.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.494+0000 7ff12c426640 1 -- 172.21.15.19:0/3179409568 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff1240cb2b0 msgr2=0x7ff1241f2700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:13.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.494+0000 7ff12c426640 1 --2- 172.21.15.19:0/3179409568 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff1240cb2b0 0x7ff1241f2700 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff11400c530 tx=0x7ff11400c930 comp rx=0 tx=0).stop 2024-08-31T12:18:13.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.494+0000 7ff12a9c3640 1 -- 172.21.15.19:0/3179409568 reap_dead start 2024-08-31T12:18:13.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.494+0000 7ff12a9c3640 1 -- 172.21.15.19:0/3179409568 reap_dead start 2024-08-31T12:18:13.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.494+0000 7ff12c426640 1 -- 172.21.15.19:0/3179409568 shutdown_connections 2024-08-31T12:18:13.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.494+0000 7ff12c426640 1 -- 172.21.15.19:0/3179409568 >> 172.21.15.19:0/3179409568 conn(0x7ff1240ca670 msgr2=0x7ff12414a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:18:13.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.494+0000 7ff12c426640 1 -- 172.21.15.19:0/3179409568 shutdown_connections 2024-08-31T12:18:13.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.494+0000 7ff12c426640 1 -- 172.21.15.19:0/3179409568 wait complete. 2024-08-31T12:18:13.512 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:18:13.512 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-08-31T12:18:13.512 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-31T12:18:13.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.638+0000 7fd8f92fa640 1 -- 172.21.15.19:0/2340689046 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd8f40cc400 msgr2=0x7fd8f40cc810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:13.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.638+0000 7fd8f92fa640 1 --2- 172.21.15.19:0/2340689046 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd8f40cc400 0x7fd8f40cc810 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec005fd0 tx=0x7fd8ec02a8d0 comp rx=0 tx=0).stop 2024-08-31T12:18:13.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.638+0000 7fd8f92fa640 1 -- 172.21.15.19:0/2340689046 shutdown_connections 2024-08-31T12:18:13.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.638+0000 7fd8f92fa640 1 --2- 172.21.15.19:0/2340689046 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd8f40cc400 0x7fd8f40cc810 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:13.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.638+0000 7fd8f92fa640 1 --2- 172.21.15.19:0/2340689046 >> v2:172.21.15.43:3300/0 conn(0x7fd8f40c5370 0x7fd8f406ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:13.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.638+0000 7fd8f92fa640 1 -- 172.21.15.19:0/2340689046 >> 172.21.15.19:0/2340689046 conn(0x7fd8f40ca670 msgr2=0x7fd8f40cd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:18:13.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.638+0000 7fd8f92fa640 1 -- 172.21.15.19:0/2340689046 shutdown_connections 2024-08-31T12:18:13.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.638+0000 7fd8f92fa640 1 -- 172.21.15.19:0/2340689046 wait complete. 2024-08-31T12:18:13.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.640+0000 7fd8f92fa640 1 Processor -- start 2024-08-31T12:18:13.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.640+0000 7fd8f92fa640 1 -- start start 2024-08-31T12:18:13.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.641+0000 7fd8f92fa640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd8f40c5370 0x7fd8f4153250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:13.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.641+0000 7fd8f92fa640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd8f40cb2b0 0x7fd8f40c84d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:13.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.641+0000 7fd8f92fa640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd8f40cc400 0x7fd8f41e24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:13.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.641+0000 7fd8f92fa640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f40c72e0 con 0x7fd8f40cb2b0 2024-08-31T12:18:13.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.641+0000 7fd8f92fa640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f40ccd50 con 0x7fd8f40c5370 2024-08-31T12:18:13.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.641+0000 7fd8f92fa640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fd8f40cb8c0 con 0x7fd8f40cc400 2024-08-31T12:18:13.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.641+0000 7fd8f27fc640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd8f40cb2b0 0x7fd8f40c84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:13.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.641+0000 7fd8f27fc640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd8f40cb2b0 0x7fd8f40c84d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:34658/0 (socket says 172.21.15.19:34658) 2024-08-31T12:18:13.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.641+0000 7fd8f27fc640 1 -- 172.21.15.19:0/536880173 learned_addr learned my addr 172.21.15.19:0/536880173 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:18:13.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.641+0000 7fd8f2ffd640 1 --2- 172.21.15.19:0/536880173 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd8f40c5370 0x7fd8f4153250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:13.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.641+0000 7fd8f37fe640 1 --2- 172.21.15.19:0/536880173 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd8f40cc400 0x7fd8f41e24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:13.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.641+0000 7fd8f27fc640 1 -- 172.21.15.19:0/536880173 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd8f40cc400 msgr2=0x7fd8f41e24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:13.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.641+0000 7fd8f27fc640 1 --2- 172.21.15.19:0/536880173 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd8f40cc400 0x7fd8f41e24e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:13.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.641+0000 7fd8f27fc640 1 -- 172.21.15.19:0/536880173 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd8f40c5370 msgr2=0x7fd8f4153250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:13.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.641+0000 7fd8f27fc640 1 --2- 172.21.15.19:0/536880173 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd8f40c5370 0x7fd8f4153250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:13.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.641+0000 7fd8f27fc640 1 -- 172.21.15.19:0/536880173 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8f40c8a10 con 0x7fd8f40cb2b0 2024-08-31T12:18:13.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.641+0000 7fd8f2ffd640 1 --2- 172.21.15.19:0/536880173 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd8f40c5370 0x7fd8f4153250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:18:13.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.642+0000 7fd8f27fc640 1 --2- 172.21.15.19:0/536880173 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd8f40cb2b0 0x7fd8f40c84d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd8e800c530 tx=0x7fd8e800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:13.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.642+0000 7fd8db7fe640 1 -- 172.21.15.19:0/536880173 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e8016020 con 0x7fd8f40cb2b0 2024-08-31T12:18:13.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.642+0000 7fd8db7fe640 1 -- 172.21.15.19:0/536880173 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd8e8012940 con 0x7fd8f40cb2b0 2024-08-31T12:18:13.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.642+0000 7fd8f92fa640 1 -- 172.21.15.19:0/536880173 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8f41f5400 con 0x7fd8f40cb2b0 2024-08-31T12:18:13.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.642+0000 7fd8db7fe640 1 -- 172.21.15.19:0/536880173 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e8012d90 con 0x7fd8f40cb2b0 2024-08-31T12:18:13.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.642+0000 7fd8f92fa640 1 -- 172.21.15.19:0/536880173 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8f4147c80 con 0x7fd8f40cb2b0 2024-08-31T12:18:13.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.643+0000 7fd8f92fa640 1 --2- 172.21.15.19:0/536880173 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd8f4148710 0x7fd8f40c9330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:13.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.643+0000 7fd8f92fa640 1 -- 172.21.15.19:0/536880173 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fd8f40c8ee0 con 0x7fd8f4148710 2024-08-31T12:18:13.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.643+0000 7fd8f2ffd640 1 --2- 172.21.15.19:0/536880173 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd8f4148710 0x7fd8f40c9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:13.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.643+0000 7fd8f2ffd640 1 -- 172.21.15.19:0/536880173 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd8f406e760 con 0x7fd8f4148710 2024-08-31T12:18:13.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.644+0000 7fd8f2ffd640 1 --2- 172.21.15.19:0/536880173 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd8f4148710 0x7fd8f40c9330 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd8e40119b0 tx=0x7fd8e4011e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:13.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.644+0000 7fd8db7fe640 1 -- 172.21.15.19:0/536880173 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e401f070 con 0x7fd8f4148710 2024-08-31T12:18:13.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.644+0000 7fd8db7fe640 1 -- 172.21.15.19:0/536880173 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fd8e8019020 con 0x7fd8f40cb2b0 2024-08-31T12:18:13.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.644+0000 7fd8db7fe640 1 --2- 172.21.15.19:0/536880173 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd8e0077980 0x7fd8e0079e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:13.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.645+0000 7fd8db7fe640 1 -- 172.21.15.19:0/536880173 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fd8e80993f0 con 0x7fd8f40cb2b0 2024-08-31T12:18:13.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.645+0000 7fd8f37fe640 1 --2- 172.21.15.19:0/536880173 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd8e0077980 0x7fd8e0079e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:13.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.645+0000 7fd8db7fe640 1 -- 172.21.15.19:0/536880173 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd8e400fd60 con 0x7fd8f4148710 2024-08-31T12:18:13.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.645+0000 7fd8db7fe640 1 -- 172.21.15.19:0/536880173 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd8f4148710 msgr2=0x7fd8f40c9330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:13.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.645+0000 7fd8db7fe640 1 --2- 172.21.15.19:0/536880173 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd8f4148710 0x7fd8f40c9330 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd8e40119b0 tx=0x7fd8e4011e70 comp rx=0 tx=0).stop 2024-08-31T12:18:13.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.645+0000 7fd8db7fe640 1 -- 172.21.15.19:0/536880173 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd8f4148710 msgr2=0x7fd8f40c9330 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:18:13.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.645+0000 7fd8db7fe640 1 --2- 172.21.15.19:0/536880173 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd8f4148710 0x7fd8f40c9330 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:13.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.645+0000 7fd8f37fe640 1 --2- 172.21.15.19:0/536880173 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd8e0077980 0x7fd8e0079e90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec034000 tx=0x7fd8ec004ab0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:13.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.652+0000 7fd8f92fa640 1 --2- 172.21.15.19:0/536880173 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd8f40c9870 0x7fd8f406cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:13.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.652+0000 7fd8f92fa640 1 -- 172.21.15.19:0/536880173 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fd8f4153790 con 0x7fd8f40c9870 2024-08-31T12:18:13.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.652+0000 7fd8f2ffd640 1 --2- 172.21.15.19:0/536880173 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd8f40c9870 0x7fd8f406cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:13.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.652+0000 7fd8f2ffd640 1 -- 172.21.15.19:0/536880173 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd8f40c4cc0 con 0x7fd8f40c9870 2024-08-31T12:18:13.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.653+0000 7fd8f2ffd640 1 --2- 172.21.15.19:0/536880173 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd8f40c9870 0x7fd8f406cae0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd8e40119b0 tx=0x7fd8e40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:13.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.653+0000 7fd8db7fe640 1 -- 172.21.15.19:0/536880173 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e4012e60 con 0x7fd8f40c9870 2024-08-31T12:18:13.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.653+0000 7fd8db7fe640 1 -- 172.21.15.19:0/536880173 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4541 (secure 0 0 0) 0x7fd8e4004020 con 0x7fd8f40c9870 2024-08-31T12:18:13.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.653+0000 7fd8db7fe640 1 -- 172.21.15.19:0/536880173 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd8f40c9870 msgr2=0x7fd8f406cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:13.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.653+0000 7fd8db7fe640 1 --2- 172.21.15.19:0/536880173 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd8f40c9870 0x7fd8f406cae0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd8e40119b0 tx=0x7fd8e40046c0 comp rx=0 tx=0).stop 2024-08-31T12:18:13.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.653+0000 7fd8db7fe640 1 -- 172.21.15.19:0/536880173 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd8f40c9870 msgr2=0x7fd8f406cae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:18:13.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.653+0000 7fd8db7fe640 1 --2- 172.21.15.19:0/536880173 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd8f40c9870 0x7fd8f406cae0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:13.656 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:18:13.656 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:18:13.656 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:18:13.656 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:18:13.656 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 20, 2024-08-31T12:18:13.656 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 21775, 2024-08-31T12:18:13.656 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:18:13.656 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:18:13.656 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:18:13.656 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:18:13.656 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:13.657 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 15, 2024-08-31T12:18:13.657 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:18:13.657 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:18:13.657 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:18:13.657 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:18:13.657 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:18:13.657 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:18:13.657 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:18:13.657 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:18:13.657 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:18:13.657 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:18:13.657 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:18:13.657 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:18:13.658 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:18:13.658 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:18:13.658 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:13.658 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:18:13.658 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:18:13.658 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:18:13.658 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:18:13.658 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:18:13.658 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:18:13.658 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:18:13.658 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:18:13.658 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:18:13.658 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:18:13.658 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:18:13.659 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:18:13.659 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:18:13.659 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:13.659 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:13.659 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:18:13.659 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:18:13.659 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:18:13.659 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:18:13.659 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:18:13.659 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:18:13.659 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:18:13.659 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:18:13.659 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:18:13.659 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:18:13.660 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:18:13.660 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:18:13.660 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:18:13.660 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:18:13.660 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:18:13.660 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:18:13.660 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:18:13.660 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:18:13.660 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:18:13.660 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:18:13.660 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:18:13.660 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:18:13.660 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:18:13.661 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:18:13.661 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:18:13.661 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:18:13.661 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:18:13.661 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:18:13.661 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:13.661 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:18:13.661 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:13.661 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:18:13.661 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:13.661 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:18:13.661 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:18:13.661 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:18:13.662 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:18:13.662 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:13.662 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:18:13.662 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:18:13.662 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:13.662 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:13.662 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:13.662 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:18:13.662 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:18:13.662 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:13.662 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:13.662 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:13.662 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:13.662 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:18:13.663 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:18:13.663 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:18:13.663 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:18:13.663 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:18:13.663 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:13.663 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:13.663 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:18:13.663 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:18:13.663 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:18:13.663 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:18:13.663 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:13.663 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:18:13.663 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:18:13.664 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:13.664 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:13.664 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:13.664 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:18:13.664 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:18:13.664 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:13.664 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:13.664 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:13.664 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:13.664 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:18:13.664 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:18:13.664 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:18:13.664 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:18:13.665 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:18:13.665 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:13.665 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:13.665 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:18:13.665 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:18:13.665 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:18:13.665 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:18:13.665 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:13.665 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:18:13.665 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:18:13.665 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:13.665 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:13.665 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:13.666 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:18:13.666 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:18:13.666 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:13.666 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:13.666 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:13.666 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:13.666 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:18:13.666 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:18:13.666 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:18:13.666 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:18:13.666 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:18:13.666 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:13.667 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:13.667 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:13.667 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:18:13.667 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:18:13.667 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:13.667 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:18:13.667 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:18:13.667 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:18:13.667 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:13.667 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:13.667 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:18:13.667 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:13.667 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:18:13.667 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:18:13.668 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:18:13.668 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:13.668 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:13.668 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:13.668 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:18:13.668 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:18:13.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.653+0000 7fd8f92fa640 1 -- 172.21.15.19:0/536880173 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd8e0077980 msgr2=0x7fd8e0079e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:13.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.653+0000 7fd8f92fa640 1 --2- 172.21.15.19:0/536880173 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd8e0077980 0x7fd8e0079e90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec034000 tx=0x7fd8ec004ab0 comp rx=0 tx=0).stop 2024-08-31T12:18:13.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.653+0000 7fd8f92fa640 1 -- 172.21.15.19:0/536880173 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd8f40cb2b0 msgr2=0x7fd8f40c84d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:13.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.653+0000 7fd8f92fa640 1 --2- 172.21.15.19:0/536880173 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd8f40cb2b0 0x7fd8f40c84d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd8e800c530 tx=0x7fd8e800c9f0 comp rx=0 tx=0).stop 2024-08-31T12:18:13.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.653+0000 7fd8f37fe640 1 -- 172.21.15.19:0/536880173 reap_dead start 2024-08-31T12:18:13.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.653+0000 7fd8f37fe640 1 -- 172.21.15.19:0/536880173 reap_dead start 2024-08-31T12:18:13.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.653+0000 7fd8f92fa640 1 -- 172.21.15.19:0/536880173 shutdown_connections 2024-08-31T12:18:13.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.653+0000 7fd8f92fa640 1 -- 172.21.15.19:0/536880173 >> 172.21.15.19:0/536880173 conn(0x7fd8f40ca670 msgr2=0x7fd8f40635a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:18:13.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.653+0000 7fd8f92fa640 1 -- 172.21.15.19:0/536880173 shutdown_connections 2024-08-31T12:18:13.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.653+0000 7fd8f92fa640 1 -- 172.21.15.19:0/536880173 wait complete. 2024-08-31T12:18:13.670 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:18:13.671 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-08-31T12:18:13.671 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-08-31T12:18:13.671 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-08-31T12:18:13.671 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-08-31T12:18:13.671 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-08-31T12:18:13.671 INFO:teuthology.orchestra.run:waiting for 300 2024-08-31T12:18:13.689 INFO:tasks.ceph.mon.b:Stopped 2024-08-31T12:18:13.689 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-08-31T12:18:13.689 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:18:13.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.815+0000 7fd862d00640 1 -- 172.21.15.19:0/1873290303 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd84c0039b0 msgr2=0x7fd84c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:13.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.815+0000 7fd862d00640 1 --2- 172.21.15.19:0/1873290303 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd84c0039b0 0x7fd84c003e20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd85000c640 tx=0x7fd84c005830 comp rx=0 tx=0).stop 2024-08-31T12:18:13.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.815+0000 7fd86129d640 1 -- 172.21.15.19:0/1873290303 reap_dead start 2024-08-31T12:18:13.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.815+0000 7fd862d00640 1 -- 172.21.15.19:0/1873290303 shutdown_connections 2024-08-31T12:18:13.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.815+0000 7fd862d00640 1 -- 172.21.15.19:0/1873290303 >> 172.21.15.19:0/1873290303 conn(0x7fd85c074810 msgr2=0x7fd85c068aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:18:13.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.815+0000 7fd862d00640 1 -- 172.21.15.19:0/1873290303 shutdown_connections 2024-08-31T12:18:13.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.815+0000 7fd862d00640 1 -- 172.21.15.19:0/1873290303 wait complete. 2024-08-31T12:18:13.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.817+0000 7fd862d00640 1 Processor -- start 2024-08-31T12:18:13.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.817+0000 7fd862d00640 1 -- start start 2024-08-31T12:18:13.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.818+0000 7fd862d00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd84c0039b0 0x7fd85c15dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:13.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.818+0000 7fd862d00640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd85c1f4c10 0x7fd85c0c5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:13.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.818+0000 7fd862d00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd85c1501a0 0x7fd85c1505b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:13.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.818+0000 7fd862d00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fd85c15ba70 con 0x7fd84c0039b0 2024-08-31T12:18:13.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.818+0000 7fd862d00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd85c15f050 con 0x7fd85c1501a0 2024-08-31T12:18:13.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.818+0000 7fd862d00640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fd85c0cc780 con 0x7fd85c1f4c10 2024-08-31T12:18:13.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.819+0000 7fd860a9c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd84c0039b0 0x7fd85c15dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:13.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.819+0000 7fd86129d640 1 -- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd85c1501a0 msgr2=0x7fd85c1505b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3300/0 2024-08-31T12:18:13.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.819+0000 7fd86129d640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd85c1501a0 0x7fd85c1505b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:18:13.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.819+0000 7fd860a9c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd84c0039b0 0x7fd85c15dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:34702/0 (socket says 172.21.15.19:34702) 2024-08-31T12:18:13.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.819+0000 7fd860a9c640 1 -- 172.21.15.19:0/3642841914 learned_addr learned my addr 172.21.15.19:0/3642841914 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:18:13.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.819+0000 7fd85bfff640 1 --2- 172.21.15.19:0/3642841914 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd85c1f4c10 0x7fd85c0c5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:13.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.819+0000 7fd860a9c640 1 -- 172.21.15.19:0/3642841914 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd85c1f4c10 msgr2=0x7fd85c0c5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:13.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.819+0000 7fd860a9c640 1 --2- 172.21.15.19:0/3642841914 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd85c1f4c10 0x7fd85c0c5880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:13.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.819+0000 7fd860a9c640 1 -- 172.21.15.19:0/3642841914 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd85c1501a0 msgr2=0x7fd85c1505b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:18:13.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.819+0000 7fd860a9c640 1 --2- 172.21.15.19:0/3642841914 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd85c1501a0 0x7fd85c1505b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:13.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.819+0000 7fd860a9c640 1 -- 172.21.15.19:0/3642841914 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd85c15db10 con 0x7fd84c0039b0 2024-08-31T12:18:13.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.819+0000 7fd860a9c640 1 --2- 172.21.15.19:0/3642841914 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd84c0039b0 0x7fd85c15dde0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd85002b330 tx=0x7fd850006620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:13.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.819+0000 7fd8597fa640 1 -- 172.21.15.19:0/3642841914 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd850039070 con 0x7fd84c0039b0 2024-08-31T12:18:13.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.819+0000 7fd8597fa640 1 -- 172.21.15.19:0/3642841914 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd85002b950 con 0x7fd84c0039b0 2024-08-31T12:18:13.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.819+0000 7fd862d00640 1 -- 172.21.15.19:0/3642841914 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd85c1f20a0 con 0x7fd84c0039b0 2024-08-31T12:18:13.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.819+0000 7fd8597fa640 1 -- 172.21.15.19:0/3642841914 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd85002bda0 con 0x7fd84c0039b0 2024-08-31T12:18:13.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.820+0000 7fd862d00640 1 -- 172.21.15.19:0/3642841914 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd85c0ca500 con 0x7fd84c0039b0 2024-08-31T12:18:13.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.821+0000 7fd862d00640 1 -- 172.21.15.19:0/3642841914 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd85c155660 con 0x7fd84c0039b0 2024-08-31T12:18:13.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.821+0000 7fd8597fa640 1 -- 172.21.15.19:0/3642841914 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fd850045020 con 0x7fd84c0039b0 2024-08-31T12:18:13.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.822+0000 7fd8597fa640 1 --2- 172.21.15.19:0/3642841914 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd840077a30 0x7fd840079f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:13.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.822+0000 7fd8597fa640 1 -- 172.21.15.19:0/3642841914 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fd8500b9620 con 0x7fd84c0039b0 2024-08-31T12:18:13.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.822+0000 7fd85bfff640 1 --2- 172.21.15.19:0/3642841914 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd840077a30 0x7fd840079f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:13.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.823+0000 7fd85bfff640 1 --2- 172.21.15.19:0/3642841914 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd840077a30 0x7fd840079f40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd85c158ac0 tx=0x7fd848005ea0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:13.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:13.828+0000 7fd8597fa640 1 -- 172.21.15.19:0/3642841914 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fd85c1f20a0 con 0x7fd84c0039b0 2024-08-31T12:18:14.104 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:14.102+0000 7fd862d00640 1 -- 172.21.15.19:0/3642841914 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd85c158b40 con 0x7fd84c0039b0 2024-08-31T12:18:14.104 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:14.103+0000 7fd8597fa640 1 -- 172.21.15.19:0/3642841914 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fd85c0cb690 con 0x7fd84c0039b0 2024-08-31T12:18:14.105 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:18:14.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:14.105+0000 7fd862d00640 1 -- 172.21.15.19:0/3642841914 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd840077a30 msgr2=0x7fd840079f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:14.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:14.105+0000 7fd862d00640 1 --2- 172.21.15.19:0/3642841914 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd840077a30 0x7fd840079f40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd85c158ac0 tx=0x7fd848005ea0 comp rx=0 tx=0).stop 2024-08-31T12:18:14.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:14.105+0000 7fd862d00640 1 -- 172.21.15.19:0/3642841914 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd84c0039b0 msgr2=0x7fd85c15dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:14.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:14.105+0000 7fd862d00640 1 --2- 172.21.15.19:0/3642841914 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd84c0039b0 0x7fd85c15dde0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd85002b330 tx=0x7fd850006620 comp rx=0 tx=0).stop 2024-08-31T12:18:14.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:14.105+0000 7fd862d00640 1 -- 172.21.15.19:0/3642841914 shutdown_connections 2024-08-31T12:18:14.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:14.105+0000 7fd862d00640 1 --2- 172.21.15.19:0/3642841914 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd840077a30 0x7fd840079f40 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:14.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:14.105+0000 7fd862d00640 1 --2- 172.21.15.19:0/3642841914 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd85c1501a0 0x7fd85c1505b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:14.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:14.105+0000 7fd862d00640 1 --2- 172.21.15.19:0/3642841914 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd85c1f4c10 0x7fd85c0c5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:14.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:14.105+0000 7fd862d00640 1 --2- 172.21.15.19:0/3642841914 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd84c0039b0 0x7fd85c15dde0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:14.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:14.105+0000 7fd862d00640 1 -- 172.21.15.19:0/3642841914 >> 172.21.15.19:0/3642841914 conn(0x7fd85c074810 msgr2=0x7fd85c0c6680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:18:14.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:14.105+0000 7fd862d00640 1 -- 172.21.15.19:0/3642841914 shutdown_connections 2024-08-31T12:18:14.108 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:14.105+0000 7fd862d00640 1 -- 172.21.15.19:0/3642841914 wait complete. 2024-08-31T12:18:17.118 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:18:17.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.245+0000 7fcee17ad640 1 -- 172.21.15.19:0/3905987059 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcedc0caa20 msgr2=0x7fcedc164740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:17.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.245+0000 7fcee17ad640 1 --2- 172.21.15.19:0/3905987059 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcedc0caa20 0x7fcedc164740 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fced4005fd0 tx=0x7fced402ab20 comp rx=0 tx=0).stop 2024-08-31T12:18:17.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.245+0000 7fcee17ad640 1 -- 172.21.15.19:0/3905987059 shutdown_connections 2024-08-31T12:18:17.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.245+0000 7fcee17ad640 1 --2- 172.21.15.19:0/3905987059 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcedc0caa20 0x7fcedc164740 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:17.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.245+0000 7fcee17ad640 1 -- 172.21.15.19:0/3905987059 >> 172.21.15.19:0/3905987059 conn(0x7fcedc072c80 msgr2=0x7fcedc073090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:18:17.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.245+0000 7fcee17ad640 1 -- 172.21.15.19:0/3905987059 shutdown_connections 2024-08-31T12:18:17.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.245+0000 7fcee17ad640 1 -- 172.21.15.19:0/3905987059 wait complete. 2024-08-31T12:18:17.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.247+0000 7fcee17ad640 1 Processor -- start 2024-08-31T12:18:17.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.247+0000 7fcee17ad640 1 -- start start 2024-08-31T12:18:17.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.248+0000 7fcee17ad640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcedc071b40 0x7fcedc156f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:17.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.248+0000 7fcee17ad640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcedc0caa20 0x7fcedc0cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:17.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.248+0000 7fcee17ad640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcedc153230 0x7fcedc0b35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:17.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.248+0000 7fcee17ad640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fcedc0721e0 con 0x7fcedc0caa20 2024-08-31T12:18:17.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.248+0000 7fcee17ad640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fcedc0cae30 con 0x7fcedc071b40 2024-08-31T12:18:17.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.248+0000 7fcee17ad640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fcedc072360 con 0x7fcedc153230 2024-08-31T12:18:17.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.248+0000 7fced3fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcedc0caa20 0x7fcedc0cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:17.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.248+0000 7fcedaffd640 1 -- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcedc071b40 msgr2=0x7fcedc156f90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3300/0 2024-08-31T12:18:17.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.248+0000 7fcedaffd640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcedc071b40 0x7fcedc156f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:18:17.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.248+0000 7fced3fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcedc0caa20 0x7fcedc0cedb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:34716/0 (socket says 172.21.15.19:34716) 2024-08-31T12:18:17.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.248+0000 7fced3fff640 1 -- 172.21.15.19:0/4018468961 learned_addr learned my addr 172.21.15.19:0/4018468961 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:18:17.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.248+0000 7fcedb7fe640 1 --2- 172.21.15.19:0/4018468961 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcedc153230 0x7fcedc0b35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:17.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.249+0000 7fced3fff640 1 -- 172.21.15.19:0/4018468961 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcedc153230 msgr2=0x7fcedc0b35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:17.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.249+0000 7fced3fff640 1 --2- 172.21.15.19:0/4018468961 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcedc153230 0x7fcedc0b35a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:17.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.249+0000 7fced3fff640 1 -- 172.21.15.19:0/4018468961 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcedc071b40 msgr2=0x7fcedc156f90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:18:17.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.249+0000 7fced3fff640 1 --2- 172.21.15.19:0/4018468961 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcedc071b40 0x7fcedc156f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:17.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.249+0000 7fced3fff640 1 -- 172.21.15.19:0/4018468961 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcedc1574d0 con 0x7fcedc0caa20 2024-08-31T12:18:17.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.249+0000 7fced3fff640 1 --2- 172.21.15.19:0/4018468961 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcedc0caa20 0x7fcedc0cedb0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcec400c370 tx=0x7fcec400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:17.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.249+0000 7fced37fe640 1 -- 172.21.15.19:0/4018468961 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcec4017070 con 0x7fcedc0caa20 2024-08-31T12:18:17.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.249+0000 7fced37fe640 1 -- 172.21.15.19:0/4018468961 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcec40053c0 con 0x7fcedc0caa20 2024-08-31T12:18:17.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.249+0000 7fcee17ad640 1 -- 172.21.15.19:0/4018468961 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcedc1581f0 con 0x7fcedc0caa20 2024-08-31T12:18:17.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.249+0000 7fced37fe640 1 -- 172.21.15.19:0/4018468961 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcec4003850 con 0x7fcedc0caa20 2024-08-31T12:18:17.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.249+0000 7fcee17ad640 1 -- 172.21.15.19:0/4018468961 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcedc1587b0 con 0x7fcedc0caa20 2024-08-31T12:18:17.256 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.250+0000 7fcee17ad640 1 -- 172.21.15.19:0/4018468961 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcedc070ad0 con 0x7fcedc0caa20 2024-08-31T12:18:17.256 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.251+0000 7fced37fe640 1 -- 172.21.15.19:0/4018468961 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fcec401a070 con 0x7fcedc0caa20 2024-08-31T12:18:17.256 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.252+0000 7fced37fe640 1 --2- 172.21.15.19:0/4018468961 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fcebc077a30 0x7fcebc079f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:17.256 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.252+0000 7fced37fe640 1 -- 172.21.15.19:0/4018468961 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fcec4098990 con 0x7fcedc0caa20 2024-08-31T12:18:17.256 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.252+0000 7fcedaffd640 1 --2- 172.21.15.19:0/4018468961 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fcebc077a30 0x7fcebc079f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:17.256 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.253+0000 7fcedaffd640 1 --2- 172.21.15.19:0/4018468961 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fcebc077a30 0x7fcebc079f40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fcecc006fd0 tx=0x7fcecc006d60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:17.259 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.258+0000 7fced37fe640 1 -- 172.21.15.19:0/4018468961 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fcec4061710 con 0x7fcedc0caa20 2024-08-31T12:18:17.549 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.547+0000 7fcee17ad640 1 -- 172.21.15.19:0/4018468961 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fcedc0ca110 con 0x7fcedc0caa20 2024-08-31T12:18:17.550 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.548+0000 7fced37fe640 1 -- 172.21.15.19:0/4018468961 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fcec4061010 con 0x7fcedc0caa20 2024-08-31T12:18:17.550 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":19,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:18:17.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.550+0000 7fcee17ad640 1 -- 172.21.15.19:0/4018468961 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fcebc077a30 msgr2=0x7fcebc079f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:17.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.550+0000 7fcee17ad640 1 --2- 172.21.15.19:0/4018468961 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fcebc077a30 0x7fcebc079f40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fcecc006fd0 tx=0x7fcecc006d60 comp rx=0 tx=0).stop 2024-08-31T12:18:17.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.550+0000 7fcee17ad640 1 -- 172.21.15.19:0/4018468961 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcedc0caa20 msgr2=0x7fcedc0cedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:17.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.550+0000 7fcee17ad640 1 --2- 172.21.15.19:0/4018468961 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcedc0caa20 0x7fcedc0cedb0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcec400c370 tx=0x7fcec400c830 comp rx=0 tx=0).stop 2024-08-31T12:18:17.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.550+0000 7fcee17ad640 1 -- 172.21.15.19:0/4018468961 shutdown_connections 2024-08-31T12:18:17.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.550+0000 7fcee17ad640 1 --2- 172.21.15.19:0/4018468961 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fcebc077a30 0x7fcebc079f40 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:17.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.550+0000 7fcee17ad640 1 --2- 172.21.15.19:0/4018468961 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcedc153230 0x7fcedc0b35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:17.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.550+0000 7fcee17ad640 1 --2- 172.21.15.19:0/4018468961 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcedc0caa20 0x7fcedc0cedb0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:17.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.550+0000 7fcee17ad640 1 --2- 172.21.15.19:0/4018468961 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcedc071b40 0x7fcedc156f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:17.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.550+0000 7fcee17ad640 1 -- 172.21.15.19:0/4018468961 >> 172.21.15.19:0/4018468961 conn(0x7fcedc072c80 msgr2=0x7fcedc0c46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:18:17.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.550+0000 7fcee17ad640 1 -- 172.21.15.19:0/4018468961 shutdown_connections 2024-08-31T12:18:17.554 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:17.550+0000 7fcee17ad640 1 -- 172.21.15.19:0/4018468961 wait complete. 2024-08-31T12:18:20.564 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:18:20.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.681+0000 7fe32f6de640 1 -- 172.21.15.19:0/3859204313 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe3280c7ac0 msgr2=0x7fe328152040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:20.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.681+0000 7fe32f6de640 1 --2- 172.21.15.19:0/3859204313 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe3280c7ac0 0x7fe328152040 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe3240098e0 tx=0x7fe32402a5d0 comp rx=0 tx=0).stop 2024-08-31T12:18:20.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.681+0000 7fe32f6de640 1 -- 172.21.15.19:0/3859204313 shutdown_connections 2024-08-31T12:18:20.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.681+0000 7fe32f6de640 1 --2- 172.21.15.19:0/3859204313 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe3280c89a0 0x7fe3280c8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:20.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.681+0000 7fe32f6de640 1 --2- 172.21.15.19:0/3859204313 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe3280c7ac0 0x7fe328152040 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:20.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.681+0000 7fe32f6de640 1 -- 172.21.15.19:0/3859204313 >> 172.21.15.19:0/3859204313 conn(0x7fe32806cae0 msgr2=0x7fe32806cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:18:20.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.682+0000 7fe32f6de640 1 -- 172.21.15.19:0/3859204313 shutdown_connections 2024-08-31T12:18:20.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.682+0000 7fe32f6de640 1 -- 172.21.15.19:0/3859204313 wait complete. 2024-08-31T12:18:20.685 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.684+0000 7fe32f6de640 1 Processor -- start 2024-08-31T12:18:20.685 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.684+0000 7fe32f6de640 1 -- start start 2024-08-31T12:18:20.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.685+0000 7fe32f6de640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe3280b28d0 0x7fe3280745b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:20.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.685+0000 7fe32f6de640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe3280c7ac0 0x7fe328153a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:20.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.685+0000 7fe32f6de640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe3280c89a0 0x7fe3281f5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:20.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.685+0000 7fe32f6de640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fe3280c92f0 con 0x7fe3280b28d0 2024-08-31T12:18:20.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.685+0000 7fe32f6de640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe328068fd0 con 0x7fe3280c7ac0 2024-08-31T12:18:20.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.685+0000 7fe32f6de640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fe32806da00 con 0x7fe3280c89a0 2024-08-31T12:18:20.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.685+0000 7fe32d47a640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe3280b28d0 0x7fe3280745b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:20.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.685+0000 7fe32d47a640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe3280b28d0 0x7fe3280745b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:55472/0 (socket says 172.21.15.19:55472) 2024-08-31T12:18:20.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.685+0000 7fe32d47a640 1 -- 172.21.15.19:0/2161972775 learned_addr learned my addr 172.21.15.19:0/2161972775 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:18:20.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.685+0000 7fe32cc79640 1 -- 172.21.15.19:0/2161972775 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe3280c7ac0 msgr2=0x7fe328153a80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3300/0 2024-08-31T12:18:20.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.685+0000 7fe32cc79640 1 --2- 172.21.15.19:0/2161972775 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe3280c7ac0 0x7fe328153a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:18:20.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.685+0000 7fe32dc7b640 1 --2- 172.21.15.19:0/2161972775 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe3280c89a0 0x7fe3281f5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:20.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.685+0000 7fe32d47a640 1 -- 172.21.15.19:0/2161972775 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe3280c89a0 msgr2=0x7fe3281f5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:20.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.685+0000 7fe32d47a640 1 --2- 172.21.15.19:0/2161972775 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe3280c89a0 0x7fe3281f5890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:20.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.685+0000 7fe32d47a640 1 -- 172.21.15.19:0/2161972775 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe3280c7ac0 msgr2=0x7fe328153a80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:18:20.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.685+0000 7fe32d47a640 1 --2- 172.21.15.19:0/2161972775 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe3280c7ac0 0x7fe328153a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:20.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.685+0000 7fe32d47a640 1 -- 172.21.15.19:0/2161972775 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3281516a0 con 0x7fe3280b28d0 2024-08-31T12:18:20.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.686+0000 7fe32d47a640 1 --2- 172.21.15.19:0/2161972775 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe3280b28d0 0x7fe3280745b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe324009a00 tx=0x7fe3240316b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:20.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.686+0000 7fe31dffb640 1 -- 172.21.15.19:0/2161972775 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe32403f070 con 0x7fe3280b28d0 2024-08-31T12:18:20.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.686+0000 7fe31dffb640 1 -- 172.21.15.19:0/2161972775 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe324031950 con 0x7fe3280b28d0 2024-08-31T12:18:20.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.686+0000 7fe31dffb640 1 -- 172.21.15.19:0/2161972775 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe324031da0 con 0x7fe3280b28d0 2024-08-31T12:18:20.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.686+0000 7fe32f6de640 1 -- 172.21.15.19:0/2161972775 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe328154580 con 0x7fe3280b28d0 2024-08-31T12:18:20.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.686+0000 7fe32f6de640 1 -- 172.21.15.19:0/2161972775 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe32814c980 con 0x7fe3280b28d0 2024-08-31T12:18:20.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.687+0000 7fe32f6de640 1 -- 172.21.15.19:0/2161972775 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3280c3e90 con 0x7fe3280b28d0 2024-08-31T12:18:20.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.688+0000 7fe31dffb640 1 -- 172.21.15.19:0/2161972775 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fe324042070 con 0x7fe3280b28d0 2024-08-31T12:18:20.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.688+0000 7fe31dffb640 1 --2- 172.21.15.19:0/2161972775 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe30c077a30 0x7fe30c079f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:20.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.689+0000 7fe31dffb640 1 -- 172.21.15.19:0/2161972775 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fe3240ba300 con 0x7fe3280b28d0 2024-08-31T12:18:20.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.689+0000 7fe32cc79640 1 --2- 172.21.15.19:0/2161972775 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe30c077a30 0x7fe30c079f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:20.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.689+0000 7fe32cc79640 1 --2- 172.21.15.19:0/2161972775 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe30c077a30 0x7fe30c079f40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe318000c00 tx=0x7fe3180079f0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:20.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.693+0000 7fe31dffb640 1 -- 172.21.15.19:0/2161972775 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fe324082f60 con 0x7fe3280b28d0 2024-08-31T12:18:20.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.994+0000 7fe32f6de640 1 -- 172.21.15.19:0/2161972775 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe328152330 con 0x7fe3280b28d0 2024-08-31T12:18:20.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.994+0000 7fe31dffb640 1 -- 172.21.15.19:0/2161972775 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fe324082860 con 0x7fe3280b28d0 2024-08-31T12:18:20.997 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:18:20.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.997+0000 7fe32f6de640 1 -- 172.21.15.19:0/2161972775 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe30c077a30 msgr2=0x7fe30c079f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:20.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.997+0000 7fe32f6de640 1 --2- 172.21.15.19:0/2161972775 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe30c077a30 0x7fe30c079f40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe318000c00 tx=0x7fe3180079f0 comp rx=0 tx=0).stop 2024-08-31T12:18:20.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.997+0000 7fe32f6de640 1 -- 172.21.15.19:0/2161972775 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe3280b28d0 msgr2=0x7fe3280745b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:20.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.997+0000 7fe32f6de640 1 --2- 172.21.15.19:0/2161972775 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe3280b28d0 0x7fe3280745b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe324009a00 tx=0x7fe3240316b0 comp rx=0 tx=0).stop 2024-08-31T12:18:20.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.997+0000 7fe32f6de640 1 -- 172.21.15.19:0/2161972775 shutdown_connections 2024-08-31T12:18:20.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.997+0000 7fe32f6de640 1 --2- 172.21.15.19:0/2161972775 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe30c077a30 0x7fe30c079f40 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:20.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.997+0000 7fe32f6de640 1 --2- 172.21.15.19:0/2161972775 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe3280c89a0 0x7fe3281f5890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:20.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.997+0000 7fe32f6de640 1 --2- 172.21.15.19:0/2161972775 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe3280c7ac0 0x7fe328153a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:21.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.997+0000 7fe32f6de640 1 --2- 172.21.15.19:0/2161972775 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe3280b28d0 0x7fe3280745b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:21.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.997+0000 7fe32f6de640 1 -- 172.21.15.19:0/2161972775 >> 172.21.15.19:0/2161972775 conn(0x7fe32806cae0 msgr2=0x7fe328142940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:18:21.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.997+0000 7fe32f6de640 1 -- 172.21.15.19:0/2161972775 shutdown_connections 2024-08-31T12:18:21.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:20.997+0000 7fe32f6de640 1 -- 172.21.15.19:0/2161972775 wait complete. 2024-08-31T12:18:24.012 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:18:25.948 DEBUG:teuthology.orchestra.run.smithi019:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:18:25.951 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:18:28.544 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.542+0000 7fd432924640 1 -- 172.21.15.19:0/136570325 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd4180036a0 msgr2=0x7fd418005b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:28.545 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.542+0000 7fd432924640 1 --2- 172.21.15.19:0/136570325 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd4180036a0 0x7fd418005b40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd41c006b20 tx=0x7fd41c02b510 comp rx=0 tx=0).stop 2024-08-31T12:18:28.545 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.542+0000 7fd430ec1640 1 -- 172.21.15.19:0/136570325 reap_dead start 2024-08-31T12:18:28.545 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.542+0000 7fd432924640 1 -- 172.21.15.19:0/136570325 shutdown_connections 2024-08-31T12:18:28.545 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.542+0000 7fd432924640 1 -- 172.21.15.19:0/136570325 >> 172.21.15.19:0/136570325 conn(0x7fd42c074810 msgr2=0x7fd42c068aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:18:28.545 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.542+0000 7fd432924640 1 -- 172.21.15.19:0/136570325 shutdown_connections 2024-08-31T12:18:28.545 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.542+0000 7fd432924640 1 -- 172.21.15.19:0/136570325 wait complete. 2024-08-31T12:18:28.546 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.544+0000 7fd432924640 1 Processor -- start 2024-08-31T12:18:28.546 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.544+0000 7fd432924640 1 -- start start 2024-08-31T12:18:28.549 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.546+0000 7fd432924640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4180036a0 0x7fd42c0c8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:28.549 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.546+0000 7fd432924640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd42c158010 0x7fd42c155b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:28.549 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.546+0000 7fd432924640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd42c1560b0 0x7fd42c0c9860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:28.550 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.546+0000 7fd432924640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fd42c0cc780 con 0x7fd42c158010 2024-08-31T12:18:28.550 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.546+0000 7fd432924640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd42c15f050 con 0x7fd4180036a0 2024-08-31T12:18:28.550 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.546+0000 7fd432924640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fd42c0cc900 con 0x7fd42c1560b0 2024-08-31T12:18:28.550 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.546+0000 7fd42bfff640 1 -- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4180036a0 msgr2=0x7fd42c0c8480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3300/0 2024-08-31T12:18:28.550 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.546+0000 7fd42bfff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4180036a0 0x7fd42c0c8480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:18:28.550 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.546+0000 7fd42b7fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd42c158010 0x7fd42c155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:28.550 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.546+0000 7fd42b7fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd42c158010 0x7fd42c155b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:39012/0 (socket says 172.21.15.19:39012) 2024-08-31T12:18:28.550 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.546+0000 7fd42b7fe640 1 -- 172.21.15.19:0/4107480715 learned_addr learned my addr 172.21.15.19:0/4107480715 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:18:28.550 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.546+0000 7fd430ec1640 1 --2- 172.21.15.19:0/4107480715 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd42c1560b0 0x7fd42c0c9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:28.550 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.546+0000 7fd42b7fe640 1 -- 172.21.15.19:0/4107480715 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd42c1560b0 msgr2=0x7fd42c0c9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:28.550 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.546+0000 7fd42b7fe640 1 --2- 172.21.15.19:0/4107480715 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd42c1560b0 0x7fd42c0c9860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:28.550 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.546+0000 7fd42b7fe640 1 -- 172.21.15.19:0/4107480715 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4180036a0 msgr2=0x7fd42c0c8480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:18:28.551 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.546+0000 7fd42b7fe640 1 --2- 172.21.15.19:0/4107480715 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4180036a0 0x7fd42c0c8480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:28.551 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.546+0000 7fd42b7fe640 1 -- 172.21.15.19:0/4107480715 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd42c0c89c0 con 0x7fd42c158010 2024-08-31T12:18:28.551 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.547+0000 7fd42b7fe640 1 --2- 172.21.15.19:0/4107480715 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd42c158010 0x7fd42c155b70 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd420001490 tx=0x7fd42000be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:28.551 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.547+0000 7fd428ff9640 1 -- 172.21.15.19:0/4107480715 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd420019070 con 0x7fd42c158010 2024-08-31T12:18:28.551 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.547+0000 7fd428ff9640 1 -- 172.21.15.19:0/4107480715 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd42000cd80 con 0x7fd42c158010 2024-08-31T12:18:28.551 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.547+0000 7fd432924640 1 -- 172.21.15.19:0/4107480715 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd42c14d050 con 0x7fd42c158010 2024-08-31T12:18:28.551 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.547+0000 7fd428ff9640 1 -- 172.21.15.19:0/4107480715 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4200146a0 con 0x7fd42c158010 2024-08-31T12:18:28.551 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.547+0000 7fd432924640 1 -- 172.21.15.19:0/4107480715 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd42c15b240 con 0x7fd42c158010 2024-08-31T12:18:28.552 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.548+0000 7fd432924640 1 -- 172.21.15.19:0/4107480715 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd42c155660 con 0x7fd42c158010 2024-08-31T12:18:28.552 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.549+0000 7fd428ff9640 1 -- 172.21.15.19:0/4107480715 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fd42001c070 con 0x7fd42c158010 2024-08-31T12:18:28.552 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.549+0000 7fd428ff9640 1 --2- 172.21.15.19:0/4107480715 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd4180787a0 0x7fd41807acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:28.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.549+0000 7fd428ff9640 1 -- 172.21.15.19:0/4107480715 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fd420098fa0 con 0x7fd42c158010 2024-08-31T12:18:28.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.550+0000 7fd42bfff640 1 --2- 172.21.15.19:0/4107480715 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd4180787a0 0x7fd41807acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:28.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.550+0000 7fd42bfff640 1 --2- 172.21.15.19:0/4107480715 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd4180787a0 0x7fd41807acb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd41c036000 tx=0x7fd41c001120 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:28.556 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.555+0000 7fd428ff9640 1 -- 172.21.15.19:0/4107480715 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fd420052cd0 con 0x7fd42c158010 2024-08-31T12:18:28.854 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.851+0000 7fd432924640 1 -- 172.21.15.19:0/4107480715 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd42c0c7be0 con 0x7fd42c158010 2024-08-31T12:18:28.854 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.852+0000 7fd428ff9640 1 -- 172.21.15.19:0/4107480715 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7fd42006a1d0 con 0x7fd42c158010 2024-08-31T12:18:28.854 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":24,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:18:28.856 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.854+0000 7fd432924640 1 -- 172.21.15.19:0/4107480715 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd4180787a0 msgr2=0x7fd41807acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:28.856 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.854+0000 7fd432924640 1 --2- 172.21.15.19:0/4107480715 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd4180787a0 0x7fd41807acb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd41c036000 tx=0x7fd41c001120 comp rx=0 tx=0).stop 2024-08-31T12:18:28.856 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.854+0000 7fd432924640 1 -- 172.21.15.19:0/4107480715 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd42c158010 msgr2=0x7fd42c155b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:28.856 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.854+0000 7fd432924640 1 --2- 172.21.15.19:0/4107480715 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd42c158010 0x7fd42c155b70 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd420001490 tx=0x7fd42000be10 comp rx=0 tx=0).stop 2024-08-31T12:18:28.856 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.854+0000 7fd432924640 1 -- 172.21.15.19:0/4107480715 shutdown_connections 2024-08-31T12:18:28.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.854+0000 7fd432924640 1 --2- 172.21.15.19:0/4107480715 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd4180787a0 0x7fd41807acb0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:28.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.854+0000 7fd432924640 1 --2- 172.21.15.19:0/4107480715 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd42c1560b0 0x7fd42c0c9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:28.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.854+0000 7fd432924640 1 --2- 172.21.15.19:0/4107480715 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd42c158010 0x7fd42c155b70 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:28.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.854+0000 7fd432924640 1 --2- 172.21.15.19:0/4107480715 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4180036a0 0x7fd42c0c8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:28.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.854+0000 7fd432924640 1 -- 172.21.15.19:0/4107480715 >> 172.21.15.19:0/4107480715 conn(0x7fd42c074810 msgr2=0x7fd42c1539f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:18:28.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.854+0000 7fd432924640 1 -- 172.21.15.19:0/4107480715 shutdown_connections 2024-08-31T12:18:28.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:28.854+0000 7fd432924640 1 -- 172.21.15.19:0/4107480715 wait complete. 2024-08-31T12:18:28.869 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-08-31T12:18:28.870 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-08-31T12:18:28.870 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-31T12:18:29.002 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.000+0000 7fe872cc5640 1 -- 172.21.15.19:0/795925345 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe86c14e700 msgr2=0x7fe86c06dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:29.002 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.000+0000 7fe872cc5640 1 --2- 172.21.15.19:0/795925345 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe86c14e700 0x7fe86c06dbf0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe86c0c6860 tx=0x7fe86002a580 comp rx=0 tx=0).stop 2024-08-31T12:18:29.002 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.000+0000 7fe872cc5640 1 -- 172.21.15.19:0/795925345 shutdown_connections 2024-08-31T12:18:29.002 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.000+0000 7fe872cc5640 1 --2- 172.21.15.19:0/795925345 >> v2:172.21.15.43:3300/0 conn(0x7fe86c06e130 0x7fe86c15b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:29.003 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.000+0000 7fe872cc5640 1 --2- 172.21.15.19:0/795925345 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe86c14e700 0x7fe86c06dbf0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:29.003 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.000+0000 7fe872cc5640 1 -- 172.21.15.19:0/795925345 >> 172.21.15.19:0/795925345 conn(0x7fe86c0cda10 msgr2=0x7fe86c0c9e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:18:29.003 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.000+0000 7fe872cc5640 1 -- 172.21.15.19:0/795925345 shutdown_connections 2024-08-31T12:18:29.003 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.000+0000 7fe872cc5640 1 -- 172.21.15.19:0/795925345 wait complete. 2024-08-31T12:18:29.003 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.002+0000 7fe872cc5640 1 Processor -- start 2024-08-31T12:18:29.004 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.002+0000 7fe872cc5640 1 -- start start 2024-08-31T12:18:29.006 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.003+0000 7fe872cc5640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe86c06e130 0x7fe86c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:29.006 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.003+0000 7fe872cc5640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe86c14e700 0x7fe86c149a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:29.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.003+0000 7fe872cc5640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe86c15b7d0 0x7fe86c148a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:29.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.003+0000 7fe872cc5640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fe86c158970 con 0x7fe86c15b7d0 2024-08-31T12:18:29.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.003+0000 7fe872cc5640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe86c15bbe0 con 0x7fe86c06e130 2024-08-31T12:18:29.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.003+0000 7fe872cc5640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fe86c0c49c0 con 0x7fe86c14e700 2024-08-31T12:18:29.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.004+0000 7fe870a61640 1 -- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe86c06e130 msgr2=0x7fe86c06d810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3300/0 2024-08-31T12:18:29.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.004+0000 7fe871262640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe86c15b7d0 0x7fe86c148a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:29.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.004+0000 7fe870a61640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe86c06e130 0x7fe86c06d810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:18:29.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.004+0000 7fe871262640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe86c15b7d0 0x7fe86c148a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:39028/0 (socket says 172.21.15.19:39028) 2024-08-31T12:18:29.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.004+0000 7fe871262640 1 -- 172.21.15.19:0/2077703139 learned_addr learned my addr 172.21.15.19:0/2077703139 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:18:29.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.004+0000 7fe86bfff640 1 --2- 172.21.15.19:0/2077703139 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe86c14e700 0x7fe86c149a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:29.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.004+0000 7fe871262640 1 -- 172.21.15.19:0/2077703139 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe86c14e700 msgr2=0x7fe86c149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:29.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.004+0000 7fe871262640 1 --2- 172.21.15.19:0/2077703139 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe86c14e700 0x7fe86c149a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:29.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.004+0000 7fe871262640 1 -- 172.21.15.19:0/2077703139 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe86c06e130 msgr2=0x7fe86c06d810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:18:29.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.004+0000 7fe871262640 1 --2- 172.21.15.19:0/2077703139 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe86c06e130 0x7fe86c06d810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:29.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.004+0000 7fe871262640 1 -- 172.21.15.19:0/2077703139 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe86c1effd0 con 0x7fe86c15b7d0 2024-08-31T12:18:29.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.004+0000 7fe871262640 1 --2- 172.21.15.19:0/2077703139 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe86c15b7d0 0x7fe86c148a10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe86400c700 tx=0x7fe86400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:29.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.004+0000 7fe8697fa640 1 -- 172.21.15.19:0/2077703139 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe864003b70 con 0x7fe86c15b7d0 2024-08-31T12:18:29.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.004+0000 7fe8697fa640 1 -- 172.21.15.19:0/2077703139 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe864003d10 con 0x7fe86c15b7d0 2024-08-31T12:18:29.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.004+0000 7fe872cc5640 1 -- 172.21.15.19:0/2077703139 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe86c06cd30 con 0x7fe86c15b7d0 2024-08-31T12:18:29.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.004+0000 7fe8697fa640 1 -- 172.21.15.19:0/2077703139 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86400aab0 con 0x7fe86c15b7d0 2024-08-31T12:18:29.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.004+0000 7fe872cc5640 1 -- 172.21.15.19:0/2077703139 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe86c06d2f0 con 0x7fe86c15b7d0 2024-08-31T12:18:29.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.005+0000 7fe872cc5640 1 --2- 172.21.15.19:0/2077703139 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe86c1e3000 0x7fe86c147300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:29.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.006+0000 7fe872cc5640 1 -- 172.21.15.19:0/2077703139 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fe86c0ca7e0 con 0x7fe86c1e3000 2024-08-31T12:18:29.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.006+0000 7fe870a61640 1 --2- 172.21.15.19:0/2077703139 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe86c1e3000 0x7fe86c147300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:29.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.006+0000 7fe870a61640 1 -- 172.21.15.19:0/2077703139 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe86c1efcf0 con 0x7fe86c1e3000 2024-08-31T12:18:29.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.006+0000 7fe870a61640 1 --2- 172.21.15.19:0/2077703139 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe86c1e3000 0x7fe86c147300 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe86c0c8250 tx=0x7fe860005a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:29.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.006+0000 7fe8697fa640 1 -- 172.21.15.19:0/2077703139 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe860035040 con 0x7fe86c1e3000 2024-08-31T12:18:29.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.007+0000 7fe8697fa640 1 -- 172.21.15.19:0/2077703139 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fe86400ac50 con 0x7fe86c15b7d0 2024-08-31T12:18:29.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.007+0000 7fe8697fa640 1 --2- 172.21.15.19:0/2077703139 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe850077950 0x7fe850079e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:29.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.007+0000 7fe8697fa640 1 -- 172.21.15.19:0/2077703139 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fe864099540 con 0x7fe86c15b7d0 2024-08-31T12:18:29.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.008+0000 7fe86bfff640 1 --2- 172.21.15.19:0/2077703139 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe850077950 0x7fe850079e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:29.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.008+0000 7fe8697fa640 1 -- 172.21.15.19:0/2077703139 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe860004220 con 0x7fe86c1e3000 2024-08-31T12:18:29.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.008+0000 7fe8697fa640 1 -- 172.21.15.19:0/2077703139 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe86c1e3000 msgr2=0x7fe86c147300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:29.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.008+0000 7fe8697fa640 1 --2- 172.21.15.19:0/2077703139 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe86c1e3000 0x7fe86c147300 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe86c0c8250 tx=0x7fe860005a60 comp rx=0 tx=0).stop 2024-08-31T12:18:29.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.008+0000 7fe8697fa640 1 -- 172.21.15.19:0/2077703139 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe86c1e3000 msgr2=0x7fe86c147300 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:18:29.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.008+0000 7fe8697fa640 1 --2- 172.21.15.19:0/2077703139 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe86c1e3000 0x7fe86c147300 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:29.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.008+0000 7fe86bfff640 1 --2- 172.21.15.19:0/2077703139 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe850077950 0x7fe850079e60 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe858006fd0 tx=0x7fe858008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:29.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.017+0000 7fe872cc5640 1 --2- 172.21.15.19:0/2077703139 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe86c15b070 0x7fe86c147840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:29.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.017+0000 7fe872cc5640 1 -- 172.21.15.19:0/2077703139 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fe86c0745f0 con 0x7fe86c15b070 2024-08-31T12:18:29.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.017+0000 7fe870a61640 1 --2- 172.21.15.19:0/2077703139 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe86c15b070 0x7fe86c147840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:29.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.018+0000 7fe870a61640 1 -- 172.21.15.19:0/2077703139 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe86c06cae0 con 0x7fe86c15b070 2024-08-31T12:18:29.020 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.018+0000 7fe870a61640 1 --2- 172.21.15.19:0/2077703139 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe86c15b070 0x7fe86c147840 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe860009230 tx=0x7fe8600038f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:29.020 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.018+0000 7fe8697fa640 1 -- 172.21.15.19:0/2077703139 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe860003c80 con 0x7fe86c15b070 2024-08-31T12:18:29.021 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.018+0000 7fe8697fa640 1 -- 172.21.15.19:0/2077703139 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5032 (secure 0 0 0) 0x7fe860003c80 con 0x7fe86c15b070 2024-08-31T12:18:29.021 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.018+0000 7fe8697fa640 1 -- 172.21.15.19:0/2077703139 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe86c15b070 msgr2=0x7fe86c147840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:29.021 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.018+0000 7fe8697fa640 1 --2- 172.21.15.19:0/2077703139 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe86c15b070 0x7fe86c147840 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe860009230 tx=0x7fe8600038f0 comp rx=0 tx=0).stop 2024-08-31T12:18:29.021 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.018+0000 7fe8697fa640 1 -- 172.21.15.19:0/2077703139 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe86c15b070 msgr2=0x7fe86c147840 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:18:29.021 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.018+0000 7fe8697fa640 1 --2- 172.21.15.19:0/2077703139 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe86c15b070 0x7fe86c147840 secure :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe860009230 tx=0x7fe8600038f0 comp rx=0 tx=0).stop 2024-08-31T12:18:29.021 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:18:29.021 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:18:29.021 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:18:29.021 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "leader", 2024-08-31T12:18:29.022 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 24, 2024-08-31T12:18:29.022 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 89010, 2024-08-31T12:18:29.022 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:18:29.022 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:18:29.022 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:18:29.022 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:29.022 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 1, 2024-08-31T12:18:29.022 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:18:29.022 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:18:29.022 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:18:29.022 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:18:29.022 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:18:29.022 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:18:29.023 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:18:29.023 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:18:29.023 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:18:29.023 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:18:29.023 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:18:29.023 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:18:29.023 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:18:29.023 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:18:29.023 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:29.023 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:18:29.023 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:18:29.023 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:18:29.024 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:18:29.024 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:18:29.024 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:18:29.024 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:18:29.024 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:18:29.024 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:18:29.024 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:18:29.024 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:18:29.024 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:18:29.024 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:18:29.024 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:29.024 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:29.025 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:18:29.025 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:18:29.025 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:18:29.025 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:18:29.025 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:18:29.025 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:18:29.025 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:18:29.025 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:18:29.025 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:18:29.025 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:18:29.025 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:18:29.025 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:18:29.025 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:18:29.026 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:18:29.026 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:18:29.026 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:18:29.026 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:18:29.026 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:18:29.026 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:18:29.026 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:18:29.026 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:18:29.026 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:18:29.026 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:18:29.026 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:18:29.026 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:18:29.027 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:18:29.027 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:18:29.027 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:18:29.027 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:29.027 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:18:29.027 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:29.027 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:18:29.027 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:29.027 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:18:29.027 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:18:29.027 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:18:29.027 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:18:29.027 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:29.028 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:18:29.028 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:18:29.028 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:29.028 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:29.028 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:29.028 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:18:29.028 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:18:29.028 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:29.028 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:29.028 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:29.028 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:29.028 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:18:29.029 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:18:29.029 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:18:29.029 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:18:29.029 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:18:29.029 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:29.029 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:29.029 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:18:29.029 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:18:29.029 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:18:29.029 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:18:29.030 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:29.030 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:18:29.030 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:18:29.030 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:29.030 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:29.030 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:29.030 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:18:29.030 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:18:29.030 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:29.030 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:29.030 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:29.030 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:29.030 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:18:29.030 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:18:29.031 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:18:29.031 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:18:29.031 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:18:29.031 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:29.031 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:29.031 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:18:29.031 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:18:29.031 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:18:29.031 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:18:29.031 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:29.031 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:18:29.031 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:18:29.032 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:29.032 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:29.032 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:29.032 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:18:29.032 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:18:29.032 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:29.032 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:29.032 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:29.032 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:29.032 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:18:29.032 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:18:29.032 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:18:29.033 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:18:29.033 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:18:29.033 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:29.033 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:29.033 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:29.033 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:18:29.033 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:18:29.033 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:29.033 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:18:29.033 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:18:29.033 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:18:29.033 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:29.034 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:29.034 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:18:29.034 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:29.034 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:18:29.034 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:18:29.034 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 5 2024-08-31T12:18:29.034 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:29.034 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:29.034 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:18:29.034 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:29.034 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:18:29.034 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:18:29.034 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 8 2024-08-31T12:18:29.034 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:29.035 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:29.035 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:18:29.035 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:29.035 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:18:29.035 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:18:29.035 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 6 2024-08-31T12:18:29.035 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:29.035 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:29.035 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:18:29.035 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:29.035 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:18:29.035 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:18:29.035 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:18:29.035 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:29.036 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:29.036 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:29.036 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:18:29.036 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:18:29.038 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.019+0000 7fe872cc5640 1 -- 172.21.15.19:0/2077703139 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe850077950 msgr2=0x7fe850079e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:29.038 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.019+0000 7fe872cc5640 1 --2- 172.21.15.19:0/2077703139 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe850077950 0x7fe850079e60 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe858006fd0 tx=0x7fe858008040 comp rx=0 tx=0).stop 2024-08-31T12:18:29.038 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.019+0000 7fe872cc5640 1 -- 172.21.15.19:0/2077703139 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe86c15b7d0 msgr2=0x7fe86c148a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:29.038 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.019+0000 7fe872cc5640 1 --2- 172.21.15.19:0/2077703139 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe86c15b7d0 0x7fe86c148a10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe86400c700 tx=0x7fe86400cbc0 comp rx=0 tx=0).stop 2024-08-31T12:18:29.038 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.019+0000 7fe871262640 1 -- 172.21.15.19:0/2077703139 reap_dead start 2024-08-31T12:18:29.038 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.019+0000 7fe871262640 1 -- 172.21.15.19:0/2077703139 reap_dead start 2024-08-31T12:18:29.038 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.019+0000 7fe872cc5640 1 -- 172.21.15.19:0/2077703139 shutdown_connections 2024-08-31T12:18:29.038 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.019+0000 7fe872cc5640 1 -- 172.21.15.19:0/2077703139 >> 172.21.15.19:0/2077703139 conn(0x7fe86c0cda10 msgr2=0x7fe86c0c9d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:18:29.038 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.019+0000 7fe872cc5640 1 -- 172.21.15.19:0/2077703139 shutdown_connections 2024-08-31T12:18:29.038 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.019+0000 7fe872cc5640 1 -- 172.21.15.19:0/2077703139 wait complete. 2024-08-31T12:18:29.039 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-08-31T12:18:29.039 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-08-31T12:18:29.039 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-31T12:18:29.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.164+0000 7f34d1e8d640 1 -- 172.21.15.19:0/3356578775 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f34cc0cb2f0 msgr2=0x7f34cc0cc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:29.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.164+0000 7f34d1e8d640 1 --2- 172.21.15.19:0/3356578775 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f34cc0cb2f0 0x7f34cc0cc4d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f34cc0ce650 tx=0x7f34c402aac0 comp rx=0 tx=0).stop 2024-08-31T12:18:29.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.165+0000 7f34d1e8d640 1 -- 172.21.15.19:0/3356578775 shutdown_connections 2024-08-31T12:18:29.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.165+0000 7f34d1e8d640 1 --2- 172.21.15.19:0/3356578775 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f34cc0cb2f0 0x7f34cc0cc4d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:29.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.165+0000 7f34d1e8d640 1 --2- 172.21.15.19:0/3356578775 >> v2:172.21.15.43:3300/0 conn(0x7f34cc0c5c20 0x7f34cc0cbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:29.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.165+0000 7f34d1e8d640 1 -- 172.21.15.19:0/3356578775 >> 172.21.15.19:0/3356578775 conn(0x7f34cc074a90 msgr2=0x7f34cc068aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:18:29.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.165+0000 7f34d1e8d640 1 -- 172.21.15.19:0/3356578775 shutdown_connections 2024-08-31T12:18:29.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.165+0000 7f34d1e8d640 1 -- 172.21.15.19:0/3356578775 wait complete. 2024-08-31T12:18:29.168 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.166+0000 7f34d1e8d640 1 Processor -- start 2024-08-31T12:18:29.168 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.167+0000 7f34d1e8d640 1 -- start start 2024-08-31T12:18:29.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.168+0000 7f34d1e8d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f34cc0c5c20 0x7f34cc0c8130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:29.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.168+0000 7f34d1e8d640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34cc0cb2f0 0x7f34cc14ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:29.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.168+0000 7f34d1e8d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f34cc155520 0x7f34cc150450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:29.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.168+0000 7f34d1e8d640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f34cc0d1140 con 0x7f34cc155520 2024-08-31T12:18:29.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.168+0000 7f34d1e8d640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f34cc0c6750 con 0x7f34cc0cb2f0 2024-08-31T12:18:29.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.168+0000 7f34d1e8d640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f34cc15b730 con 0x7f34cc0c5c20 2024-08-31T12:18:29.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.168+0000 7f34cbfff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f34cc155520 0x7f34cc150450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:29.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.168+0000 7f34caffd640 1 -- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34cc0cb2f0 msgr2=0x7f34cc14ff10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3300/0 2024-08-31T12:18:29.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.168+0000 7f34caffd640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34cc0cb2f0 0x7f34cc14ff10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:18:29.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.168+0000 7f34cbfff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f34cc155520 0x7f34cc150450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:39072/0 (socket says 172.21.15.19:39072) 2024-08-31T12:18:29.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.168+0000 7f34cbfff640 1 -- 172.21.15.19:0/2530617135 learned_addr learned my addr 172.21.15.19:0/2530617135 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:18:29.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.168+0000 7f34cb7fe640 1 --2- 172.21.15.19:0/2530617135 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f34cc0c5c20 0x7f34cc0c8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:29.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.168+0000 7f34cbfff640 1 -- 172.21.15.19:0/2530617135 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f34cc0c5c20 msgr2=0x7f34cc0c8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:29.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.168+0000 7f34cbfff640 1 --2- 172.21.15.19:0/2530617135 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f34cc0c5c20 0x7f34cc0c8130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:29.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.168+0000 7f34cbfff640 1 -- 172.21.15.19:0/2530617135 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34cc0cb2f0 msgr2=0x7f34cc14ff10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:18:29.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.168+0000 7f34cbfff640 1 --2- 172.21.15.19:0/2530617135 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34cc0cb2f0 0x7f34cc14ff10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:29.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.168+0000 7f34cbfff640 1 -- 172.21.15.19:0/2530617135 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34cc0c8670 con 0x7f34cc155520 2024-08-31T12:18:29.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.168+0000 7f34cb7fe640 1 --2- 172.21.15.19:0/2530617135 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f34cc0c5c20 0x7f34cc0c8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:18:29.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.168+0000 7f34cbfff640 1 --2- 172.21.15.19:0/2530617135 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f34cc155520 0x7f34cc150450 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f34c4009d60 tx=0x7f34c4003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:29.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.169+0000 7f34b3fff640 1 -- 172.21.15.19:0/2530617135 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c4044070 con 0x7f34cc155520 2024-08-31T12:18:29.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.169+0000 7f34b3fff640 1 -- 172.21.15.19:0/2530617135 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f34c4004250 con 0x7f34cc155520 2024-08-31T12:18:29.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.169+0000 7f34d1e8d640 1 -- 172.21.15.19:0/2530617135 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34cc0ca910 con 0x7f34cc155520 2024-08-31T12:18:29.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.169+0000 7f34b3fff640 1 -- 172.21.15.19:0/2530617135 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c40046a0 con 0x7f34cc155520 2024-08-31T12:18:29.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.169+0000 7f34d1e8d640 1 -- 172.21.15.19:0/2530617135 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34cc0d04d0 con 0x7f34cc155520 2024-08-31T12:18:29.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.170+0000 7f34d1e8d640 1 --2- 172.21.15.19:0/2530617135 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f34cc0cf9c0 0x7f34cc0c9180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:29.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.170+0000 7f34d1e8d640 1 -- 172.21.15.19:0/2530617135 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f34cc0c8b40 con 0x7f34cc0cf9c0 2024-08-31T12:18:29.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.170+0000 7f34cb7fe640 1 --2- 172.21.15.19:0/2530617135 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f34cc0cf9c0 0x7f34cc0c9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:29.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.170+0000 7f34cb7fe640 1 -- 172.21.15.19:0/2530617135 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f34cc069ab0 con 0x7f34cc0cf9c0 2024-08-31T12:18:29.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.171+0000 7f34cb7fe640 1 --2- 172.21.15.19:0/2530617135 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f34cc0cf9c0 0x7f34cc0c9180 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f34bc00d700 tx=0x7f34bc00dbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:29.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.171+0000 7f34b3fff640 1 -- 172.21.15.19:0/2530617135 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34bc00b730 con 0x7f34cc0cf9c0 2024-08-31T12:18:29.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.171+0000 7f34b3fff640 1 -- 172.21.15.19:0/2530617135 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f34c4036400 con 0x7f34cc155520 2024-08-31T12:18:29.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.172+0000 7f34b3fff640 1 --2- 172.21.15.19:0/2530617135 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f34b8077950 0x7f34b8079e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:29.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.172+0000 7f34b3fff640 1 -- 172.21.15.19:0/2530617135 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f34c40b9db0 con 0x7f34cc155520 2024-08-31T12:18:29.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.172+0000 7f34b3fff640 1 -- 172.21.15.19:0/2530617135 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f34bc019070 con 0x7f34cc0cf9c0 2024-08-31T12:18:29.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.172+0000 7f34b3fff640 1 -- 172.21.15.19:0/2530617135 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f34cc0cf9c0 msgr2=0x7f34cc0c9180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:29.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.172+0000 7f34b3fff640 1 --2- 172.21.15.19:0/2530617135 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f34cc0cf9c0 0x7f34cc0c9180 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f34bc00d700 tx=0x7f34bc00dbc0 comp rx=0 tx=0).stop 2024-08-31T12:18:29.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.172+0000 7f34b3fff640 1 -- 172.21.15.19:0/2530617135 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f34cc0cf9c0 msgr2=0x7f34cc0c9180 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:18:29.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.172+0000 7f34b3fff640 1 --2- 172.21.15.19:0/2530617135 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f34cc0cf9c0 0x7f34cc0c9180 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:29.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.172+0000 7f34caffd640 1 --2- 172.21.15.19:0/2530617135 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f34b8077950 0x7f34b8079e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:29.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.173+0000 7f34caffd640 1 --2- 172.21.15.19:0/2530617135 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f34b8077950 0x7f34b8079e60 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f34c0006fd0 tx=0x7f34c0006d60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:29.183 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.181+0000 7f34d1e8d640 1 --2- 172.21.15.19:0/2530617135 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f34cc0cfdd0 0x7f34cc0c96c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:29.183 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.181+0000 7f34d1e8d640 1 -- 172.21.15.19:0/2530617135 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f34cc0c7e50 con 0x7f34cc0cfdd0 2024-08-31T12:18:29.184 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.182+0000 7f34cb7fe640 1 --2- 172.21.15.19:0/2530617135 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f34cc0cfdd0 0x7f34cc0c96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:29.184 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.182+0000 7f34cb7fe640 1 -- 172.21.15.19:0/2530617135 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f34cc06ece0 con 0x7f34cc0cfdd0 2024-08-31T12:18:29.184 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.182+0000 7f34cb7fe640 1 --2- 172.21.15.19:0/2530617135 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f34cc0cfdd0 0x7f34cc0c96c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f34bc00d700 tx=0x7f34bc0047a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:29.184 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.182+0000 7f34b3fff640 1 -- 172.21.15.19:0/2530617135 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34bc01eb10 con 0x7f34cc0cfdd0 2024-08-31T12:18:29.185 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.182+0000 7f34b3fff640 1 -- 172.21.15.19:0/2530617135 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4696 (secure 0 0 0) 0x7f34bc01eb10 con 0x7f34cc0cfdd0 2024-08-31T12:18:29.185 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.182+0000 7f34b3fff640 1 -- 172.21.15.19:0/2530617135 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f34cc0cfdd0 msgr2=0x7f34cc0c96c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:29.185 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.182+0000 7f34b3fff640 1 --2- 172.21.15.19:0/2530617135 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f34cc0cfdd0 0x7f34cc0c96c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f34bc00d700 tx=0x7f34bc0047a0 comp rx=0 tx=0).stop 2024-08-31T12:18:29.185 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.182+0000 7f34b3fff640 1 -- 172.21.15.19:0/2530617135 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f34cc0cfdd0 msgr2=0x7f34cc0c96c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:18:29.185 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.182+0000 7f34b3fff640 1 --2- 172.21.15.19:0/2530617135 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f34cc0cfdd0 0x7f34cc0c96c0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:29.185 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:18:29.186 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:18:29.186 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:18:29.186 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:18:29.186 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 24, 2024-08-31T12:18:29.186 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 37305, 2024-08-31T12:18:29.186 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:18:29.186 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:18:29.186 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:18:29.186 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:29.186 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 1, 2024-08-31T12:18:29.186 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:18:29.187 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:18:29.187 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:18:29.187 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:18:29.187 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:18:29.187 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:18:29.187 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:18:29.187 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:18:29.187 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:18:29.187 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:18:29.187 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:18:29.187 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:18:29.187 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:18:29.187 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:18:29.188 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:29.188 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:18:29.188 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:18:29.188 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:18:29.188 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:18:29.188 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:18:29.188 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:18:29.188 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:18:29.188 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:18:29.188 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:18:29.188 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:18:29.188 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:18:29.189 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:18:29.189 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:18:29.189 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:29.189 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:29.189 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:18:29.189 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:18:29.189 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:18:29.189 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:18:29.189 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:18:29.189 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:18:29.189 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:18:29.189 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:18:29.189 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:18:29.190 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:18:29.190 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:18:29.190 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:18:29.190 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:18:29.190 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:18:29.190 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:18:29.190 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:18:29.190 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:18:29.190 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:18:29.190 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:18:29.190 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:18:29.190 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:18:29.190 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:18:29.190 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:18:29.190 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:18:29.190 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:18:29.191 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:18:29.191 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:18:29.191 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:18:29.191 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:29.191 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:18:29.191 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:29.191 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:18:29.191 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:29.191 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:18:29.191 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:18:29.191 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:18:29.191 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:18:29.191 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:29.191 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:18:29.191 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:18:29.192 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:29.192 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:29.192 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:29.192 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:18:29.192 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:18:29.192 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:29.192 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:29.192 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:29.192 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:29.192 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:18:29.192 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:18:29.192 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:18:29.193 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:18:29.193 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:18:29.193 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:29.193 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:29.193 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:18:29.193 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:18:29.193 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:18:29.193 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:18:29.193 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:29.193 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:18:29.193 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:18:29.193 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:29.193 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:29.193 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:29.194 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:18:29.194 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:18:29.194 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:29.194 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:29.194 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:29.194 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:29.194 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:18:29.194 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:18:29.194 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:18:29.194 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:18:29.194 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:18:29.194 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:29.194 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:29.195 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:18:29.195 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:18:29.195 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:18:29.195 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:18:29.195 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:29.195 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:18:29.195 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:18:29.195 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:29.195 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:29.195 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:29.195 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:18:29.195 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:18:29.195 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:29.195 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:29.195 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:29.196 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:29.196 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:18:29.196 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:18:29.196 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:18:29.196 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:18:29.196 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:18:29.196 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:29.196 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:29.196 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:29.196 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:18:29.196 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:18:29.196 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:29.196 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:18:29.196 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:18:29.196 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:18:29.197 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:29.197 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:29.197 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:18:29.197 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:29.197 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:18:29.197 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:18:29.197 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:18:29.197 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:29.197 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:29.197 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:18:29.197 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:29.197 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:18:29.197 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:18:29.197 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:18:29.198 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:29.198 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:29.198 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:29.198 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:18:29.198 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:18:29.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.183+0000 7f34d1e8d640 1 -- 172.21.15.19:0/2530617135 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f34b8077950 msgr2=0x7f34b8079e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:29.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.183+0000 7f34d1e8d640 1 --2- 172.21.15.19:0/2530617135 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f34b8077950 0x7f34b8079e60 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f34c0006fd0 tx=0x7f34c0006d60 comp rx=0 tx=0).stop 2024-08-31T12:18:29.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.183+0000 7f34d1e8d640 1 -- 172.21.15.19:0/2530617135 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f34cc155520 msgr2=0x7f34cc150450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:29.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.183+0000 7f34cbfff640 1 -- 172.21.15.19:0/2530617135 reap_dead start 2024-08-31T12:18:29.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.183+0000 7f34d1e8d640 1 --2- 172.21.15.19:0/2530617135 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f34cc155520 0x7f34cc150450 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f34c4009d60 tx=0x7f34c4003f10 comp rx=0 tx=0).stop 2024-08-31T12:18:29.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.183+0000 7f34d1e8d640 1 -- 172.21.15.19:0/2530617135 shutdown_connections 2024-08-31T12:18:29.200 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.183+0000 7f34d1e8d640 1 --2- 172.21.15.19:0/2530617135 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f34cc155520 0x7f34cc150450 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:29.200 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.183+0000 7f34d1e8d640 1 -- 172.21.15.19:0/2530617135 >> 172.21.15.19:0/2530617135 conn(0x7f34cc074a90 msgr2=0x7f34cc152580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:18:29.200 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.183+0000 7f34d1e8d640 1 -- 172.21.15.19:0/2530617135 shutdown_connections 2024-08-31T12:18:29.200 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:29.183+0000 7f34d1e8d640 1 -- 172.21.15.19:0/2530617135 wait complete. 2024-08-31T12:18:29.200 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-08-31T12:18:29.200 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-08-31T12:18:49.201 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-08-31T12:18:49.202 INFO:tasks.ceph.mon.b:Restarting daemon 2024-08-31T12:18:49.202 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-08-31T12:18:49.205 INFO:tasks.ceph.mon.b:Started 2024-08-31T12:18:49.205 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-08-31T12:18:49.206 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:18:49.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.340+0000 7f47d19f8640 1 -- 172.21.15.19:0/4217007695 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f47cc0c86b0 msgr2=0x7f47cc1587e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:49.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.340+0000 7f47d19f8640 1 --2- 172.21.15.19:0/4217007695 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f47cc0c86b0 0x7f47cc1587e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f47bc005fd0 tx=0x7f47bc02a5d0 comp rx=0 tx=0).stop 2024-08-31T12:18:49.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.340+0000 7f47d19f8640 1 -- 172.21.15.19:0/4217007695 shutdown_connections 2024-08-31T12:18:49.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.340+0000 7f47d19f8640 1 --2- 172.21.15.19:0/4217007695 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f47cc0c20a0 0x7f47cc14e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:49.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.340+0000 7f47d19f8640 1 --2- 172.21.15.19:0/4217007695 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f47cc0c86b0 0x7f47cc1587e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:49.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.340+0000 7f47d19f8640 1 -- 172.21.15.19:0/4217007695 >> 172.21.15.19:0/4217007695 conn(0x7f47cc06cae0 msgr2=0x7f47cc06cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:18:49.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.340+0000 7f47d19f8640 1 -- 172.21.15.19:0/4217007695 shutdown_connections 2024-08-31T12:18:49.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.340+0000 7f47d19f8640 1 -- 172.21.15.19:0/4217007695 wait complete. 2024-08-31T12:18:49.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.342+0000 7f47d19f8640 1 Processor -- start 2024-08-31T12:18:49.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.342+0000 7f47d19f8640 1 -- start start 2024-08-31T12:18:49.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.343+0000 7f47d19f8640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f47cc0c20a0 0x7f47cc0aa5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:49.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.344+0000 7f47d19f8640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f47cc0c24b0 0x7f47cc1de9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:49.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.344+0000 7f47d19f8640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f47cc0c86b0 0x7f47cc1dfc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:49.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.344+0000 7f47d19f8640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f47cc152330 con 0x7f47cc0c86b0 2024-08-31T12:18:49.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.344+0000 7f47d19f8640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f47cc0b31e0 con 0x7f47cc0c24b0 2024-08-31T12:18:49.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.344+0000 7f47d19f8640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f47cc070530 con 0x7f47cc0c20a0 2024-08-31T12:18:49.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.344+0000 7f47cb7fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f47cc0c86b0 0x7f47cc1dfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:49.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.344+0000 7f47caffd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f47cc0c20a0 0x7f47cc0aa5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:49.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.344+0000 7f47cb7fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f47cc0c86b0 0x7f47cc1dfc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:53112/0 (socket says 172.21.15.19:53112) 2024-08-31T12:18:49.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.344+0000 7f47caffd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f47cc0c20a0 0x7f47cc0aa5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.19:44790/0 (socket says 172.21.15.19:44790) 2024-08-31T12:18:49.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.344+0000 7f47ca7fc640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f47cc0c24b0 0x7f47cc1de9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:49.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.344+0000 7f47cb7fe640 1 -- 172.21.15.19:0/1648732439 learned_addr learned my addr 172.21.15.19:0/1648732439 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:18:49.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.344+0000 7f47ca7fc640 1 -- 172.21.15.19:0/1648732439 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f47cc0c24b0 msgr2=0x7f47cc1de9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-08-31T12:18:49.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.344+0000 7f47ca7fc640 1 -- 172.21.15.19:0/1648732439 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f47cc0c24b0 msgr2=0x7f47cc1de9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-31T12:18:49.349 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.344+0000 7f47ca7fc640 1 --2- 172.21.15.19:0/1648732439 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f47cc0c24b0 0x7f47cc1de9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-31T12:18:49.349 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.344+0000 7f47ca7fc640 1 --2- 172.21.15.19:0/1648732439 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f47cc0c24b0 0x7f47cc1de9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:18:49.349 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.344+0000 7f47caffd640 1 -- 172.21.15.19:0/1648732439 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f47cc0c24b0 msgr2=0x7f47cc1de9f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:18:49.349 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.344+0000 7f47caffd640 1 --2- 172.21.15.19:0/1648732439 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f47cc0c24b0 0x7f47cc1de9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:49.349 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.344+0000 7f47caffd640 1 -- 172.21.15.19:0/1648732439 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f47cc0c86b0 msgr2=0x7f47cc1dfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:49.349 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.344+0000 7f47caffd640 1 --2- 172.21.15.19:0/1648732439 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f47cc0c86b0 0x7f47cc1dfc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:49.349 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.344+0000 7f47caffd640 1 -- 172.21.15.19:0/1648732439 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47cc1def30 con 0x7f47cc0c20a0 2024-08-31T12:18:49.349 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.344+0000 7f47caffd640 1 --2- 172.21.15.19:0/1648732439 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f47cc0c20a0 0x7f47cc0aa5e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f47bc031000 tx=0x7f47bc005750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:49.350 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.345+0000 7f47b3fff640 1 -- 172.21.15.19:0/1648732439 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47bc03e030 con 0x7f47cc0c20a0 2024-08-31T12:18:49.350 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.345+0000 7f47b3fff640 1 -- 172.21.15.19:0/1648732439 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f47bc003a70 con 0x7f47cc0c20a0 2024-08-31T12:18:49.350 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.345+0000 7f47d19f8640 1 -- 172.21.15.19:0/1648732439 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47cc0b2180 con 0x7f47cc0c20a0 2024-08-31T12:18:49.350 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.345+0000 7f47b3fff640 1 -- 172.21.15.19:0/1648732439 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47bc035040 con 0x7f47cc0c20a0 2024-08-31T12:18:49.350 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.345+0000 7f47d19f8640 1 -- 172.21.15.19:0/1648732439 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f47cc1eeba0 con 0x7f47cc0c20a0 2024-08-31T12:18:49.351 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.346+0000 7f47d19f8640 1 -- 172.21.15.19:0/1648732439 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47cc0cc010 con 0x7f47cc0c20a0 2024-08-31T12:18:49.351 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.346+0000 7f47b3fff640 1 -- 172.21.15.19:0/1648732439 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f47bc041030 con 0x7f47cc0c20a0 2024-08-31T12:18:49.351 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.347+0000 7f47b3fff640 1 --2- 172.21.15.19:0/1648732439 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f47b8077a50 0x7f47b8079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:49.351 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.347+0000 7f47b3fff640 1 -- 172.21.15.19:0/1648732439 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f47bc0b9f70 con 0x7f47cc0c20a0 2024-08-31T12:18:49.352 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.347+0000 7f47ca7fc640 1 --2- 172.21.15.19:0/1648732439 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f47b8077a50 0x7f47b8079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:49.352 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.348+0000 7f47ca7fc640 1 --2- 172.21.15.19:0/1648732439 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f47b8077a50 0x7f47b8079f60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f47cc1571f0 tx=0x7f47c0005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:49.354 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.352+0000 7f47b3fff640 1 -- 172.21.15.19:0/1648732439 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f47cc0cc010 con 0x7f47cc0c20a0 2024-08-31T12:18:49.637 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.635+0000 7f47d19f8640 1 -- 172.21.15.19:0/1648732439 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f47cc0c78a0 con 0x7f47cc0c20a0 2024-08-31T12:18:49.637 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.636+0000 7f47b3fff640 1 -- 172.21.15.19:0/1648732439 <== mon.2 v2:172.21.15.43:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f47bc0824f0 con 0x7f47cc0c20a0 2024-08-31T12:18:49.638 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":24,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:18:49.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.638+0000 7f47d19f8640 1 -- 172.21.15.19:0/1648732439 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f47b8077a50 msgr2=0x7f47b8079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:49.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.638+0000 7f47d19f8640 1 --2- 172.21.15.19:0/1648732439 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f47b8077a50 0x7f47b8079f60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f47cc1571f0 tx=0x7f47c0005f90 comp rx=0 tx=0).stop 2024-08-31T12:18:49.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.638+0000 7f47d19f8640 1 -- 172.21.15.19:0/1648732439 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f47cc0c20a0 msgr2=0x7f47cc0aa5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:49.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.638+0000 7f47d19f8640 1 --2- 172.21.15.19:0/1648732439 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f47cc0c20a0 0x7f47cc0aa5e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f47bc031000 tx=0x7f47bc005750 comp rx=0 tx=0).stop 2024-08-31T12:18:49.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.638+0000 7f47d19f8640 1 -- 172.21.15.19:0/1648732439 shutdown_connections 2024-08-31T12:18:49.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.638+0000 7f47d19f8640 1 --2- 172.21.15.19:0/1648732439 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f47b8077a50 0x7f47b8079f60 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:49.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.638+0000 7f47d19f8640 1 --2- 172.21.15.19:0/1648732439 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f47cc0c86b0 0x7f47cc1dfc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:49.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.638+0000 7f47d19f8640 1 --2- 172.21.15.19:0/1648732439 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f47cc0c24b0 0x7f47cc1de9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:49.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.638+0000 7f47d19f8640 1 --2- 172.21.15.19:0/1648732439 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f47cc0c20a0 0x7f47cc0aa5e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:49.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.638+0000 7f47d19f8640 1 -- 172.21.15.19:0/1648732439 >> 172.21.15.19:0/1648732439 conn(0x7f47cc06cae0 msgr2=0x7f47cc074430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:18:49.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.638+0000 7f47d19f8640 1 -- 172.21.15.19:0/1648732439 shutdown_connections 2024-08-31T12:18:49.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:49.639+0000 7f47d19f8640 1 -- 172.21.15.19:0/1648732439 wait complete. 2024-08-31T12:18:52.653 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:18:52.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.775+0000 7f6699b13640 1 -- 172.21.15.19:0/8323626 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f66940cc070 msgr2=0x7f66940cc480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:52.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.775+0000 7f6699b13640 1 --2- 172.21.15.19:0/8323626 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f66940cc070 0x7f66940cc480 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f669408e8e0 tx=0x7f668802a5f0 comp rx=0 tx=0).stop 2024-08-31T12:18:52.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.775+0000 7f6699b13640 1 -- 172.21.15.19:0/8323626 shutdown_connections 2024-08-31T12:18:52.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.775+0000 7f6699b13640 1 --2- 172.21.15.19:0/8323626 >> v2:172.21.15.43:3300/0 conn(0x7f669415b120 0x7f669415b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:52.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.775+0000 7f6699b13640 1 --2- 172.21.15.19:0/8323626 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f66940cc070 0x7f66940cc480 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:52.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.775+0000 7f6699b13640 1 --2- 172.21.15.19:0/8323626 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f66940ccbb0 0x7f66940cb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:52.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.775+0000 7f6699b13640 1 -- 172.21.15.19:0/8323626 >> 172.21.15.19:0/8323626 conn(0x7f6694074810 msgr2=0x7f6694068aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:18:52.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.775+0000 7f6699b13640 1 -- 172.21.15.19:0/8323626 shutdown_connections 2024-08-31T12:18:52.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.775+0000 7f6699b13640 1 -- 172.21.15.19:0/8323626 wait complete. 2024-08-31T12:18:52.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.776+0000 7f6699b13640 1 Processor -- start 2024-08-31T12:18:52.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.777+0000 7f6699b13640 1 -- start start 2024-08-31T12:18:52.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.778+0000 7f6699b13640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f66940cc070 0x7f6694157d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:52.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.778+0000 7f6699b13640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f66940ccbb0 0x7f66940d0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:52.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.778+0000 7f6699b13640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f669415b120 0x7f66941f2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:52.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.778+0000 7f6699b13640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f6694155660 con 0x7f66940cc070 2024-08-31T12:18:52.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.778+0000 7f6699b13640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f669415ba70 con 0x7f66940ccbb0 2024-08-31T12:18:52.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.778+0000 7f6699b13640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f66941557e0 con 0x7f669415b120 2024-08-31T12:18:52.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.778+0000 7f66937fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f66940cc070 0x7f6694157d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:52.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.778+0000 7f66937fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f66940cc070 0x7f6694157d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:53128/0 (socket says 172.21.15.19:53128) 2024-08-31T12:18:52.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.778+0000 7f66937fe640 1 -- 172.21.15.19:0/655225299 learned_addr learned my addr 172.21.15.19:0/655225299 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:18:52.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.778+0000 7f6693fff640 1 --2- 172.21.15.19:0/655225299 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f669415b120 0x7f66941f2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:52.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.778+0000 7f6692ffd640 1 --2- 172.21.15.19:0/655225299 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f66940ccbb0 0x7f66940d0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:52.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.778+0000 7f6692ffd640 1 -- 172.21.15.19:0/655225299 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f66940ccbb0 msgr2=0x7f66940d0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-08-31T12:18:52.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.778+0000 7f6692ffd640 1 -- 172.21.15.19:0/655225299 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f66940ccbb0 msgr2=0x7f66940d0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-31T12:18:52.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.778+0000 7f6692ffd640 1 --2- 172.21.15.19:0/655225299 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f66940ccbb0 0x7f66940d0ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-31T12:18:52.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.778+0000 7f6692ffd640 1 --2- 172.21.15.19:0/655225299 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f66940ccbb0 0x7f66940d0ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:18:52.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.779+0000 7f66937fe640 1 -- 172.21.15.19:0/655225299 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f669415b120 msgr2=0x7f66941f2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:52.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.779+0000 7f66937fe640 1 --2- 172.21.15.19:0/655225299 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f669415b120 0x7f66941f2470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:52.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.779+0000 7f66937fe640 1 -- 172.21.15.19:0/655225299 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f66940ccbb0 msgr2=0x7f66940d0ed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:18:52.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.779+0000 7f66937fe640 1 --2- 172.21.15.19:0/655225299 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f66940ccbb0 0x7f66940d0ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:52.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.779+0000 7f66937fe640 1 -- 172.21.15.19:0/655225299 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66941f29b0 con 0x7f66940cc070 2024-08-31T12:18:52.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.779+0000 7f66937fe640 1 --2- 172.21.15.19:0/655225299 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f66940cc070 0x7f6694157d40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f668400d660 tx=0x7f668400db20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:52.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.779+0000 7f667bfff640 1 -- 172.21.15.19:0/655225299 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6684017070 con 0x7f66940cc070 2024-08-31T12:18:52.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.779+0000 7f667bfff640 1 -- 172.21.15.19:0/655225299 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6684005310 con 0x7f66940cc070 2024-08-31T12:18:52.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.779+0000 7f6699b13640 1 -- 172.21.15.19:0/655225299 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66940c9690 con 0x7f66940cc070 2024-08-31T12:18:52.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.779+0000 7f667bfff640 1 -- 172.21.15.19:0/655225299 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6684009820 con 0x7f66940cc070 2024-08-31T12:18:52.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.779+0000 7f6699b13640 1 -- 172.21.15.19:0/655225299 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66940c9c60 con 0x7f66940cc070 2024-08-31T12:18:52.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.780+0000 7f6699b13640 1 -- 172.21.15.19:0/655225299 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f669406e1a0 con 0x7f66940cc070 2024-08-31T12:18:52.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.781+0000 7f667bfff640 1 -- 172.21.15.19:0/655225299 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f66840099c0 con 0x7f66940cc070 2024-08-31T12:18:52.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.782+0000 7f667bfff640 1 --2- 172.21.15.19:0/655225299 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6680077a80 0x7f6680079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:52.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.782+0000 7f667bfff640 1 -- 172.21.15.19:0/655225299 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f6684099690 con 0x7f66940cc070 2024-08-31T12:18:52.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.782+0000 7f6692ffd640 1 --2- 172.21.15.19:0/655225299 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6680077a80 0x7f6680079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:52.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.783+0000 7f6692ffd640 1 --2- 172.21.15.19:0/655225299 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6680077a80 0x7f6680079f90 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6688033000 tx=0x7f6688004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:52.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:52.786+0000 7f667bfff640 1 -- 172.21.15.19:0/655225299 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f6684061d40 con 0x7f66940cc070 2024-08-31T12:18:53.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:53.084+0000 7f6699b13640 1 -- 172.21.15.19:0/655225299 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f66940cbe50 con 0x7f66940cc070 2024-08-31T12:18:53.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:53.084+0000 7f667bfff640 1 -- 172.21.15.19:0/655225299 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f6684066500 con 0x7f66940cc070 2024-08-31T12:18:53.085 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":24,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":25,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:18:53.088 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:53.086+0000 7f6699b13640 1 -- 172.21.15.19:0/655225299 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6680077a80 msgr2=0x7f6680079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:53.088 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:53.086+0000 7f6699b13640 1 --2- 172.21.15.19:0/655225299 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6680077a80 0x7f6680079f90 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6688033000 tx=0x7f6688004730 comp rx=0 tx=0).stop 2024-08-31T12:18:53.088 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:53.086+0000 7f6699b13640 1 -- 172.21.15.19:0/655225299 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f66940cc070 msgr2=0x7f6694157d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:53.088 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:53.086+0000 7f6699b13640 1 --2- 172.21.15.19:0/655225299 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f66940cc070 0x7f6694157d40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f668400d660 tx=0x7f668400db20 comp rx=0 tx=0).stop 2024-08-31T12:18:53.088 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:53.086+0000 7f6699b13640 1 -- 172.21.15.19:0/655225299 shutdown_connections 2024-08-31T12:18:53.088 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:53.086+0000 7f6699b13640 1 --2- 172.21.15.19:0/655225299 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6680077a80 0x7f6680079f90 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:53.088 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:53.086+0000 7f6699b13640 1 --2- 172.21.15.19:0/655225299 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f669415b120 0x7f66941f2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:53.088 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:53.086+0000 7f6699b13640 1 --2- 172.21.15.19:0/655225299 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f66940ccbb0 0x7f66940d0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:53.088 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:53.086+0000 7f6699b13640 1 --2- 172.21.15.19:0/655225299 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f66940cc070 0x7f6694157d40 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:53.088 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:53.087+0000 7f6699b13640 1 -- 172.21.15.19:0/655225299 >> 172.21.15.19:0/655225299 conn(0x7f6694074810 msgr2=0x7f66940c6530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:18:53.089 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:53.087+0000 7f6699b13640 1 -- 172.21.15.19:0/655225299 shutdown_connections 2024-08-31T12:18:53.089 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:53.087+0000 7f6699b13640 1 -- 172.21.15.19:0/655225299 wait complete. 2024-08-31T12:18:56.007 DEBUG:teuthology.orchestra.run.smithi019:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:18:56.009 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:18:56.100 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:18:56.228 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.227+0000 7f040e762640 1 -- 172.21.15.19:0/3192020100 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f04080cac90 msgr2=0x7f04081531f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:56.228 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.227+0000 7f040e762640 1 --2- 172.21.15.19:0/3192020100 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f04080cac90 0x7f04081531f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f03fc009860 tx=0x7f03fc02a4a0 comp rx=0 tx=0).stop 2024-08-31T12:18:56.229 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.227+0000 7f040e762640 1 -- 172.21.15.19:0/3192020100 shutdown_connections 2024-08-31T12:18:56.229 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.227+0000 7f040e762640 1 --2- 172.21.15.19:0/3192020100 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f04080cac90 0x7f04081531f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:56.229 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.227+0000 7f040e762640 1 -- 172.21.15.19:0/3192020100 >> 172.21.15.19:0/3192020100 conn(0x7f04080cefb0 msgr2=0x7f04080cb340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:18:56.229 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.227+0000 7f040e762640 1 -- 172.21.15.19:0/3192020100 shutdown_connections 2024-08-31T12:18:56.229 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.227+0000 7f040e762640 1 -- 172.21.15.19:0/3192020100 wait complete. 2024-08-31T12:18:56.230 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.229+0000 7f040e762640 1 Processor -- start 2024-08-31T12:18:56.230 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.229+0000 7f040e762640 1 -- start start 2024-08-31T12:18:56.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.231+0000 7f040e762640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f04080ca7d0 0x7f04080c6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:56.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.231+0000 7f040e762640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f04080cac90 0x7f04080b3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:56.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.231+0000 7f040e762640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f040815c4b0 0x7f04080b2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:56.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.231+0000 7f040e762640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f04080714f0 con 0x7f04080ca7d0 2024-08-31T12:18:56.235 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.231+0000 7f040e762640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f04080c9bb0 con 0x7f040815c4b0 2024-08-31T12:18:56.235 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.231+0000 7f040e762640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f040815d3a0 con 0x7f04080cac90 2024-08-31T12:18:56.235 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.231+0000 7f0407fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f04080ca7d0 0x7f04080c6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:56.235 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.231+0000 7f0407fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f04080ca7d0 0x7f04080c6fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:53142/0 (socket says 172.21.15.19:53142) 2024-08-31T12:18:56.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.231+0000 7f04077fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f04080cac90 0x7f04080b3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:56.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.231+0000 7f0407fff640 1 -- 172.21.15.19:0/1721989058 learned_addr learned my addr 172.21.15.19:0/1721989058 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:18:56.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.231+0000 7f040ccff640 1 --2- 172.21.15.19:0/1721989058 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f040815c4b0 0x7f04080b2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:56.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.232+0000 7f0407fff640 1 -- 172.21.15.19:0/1721989058 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f04080cac90 msgr2=0x7f04080b3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:56.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.232+0000 7f0407fff640 1 --2- 172.21.15.19:0/1721989058 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f04080cac90 0x7f04080b3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:56.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.232+0000 7f0407fff640 1 -- 172.21.15.19:0/1721989058 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f040815c4b0 msgr2=0x7f04080b2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:56.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.232+0000 7f0407fff640 1 --2- 172.21.15.19:0/1721989058 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f040815c4b0 0x7f04080b2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:56.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.232+0000 7f0407fff640 1 -- 172.21.15.19:0/1721989058 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04081efff0 con 0x7f04080ca7d0 2024-08-31T12:18:56.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.232+0000 7f04077fe640 1 --2- 172.21.15.19:0/1721989058 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f04080cac90 0x7f04080b3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:18:56.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.232+0000 7f040ccff640 1 --2- 172.21.15.19:0/1721989058 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f040815c4b0 0x7f04080b2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:18:56.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.232+0000 7f0407fff640 1 --2- 172.21.15.19:0/1721989058 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f04080ca7d0 0x7f04080c6fb0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f03f800c3f0 tx=0x7f03f800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:56.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.232+0000 7f0404ff9640 1 -- 172.21.15.19:0/1721989058 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03f8017070 con 0x7f04080ca7d0 2024-08-31T12:18:56.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.232+0000 7f0404ff9640 1 -- 172.21.15.19:0/1721989058 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f03f80053e0 con 0x7f04080ca7d0 2024-08-31T12:18:56.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.232+0000 7f040e762640 1 -- 172.21.15.19:0/1721989058 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04080b4450 con 0x7f04080ca7d0 2024-08-31T12:18:56.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.232+0000 7f0404ff9640 1 -- 172.21.15.19:0/1721989058 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03f8003870 con 0x7f04080ca7d0 2024-08-31T12:18:56.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.232+0000 7f040e762640 1 -- 172.21.15.19:0/1721989058 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04081f0260 con 0x7f04080ca7d0 2024-08-31T12:18:56.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.233+0000 7f040e762640 1 -- 172.21.15.19:0/1721989058 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04080ca160 con 0x7f04080ca7d0 2024-08-31T12:18:56.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.234+0000 7f0404ff9640 1 -- 172.21.15.19:0/1721989058 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f03f801a070 con 0x7f04080ca7d0 2024-08-31T12:18:56.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.235+0000 7f0404ff9640 1 --2- 172.21.15.19:0/1721989058 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f03f4077a50 0x7f03f4079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:56.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.235+0000 7f0404ff9640 1 -- 172.21.15.19:0/1721989058 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f03f80989d0 con 0x7f04080ca7d0 2024-08-31T12:18:56.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.235+0000 7f04077fe640 1 --2- 172.21.15.19:0/1721989058 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f03f4077a50 0x7f03f4079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:56.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.236+0000 7f04077fe640 1 --2- 172.21.15.19:0/1721989058 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f03f4077a50 0x7f03f4079f60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f03fc034000 tx=0x7f03fc004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:56.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.240+0000 7f0404ff9640 1 -- 172.21.15.19:0/1721989058 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f03f8061070 con 0x7f04080ca7d0 2024-08-31T12:18:56.540 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.539+0000 7f040e762640 1 -- 172.21.15.19:0/1721989058 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0408150f20 con 0x7f04080ca7d0 2024-08-31T12:18:56.541 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.540+0000 7f0404ff9640 1 -- 172.21.15.19:0/1721989058 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f03f8065830 con 0x7f04080ca7d0 2024-08-31T12:18:56.541 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:18:56.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.542+0000 7f040e762640 1 -- 172.21.15.19:0/1721989058 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f03f4077a50 msgr2=0x7f03f4079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:56.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.542+0000 7f040e762640 1 --2- 172.21.15.19:0/1721989058 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f03f4077a50 0x7f03f4079f60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f03fc034000 tx=0x7f03fc004730 comp rx=0 tx=0).stop 2024-08-31T12:18:56.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.542+0000 7f040e762640 1 -- 172.21.15.19:0/1721989058 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f04080ca7d0 msgr2=0x7f04080c6fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:56.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.542+0000 7f040e762640 1 --2- 172.21.15.19:0/1721989058 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f04080ca7d0 0x7f04080c6fb0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f03f800c3f0 tx=0x7f03f800c8b0 comp rx=0 tx=0).stop 2024-08-31T12:18:56.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.542+0000 7f040e762640 1 -- 172.21.15.19:0/1721989058 shutdown_connections 2024-08-31T12:18:56.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.542+0000 7f040e762640 1 --2- 172.21.15.19:0/1721989058 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f03f4077a50 0x7f03f4079f60 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:56.544 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.542+0000 7f040e762640 1 --2- 172.21.15.19:0/1721989058 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f040815c4b0 0x7f04080b2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:56.544 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.542+0000 7f040e762640 1 --2- 172.21.15.19:0/1721989058 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f04080cac90 0x7f04080b3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:56.544 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.542+0000 7f040e762640 1 --2- 172.21.15.19:0/1721989058 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f04080ca7d0 0x7f04080c6fb0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:56.544 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.542+0000 7f040e762640 1 -- 172.21.15.19:0/1721989058 >> 172.21.15.19:0/1721989058 conn(0x7f04080cefb0 msgr2=0x7f040814e9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:18:56.544 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.542+0000 7f040e762640 1 -- 172.21.15.19:0/1721989058 shutdown_connections 2024-08-31T12:18:56.544 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.542+0000 7f040e762640 1 -- 172.21.15.19:0/1721989058 wait complete. 2024-08-31T12:18:56.556 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-08-31T12:18:56.556 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-08-31T12:18:56.556 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-31T12:18:56.677 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.676+0000 7f1316a6b640 1 -- 172.21.15.19:0/663267025 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f131015b6e0 msgr2=0x7f131015baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:56.678 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.676+0000 7f1316a6b640 1 --2- 172.21.15.19:0/663267025 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f131015b6e0 0x7f131015baf0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f13100cb2b0 tx=0x7f130802aa90 comp rx=0 tx=0).stop 2024-08-31T12:18:56.678 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.676+0000 7f1316a6b640 1 -- 172.21.15.19:0/663267025 shutdown_connections 2024-08-31T12:18:56.678 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.676+0000 7f1316a6b640 1 --2- 172.21.15.19:0/663267025 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f131015b6e0 0x7f131015baf0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:56.678 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.676+0000 7f1316a6b640 1 --2- 172.21.15.19:0/663267025 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f131006e0e0 0x7f1310155300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:56.678 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.676+0000 7f1316a6b640 1 --2- 172.21.15.19:0/663267025 >> v2:172.21.15.43:3300/0 conn(0x7f13100c8a40 0x7f13100c9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:56.678 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.676+0000 7f1316a6b640 1 -- 172.21.15.19:0/663267025 >> 172.21.15.19:0/663267025 conn(0x7f13100690d0 msgr2=0x7f13100d0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:18:56.678 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.676+0000 7f1316a6b640 1 -- 172.21.15.19:0/663267025 shutdown_connections 2024-08-31T12:18:56.678 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.676+0000 7f1316a6b640 1 -- 172.21.15.19:0/663267025 wait complete. 2024-08-31T12:18:56.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.678+0000 7f1316a6b640 1 Processor -- start 2024-08-31T12:18:56.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.679+0000 7f1316a6b640 1 -- start start 2024-08-31T12:18:56.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.680+0000 7f1316a6b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f131006e0e0 0x7f131014cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:56.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.680+0000 7f1316a6b640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13100c8a40 0x7f13100c5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:56.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.680+0000 7f1316a6b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f131015b6e0 0x7f13100cf850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:56.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.680+0000 7f1316a6b640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f13100c8330 con 0x7f131006e0e0 2024-08-31T12:18:56.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.680+0000 7f1316a6b640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f13100c8050 con 0x7f13100c8a40 2024-08-31T12:18:56.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.680+0000 7f1316a6b640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f13100c8610 con 0x7f131015b6e0 2024-08-31T12:18:56.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.681+0000 7f1314807640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f131006e0e0 0x7f131014cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:56.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.681+0000 7f1314807640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f131006e0e0 0x7f131014cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:53166/0 (socket says 172.21.15.19:53166) 2024-08-31T12:18:56.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.681+0000 7f1314807640 1 -- 172.21.15.19:0/1176803117 learned_addr learned my addr 172.21.15.19:0/1176803117 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:18:56.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.681+0000 7f130ffff640 1 --2- 172.21.15.19:0/1176803117 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13100c8a40 0x7f13100c5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:56.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.681+0000 7f1315008640 1 --2- 172.21.15.19:0/1176803117 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f131015b6e0 0x7f13100cf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:56.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.681+0000 7f1314807640 1 -- 172.21.15.19:0/1176803117 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f131015b6e0 msgr2=0x7f13100cf850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:56.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.681+0000 7f1314807640 1 --2- 172.21.15.19:0/1176803117 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f131015b6e0 0x7f13100cf850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:56.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.681+0000 7f1314807640 1 -- 172.21.15.19:0/1176803117 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13100c8a40 msgr2=0x7f13100c5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:56.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.681+0000 7f1314807640 1 --2- 172.21.15.19:0/1176803117 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f13100c8a40 0x7f13100c5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:56.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.681+0000 7f1314807640 1 -- 172.21.15.19:0/1176803117 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f131014d1c0 con 0x7f131006e0e0 2024-08-31T12:18:56.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.681+0000 7f1314807640 1 --2- 172.21.15.19:0/1176803117 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f131006e0e0 0x7f131014cc80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f130400d7b0 tx=0x7f130400dc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:56.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.681+0000 7f130d7fa640 1 -- 172.21.15.19:0/1176803117 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1304017070 con 0x7f131006e0e0 2024-08-31T12:18:56.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.681+0000 7f130d7fa640 1 -- 172.21.15.19:0/1176803117 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1304004310 con 0x7f131006e0e0 2024-08-31T12:18:56.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.681+0000 7f1316a6b640 1 -- 172.21.15.19:0/1176803117 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f131014da30 con 0x7f131006e0e0 2024-08-31T12:18:56.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.681+0000 7f130d7fa640 1 -- 172.21.15.19:0/1176803117 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1304009c30 con 0x7f131006e0e0 2024-08-31T12:18:56.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.681+0000 7f1316a6b640 1 -- 172.21.15.19:0/1176803117 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13101f1840 con 0x7f131006e0e0 2024-08-31T12:18:56.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.682+0000 7f1316a6b640 1 --2- 172.21.15.19:0/1176803117 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f13101f2040 0x7f13100c4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:56.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.682+0000 7f1316a6b640 1 -- 172.21.15.19:0/1176803117 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f131014d7e0 con 0x7f13101f2040 2024-08-31T12:18:56.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.683+0000 7f130ffff640 1 --2- 172.21.15.19:0/1176803117 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f13101f2040 0x7f13100c4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:56.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.683+0000 7f130ffff640 1 -- 172.21.15.19:0/1176803117 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f13100c9850 con 0x7f13101f2040 2024-08-31T12:18:56.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.683+0000 7f130ffff640 1 --2- 172.21.15.19:0/1176803117 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f13101f2040 0x7f13100c4630 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f12fc005fd0 tx=0x7f12fc010bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:56.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.683+0000 7f130d7fa640 1 -- 172.21.15.19:0/1176803117 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12fc00ed80 con 0x7f13101f2040 2024-08-31T12:18:56.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.684+0000 7f130d7fa640 1 -- 172.21.15.19:0/1176803117 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f1304005020 con 0x7f131006e0e0 2024-08-31T12:18:56.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.684+0000 7f130d7fa640 1 --2- 172.21.15.19:0/1176803117 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f12f4077950 0x7f12f4079e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:56.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.684+0000 7f130d7fa640 1 -- 172.21.15.19:0/1176803117 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f1304099960 con 0x7f131006e0e0 2024-08-31T12:18:56.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.684+0000 7f1315008640 1 --2- 172.21.15.19:0/1176803117 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f12f4077950 0x7f12f4079e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:56.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.684+0000 7f130d7fa640 1 -- 172.21.15.19:0/1176803117 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f12fc01c480 con 0x7f13101f2040 2024-08-31T12:18:56.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.684+0000 7f130d7fa640 1 -- 172.21.15.19:0/1176803117 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f13101f2040 msgr2=0x7f13100c4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:56.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.684+0000 7f130d7fa640 1 --2- 172.21.15.19:0/1176803117 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f13101f2040 0x7f13100c4630 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f12fc005fd0 tx=0x7f12fc010bf0 comp rx=0 tx=0).stop 2024-08-31T12:18:56.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.684+0000 7f130d7fa640 1 -- 172.21.15.19:0/1176803117 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f13101f2040 msgr2=0x7f13100c4630 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:18:56.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.684+0000 7f130d7fa640 1 --2- 172.21.15.19:0/1176803117 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f13101f2040 0x7f13100c4630 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:56.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.685+0000 7f1315008640 1 --2- 172.21.15.19:0/1176803117 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f12f4077950 0x7f12f4079e60 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1308035000 tx=0x7f1308038040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:56.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.693+0000 7f1316a6b640 1 --2- 172.21.15.19:0/1176803117 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f13100c4fe0 0x7f13100c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:56.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.693+0000 7f1316a6b640 1 -- 172.21.15.19:0/1176803117 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f13100ccf20 con 0x7f13100c4fe0 2024-08-31T12:18:56.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.693+0000 7f130ffff640 1 --2- 172.21.15.19:0/1176803117 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f13100c4fe0 0x7f13100c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:56.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.693+0000 7f130ffff640 1 -- 172.21.15.19:0/1176803117 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f13100c6100 con 0x7f13100c4fe0 2024-08-31T12:18:56.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.693+0000 7f130ffff640 1 --2- 172.21.15.19:0/1176803117 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f13100c4fe0 0x7f13100c53f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f13101eeeb0 tx=0x7f12fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:56.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.693+0000 7f130d7fa640 1 -- 172.21.15.19:0/1176803117 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12fc027070 con 0x7f13100c4fe0 2024-08-31T12:18:56.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.694+0000 7f130d7fa640 1 -- 172.21.15.19:0/1176803117 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5044 (secure 0 0 0) 0x7f12fc0212a0 con 0x7f13100c4fe0 2024-08-31T12:18:56.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.694+0000 7f130d7fa640 1 -- 172.21.15.19:0/1176803117 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f13100c4fe0 msgr2=0x7f13100c53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:56.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.694+0000 7f130d7fa640 1 --2- 172.21.15.19:0/1176803117 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f13100c4fe0 0x7f13100c53f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f13101eeeb0 tx=0x7f12fc0046c0 comp rx=0 tx=0).stop 2024-08-31T12:18:56.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.694+0000 7f130d7fa640 1 -- 172.21.15.19:0/1176803117 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f13100c4fe0 msgr2=0x7f13100c53f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:18:56.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.694+0000 7f130d7fa640 1 --2- 172.21.15.19:0/1176803117 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f13100c4fe0 0x7f13100c53f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:56.695 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:18:56.696 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:18:56.696 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:18:56.696 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "leader", 2024-08-31T12:18:56.696 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 26, 2024-08-31T12:18:56.696 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 116684, 2024-08-31T12:18:56.696 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:18:56.696 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:18:56.696 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:18:56.696 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:18:56.696 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:56.696 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 2, 2024-08-31T12:18:56.696 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:18:56.696 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:18:56.697 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:18:56.697 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:18:56.697 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:18:56.697 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:18:56.697 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:18:56.697 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:18:56.697 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:18:56.697 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:18:56.697 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:18:56.697 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:18:56.697 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:18:56.698 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:18:56.698 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:56.698 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:18:56.698 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:18:56.698 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:18:56.698 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:18:56.698 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:18:56.698 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:18:56.698 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:18:56.698 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:18:56.698 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:18:56.698 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:18:56.698 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:18:56.698 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:18:56.699 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:18:56.699 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:56.699 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:56.699 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:18:56.699 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:18:56.699 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:18:56.699 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:18:56.699 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:18:56.699 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:18:56.699 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:18:56.699 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:18:56.699 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:18:56.699 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:18:56.700 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:18:56.700 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:18:56.700 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:18:56.700 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:18:56.700 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:18:56.700 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:18:56.700 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:18:56.700 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:18:56.700 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:18:56.700 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:18:56.700 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:18:56.700 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:18:56.701 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:18:56.701 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:18:56.701 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:18:56.701 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:18:56.701 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:18:56.701 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:18:56.701 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:56.701 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:18:56.701 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:56.701 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:18:56.701 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:56.701 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:18:56.701 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:18:56.702 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:18:56.702 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:18:56.702 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:56.702 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:18:56.702 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:18:56.702 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:56.702 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:56.702 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:56.702 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:18:56.702 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:18:56.702 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:56.702 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:56.702 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:56.703 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:56.703 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:18:56.703 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:18:56.703 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:18:56.703 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:18:56.703 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:18:56.703 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:56.703 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:56.703 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:18:56.703 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:18:56.703 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:18:56.704 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:18:56.704 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:56.704 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:18:56.704 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:18:56.704 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:56.704 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:56.704 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:56.704 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:18:56.704 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:18:56.704 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:56.704 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:56.704 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:56.705 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:56.705 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:18:56.705 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:18:56.705 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:18:56.705 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:18:56.705 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:18:56.705 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:56.705 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:56.705 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:18:56.705 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:18:56.705 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:18:56.705 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:18:56.705 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:56.705 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:18:56.706 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:18:56.706 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:56.706 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:56.706 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:56.706 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:18:56.706 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:18:56.706 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:56.706 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:56.706 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:56.706 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:56.706 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:18:56.706 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:18:56.706 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:18:56.706 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:18:56.706 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:18:56.707 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:56.707 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:56.707 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:56.707 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:18:56.707 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:18:56.707 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:56.707 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:18:56.707 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:18:56.707 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:18:56.707 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:56.707 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:56.707 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:18:56.707 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:56.707 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:18:56.708 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:18:56.708 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 5 2024-08-31T12:18:56.708 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:56.708 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:56.708 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:18:56.708 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:56.708 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:18:56.708 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:18:56.708 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 8 2024-08-31T12:18:56.708 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:56.708 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:56.708 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:18:56.708 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:56.708 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:18:56.709 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:18:56.709 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 6 2024-08-31T12:18:56.709 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:56.709 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:56.709 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:18:56.709 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:56.709 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:18:56.709 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:18:56.709 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:18:56.709 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:56.709 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:56.709 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:56.709 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:18:56.709 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:18:56.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.694+0000 7f1316a6b640 1 -- 172.21.15.19:0/1176803117 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f12f4077950 msgr2=0x7f12f4079e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:56.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.694+0000 7f1316a6b640 1 --2- 172.21.15.19:0/1176803117 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f12f4077950 0x7f12f4079e60 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1308035000 tx=0x7f1308038040 comp rx=0 tx=0).stop 2024-08-31T12:18:56.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.694+0000 7f1316a6b640 1 -- 172.21.15.19:0/1176803117 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f131006e0e0 msgr2=0x7f131014cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:56.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.694+0000 7f1316a6b640 1 --2- 172.21.15.19:0/1176803117 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f131006e0e0 0x7f131014cc80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f130400d7b0 tx=0x7f130400dc70 comp rx=0 tx=0).stop 2024-08-31T12:18:56.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.694+0000 7f1315008640 1 -- 172.21.15.19:0/1176803117 reap_dead start 2024-08-31T12:18:56.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.694+0000 7f1315008640 1 -- 172.21.15.19:0/1176803117 reap_dead start 2024-08-31T12:18:56.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.694+0000 7f1316a6b640 1 -- 172.21.15.19:0/1176803117 shutdown_connections 2024-08-31T12:18:56.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.694+0000 7f1316a6b640 1 -- 172.21.15.19:0/1176803117 >> 172.21.15.19:0/1176803117 conn(0x7f13100690d0 msgr2=0x7f13101500c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:18:56.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.694+0000 7f1316a6b640 1 -- 172.21.15.19:0/1176803117 shutdown_connections 2024-08-31T12:18:56.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.694+0000 7f1316a6b640 1 -- 172.21.15.19:0/1176803117 wait complete. 2024-08-31T12:18:56.712 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:18:56.712 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-08-31T12:18:56.712 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-31T12:18:56.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.836+0000 7fbe371c9640 1 -- 172.21.15.19:0/56326239 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbe301587f0 msgr2=0x7fbe300c1ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:56.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.836+0000 7fbe371c9640 1 --2- 172.21.15.19:0/56326239 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbe301587f0 0x7fbe300c1ec0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c009830 tx=0x7fbe1c02a510 comp rx=0 tx=0).stop 2024-08-31T12:18:56.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.836+0000 7fbe371c9640 1 -- 172.21.15.19:0/56326239 shutdown_connections 2024-08-31T12:18:56.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.836+0000 7fbe371c9640 1 --2- 172.21.15.19:0/56326239 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbe301587f0 0x7fbe300c1ec0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:56.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.836+0000 7fbe371c9640 1 --2- 172.21.15.19:0/56326239 >> v2:172.21.15.43:3300/0 conn(0x7fbe300c7ac0 0x7fbe3014e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:56.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.836+0000 7fbe371c9640 1 -- 172.21.15.19:0/56326239 >> 172.21.15.19:0/56326239 conn(0x7fbe3006cae0 msgr2=0x7fbe3006cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:18:56.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.836+0000 7fbe371c9640 1 -- 172.21.15.19:0/56326239 shutdown_connections 2024-08-31T12:18:56.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.836+0000 7fbe371c9640 1 -- 172.21.15.19:0/56326239 wait complete. 2024-08-31T12:18:56.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.838+0000 7fbe371c9640 1 Processor -- start 2024-08-31T12:18:56.839 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.838+0000 7fbe371c9640 1 -- start start 2024-08-31T12:18:56.841 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.839+0000 7fbe371c9640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbe300c2400 0x7fbe30155e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:56.841 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.839+0000 7fbe371c9640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbe300c7ac0 0x7fbe3013f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:56.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.839+0000 7fbe371c9640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbe301587f0 0x7fbe3014d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:56.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.839+0000 7fbe371c9640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fbe300c3e90 con 0x7fbe300c2400 2024-08-31T12:18:56.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.839+0000 7fbe371c9640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fbe30152380 con 0x7fbe300c7ac0 2024-08-31T12:18:56.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.839+0000 7fbe371c9640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fbe300c4010 con 0x7fbe301587f0 2024-08-31T12:18:56.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.839+0000 7fbe34f65640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbe300c2400 0x7fbe30155e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:56.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.839+0000 7fbe34f65640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbe300c2400 0x7fbe30155e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:53204/0 (socket says 172.21.15.19:53204) 2024-08-31T12:18:56.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.839+0000 7fbe34f65640 1 -- 172.21.15.19:0/3079486219 learned_addr learned my addr 172.21.15.19:0/3079486219 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:18:56.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.840+0000 7fbe2ffff640 1 --2- 172.21.15.19:0/3079486219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbe300c7ac0 0x7fbe3013f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:56.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.840+0000 7fbe35766640 1 --2- 172.21.15.19:0/3079486219 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbe301587f0 0x7fbe3014d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:56.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.840+0000 7fbe34f65640 1 -- 172.21.15.19:0/3079486219 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbe301587f0 msgr2=0x7fbe3014d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:56.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.840+0000 7fbe34f65640 1 --2- 172.21.15.19:0/3079486219 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbe301587f0 0x7fbe3014d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:56.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.840+0000 7fbe34f65640 1 -- 172.21.15.19:0/3079486219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbe300c7ac0 msgr2=0x7fbe3013f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:56.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.840+0000 7fbe34f65640 1 --2- 172.21.15.19:0/3079486219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbe300c7ac0 0x7fbe3013f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:56.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.840+0000 7fbe34f65640 1 -- 172.21.15.19:0/3079486219 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe30156340 con 0x7fbe300c2400 2024-08-31T12:18:56.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.840+0000 7fbe34f65640 1 --2- 172.21.15.19:0/3079486219 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbe300c2400 0x7fbe30155e00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbe2400e9f0 tx=0x7fbe2400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:56.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.840+0000 7fbe2d7fa640 1 -- 172.21.15.19:0/3079486219 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe2400cd80 con 0x7fbe300c2400 2024-08-31T12:18:56.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.840+0000 7fbe2d7fa640 1 -- 172.21.15.19:0/3079486219 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbe24004020 con 0x7fbe300c2400 2024-08-31T12:18:56.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.840+0000 7fbe371c9640 1 -- 172.21.15.19:0/3079486219 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe301577d0 con 0x7fbe300c2400 2024-08-31T12:18:56.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.840+0000 7fbe2d7fa640 1 -- 172.21.15.19:0/3079486219 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe24004470 con 0x7fbe300c2400 2024-08-31T12:18:56.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.840+0000 7fbe371c9640 1 -- 172.21.15.19:0/3079486219 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe3014a2f0 con 0x7fbe300c2400 2024-08-31T12:18:56.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.841+0000 7fbe371c9640 1 --2- 172.21.15.19:0/3079486219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbe30156510 0x7fbe3014a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:56.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.841+0000 7fbe371c9640 1 -- 172.21.15.19:0/3079486219 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fbe30157a90 con 0x7fbe30156510 2024-08-31T12:18:56.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.842+0000 7fbe2ffff640 1 --2- 172.21.15.19:0/3079486219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbe30156510 0x7fbe3014a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:56.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.842+0000 7fbe2ffff640 1 -- 172.21.15.19:0/3079486219 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbe3014e720 con 0x7fbe30156510 2024-08-31T12:18:56.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.842+0000 7fbe2ffff640 1 --2- 172.21.15.19:0/3079486219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbe30156510 0x7fbe3014a530 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbe300b3380 tx=0x7fbe1c003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:56.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.842+0000 7fbe2d7fa640 1 -- 172.21.15.19:0/3079486219 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe1c03c620 con 0x7fbe30156510 2024-08-31T12:18:56.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.843+0000 7fbe2d7fa640 1 -- 172.21.15.19:0/3079486219 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fbe24005030 con 0x7fbe300c2400 2024-08-31T12:18:56.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.843+0000 7fbe2d7fa640 1 --2- 172.21.15.19:0/3079486219 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fbe14077980 0x7fbe14079e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:56.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.843+0000 7fbe2d7fa640 1 -- 172.21.15.19:0/3079486219 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fbe24099e90 con 0x7fbe300c2400 2024-08-31T12:18:56.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.843+0000 7fbe35766640 1 --2- 172.21.15.19:0/3079486219 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fbe14077980 0x7fbe14079e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:56.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.843+0000 7fbe2d7fa640 1 -- 172.21.15.19:0/3079486219 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbe1c040070 con 0x7fbe30156510 2024-08-31T12:18:56.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.843+0000 7fbe2d7fa640 1 -- 172.21.15.19:0/3079486219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbe30156510 msgr2=0x7fbe3014a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:56.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.843+0000 7fbe2d7fa640 1 --2- 172.21.15.19:0/3079486219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbe30156510 0x7fbe3014a530 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbe300b3380 tx=0x7fbe1c003b90 comp rx=0 tx=0).stop 2024-08-31T12:18:56.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.843+0000 7fbe2d7fa640 1 -- 172.21.15.19:0/3079486219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbe30156510 msgr2=0x7fbe3014a530 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:18:56.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.843+0000 7fbe2d7fa640 1 --2- 172.21.15.19:0/3079486219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbe30156510 0x7fbe3014a530 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbe300b3380 tx=0x7fbe1c003b90 comp rx=0 tx=0).stop 2024-08-31T12:18:56.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.844+0000 7fbe35766640 1 --2- 172.21.15.19:0/3079486219 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fbe14077980 0x7fbe14079e90 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fbe280017a0 tx=0x7fbe28001730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:56.853 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.852+0000 7fbe371c9640 1 --2- 172.21.15.19:0/3079486219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbe30142250 0x7fbe30142660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:56.853 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.852+0000 7fbe371c9640 1 -- 172.21.15.19:0/3079486219 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fbe300c78a0 con 0x7fbe30142250 2024-08-31T12:18:56.853 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.852+0000 7fbe2ffff640 1 --2- 172.21.15.19:0/3079486219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbe30142250 0x7fbe30142660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:56.854 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.853+0000 7fbe2ffff640 1 -- 172.21.15.19:0/3079486219 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbe300c9090 con 0x7fbe30142250 2024-08-31T12:18:56.854 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.853+0000 7fbe2ffff640 1 --2- 172.21.15.19:0/3079486219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbe30142250 0x7fbe30142660 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c009780 tx=0x7fbe1c005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:56.854 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.853+0000 7fbe2d7fa640 1 -- 172.21.15.19:0/3079486219 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe1c044b70 con 0x7fbe30142250 2024-08-31T12:18:56.855 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.854+0000 7fbe2d7fa640 1 -- 172.21.15.19:0/3079486219 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4539 (secure 0 0 0) 0x7fbe1c009450 con 0x7fbe30142250 2024-08-31T12:18:56.855 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.854+0000 7fbe2d7fa640 1 -- 172.21.15.19:0/3079486219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbe30142250 msgr2=0x7fbe30142660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:56.855 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.854+0000 7fbe2d7fa640 1 --2- 172.21.15.19:0/3079486219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbe30142250 0x7fbe30142660 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c009780 tx=0x7fbe1c005f90 comp rx=0 tx=0).stop 2024-08-31T12:18:56.855 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.854+0000 7fbe2d7fa640 1 -- 172.21.15.19:0/3079486219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbe30142250 msgr2=0x7fbe30142660 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:18:56.855 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.854+0000 7fbe2d7fa640 1 --2- 172.21.15.19:0/3079486219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbe30142250 0x7fbe30142660 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c009780 tx=0x7fbe1c005f90 comp rx=0 tx=0).stop 2024-08-31T12:18:56.855 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:18:56.855 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:18:56.856 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:18:56.856 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:18:56.856 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 26, 2024-08-31T12:18:56.856 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 7539, 2024-08-31T12:18:56.856 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:18:56.856 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:18:56.856 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:18:56.856 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:18:56.856 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:56.856 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 2, 2024-08-31T12:18:56.856 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:18:56.856 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:18:56.857 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:18:56.857 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:18:56.857 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:18:56.857 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:18:56.857 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:18:56.857 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:18:56.857 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:18:56.858 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:18:56.858 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:18:56.858 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:18:56.858 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:18:56.858 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:18:56.858 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:56.858 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:18:56.858 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:18:56.858 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:18:56.858 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:18:56.858 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:18:56.858 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:18:56.858 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:18:56.858 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:18:56.858 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:18:56.859 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:18:56.859 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:18:56.859 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:18:56.859 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:18:56.859 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:56.859 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:56.859 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:18:56.859 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:18:56.859 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:18:56.859 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:18:56.859 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:18:56.859 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:18:56.860 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:18:56.860 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:18:56.860 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:18:56.860 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:18:56.860 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:18:56.860 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:18:56.860 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:18:56.860 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:18:56.860 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:18:56.860 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:18:56.860 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:18:56.860 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:18:56.861 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:18:56.861 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:18:56.861 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:18:56.861 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:18:56.861 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:18:56.861 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:18:56.861 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:18:56.861 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:18:56.861 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:18:56.861 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:18:56.861 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:56.861 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:18:56.861 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:56.861 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:18:56.862 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:56.862 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:18:56.862 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:18:56.862 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:18:56.862 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:18:56.862 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:56.862 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:18:56.862 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:18:56.862 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:56.862 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:56.862 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:56.862 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:18:56.862 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:18:56.862 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:56.863 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:56.863 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:56.863 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:56.863 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:18:56.863 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:18:56.863 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:18:56.863 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:18:56.863 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:18:56.863 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:56.863 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:56.863 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:18:56.863 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:18:56.863 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:18:56.863 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:18:56.864 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:56.864 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:18:56.864 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:18:56.864 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:56.864 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:56.864 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:56.864 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:18:56.864 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:18:56.864 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:56.864 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:56.864 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:56.864 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:56.865 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:18:56.865 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:18:56.865 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:18:56.865 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:18:56.865 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:18:56.865 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:56.865 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:56.865 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:18:56.865 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:18:56.865 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:18:56.865 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:18:56.865 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:56.865 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:18:56.866 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:18:56.866 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:56.866 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:56.866 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:56.866 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:18:56.866 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:18:56.866 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:56.866 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:56.866 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:56.866 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:56.866 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:18:56.866 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:18:56.866 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:18:56.866 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:18:56.867 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:18:56.867 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:56.867 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:56.867 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:56.867 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:18:56.867 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:18:56.867 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:56.867 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:18:56.867 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:18:56.867 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:18:56.867 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:56.867 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:56.867 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:18:56.867 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:56.867 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:18:56.868 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:18:56.868 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:18:56.868 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:56.868 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:56.868 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:56.868 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:18:56.868 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:18:56.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.854+0000 7fbe371c9640 1 -- 172.21.15.19:0/3079486219 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fbe14077980 msgr2=0x7fbe14079e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:56.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.854+0000 7fbe371c9640 1 --2- 172.21.15.19:0/3079486219 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fbe14077980 0x7fbe14079e90 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fbe280017a0 tx=0x7fbe28001730 comp rx=0 tx=0).stop 2024-08-31T12:18:56.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.854+0000 7fbe371c9640 1 -- 172.21.15.19:0/3079486219 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbe300c2400 msgr2=0x7fbe30155e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:56.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.854+0000 7fbe371c9640 1 --2- 172.21.15.19:0/3079486219 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbe300c2400 0x7fbe30155e00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbe2400e9f0 tx=0x7fbe2400eeb0 comp rx=0 tx=0).stop 2024-08-31T12:18:56.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.854+0000 7fbe35766640 1 -- 172.21.15.19:0/3079486219 reap_dead start 2024-08-31T12:18:56.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.854+0000 7fbe371c9640 1 -- 172.21.15.19:0/3079486219 shutdown_connections 2024-08-31T12:18:56.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.854+0000 7fbe371c9640 1 --2- 172.21.15.19:0/3079486219 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbe300c2400 0x7fbe30155e00 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:56.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.854+0000 7fbe371c9640 1 -- 172.21.15.19:0/3079486219 >> 172.21.15.19:0/3079486219 conn(0x7fbe3006cae0 msgr2=0x7fbe30070fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:18:56.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.854+0000 7fbe371c9640 1 -- 172.21.15.19:0/3079486219 shutdown_connections 2024-08-31T12:18:56.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.854+0000 7fbe371c9640 1 -- 172.21.15.19:0/3079486219 wait complete. 2024-08-31T12:18:56.870 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:18:56.870 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-08-31T12:18:56.871 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-31T12:18:56.992 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.991+0000 7f80af139640 1 -- 172.21.15.19:0/2839748346 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f80a80c9440 msgr2=0x7f80a8070bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:56.992 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.991+0000 7f80af139640 1 --2- 172.21.15.19:0/2839748346 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f80a80c9440 0x7f80a8070bf0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f80a0009be0 tx=0x7f80a002a960 comp rx=0 tx=0).stop 2024-08-31T12:18:56.992 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.991+0000 7f80af139640 1 -- 172.21.15.19:0/2839748346 shutdown_connections 2024-08-31T12:18:56.992 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.991+0000 7f80af139640 1 --2- 172.21.15.19:0/2839748346 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f80a80c9440 0x7f80a8070bf0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:56.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.991+0000 7f80af139640 1 --2- 172.21.15.19:0/2839748346 >> v2:172.21.15.43:3300/0 conn(0x7f80a8153190 0x7f80a8159820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:56.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.991+0000 7f80af139640 1 -- 172.21.15.19:0/2839748346 >> 172.21.15.19:0/2839748346 conn(0x7f80a80728b0 msgr2=0x7f80a8072cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:18:56.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.991+0000 7f80af139640 1 -- 172.21.15.19:0/2839748346 shutdown_connections 2024-08-31T12:18:56.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.991+0000 7f80af139640 1 -- 172.21.15.19:0/2839748346 wait complete. 2024-08-31T12:18:56.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.993+0000 7f80af139640 1 Processor -- start 2024-08-31T12:18:56.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.993+0000 7f80af139640 1 -- start start 2024-08-31T12:18:56.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.994+0000 7f80af139640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f80a80c9030 0x7f80a81f1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:56.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.994+0000 7f80af139640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80a80c9440 0x7f80a80ca730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:56.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.995+0000 7f80af139640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f80a8153190 0x7f80a8143da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:56.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.995+0000 7f80af139640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f80a81535a0 con 0x7f80a80c9030 2024-08-31T12:18:56.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.995+0000 7f80af139640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f80a80c4db0 con 0x7f80a80c9440 2024-08-31T12:18:56.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.995+0000 7f80af139640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f80a815cdc0 con 0x7f80a8153190 2024-08-31T12:18:56.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.995+0000 7f80aced5640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f80a80c9030 0x7f80a81f1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:56.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.995+0000 7f80aced5640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f80a80c9030 0x7f80a81f1b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:53212/0 (socket says 172.21.15.19:53212) 2024-08-31T12:18:56.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.995+0000 7f80aced5640 1 -- 172.21.15.19:0/2955129650 learned_addr learned my addr 172.21.15.19:0/2955129650 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:18:56.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.995+0000 7f80ad6d6640 1 --2- 172.21.15.19:0/2955129650 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f80a8153190 0x7f80a8143da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:56.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.995+0000 7f80a7fff640 1 --2- 172.21.15.19:0/2955129650 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80a80c9440 0x7f80a80ca730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:56.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.995+0000 7f80aced5640 1 -- 172.21.15.19:0/2955129650 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f80a8153190 msgr2=0x7f80a8143da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:56.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.995+0000 7f80aced5640 1 --2- 172.21.15.19:0/2955129650 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f80a8153190 0x7f80a8143da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:56.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.995+0000 7f80aced5640 1 -- 172.21.15.19:0/2955129650 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80a80c9440 msgr2=0x7f80a80ca730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:56.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.995+0000 7f80aced5640 1 --2- 172.21.15.19:0/2955129650 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80a80c9440 0x7f80a80ca730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:56.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.995+0000 7f80aced5640 1 -- 172.21.15.19:0/2955129650 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80a806ec90 con 0x7f80a80c9030 2024-08-31T12:18:56.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.995+0000 7f80a7fff640 1 --2- 172.21.15.19:0/2955129650 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80a80c9440 0x7f80a80ca730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:18:56.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.995+0000 7f80aced5640 1 --2- 172.21.15.19:0/2955129650 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f80a80c9030 0x7f80a81f1b30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f809c00e7c0 tx=0x7f809c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:56.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.996+0000 7f80a57fa640 1 -- 172.21.15.19:0/2955129650 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f809c00ccc0 con 0x7f80a80c9030 2024-08-31T12:18:56.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.996+0000 7f80a57fa640 1 -- 172.21.15.19:0/2955129650 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f809c010430 con 0x7f80a80c9030 2024-08-31T12:18:56.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.996+0000 7f80af139640 1 -- 172.21.15.19:0/2955129650 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80a81f2070 con 0x7f80a80c9030 2024-08-31T12:18:56.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.996+0000 7f80a57fa640 1 -- 172.21.15.19:0/2955129650 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f809c010860 con 0x7f80a80c9030 2024-08-31T12:18:56.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.996+0000 7f80af139640 1 -- 172.21.15.19:0/2955129650 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80a814f150 con 0x7f80a80c9030 2024-08-31T12:18:57.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.997+0000 7f80af139640 1 --2- 172.21.15.19:0/2955129650 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f80a81442e0 0x7f80a806efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:57.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.997+0000 7f80af139640 1 -- 172.21.15.19:0/2955129650 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f80a80b0bc0 con 0x7f80a81442e0 2024-08-31T12:18:57.001 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.997+0000 7f80a7fff640 1 --2- 172.21.15.19:0/2955129650 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f80a81442e0 0x7f80a806efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:57.001 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.997+0000 7f80a7fff640 1 -- 172.21.15.19:0/2955129650 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f80a81f18f0 con 0x7f80a81442e0 2024-08-31T12:18:57.001 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.998+0000 7f80a7fff640 1 --2- 172.21.15.19:0/2955129650 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f80a81442e0 0x7f80a806efe0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f809400e4d0 tx=0x7f809400e990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:57.001 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.998+0000 7f80a57fa640 1 -- 172.21.15.19:0/2955129650 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f809401a070 con 0x7f80a81442e0 2024-08-31T12:18:57.001 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.998+0000 7f80a57fa640 1 -- 172.21.15.19:0/2955129650 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f809c010a00 con 0x7f80a80c9030 2024-08-31T12:18:57.001 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.998+0000 7f80a57fa640 1 --2- 172.21.15.19:0/2955129650 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f808c077950 0x7f808c079e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:57.001 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.999+0000 7f80a57fa640 1 -- 172.21.15.19:0/2955129650 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f809c09a1e0 con 0x7f80a80c9030 2024-08-31T12:18:57.001 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.999+0000 7f80ad6d6640 1 --2- 172.21.15.19:0/2955129650 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f808c077950 0x7f808c079e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:57.001 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.999+0000 7f80a57fa640 1 -- 172.21.15.19:0/2955129650 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8094005030 con 0x7f80a81442e0 2024-08-31T12:18:57.001 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.999+0000 7f80a57fa640 1 -- 172.21.15.19:0/2955129650 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f80a81442e0 msgr2=0x7f80a806efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:57.001 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.999+0000 7f80a57fa640 1 --2- 172.21.15.19:0/2955129650 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f80a81442e0 0x7f80a806efe0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f809400e4d0 tx=0x7f809400e990 comp rx=0 tx=0).stop 2024-08-31T12:18:57.001 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.999+0000 7f80a57fa640 1 -- 172.21.15.19:0/2955129650 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f80a81442e0 msgr2=0x7f80a806efe0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:18:57.002 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.999+0000 7f80a57fa640 1 --2- 172.21.15.19:0/2955129650 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f80a81442e0 0x7f80a806efe0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:57.002 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:56.999+0000 7f80ad6d6640 1 --2- 172.21.15.19:0/2955129650 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f808c077950 0x7f808c079e60 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f80a0034000 tx=0x7f80a0004ab0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:57.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.006+0000 7f80af139640 1 --2- 172.21.15.19:0/2955129650 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f80a806f520 0x7f80a81f4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:57.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.006+0000 7f80af139640 1 -- 172.21.15.19:0/2955129650 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f80a8159820 con 0x7f80a806f520 2024-08-31T12:18:57.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.006+0000 7f80a7fff640 1 --2- 172.21.15.19:0/2955129650 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f80a806f520 0x7f80a81f4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:57.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.006+0000 7f80a7fff640 1 -- 172.21.15.19:0/2955129650 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f80a80c9a20 con 0x7f80a806f520 2024-08-31T12:18:57.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.007+0000 7f80a7fff640 1 --2- 172.21.15.19:0/2955129650 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f80a806f520 0x7f80a81f4e10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f809400e4d0 tx=0x7f8094000ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:57.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.007+0000 7f80a57fa640 1 -- 172.21.15.19:0/2955129650 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8094004880 con 0x7f80a806f520 2024-08-31T12:18:57.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.007+0000 7f80a57fa640 1 -- 172.21.15.19:0/2955129650 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4707 (secure 0 0 0) 0x7f8094004880 con 0x7f80a806f520 2024-08-31T12:18:57.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.007+0000 7f80a57fa640 1 -- 172.21.15.19:0/2955129650 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f80a806f520 msgr2=0x7f80a81f4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:57.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.007+0000 7f80a57fa640 1 --2- 172.21.15.19:0/2955129650 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f80a806f520 0x7f80a81f4e10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f809400e4d0 tx=0x7f8094000ed0 comp rx=0 tx=0).stop 2024-08-31T12:18:57.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.007+0000 7f80a57fa640 1 -- 172.21.15.19:0/2955129650 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f80a806f520 msgr2=0x7f80a81f4e10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:18:57.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.007+0000 7f80a57fa640 1 --2- 172.21.15.19:0/2955129650 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f80a806f520 0x7f80a81f4e10 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:57.009 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:18:57.009 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:18:57.009 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:18:57.009 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:18:57.009 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 26, 2024-08-31T12:18:57.009 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 65129, 2024-08-31T12:18:57.009 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:18:57.009 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:18:57.009 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:18:57.009 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:18:57.009 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:57.010 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 2, 2024-08-31T12:18:57.010 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:18:57.010 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:18:57.010 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:18:57.010 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:18:57.010 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:18:57.010 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:18:57.010 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:18:57.010 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:18:57.010 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:18:57.010 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:18:57.010 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:18:57.011 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:18:57.011 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:18:57.011 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:18:57.011 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:57.011 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:18:57.011 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:18:57.011 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:18:57.011 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:18:57.011 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:18:57.011 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:18:57.011 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:18:57.011 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:18:57.012 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:18:57.012 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:18:57.012 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:18:57.012 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:18:57.012 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:18:57.012 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:57.012 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:57.012 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:18:57.012 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:18:57.012 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:18:57.012 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:18:57.013 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:18:57.013 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:18:57.013 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:18:57.013 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:18:57.013 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:18:57.013 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:18:57.013 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:18:57.013 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:18:57.013 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:18:57.013 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:18:57.013 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:18:57.013 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:18:57.013 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:18:57.014 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:18:57.014 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:18:57.014 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:18:57.014 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:18:57.014 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:18:57.014 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:18:57.014 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:18:57.014 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:18:57.014 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:18:57.014 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:18:57.014 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:18:57.014 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:57.014 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:18:57.015 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:57.015 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:18:57.015 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:57.015 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:18:57.015 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:18:57.015 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:18:57.015 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:18:57.015 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:57.015 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:18:57.015 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:18:57.015 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:57.015 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:57.015 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:57.015 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:18:57.015 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:18:57.016 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:57.016 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:57.016 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:57.016 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:57.016 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:18:57.016 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:18:57.016 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:18:57.016 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:18:57.016 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:18:57.016 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:57.016 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:57.016 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:18:57.016 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:18:57.017 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:18:57.017 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:18:57.017 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:57.017 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:18:57.017 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:18:57.017 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:57.017 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:57.017 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:57.017 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:18:57.017 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:18:57.017 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:57.017 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:57.017 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:57.017 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:57.017 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:18:57.018 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:18:57.018 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:18:57.018 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:18:57.018 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:18:57.018 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:57.018 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:57.018 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:18:57.018 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:18:57.018 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:18:57.018 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:18:57.018 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:57.018 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:18:57.019 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:18:57.019 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:57.019 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:57.019 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:57.019 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:18:57.019 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:18:57.019 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:18:57.019 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:57.019 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:57.019 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:57.019 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:18:57.019 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:18:57.019 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:18:57.020 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:18:57.020 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:18:57.020 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:57.020 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:57.020 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:57.020 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:18:57.020 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:18:57.020 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:57.020 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:18:57.020 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:18:57.020 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:18:57.020 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:57.020 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:57.021 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:18:57.021 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:57.021 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:18:57.021 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:18:57.021 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:18:57.021 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:57.021 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:18:57.021 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:18:57.021 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:18:57.021 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:18:57.021 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:18:57.021 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:18:57.021 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:18:57.022 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:18:57.022 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:18:57.022 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:18:57.022 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:18:57.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.007+0000 7f80af139640 1 -- 172.21.15.19:0/2955129650 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f808c077950 msgr2=0x7f808c079e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:57.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.007+0000 7f80af139640 1 --2- 172.21.15.19:0/2955129650 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f808c077950 0x7f808c079e60 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f80a0034000 tx=0x7f80a0004ab0 comp rx=0 tx=0).stop 2024-08-31T12:18:57.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.007+0000 7f80af139640 1 -- 172.21.15.19:0/2955129650 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f80a80c9030 msgr2=0x7f80a81f1b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:57.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.007+0000 7f80af139640 1 --2- 172.21.15.19:0/2955129650 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f80a80c9030 0x7f80a81f1b30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f809c00e7c0 tx=0x7f809c00ec80 comp rx=0 tx=0).stop 2024-08-31T12:18:57.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.007+0000 7f80ad6d6640 1 -- 172.21.15.19:0/2955129650 reap_dead start 2024-08-31T12:18:57.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.007+0000 7f80ad6d6640 1 -- 172.21.15.19:0/2955129650 reap_dead start 2024-08-31T12:18:57.024 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.007+0000 7f80af139640 1 -- 172.21.15.19:0/2955129650 shutdown_connections 2024-08-31T12:18:57.024 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.008+0000 7f80af139640 1 -- 172.21.15.19:0/2955129650 >> 172.21.15.19:0/2955129650 conn(0x7f80a80728b0 msgr2=0x7f80a814f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:18:57.024 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.008+0000 7f80af139640 1 -- 172.21.15.19:0/2955129650 shutdown_connections 2024-08-31T12:18:57.024 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.008+0000 7f80af139640 1 -- 172.21.15.19:0/2955129650 wait complete. 2024-08-31T12:18:57.024 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:18:57.024 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-08-31T12:18:57.024 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-08-31T12:18:57.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.146+0000 7f38cc87f640 1 -- 172.21.15.19:0/707394276 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f38c80c3640 msgr2=0x7f38c80c3a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:57.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.146+0000 7f38cc87f640 1 --2- 172.21.15.19:0/707394276 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f38c80c3640 0x7f38c80c3a50 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f38c80c33f0 tx=0x7f38bc02a540 comp rx=0 tx=0).stop 2024-08-31T12:18:57.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.146+0000 7f38cc87f640 1 -- 172.21.15.19:0/707394276 shutdown_connections 2024-08-31T12:18:57.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.146+0000 7f38cc87f640 1 --2- 172.21.15.19:0/707394276 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f38c8157920 0x7f38c8157d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:57.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.146+0000 7f38cc87f640 1 --2- 172.21.15.19:0/707394276 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f38c80c3640 0x7f38c80c3a50 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:57.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.146+0000 7f38cc87f640 1 --2- 172.21.15.19:0/707394276 >> v2:172.21.15.43:3300/0 conn(0x7f38c8070310 0x7f38c8151560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:57.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.146+0000 7f38cc87f640 1 -- 172.21.15.19:0/707394276 >> 172.21.15.19:0/707394276 conn(0x7f38c806cae0 msgr2=0x7f38c806cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:18:57.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.146+0000 7f38cc87f640 1 -- 172.21.15.19:0/707394276 shutdown_connections 2024-08-31T12:18:57.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.146+0000 7f38cc87f640 1 -- 172.21.15.19:0/707394276 wait complete. 2024-08-31T12:18:57.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.149+0000 7f38cc87f640 1 Processor -- start 2024-08-31T12:18:57.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.149+0000 7f38cc87f640 1 -- start start 2024-08-31T12:18:57.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.150+0000 7f38cc87f640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f38c8070310 0x7f38c814ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:57.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.151+0000 7f38cc87f640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38c80c3640 0x7f38c813f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:57.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.151+0000 7f38cc87f640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f38c8157920 0x7f38c813f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:57.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.151+0000 7f38cc87f640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f38c806f2f0 con 0x7f38c8070310 2024-08-31T12:18:57.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.151+0000 7f38cc87f640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f38c80cc930 con 0x7f38c80c3640 2024-08-31T12:18:57.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.151+0000 7f38cc87f640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f38c806f5d0 con 0x7f38c8157920 2024-08-31T12:18:57.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.151+0000 7f38c659c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f38c8070310 0x7f38c814ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:57.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.151+0000 7f38c659c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f38c8070310 0x7f38c814ff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:53244/0 (socket says 172.21.15.19:53244) 2024-08-31T12:18:57.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.151+0000 7f38c659c640 1 -- 172.21.15.19:0/2844753598 learned_addr learned my addr 172.21.15.19:0/2844753598 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:18:57.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.151+0000 7f38c5d9b640 1 --2- 172.21.15.19:0/2844753598 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38c80c3640 0x7f38c813f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:57.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.151+0000 7f38c6d9d640 1 --2- 172.21.15.19:0/2844753598 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f38c8157920 0x7f38c813f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:57.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.151+0000 7f38c659c640 1 -- 172.21.15.19:0/2844753598 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f38c8157920 msgr2=0x7f38c813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:57.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.151+0000 7f38c659c640 1 --2- 172.21.15.19:0/2844753598 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f38c8157920 0x7f38c813f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:57.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.151+0000 7f38c659c640 1 -- 172.21.15.19:0/2844753598 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38c80c3640 msgr2=0x7f38c813f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:57.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.151+0000 7f38c659c640 1 --2- 172.21.15.19:0/2844753598 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38c80c3640 0x7f38c813f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:57.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.151+0000 7f38c659c640 1 -- 172.21.15.19:0/2844753598 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38c81f3840 con 0x7f38c8070310 2024-08-31T12:18:57.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.151+0000 7f38c5d9b640 1 --2- 172.21.15.19:0/2844753598 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38c80c3640 0x7f38c813f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:18:57.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.151+0000 7f38c659c640 1 --2- 172.21.15.19:0/2844753598 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f38c8070310 0x7f38c814ff80 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f38b800e9f0 tx=0x7f38b800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:57.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.152+0000 7f38b6ffd640 1 -- 172.21.15.19:0/2844753598 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b800ccc0 con 0x7f38c8070310 2024-08-31T12:18:57.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.152+0000 7f38b6ffd640 1 -- 172.21.15.19:0/2844753598 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f38b8010430 con 0x7f38c8070310 2024-08-31T12:18:57.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.152+0000 7f38cc87f640 1 -- 172.21.15.19:0/2844753598 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38c8140590 con 0x7f38c8070310 2024-08-31T12:18:57.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.152+0000 7f38b6ffd640 1 -- 172.21.15.19:0/2844753598 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b8010840 con 0x7f38c8070310 2024-08-31T12:18:57.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.152+0000 7f38cc87f640 1 -- 172.21.15.19:0/2844753598 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38c8070aa0 con 0x7f38c8070310 2024-08-31T12:18:57.155 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.153+0000 7f38cc87f640 1 -- 172.21.15.19:0/2844753598 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38c806f8b0 con 0x7f38c8070310 2024-08-31T12:18:57.155 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.154+0000 7f38b6ffd640 1 -- 172.21.15.19:0/2844753598 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f38b80109e0 con 0x7f38c8070310 2024-08-31T12:18:57.155 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.154+0000 7f38b6ffd640 1 --2- 172.21.15.19:0/2844753598 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f38a8077a50 0x7f38a8079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:18:57.155 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.155+0000 7f38b6ffd640 1 -- 172.21.15.19:0/2844753598 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f38b809a0e0 con 0x7f38c8070310 2024-08-31T12:18:57.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.155+0000 7f38c5d9b640 1 --2- 172.21.15.19:0/2844753598 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f38a8077a50 0x7f38a8079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:18:57.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.156+0000 7f38c5d9b640 1 --2- 172.21.15.19:0/2844753598 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f38a8077a50 0x7f38a8079f60 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f38bc033000 tx=0x7f38bc004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:18:57.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.160+0000 7f38b6ffd640 1 -- 172.21.15.19:0/2844753598 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f38b8062e60 con 0x7f38c8070310 2024-08-31T12:18:57.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.465+0000 7f38cc87f640 1 -- 172.21.15.19:0/2844753598 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f38c81516f0 con 0x7f38c8070310 2024-08-31T12:18:57.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.466+0000 7f38b6ffd640 1 -- 172.21.15.19:0/2844753598 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f38b8062760 con 0x7f38c8070310 2024-08-31T12:18:57.471 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.470+0000 7f38cc87f640 1 -- 172.21.15.19:0/2844753598 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f38a8077a50 msgr2=0x7f38a8079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:57.471 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.470+0000 7f38cc87f640 1 --2- 172.21.15.19:0/2844753598 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f38a8077a50 0x7f38a8079f60 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f38bc033000 tx=0x7f38bc004730 comp rx=0 tx=0).stop 2024-08-31T12:18:57.471 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.470+0000 7f38cc87f640 1 -- 172.21.15.19:0/2844753598 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f38c8070310 msgr2=0x7f38c814ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:18:57.471 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.470+0000 7f38cc87f640 1 --2- 172.21.15.19:0/2844753598 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f38c8070310 0x7f38c814ff80 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f38b800e9f0 tx=0x7f38b800eeb0 comp rx=0 tx=0).stop 2024-08-31T12:18:57.471 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.470+0000 7f38cc87f640 1 -- 172.21.15.19:0/2844753598 shutdown_connections 2024-08-31T12:18:57.471 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.470+0000 7f38cc87f640 1 --2- 172.21.15.19:0/2844753598 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f38a8077a50 0x7f38a8079f60 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:57.471 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.470+0000 7f38cc87f640 1 --2- 172.21.15.19:0/2844753598 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f38c8157920 0x7f38c813f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:57.471 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.470+0000 7f38cc87f640 1 --2- 172.21.15.19:0/2844753598 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38c80c3640 0x7f38c813f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:57.471 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.470+0000 7f38cc87f640 1 --2- 172.21.15.19:0/2844753598 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f38c8070310 0x7f38c814ff80 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:18:57.471 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.470+0000 7f38cc87f640 1 -- 172.21.15.19:0/2844753598 >> 172.21.15.19:0/2844753598 conn(0x7f38c806cae0 msgr2=0x7f38c80635a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:18:57.471 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.470+0000 7f38cc87f640 1 -- 172.21.15.19:0/2844753598 shutdown_connections 2024-08-31T12:18:57.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:18:57.470+0000 7f38cc87f640 1 -- 172.21.15.19:0/2844753598 wait complete. 2024-08-31T12:18:57.483 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2024-08-31T12:19:07.484 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-08-31T12:19:07.484 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:19:07.614 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.613+0000 7f9ea242f640 1 -- 172.21.15.19:0/2270964135 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9e9c0c72a0 msgr2=0x7f9e9c06cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:07.614 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.613+0000 7f9ea242f640 1 --2- 172.21.15.19:0/2270964135 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9e9c0c72a0 0x7f9e9c06cf70 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c0752c0 tx=0x7f9e8c02a580 comp rx=0 tx=0).stop 2024-08-31T12:19:07.614 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.613+0000 7f9ea242f640 1 -- 172.21.15.19:0/2270964135 shutdown_connections 2024-08-31T12:19:07.614 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.613+0000 7f9ea242f640 1 --2- 172.21.15.19:0/2270964135 >> v2:172.21.15.43:3300/0 conn(0x7f9e9c0c8390 0x7f9e9c0c87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:07.614 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.613+0000 7f9ea242f640 1 --2- 172.21.15.19:0/2270964135 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9e9c0c7d60 0x7f9e9c06d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:07.615 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.613+0000 7f9ea242f640 1 --2- 172.21.15.19:0/2270964135 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9e9c0c72a0 0x7f9e9c06cf70 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:07.615 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.613+0000 7f9ea242f640 1 -- 172.21.15.19:0/2270964135 >> 172.21.15.19:0/2270964135 conn(0x7f9e9c0cd7d0 msgr2=0x7f9e9c0cdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:19:07.615 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.613+0000 7f9ea242f640 1 -- 172.21.15.19:0/2270964135 shutdown_connections 2024-08-31T12:19:07.615 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.613+0000 7f9ea242f640 1 -- 172.21.15.19:0/2270964135 wait complete. 2024-08-31T12:19:07.615 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.615+0000 7f9ea242f640 1 Processor -- start 2024-08-31T12:19:07.615 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.615+0000 7f9ea242f640 1 -- start start 2024-08-31T12:19:07.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.616+0000 7f9ea242f640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9e9c0c72a0 0x7f9e9c147590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:07.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.616+0000 7f9ea242f640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e9c0c7d60 0x7f9e9c1f2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:07.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.616+0000 7f9ea242f640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9e9c0c8390 0x7f9e9c156430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:07.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.616+0000 7f9ea242f640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9c0c9a20 con 0x7f9e9c0c72a0 2024-08-31T12:19:07.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.616+0000 7f9ea242f640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9c0c9740 con 0x7f9e9c0c7d60 2024-08-31T12:19:07.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.616+0000 7f9ea242f640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f9e9c0c4350 con 0x7f9e9c0c8390 2024-08-31T12:19:07.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.616+0000 7f9e9bfff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9e9c0c72a0 0x7f9e9c147590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:07.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.616+0000 7f9e9bfff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9e9c0c72a0 0x7f9e9c147590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:57528/0 (socket says 172.21.15.19:57528) 2024-08-31T12:19:07.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.616+0000 7f9e9bfff640 1 -- 172.21.15.19:0/733506166 learned_addr learned my addr 172.21.15.19:0/733506166 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:19:07.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.616+0000 7f9ea09cc640 1 --2- 172.21.15.19:0/733506166 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9e9c0c8390 0x7f9e9c156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:07.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.616+0000 7f9e9b7fe640 1 --2- 172.21.15.19:0/733506166 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e9c0c7d60 0x7f9e9c1f2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:07.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.617+0000 7f9e9bfff640 1 -- 172.21.15.19:0/733506166 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9e9c0c8390 msgr2=0x7f9e9c156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:07.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.617+0000 7f9e9bfff640 1 --2- 172.21.15.19:0/733506166 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9e9c0c8390 0x7f9e9c156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:07.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.617+0000 7f9e9bfff640 1 -- 172.21.15.19:0/733506166 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e9c0c7d60 msgr2=0x7f9e9c1f2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:07.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.617+0000 7f9e9bfff640 1 --2- 172.21.15.19:0/733506166 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e9c0c7d60 0x7f9e9c1f2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:07.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.617+0000 7f9e9bfff640 1 -- 172.21.15.19:0/733506166 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e9c147ad0 con 0x7f9e9c0c72a0 2024-08-31T12:19:07.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.617+0000 7f9e9bfff640 1 --2- 172.21.15.19:0/733506166 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9e9c0c72a0 0x7f9e9c147590 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c034040 tx=0x7f9e8c003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:07.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.617+0000 7f9e98ff9640 1 -- 172.21.15.19:0/733506166 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c03e030 con 0x7f9e9c0c72a0 2024-08-31T12:19:07.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.617+0000 7f9e98ff9640 1 -- 172.21.15.19:0/733506166 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9e8c003e00 con 0x7f9e9c0c72a0 2024-08-31T12:19:07.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.617+0000 7f9ea242f640 1 -- 172.21.15.19:0/733506166 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e9c1eeea0 con 0x7f9e9c0c72a0 2024-08-31T12:19:07.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.617+0000 7f9e98ff9640 1 -- 172.21.15.19:0/733506166 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c035040 con 0x7f9e9c0c72a0 2024-08-31T12:19:07.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.617+0000 7f9ea242f640 1 -- 172.21.15.19:0/733506166 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e9c14d150 con 0x7f9e9c0c72a0 2024-08-31T12:19:07.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.618+0000 7f9ea242f640 1 -- 172.21.15.19:0/733506166 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e9c0c87a0 con 0x7f9e9c0c72a0 2024-08-31T12:19:07.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.619+0000 7f9e98ff9640 1 -- 172.21.15.19:0/733506166 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f9e8c041030 con 0x7f9e9c0c72a0 2024-08-31T12:19:07.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.620+0000 7f9e98ff9640 1 --2- 172.21.15.19:0/733506166 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9e88077a50 0x7f9e88079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:07.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.620+0000 7f9e98ff9640 1 -- 172.21.15.19:0/733506166 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f9e8c0ba220 con 0x7f9e9c0c72a0 2024-08-31T12:19:07.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.620+0000 7f9e9b7fe640 1 --2- 172.21.15.19:0/733506166 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9e88077a50 0x7f9e88079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:07.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.621+0000 7f9e9b7fe640 1 --2- 172.21.15.19:0/733506166 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9e88077a50 0x7f9e88079f60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c150910 tx=0x7f9e90005ef0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:07.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.625+0000 7f9e98ff9640 1 -- 172.21.15.19:0/733506166 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f9e9c147ad0 con 0x7f9e9c0c72a0 2024-08-31T12:19:07.907 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.906+0000 7f9ea242f640 1 -- 172.21.15.19:0/733506166 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9e9c14cb90 con 0x7f9e9c0c72a0 2024-08-31T12:19:07.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.907+0000 7f9e98ff9640 1 -- 172.21.15.19:0/733506166 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f9e8c08b3c0 con 0x7f9e9c0c72a0 2024-08-31T12:19:07.908 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:19:07.910 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.909+0000 7f9ea242f640 1 -- 172.21.15.19:0/733506166 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9e88077a50 msgr2=0x7f9e88079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:07.910 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.909+0000 7f9ea242f640 1 --2- 172.21.15.19:0/733506166 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9e88077a50 0x7f9e88079f60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c150910 tx=0x7f9e90005ef0 comp rx=0 tx=0).stop 2024-08-31T12:19:07.910 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.909+0000 7f9ea242f640 1 -- 172.21.15.19:0/733506166 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9e9c0c72a0 msgr2=0x7f9e9c147590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:07.910 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.909+0000 7f9ea242f640 1 --2- 172.21.15.19:0/733506166 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9e9c0c72a0 0x7f9e9c147590 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c034040 tx=0x7f9e8c003910 comp rx=0 tx=0).stop 2024-08-31T12:19:07.910 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.909+0000 7f9ea242f640 1 -- 172.21.15.19:0/733506166 shutdown_connections 2024-08-31T12:19:07.910 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.909+0000 7f9ea242f640 1 --2- 172.21.15.19:0/733506166 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9e88077a50 0x7f9e88079f60 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:07.911 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.909+0000 7f9ea242f640 1 --2- 172.21.15.19:0/733506166 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9e9c0c8390 0x7f9e9c156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:07.911 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.909+0000 7f9ea242f640 1 --2- 172.21.15.19:0/733506166 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e9c0c7d60 0x7f9e9c1f2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:07.911 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.909+0000 7f9ea242f640 1 --2- 172.21.15.19:0/733506166 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9e9c0c72a0 0x7f9e9c147590 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:07.911 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.909+0000 7f9ea242f640 1 -- 172.21.15.19:0/733506166 >> 172.21.15.19:0/733506166 conn(0x7f9e9c0cd7d0 msgr2=0x7f9e9c1520d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:19:07.911 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.909+0000 7f9ea242f640 1 -- 172.21.15.19:0/733506166 shutdown_connections 2024-08-31T12:19:07.911 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:07.909+0000 7f9ea242f640 1 -- 172.21.15.19:0/733506166 wait complete. 2024-08-31T12:19:07.921 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-08-31T12:19:07.921 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-08-31T12:19:07.921 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-08-31T12:19:07.922 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-31T12:19:08.046 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.045+0000 7f3b3b96e640 1 -- 172.21.15.19:0/1105309919 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b340c7c50 msgr2=0x7f3b340b2860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:08.046 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.045+0000 7f3b3b96e640 1 --2- 172.21.15.19:0/1105309919 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b340c7c50 0x7f3b340b2860 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3b34071da0 tx=0x7f3b2402a4a0 comp rx=0 tx=0).stop 2024-08-31T12:19:08.046 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.045+0000 7f3b3b96e640 1 -- 172.21.15.19:0/1105309919 shutdown_connections 2024-08-31T12:19:08.046 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.045+0000 7f3b3b96e640 1 --2- 172.21.15.19:0/1105309919 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b340c7c50 0x7f3b340b2860 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:08.046 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.045+0000 7f3b3b96e640 1 --2- 172.21.15.19:0/1105309919 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3b34068aa0 0x7f3b34151ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:08.046 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.045+0000 7f3b3b96e640 1 -- 172.21.15.19:0/1105309919 >> 172.21.15.19:0/1105309919 conn(0x7f3b3406cae0 msgr2=0x7f3b3406cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:19:08.046 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.045+0000 7f3b3b96e640 1 -- 172.21.15.19:0/1105309919 shutdown_connections 2024-08-31T12:19:08.046 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.045+0000 7f3b3b96e640 1 -- 172.21.15.19:0/1105309919 wait complete. 2024-08-31T12:19:08.048 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.047+0000 7f3b3b96e640 1 Processor -- start 2024-08-31T12:19:08.048 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.047+0000 7f3b3b96e640 1 -- start start 2024-08-31T12:19:08.051 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.049+0000 7f3b3b96e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3b34068aa0 0x7f3b3414f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:08.051 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.049+0000 7f3b3b96e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b340b2da0 0x7f3b3414b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:08.051 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.049+0000 7f3b3b96e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3b340c7c50 0x7f3b34143770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:08.051 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.049+0000 7f3b3b96e640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f3b34069110 con 0x7f3b340b2da0 2024-08-31T12:19:08.051 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.049+0000 7f3b3b96e640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3b34068f90 con 0x7f3b34068aa0 2024-08-31T12:19:08.051 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.049+0000 7f3b3b96e640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f3b340c39c0 con 0x7f3b340c7c50 2024-08-31T12:19:08.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.049+0000 7f3b38f09640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b340b2da0 0x7f3b3414b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:08.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.049+0000 7f3b38f09640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b340b2da0 0x7f3b3414b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:57556/0 (socket says 172.21.15.19:57556) 2024-08-31T12:19:08.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.049+0000 7f3b38f09640 1 -- 172.21.15.19:0/4113092313 learned_addr learned my addr 172.21.15.19:0/4113092313 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:19:08.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.049+0000 7f3b3970a640 1 --2- 172.21.15.19:0/4113092313 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3b34068aa0 0x7f3b3414f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:08.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.049+0000 7f3b39f0b640 1 --2- 172.21.15.19:0/4113092313 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3b340c7c50 0x7f3b34143770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:08.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.049+0000 7f3b38f09640 1 -- 172.21.15.19:0/4113092313 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3b340c7c50 msgr2=0x7f3b34143770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:08.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.049+0000 7f3b38f09640 1 --2- 172.21.15.19:0/4113092313 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3b340c7c50 0x7f3b34143770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:08.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.049+0000 7f3b38f09640 1 -- 172.21.15.19:0/4113092313 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3b34068aa0 msgr2=0x7f3b3414f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:08.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.049+0000 7f3b38f09640 1 --2- 172.21.15.19:0/4113092313 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3b34068aa0 0x7f3b3414f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:08.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.049+0000 7f3b38f09640 1 -- 172.21.15.19:0/4113092313 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b3414e120 con 0x7f3b340b2da0 2024-08-31T12:19:08.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.050+0000 7f3b38f09640 1 --2- 172.21.15.19:0/4113092313 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b340b2da0 0x7f3b3414b060 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3b24009850 tx=0x7f3b240039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:08.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.050+0000 7f3b29ffb640 1 -- 172.21.15.19:0/4113092313 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b2403e030 con 0x7f3b340b2da0 2024-08-31T12:19:08.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.050+0000 7f3b29ffb640 1 -- 172.21.15.19:0/4113092313 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3b240047a0 con 0x7f3b340b2da0 2024-08-31T12:19:08.053 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.050+0000 7f3b3b96e640 1 -- 172.21.15.19:0/4113092313 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b3414c660 con 0x7f3b340b2da0 2024-08-31T12:19:08.053 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.050+0000 7f3b29ffb640 1 -- 172.21.15.19:0/4113092313 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b24035040 con 0x7f3b340b2da0 2024-08-31T12:19:08.053 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.050+0000 7f3b3b96e640 1 -- 172.21.15.19:0/4113092313 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b340cb2a0 con 0x7f3b340b2da0 2024-08-31T12:19:08.054 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.051+0000 7f3b3b96e640 1 --2- 172.21.15.19:0/4113092313 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b340cbaa0 0x7f3b34143cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:08.054 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.051+0000 7f3b3b96e640 1 -- 172.21.15.19:0/4113092313 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f3b341efe00 con 0x7f3b340cbaa0 2024-08-31T12:19:08.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.051+0000 7f3b3970a640 1 --2- 172.21.15.19:0/4113092313 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b340cbaa0 0x7f3b34143cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:08.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.052+0000 7f3b3970a640 1 -- 172.21.15.19:0/4113092313 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3b34158590 con 0x7f3b340cbaa0 2024-08-31T12:19:08.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.052+0000 7f3b3970a640 1 --2- 172.21.15.19:0/4113092313 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b340cbaa0 0x7f3b34143cb0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3b3000f9d0 tx=0x7f3b3000fe90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:08.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.052+0000 7f3b29ffb640 1 -- 172.21.15.19:0/4113092313 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b30005020 con 0x7f3b340cbaa0 2024-08-31T12:19:08.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.052+0000 7f3b29ffb640 1 -- 172.21.15.19:0/4113092313 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f3b24041030 con 0x7f3b340b2da0 2024-08-31T12:19:08.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.053+0000 7f3b29ffb640 1 --2- 172.21.15.19:0/4113092313 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3b18077950 0x7f3b18079e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:08.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.053+0000 7f3b29ffb640 1 -- 172.21.15.19:0/4113092313 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f3b240b9f60 con 0x7f3b340b2da0 2024-08-31T12:19:08.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.053+0000 7f3b29ffb640 1 -- 172.21.15.19:0/4113092313 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3b300051c0 con 0x7f3b340cbaa0 2024-08-31T12:19:08.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.053+0000 7f3b29ffb640 1 -- 172.21.15.19:0/4113092313 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b340cbaa0 msgr2=0x7f3b34143cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:08.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.053+0000 7f3b29ffb640 1 --2- 172.21.15.19:0/4113092313 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b340cbaa0 0x7f3b34143cb0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3b3000f9d0 tx=0x7f3b3000fe90 comp rx=0 tx=0).stop 2024-08-31T12:19:08.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.053+0000 7f3b29ffb640 1 -- 172.21.15.19:0/4113092313 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b340cbaa0 msgr2=0x7f3b34143cb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:19:08.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.053+0000 7f3b29ffb640 1 --2- 172.21.15.19:0/4113092313 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b340cbaa0 0x7f3b34143cb0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:08.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.053+0000 7f3b39f0b640 1 --2- 172.21.15.19:0/4113092313 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3b18077950 0x7f3b18079e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:08.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.054+0000 7f3b39f0b640 1 --2- 172.21.15.19:0/4113092313 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3b18077950 0x7f3b18079e60 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3b3406d1e0 tx=0x7f3b2c00d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:08.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.062+0000 7f3b3b96e640 1 --2- 172.21.15.19:0/4113092313 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b340cc9a0 0x7f3b341f2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:08.064 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.062+0000 7f3b3b96e640 1 -- 172.21.15.19:0/4113092313 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f3b340cae20 con 0x7f3b340cc9a0 2024-08-31T12:19:08.064 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.063+0000 7f3b3970a640 1 --2- 172.21.15.19:0/4113092313 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b340cc9a0 0x7f3b341f2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:08.064 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.063+0000 7f3b3970a640 1 -- 172.21.15.19:0/4113092313 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3b34151ec0 con 0x7f3b340cc9a0 2024-08-31T12:19:08.064 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.063+0000 7f3b3970a640 1 --2- 172.21.15.19:0/4113092313 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b340cc9a0 0x7f3b341f2d30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3b30019000 tx=0x7f3b300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:08.065 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.063+0000 7f3b29ffb640 1 -- 172.21.15.19:0/4113092313 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b3001f880 con 0x7f3b340cc9a0 2024-08-31T12:19:08.065 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.064+0000 7f3b29ffb640 1 -- 172.21.15.19:0/4113092313 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5045 (secure 0 0 0) 0x7f3b30018440 con 0x7f3b340cc9a0 2024-08-31T12:19:08.065 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.064+0000 7f3b29ffb640 1 -- 172.21.15.19:0/4113092313 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b340cc9a0 msgr2=0x7f3b341f2d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:08.065 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.064+0000 7f3b29ffb640 1 --2- 172.21.15.19:0/4113092313 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b340cc9a0 0x7f3b341f2d30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3b30019000 tx=0x7f3b300046c0 comp rx=0 tx=0).stop 2024-08-31T12:19:08.065 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.064+0000 7f3b29ffb640 1 -- 172.21.15.19:0/4113092313 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b340cc9a0 msgr2=0x7f3b341f2d30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:19:08.065 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.064+0000 7f3b29ffb640 1 --2- 172.21.15.19:0/4113092313 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b340cc9a0 0x7f3b341f2d30 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:08.065 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:19:08.065 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:19:08.066 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:19:08.066 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "leader", 2024-08-31T12:19:08.066 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 26, 2024-08-31T12:19:08.066 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 128054, 2024-08-31T12:19:08.066 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:19:08.066 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:19:08.066 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:19:08.066 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:19:08.066 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:08.066 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 13, 2024-08-31T12:19:08.066 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:19:08.066 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:19:08.066 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:19:08.067 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:19:08.067 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:19:08.067 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:19:08.067 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:19:08.067 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:19:08.067 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:19:08.067 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:19:08.067 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:19:08.067 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:19:08.067 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:19:08.067 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:19:08.067 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:08.067 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:19:08.068 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:19:08.068 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:19:08.068 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:19:08.068 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:19:08.068 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:19:08.068 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:19:08.068 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:19:08.068 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:19:08.068 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:19:08.068 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:19:08.068 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:19:08.068 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:19:08.068 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:08.069 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:08.069 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:19:08.069 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:19:08.069 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:19:08.069 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:19:08.069 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:19:08.069 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:19:08.069 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:19:08.069 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:19:08.069 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:19:08.069 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:19:08.069 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:19:08.069 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:19:08.070 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:19:08.070 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:19:08.070 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:19:08.070 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:19:08.070 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:19:08.070 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:19:08.070 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:19:08.070 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:19:08.070 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:19:08.070 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:19:08.070 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:19:08.070 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:19:08.070 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:19:08.070 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:19:08.071 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:19:08.071 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:19:08.071 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:08.071 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:19:08.071 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:08.071 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:19:08.071 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:08.071 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:19:08.071 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:19:08.071 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:19:08.071 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:19:08.071 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:08.071 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:19:08.071 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:19:08.072 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:08.072 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:08.072 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:08.072 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:19:08.072 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:19:08.072 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:08.072 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:08.072 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:08.072 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:08.072 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:19:08.072 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:19:08.072 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:19:08.072 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:19:08.073 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:19:08.073 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:08.073 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:08.073 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:19:08.073 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:19:08.073 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:19:08.073 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:19:08.073 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:08.073 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:19:08.073 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:19:08.073 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:08.073 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:08.073 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:08.074 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:19:08.074 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:19:08.074 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:08.074 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:08.074 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:08.074 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:08.074 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:19:08.074 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:19:08.074 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:19:08.074 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:19:08.074 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:19:08.074 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:08.074 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:08.074 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:19:08.075 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:19:08.075 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:19:08.075 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:19:08.075 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:08.075 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:19:08.075 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:19:08.075 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:08.075 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:08.075 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:08.075 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:19:08.075 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:19:08.075 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:08.075 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:08.076 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:08.076 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:08.076 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:19:08.076 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:19:08.076 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:19:08.076 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:19:08.076 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:19:08.076 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:08.076 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:08.076 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:08.076 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:19:08.076 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:19:08.076 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:08.077 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:19:08.077 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:19:08.077 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:19:08.077 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:08.077 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:08.077 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:19:08.077 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:08.077 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:19:08.077 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:19:08.077 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 5 2024-08-31T12:19:08.077 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:08.077 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:08.078 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:19:08.078 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:08.078 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:19:08.078 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:19:08.078 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 8 2024-08-31T12:19:08.078 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:08.078 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:08.078 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:19:08.078 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:08.078 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:19:08.078 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:19:08.078 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 6 2024-08-31T12:19:08.078 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:08.079 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:08.079 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:19:08.079 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:08.079 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:19:08.079 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:19:08.079 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:19:08.079 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:08.079 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:08.079 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:08.079 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:19:08.079 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:19:08.081 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.064+0000 7f3b3b96e640 1 -- 172.21.15.19:0/4113092313 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3b18077950 msgr2=0x7f3b18079e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:08.081 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.064+0000 7f3b3b96e640 1 --2- 172.21.15.19:0/4113092313 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3b18077950 0x7f3b18079e60 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3b3406d1e0 tx=0x7f3b2c00d040 comp rx=0 tx=0).stop 2024-08-31T12:19:08.081 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.064+0000 7f3b3b96e640 1 -- 172.21.15.19:0/4113092313 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b340b2da0 msgr2=0x7f3b3414b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:08.081 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.064+0000 7f3b3b96e640 1 --2- 172.21.15.19:0/4113092313 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b340b2da0 0x7f3b3414b060 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3b24009850 tx=0x7f3b240039e0 comp rx=0 tx=0).stop 2024-08-31T12:19:08.081 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.064+0000 7f3b39f0b640 1 -- 172.21.15.19:0/4113092313 reap_dead start 2024-08-31T12:19:08.081 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.064+0000 7f3b3b96e640 1 -- 172.21.15.19:0/4113092313 shutdown_connections 2024-08-31T12:19:08.081 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.064+0000 7f3b3b96e640 1 --2- 172.21.15.19:0/4113092313 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b340b2da0 0x7f3b3414b060 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:08.081 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.064+0000 7f3b3b96e640 1 -- 172.21.15.19:0/4113092313 >> 172.21.15.19:0/4113092313 conn(0x7f3b3406cae0 msgr2=0x7f3b34070bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:19:08.081 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.064+0000 7f3b3b96e640 1 -- 172.21.15.19:0/4113092313 shutdown_connections 2024-08-31T12:19:08.082 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.064+0000 7f3b3b96e640 1 -- 172.21.15.19:0/4113092313 wait complete. 2024-08-31T12:19:08.082 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:19:08.082 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-08-31T12:19:08.082 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-31T12:19:08.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.210+0000 7faaf501d640 1 -- 172.21.15.19:0/3696422152 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faaf0157d80 msgr2=0x7faaf0158190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:08.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.210+0000 7faaf501d640 1 --2- 172.21.15.19:0/3696422152 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faaf0157d80 0x7faaf0158190 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7faae4005fd0 tx=0x7faae402a5d0 comp rx=0 tx=0).stop 2024-08-31T12:19:08.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.210+0000 7faaf501d640 1 -- 172.21.15.19:0/3696422152 shutdown_connections 2024-08-31T12:19:08.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.210+0000 7faaf501d640 1 --2- 172.21.15.19:0/3696422152 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faaf0157d80 0x7faaf0158190 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:08.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.210+0000 7faaf501d640 1 --2- 172.21.15.19:0/3696422152 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faaf0141290 0x7faaf0140ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:08.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.210+0000 7faaf501d640 1 -- 172.21.15.19:0/3696422152 >> 172.21.15.19:0/3696422152 conn(0x7faaf00cc950 msgr2=0x7faaf00ccd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:19:08.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.210+0000 7faaf501d640 1 -- 172.21.15.19:0/3696422152 shutdown_connections 2024-08-31T12:19:08.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.210+0000 7faaf501d640 1 -- 172.21.15.19:0/3696422152 wait complete. 2024-08-31T12:19:08.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.212+0000 7faaf501d640 1 Processor -- start 2024-08-31T12:19:08.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.212+0000 7faaf501d640 1 -- start start 2024-08-31T12:19:08.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.214+0000 7faaf501d640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faaf006ced0 0x7faaf01f1c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:08.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.214+0000 7faaf501d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faaf0141290 0x7faaf013e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:08.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.214+0000 7faaf501d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faaf0157d80 0x7faaf01e55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:08.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.214+0000 7faaf501d640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7faaf00c6520 con 0x7faaf0157d80 2024-08-31T12:19:08.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.214+0000 7faaf501d640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7faaf00c6240 con 0x7faaf006ced0 2024-08-31T12:19:08.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.214+0000 7faaf501d640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7faaf00c7550 con 0x7faaf0141290 2024-08-31T12:19:08.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.214+0000 7faaef59e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faaf0157d80 0x7faaf01e55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:08.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.214+0000 7faaef59e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faaf0157d80 0x7faaf01e55c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:44458/0 (socket says 172.21.15.19:44458) 2024-08-31T12:19:08.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.214+0000 7faaef59e640 1 -- 172.21.15.19:0/1710982369 learned_addr learned my addr 172.21.15.19:0/1710982369 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:19:08.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.214+0000 7faaee59c640 1 --2- 172.21.15.19:0/1710982369 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faaf0141290 0x7faaf013e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:08.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.214+0000 7faaeed9d640 1 --2- 172.21.15.19:0/1710982369 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faaf006ced0 0x7faaf01f1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:08.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.214+0000 7faaef59e640 1 -- 172.21.15.19:0/1710982369 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faaf0141290 msgr2=0x7faaf013e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:08.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.214+0000 7faaef59e640 1 --2- 172.21.15.19:0/1710982369 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faaf0141290 0x7faaf013e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:08.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.214+0000 7faaef59e640 1 -- 172.21.15.19:0/1710982369 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faaf006ced0 msgr2=0x7faaf01f1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:08.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.214+0000 7faaef59e640 1 --2- 172.21.15.19:0/1710982369 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faaf006ced0 0x7faaf01f1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:08.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.214+0000 7faaef59e640 1 -- 172.21.15.19:0/1710982369 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaf01f2150 con 0x7faaf0157d80 2024-08-31T12:19:08.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.214+0000 7faaeed9d640 1 --2- 172.21.15.19:0/1710982369 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faaf006ced0 0x7faaf01f1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:19:08.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.214+0000 7faaef59e640 1 --2- 172.21.15.19:0/1710982369 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faaf0157d80 0x7faaf01e55c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7faae8002230 tx=0x7faae800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:08.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.215+0000 7faad77fe640 1 -- 172.21.15.19:0/1710982369 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae8003b70 con 0x7faaf0157d80 2024-08-31T12:19:08.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.215+0000 7faad77fe640 1 -- 172.21.15.19:0/1710982369 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faae8003d10 con 0x7faaf0157d80 2024-08-31T12:19:08.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.215+0000 7faaf501d640 1 -- 172.21.15.19:0/1710982369 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faaf00c0ac0 con 0x7faaf0157d80 2024-08-31T12:19:08.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.215+0000 7faad77fe640 1 -- 172.21.15.19:0/1710982369 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae8005330 con 0x7faaf0157d80 2024-08-31T12:19:08.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.215+0000 7faaf501d640 1 -- 172.21.15.19:0/1710982369 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faaf00c1040 con 0x7faaf0157d80 2024-08-31T12:19:08.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.216+0000 7faaf501d640 1 --2- 172.21.15.19:0/1710982369 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faaf01f2320 0x7faaf01e5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:08.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.216+0000 7faaf501d640 1 -- 172.21.15.19:0/1710982369 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7faaf00c5250 con 0x7faaf01f2320 2024-08-31T12:19:08.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.216+0000 7faaeed9d640 1 --2- 172.21.15.19:0/1710982369 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faaf01f2320 0x7faaf01e5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:08.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.216+0000 7faaeed9d640 1 -- 172.21.15.19:0/1710982369 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faaf00c4900 con 0x7faaf01f2320 2024-08-31T12:19:08.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.217+0000 7faaeed9d640 1 --2- 172.21.15.19:0/1710982369 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faaf01f2320 0x7faaf01e5b00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7faae0011c90 tx=0x7faae000f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:08.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.217+0000 7faad77fe640 1 -- 172.21.15.19:0/1710982369 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7faae80056e0 con 0x7faaf0157d80 2024-08-31T12:19:08.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.217+0000 7faad77fe640 1 --2- 172.21.15.19:0/1710982369 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7faadc077a50 0x7faadc079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:08.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.217+0000 7faad77fe640 1 -- 172.21.15.19:0/1710982369 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae001e070 con 0x7faaf01f2320 2024-08-31T12:19:08.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.217+0000 7faad77fe640 1 -- 172.21.15.19:0/1710982369 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7faae8099540 con 0x7faaf0157d80 2024-08-31T12:19:08.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.217+0000 7faaee59c640 1 --2- 172.21.15.19:0/1710982369 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7faadc077a50 0x7faadc079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:08.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.218+0000 7faad77fe640 1 -- 172.21.15.19:0/1710982369 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7faae000e880 con 0x7faaf01f2320 2024-08-31T12:19:08.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.218+0000 7faad77fe640 1 -- 172.21.15.19:0/1710982369 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faaf01f2320 msgr2=0x7faaf01e5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:08.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.218+0000 7faad77fe640 1 --2- 172.21.15.19:0/1710982369 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faaf01f2320 0x7faaf01e5b00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7faae0011c90 tx=0x7faae000f790 comp rx=0 tx=0).stop 2024-08-31T12:19:08.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.218+0000 7faad77fe640 1 -- 172.21.15.19:0/1710982369 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faaf01f2320 msgr2=0x7faaf01e5b00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:19:08.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.218+0000 7faad77fe640 1 --2- 172.21.15.19:0/1710982369 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faaf01f2320 0x7faaf01e5b00 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7faae0011c90 tx=0x7faae000f790 comp rx=0 tx=0).stop 2024-08-31T12:19:08.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.218+0000 7faaee59c640 1 --2- 172.21.15.19:0/1710982369 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7faadc077a50 0x7faadc079f60 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7faae4033000 tx=0x7faae4004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:08.225 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.224+0000 7faaf501d640 1 --2- 172.21.15.19:0/1710982369 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faaf00c1210 0x7faaf00c1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:08.225 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.224+0000 7faaf501d640 1 -- 172.21.15.19:0/1710982369 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7faaf00c5cb0 con 0x7faaf00c1210 2024-08-31T12:19:08.225 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.224+0000 7faaeed9d640 1 --2- 172.21.15.19:0/1710982369 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faaf00c1210 0x7faaf00c1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:08.226 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.225+0000 7faaeed9d640 1 -- 172.21.15.19:0/1710982369 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7faaf015c020 con 0x7faaf00c1210 2024-08-31T12:19:08.226 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.225+0000 7faaeed9d640 1 --2- 172.21.15.19:0/1710982369 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faaf00c1210 0x7faaf00c1da0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7faae0011c90 tx=0x7faae00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:08.226 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.225+0000 7faad77fe640 1 -- 172.21.15.19:0/1710982369 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae001e040 con 0x7faaf00c1210 2024-08-31T12:19:08.227 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.226+0000 7faad77fe640 1 -- 172.21.15.19:0/1710982369 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4541 (secure 0 0 0) 0x7faae0019430 con 0x7faaf00c1210 2024-08-31T12:19:08.227 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.226+0000 7faad77fe640 1 -- 172.21.15.19:0/1710982369 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faaf00c1210 msgr2=0x7faaf00c1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:08.227 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.226+0000 7faad77fe640 1 --2- 172.21.15.19:0/1710982369 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faaf00c1210 0x7faaf00c1da0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7faae0011c90 tx=0x7faae00046c0 comp rx=0 tx=0).stop 2024-08-31T12:19:08.227 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.226+0000 7faad77fe640 1 -- 172.21.15.19:0/1710982369 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faaf00c1210 msgr2=0x7faaf00c1da0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:19:08.227 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.226+0000 7faad77fe640 1 --2- 172.21.15.19:0/1710982369 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faaf00c1210 0x7faaf00c1da0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:08.227 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:19:08.227 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:19:08.227 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:19:08.227 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:19:08.228 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 26, 2024-08-31T12:19:08.228 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 18911, 2024-08-31T12:19:08.228 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:19:08.228 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:19:08.228 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:19:08.228 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:19:08.228 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:08.228 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 13, 2024-08-31T12:19:08.228 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:19:08.228 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:19:08.228 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:19:08.228 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:19:08.229 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:19:08.229 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:19:08.229 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:19:08.229 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:19:08.229 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:19:08.229 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:19:08.229 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:19:08.229 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:19:08.229 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:19:08.229 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:19:08.229 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:08.229 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:19:08.230 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:19:08.230 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:19:08.230 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:19:08.230 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:19:08.230 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:19:08.230 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:19:08.230 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:19:08.230 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:19:08.230 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:19:08.230 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:19:08.230 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:19:08.230 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:19:08.230 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:08.230 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:08.231 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:19:08.231 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:19:08.231 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:19:08.231 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:19:08.231 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:19:08.231 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:19:08.231 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:19:08.231 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:19:08.231 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:19:08.231 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:19:08.231 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:19:08.231 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:19:08.231 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:19:08.232 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:19:08.232 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:19:08.232 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:19:08.232 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:19:08.232 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:19:08.232 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:19:08.232 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:19:08.232 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:19:08.232 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:19:08.232 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:19:08.232 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:19:08.232 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:19:08.232 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:19:08.233 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:19:08.233 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:19:08.233 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:08.233 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:19:08.233 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:08.233 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:19:08.233 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:08.233 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:19:08.233 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:19:08.233 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:19:08.233 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:19:08.233 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:08.234 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:19:08.234 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:19:08.234 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:08.234 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:08.234 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:08.234 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:19:08.234 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:19:08.234 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:08.234 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:08.234 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:08.234 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:08.234 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:19:08.234 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:19:08.234 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:19:08.235 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:19:08.235 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:19:08.235 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:08.235 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:08.235 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:19:08.235 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:19:08.235 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:19:08.235 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:19:08.235 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:08.235 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:19:08.235 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:19:08.235 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:08.235 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:08.235 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:08.236 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:19:08.236 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:19:08.236 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:08.236 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:08.236 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:08.236 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:08.236 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:19:08.236 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:19:08.236 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:19:08.236 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:19:08.236 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:19:08.236 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:08.236 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:08.237 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:19:08.237 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:19:08.237 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:19:08.237 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:19:08.237 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:08.237 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:19:08.237 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:19:08.237 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:08.237 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:08.237 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:08.237 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:19:08.237 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:19:08.237 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:08.238 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:08.238 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:08.238 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:08.238 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:19:08.238 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:19:08.238 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:19:08.238 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:19:08.238 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:19:08.238 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:08.238 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:08.238 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:08.238 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:19:08.238 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:19:08.239 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:08.239 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:19:08.239 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:19:08.239 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:19:08.239 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:08.239 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:08.239 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:19:08.239 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:08.239 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:19:08.239 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:19:08.239 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:19:08.239 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:08.239 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:08.239 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:08.240 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:19:08.240 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:19:08.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.226+0000 7faaf501d640 1 -- 172.21.15.19:0/1710982369 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7faadc077a50 msgr2=0x7faadc079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:08.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.226+0000 7faaf501d640 1 --2- 172.21.15.19:0/1710982369 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7faadc077a50 0x7faadc079f60 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7faae4033000 tx=0x7faae4004730 comp rx=0 tx=0).stop 2024-08-31T12:19:08.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.226+0000 7faaf501d640 1 -- 172.21.15.19:0/1710982369 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faaf0157d80 msgr2=0x7faaf01e55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:08.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.226+0000 7faaf501d640 1 --2- 172.21.15.19:0/1710982369 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faaf0157d80 0x7faaf01e55c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7faae8002230 tx=0x7faae800cb90 comp rx=0 tx=0).stop 2024-08-31T12:19:08.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.226+0000 7faaef59e640 1 -- 172.21.15.19:0/1710982369 reap_dead start 2024-08-31T12:19:08.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.226+0000 7faaef59e640 1 -- 172.21.15.19:0/1710982369 reap_dead start 2024-08-31T12:19:08.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.226+0000 7faaf501d640 1 -- 172.21.15.19:0/1710982369 shutdown_connections 2024-08-31T12:19:08.242 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.226+0000 7faaf501d640 1 -- 172.21.15.19:0/1710982369 >> 172.21.15.19:0/1710982369 conn(0x7faaf00cc950 msgr2=0x7faaf00cbe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:19:08.242 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.226+0000 7faaf501d640 1 -- 172.21.15.19:0/1710982369 shutdown_connections 2024-08-31T12:19:08.242 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.226+0000 7faaf501d640 1 -- 172.21.15.19:0/1710982369 wait complete. 2024-08-31T12:19:08.242 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:19:08.242 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-08-31T12:19:08.242 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-31T12:19:08.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.363+0000 7fcdaa713640 1 -- 172.21.15.19:0/1573636493 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcda40c9030 msgr2=0x7fcda40cb040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:08.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.363+0000 7fcdaa713640 1 --2- 172.21.15.19:0/1573636493 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcda40c9030 0x7fcda40cb040 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fcd98005fd0 tx=0x7fcd9802a620 comp rx=0 tx=0).stop 2024-08-31T12:19:08.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.364+0000 7fcdaa713640 1 -- 172.21.15.19:0/1573636493 shutdown_connections 2024-08-31T12:19:08.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.364+0000 7fcdaa713640 1 --2- 172.21.15.19:0/1573636493 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcda40c9030 0x7fcda40cb040 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:08.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.364+0000 7fcdaa713640 1 --2- 172.21.15.19:0/1573636493 >> v2:172.21.15.43:3300/0 conn(0x7fcda4153190 0x7fcda4159820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:08.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.364+0000 7fcdaa713640 1 -- 172.21.15.19:0/1573636493 >> 172.21.15.19:0/1573636493 conn(0x7fcda40728b0 msgr2=0x7fcda4072cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:19:08.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.364+0000 7fcdaa713640 1 -- 172.21.15.19:0/1573636493 shutdown_connections 2024-08-31T12:19:08.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.364+0000 7fcdaa713640 1 -- 172.21.15.19:0/1573636493 wait complete. 2024-08-31T12:19:08.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.365+0000 7fcdaa713640 1 Processor -- start 2024-08-31T12:19:08.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.366+0000 7fcdaa713640 1 -- start start 2024-08-31T12:19:08.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.367+0000 7fcdaa713640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcda40c9030 0x7fcda41efac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:08.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.367+0000 7fcdaa713640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcda40c9440 0x7fcda4146260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:08.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.367+0000 7fcdaa713640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcda4153190 0x7fcda415b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:08.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.367+0000 7fcdaa713640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fcda41535a0 con 0x7fcda4153190 2024-08-31T12:19:08.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.367+0000 7fcdaa713640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fcda40c4db0 con 0x7fcda40c9030 2024-08-31T12:19:08.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.367+0000 7fcdaa713640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fcda415cdc0 con 0x7fcda40c9440 2024-08-31T12:19:08.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.367+0000 7fcda8cb0640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcda4153190 0x7fcda415b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:08.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.367+0000 7fcda3fff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcda40c9030 0x7fcda41efac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:08.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.367+0000 7fcda8cb0640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcda4153190 0x7fcda415b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:44480/0 (socket says 172.21.15.19:44480) 2024-08-31T12:19:08.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.367+0000 7fcda8cb0640 1 -- 172.21.15.19:0/3768992562 learned_addr learned my addr 172.21.15.19:0/3768992562 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:19:08.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.367+0000 7fcda37fe640 1 --2- 172.21.15.19:0/3768992562 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcda40c9440 0x7fcda4146260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:08.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.367+0000 7fcda3fff640 1 -- 172.21.15.19:0/3768992562 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcda40c9440 msgr2=0x7fcda4146260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:08.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.367+0000 7fcda3fff640 1 --2- 172.21.15.19:0/3768992562 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcda40c9440 0x7fcda4146260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:08.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.367+0000 7fcda3fff640 1 -- 172.21.15.19:0/3768992562 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcda4153190 msgr2=0x7fcda415b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:08.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.367+0000 7fcda3fff640 1 --2- 172.21.15.19:0/3768992562 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcda4153190 0x7fcda415b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:08.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.367+0000 7fcda3fff640 1 -- 172.21.15.19:0/3768992562 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcda41f0000 con 0x7fcda40c9030 2024-08-31T12:19:08.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.367+0000 7fcda8cb0640 1 --2- 172.21.15.19:0/3768992562 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcda4153190 0x7fcda415b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:19:08.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.368+0000 7fcda3fff640 1 --2- 172.21.15.19:0/3768992562 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcda40c9030 0x7fcda41efac0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fcd9400e750 tx=0x7fcd9400ec10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:08.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.368+0000 7fcda0ff9640 1 -- 172.21.15.19:0/3768992562 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd9400cc60 con 0x7fcda40c9030 2024-08-31T12:19:08.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.368+0000 7fcda0ff9640 1 -- 172.21.15.19:0/3768992562 <== mon.1 v2:172.21.15.43:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcd9400ee60 con 0x7fcda40c9030 2024-08-31T12:19:08.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.368+0000 7fcdaa713640 1 -- 172.21.15.19:0/3768992562 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcda414fe60 con 0x7fcda40c9030 2024-08-31T12:19:08.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.368+0000 7fcda0ff9640 1 -- 172.21.15.19:0/3768992562 <== mon.1 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd94004260 con 0x7fcda40c9030 2024-08-31T12:19:08.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.368+0000 7fcdaa713640 1 -- 172.21.15.19:0/3768992562 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcda40c8110 con 0x7fcda40c9030 2024-08-31T12:19:08.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.369+0000 7fcdaa713640 1 --2- 172.21.15.19:0/3768992562 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcda40c8630 0x7fcda415b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:08.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.369+0000 7fcdaa713640 1 -- 172.21.15.19:0/3768992562 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fcda414fb60 con 0x7fcda40c8630 2024-08-31T12:19:08.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.369+0000 7fcda37fe640 1 --2- 172.21.15.19:0/3768992562 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcda40c8630 0x7fcda415b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:08.373 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.370+0000 7fcda37fe640 1 -- 172.21.15.19:0/3768992562 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcda40ca790 con 0x7fcda40c8630 2024-08-31T12:19:08.373 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.370+0000 7fcda37fe640 1 --2- 172.21.15.19:0/3768992562 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcda40c8630 0x7fcda415b630 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcda40c5700 tx=0x7fcd98005340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:08.373 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.370+0000 7fcda0ff9640 1 -- 172.21.15.19:0/3768992562 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd98040070 con 0x7fcda40c8630 2024-08-31T12:19:08.373 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.370+0000 7fcda0ff9640 1 -- 172.21.15.19:0/3768992562 <== mon.1 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fcd940035f0 con 0x7fcda40c9030 2024-08-31T12:19:08.373 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.371+0000 7fcda0ff9640 1 --2- 172.21.15.19:0/3768992562 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fcd90077980 0x7fcd90079e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:08.373 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.371+0000 7fcda0ff9640 1 -- 172.21.15.19:0/3768992562 <== mon.1 v2:172.21.15.43:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fcd94099c70 con 0x7fcda40c9030 2024-08-31T12:19:08.373 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.371+0000 7fcda8cb0640 1 --2- 172.21.15.19:0/3768992562 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fcd90077980 0x7fcd90079e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:08.373 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.371+0000 7fcda0ff9640 1 -- 172.21.15.19:0/3768992562 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcd980049a0 con 0x7fcda40c8630 2024-08-31T12:19:08.373 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.371+0000 7fcda0ff9640 1 -- 172.21.15.19:0/3768992562 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcda40c8630 msgr2=0x7fcda415b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:08.373 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.371+0000 7fcda0ff9640 1 --2- 172.21.15.19:0/3768992562 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcda40c8630 0x7fcda415b630 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcda40c5700 tx=0x7fcd98005340 comp rx=0 tx=0).stop 2024-08-31T12:19:08.373 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.371+0000 7fcda0ff9640 1 -- 172.21.15.19:0/3768992562 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcda40c8630 msgr2=0x7fcda415b630 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:19:08.373 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.371+0000 7fcda0ff9640 1 --2- 172.21.15.19:0/3768992562 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcda40c8630 0x7fcda415b630 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:08.374 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.372+0000 7fcda8cb0640 1 --2- 172.21.15.19:0/3768992562 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fcd90077980 0x7fcd90079e90 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fcda4159740 tx=0x7fcd9c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:08.381 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.381+0000 7fcdaa713640 1 --2- 172.21.15.19:0/3768992562 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcda415bdf0 0x7fcda41f3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:08.381 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.381+0000 7fcdaa713640 1 -- 172.21.15.19:0/3768992562 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fcda4159820 con 0x7fcda415bdf0 2024-08-31T12:19:08.382 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.381+0000 7fcda37fe640 1 --2- 172.21.15.19:0/3768992562 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcda415bdf0 0x7fcda41f3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:08.382 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.381+0000 7fcda37fe640 1 -- 172.21.15.19:0/3768992562 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcda40c9a20 con 0x7fcda415bdf0 2024-08-31T12:19:08.382 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.382+0000 7fcda37fe640 1 --2- 172.21.15.19:0/3768992562 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcda415bdf0 0x7fcda41f3030 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcd98009850 tx=0x7fcd98009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:08.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.382+0000 7fcda0ff9640 1 -- 172.21.15.19:0/3768992562 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd98035040 con 0x7fcda415bdf0 2024-08-31T12:19:08.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.382+0000 7fcda0ff9640 1 -- 172.21.15.19:0/3768992562 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4708 (secure 0 0 0) 0x7fcd9803c440 con 0x7fcda415bdf0 2024-08-31T12:19:08.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.382+0000 7fcda0ff9640 1 -- 172.21.15.19:0/3768992562 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcda415bdf0 msgr2=0x7fcda41f3030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:08.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.382+0000 7fcda0ff9640 1 --2- 172.21.15.19:0/3768992562 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcda415bdf0 0x7fcda41f3030 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcd98009850 tx=0x7fcd98009260 comp rx=0 tx=0).stop 2024-08-31T12:19:08.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.382+0000 7fcda0ff9640 1 -- 172.21.15.19:0/3768992562 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcda415bdf0 msgr2=0x7fcda41f3030 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:19:08.384 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.382+0000 7fcda0ff9640 1 --2- 172.21.15.19:0/3768992562 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcda415bdf0 0x7fcda41f3030 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:08.384 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:19:08.384 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:19:08.384 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:19:08.384 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:19:08.384 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 26, 2024-08-31T12:19:08.384 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 76504, 2024-08-31T12:19:08.384 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:19:08.384 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:19:08.384 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:19:08.385 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:19:08.385 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:08.385 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 14, 2024-08-31T12:19:08.385 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:19:08.385 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:19:08.385 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:19:08.385 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:19:08.385 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:19:08.385 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:19:08.385 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:19:08.385 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:19:08.385 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:19:08.386 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:19:08.386 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:19:08.386 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:19:08.386 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:19:08.386 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:19:08.386 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:08.386 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:19:08.386 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:19:08.386 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:19:08.386 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:19:08.386 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:19:08.386 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:19:08.386 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:19:08.387 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:19:08.387 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:19:08.387 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:19:08.387 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:19:08.387 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:19:08.387 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:19:08.387 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:08.387 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:08.387 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:19:08.387 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:19:08.387 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:19:08.387 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:19:08.387 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:19:08.387 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:19:08.388 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:19:08.388 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:19:08.388 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:19:08.388 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:19:08.388 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:19:08.388 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:19:08.388 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:19:08.388 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:19:08.388 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:19:08.388 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:19:08.388 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:19:08.388 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:19:08.388 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:19:08.389 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:19:08.389 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:19:08.389 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:19:08.389 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:19:08.389 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:19:08.389 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:19:08.389 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:19:08.389 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:19:08.389 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:19:08.389 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:08.389 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:19:08.389 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:08.389 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:19:08.389 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:08.390 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:19:08.390 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:19:08.390 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:19:08.390 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:19:08.390 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:08.390 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:19:08.390 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:19:08.390 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:08.390 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:08.390 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:08.390 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:19:08.390 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:19:08.391 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:08.391 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:08.391 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:08.391 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:08.391 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:19:08.391 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:19:08.391 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:19:08.391 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:19:08.391 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:19:08.391 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:08.391 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:08.391 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:19:08.391 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:19:08.392 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:19:08.392 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:19:08.392 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:08.392 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:19:08.392 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:19:08.392 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:08.392 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:08.392 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:08.392 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:19:08.392 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:19:08.392 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:08.392 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:08.392 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:08.392 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:08.393 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:19:08.393 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:19:08.393 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:19:08.393 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:19:08.393 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:19:08.393 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:08.393 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:08.393 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:19:08.393 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:19:08.393 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:19:08.393 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:19:08.393 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:08.393 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:19:08.394 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:19:08.394 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:08.394 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:08.394 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:08.394 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:19:08.394 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:19:08.394 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:08.394 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:08.394 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:08.394 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:08.394 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:19:08.395 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:19:08.395 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:19:08.395 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:19:08.395 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:19:08.395 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:08.395 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:08.395 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:08.395 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:19:08.395 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:19:08.395 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:08.395 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:19:08.395 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:19:08.395 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:19:08.396 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:08.396 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:08.396 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:19:08.396 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:08.396 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:19:08.396 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:19:08.396 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:19:08.396 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:08.396 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:08.396 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:19:08.396 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:08.396 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:19:08.396 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:19:08.397 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:19:08.397 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:08.397 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:08.397 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:08.397 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:19:08.397 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:19:08.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.383+0000 7fcdaa713640 1 -- 172.21.15.19:0/3768992562 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fcd90077980 msgr2=0x7fcd90079e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:08.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.383+0000 7fcdaa713640 1 --2- 172.21.15.19:0/3768992562 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fcd90077980 0x7fcd90079e90 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fcda4159740 tx=0x7fcd9c009040 comp rx=0 tx=0).stop 2024-08-31T12:19:08.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.383+0000 7fcdaa713640 1 -- 172.21.15.19:0/3768992562 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcda40c9030 msgr2=0x7fcda41efac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:08.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.383+0000 7fcdaa713640 1 --2- 172.21.15.19:0/3768992562 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcda40c9030 0x7fcda41efac0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fcd9400e750 tx=0x7fcd9400ec10 comp rx=0 tx=0).stop 2024-08-31T12:19:08.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.383+0000 7fcda8cb0640 1 -- 172.21.15.19:0/3768992562 reap_dead start 2024-08-31T12:19:08.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.383+0000 7fcda8cb0640 1 -- 172.21.15.19:0/3768992562 reap_dead start 2024-08-31T12:19:08.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.383+0000 7fcdaa713640 1 -- 172.21.15.19:0/3768992562 shutdown_connections 2024-08-31T12:19:08.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.383+0000 7fcdaa713640 1 -- 172.21.15.19:0/3768992562 >> 172.21.15.19:0/3768992562 conn(0x7fcda40728b0 msgr2=0x7fcda4072cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:19:08.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.383+0000 7fcdaa713640 1 -- 172.21.15.19:0/3768992562 shutdown_connections 2024-08-31T12:19:08.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.383+0000 7fcdaa713640 1 -- 172.21.15.19:0/3768992562 wait complete. 2024-08-31T12:19:08.399 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:19:08.400 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-08-31T12:19:08.400 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-08-31T12:19:08.400 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-08-31T12:19:08.400 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-08-31T12:19:08.400 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-08-31T12:19:08.400 INFO:teuthology.orchestra.run:waiting for 300 2024-08-31T12:19:08.412 INFO:tasks.ceph.mon.b:Stopped 2024-08-31T12:19:08.412 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-08-31T12:19:08.412 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:19:08.537 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.536+0000 7f97a4fdf640 1 -- 172.21.15.19:0/2896366663 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f97a00cb280 msgr2=0x7f97a015b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:08.537 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.536+0000 7f97a4fdf640 1 --2- 172.21.15.19:0/2896366663 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f97a00cb280 0x7f97a015b120 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9794005fd0 tx=0x7f979402a5f0 comp rx=0 tx=0).stop 2024-08-31T12:19:08.537 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.536+0000 7f97a4fdf640 1 -- 172.21.15.19:0/2896366663 shutdown_connections 2024-08-31T12:19:08.537 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.536+0000 7f97a4fdf640 1 --2- 172.21.15.19:0/2896366663 >> v2:172.21.15.43:3300/0 conn(0x7f97a015b660 0x7f97a0165aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:08.537 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.536+0000 7f97a4fdf640 1 --2- 172.21.15.19:0/2896366663 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f97a00cb280 0x7f97a015b120 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:08.538 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.536+0000 7f97a4fdf640 1 -- 172.21.15.19:0/2896366663 >> 172.21.15.19:0/2896366663 conn(0x7f97a0074810 msgr2=0x7f97a0068aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:19:08.538 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.536+0000 7f97a4fdf640 1 -- 172.21.15.19:0/2896366663 shutdown_connections 2024-08-31T12:19:08.538 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.536+0000 7f97a4fdf640 1 -- 172.21.15.19:0/2896366663 wait complete. 2024-08-31T12:19:08.539 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.538+0000 7f97a4fdf640 1 Processor -- start 2024-08-31T12:19:08.539 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.538+0000 7f97a4fdf640 1 -- start start 2024-08-31T12:19:08.542 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.540+0000 7f97a4fdf640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f97a00cb280 0x7f97a00ca4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:08.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.540+0000 7f97a4fdf640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f97a00ccbb0 0x7f97a00cf970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:08.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.540+0000 7f97a4fdf640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f97a015b660 0x7f97a015a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:08.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.540+0000 7f97a4fdf640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f97a015ba70 con 0x7f97a00ccbb0 2024-08-31T12:19:08.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.540+0000 7f97a4fdf640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f97a015f050 con 0x7f97a00cb280 2024-08-31T12:19:08.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.540+0000 7f97a4fdf640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f97a00cc780 con 0x7f97a015b660 2024-08-31T12:19:08.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.540+0000 7f979e59c640 1 -- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f97a00cb280 msgr2=0x7f97a00ca4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3300/0 2024-08-31T12:19:08.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.540+0000 7f979e59c640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f97a00cb280 0x7f97a00ca4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:19:08.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.540+0000 7f979dd9b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f97a00ccbb0 0x7f97a00cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:08.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.540+0000 7f979dd9b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f97a00ccbb0 0x7f97a00cf970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:44496/0 (socket says 172.21.15.19:44496) 2024-08-31T12:19:08.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.540+0000 7f979dd9b640 1 -- 172.21.15.19:0/696326421 learned_addr learned my addr 172.21.15.19:0/696326421 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:19:08.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.540+0000 7f979ed9d640 1 --2- 172.21.15.19:0/696326421 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f97a015b660 0x7f97a015a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:08.544 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.541+0000 7f979dd9b640 1 -- 172.21.15.19:0/696326421 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f97a015b660 msgr2=0x7f97a015a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:08.544 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.541+0000 7f979dd9b640 1 --2- 172.21.15.19:0/696326421 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f97a015b660 0x7f97a015a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:08.544 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.541+0000 7f979dd9b640 1 -- 172.21.15.19:0/696326421 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f97a00cb280 msgr2=0x7f97a00ca4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:19:08.544 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.541+0000 7f979dd9b640 1 --2- 172.21.15.19:0/696326421 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f97a00cb280 0x7f97a00ca4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:08.544 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.541+0000 7f979dd9b640 1 -- 172.21.15.19:0/696326421 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97a0158470 con 0x7f97a00ccbb0 2024-08-31T12:19:08.545 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.541+0000 7f979dd9b640 1 --2- 172.21.15.19:0/696326421 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f97a00ccbb0 0x7f97a00cf970 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9794000c00 tx=0x7f97940039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:08.545 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.541+0000 7f978effd640 1 -- 172.21.15.19:0/696326421 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f979403e030 con 0x7f97a00ccbb0 2024-08-31T12:19:08.545 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.541+0000 7f97a4fdf640 1 -- 172.21.15.19:0/696326421 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97a00cfeb0 con 0x7f97a00ccbb0 2024-08-31T12:19:08.545 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.541+0000 7f978effd640 1 -- 172.21.15.19:0/696326421 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f97940047a0 con 0x7f97a00ccbb0 2024-08-31T12:19:08.545 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.541+0000 7f978effd640 1 -- 172.21.15.19:0/696326421 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9794035040 con 0x7f97a00ccbb0 2024-08-31T12:19:08.545 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.541+0000 7f97a4fdf640 1 -- 172.21.15.19:0/696326421 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97a0158190 con 0x7f97a00ccbb0 2024-08-31T12:19:08.546 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.542+0000 7f97a4fdf640 1 -- 172.21.15.19:0/696326421 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97a0155660 con 0x7f97a00ccbb0 2024-08-31T12:19:08.546 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.543+0000 7f978effd640 1 -- 172.21.15.19:0/696326421 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f9794041030 con 0x7f97a00ccbb0 2024-08-31T12:19:08.547 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.544+0000 7f978effd640 1 --2- 172.21.15.19:0/696326421 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9780077a30 0x7f9780079f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:08.547 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.544+0000 7f978effd640 1 -- 172.21.15.19:0/696326421 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f97940b9eb0 con 0x7f97a00ccbb0 2024-08-31T12:19:08.547 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.544+0000 7f979e59c640 1 --2- 172.21.15.19:0/696326421 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9780077a30 0x7f9780079f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:08.547 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.545+0000 7f979e59c640 1 --2- 172.21.15.19:0/696326421 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9780077a30 0x7f9780079f40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9790006fd0 tx=0x7f9790006d60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:08.550 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.549+0000 7f978effd640 1 -- 172.21.15.19:0/696326421 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f97a0158470 con 0x7f97a00ccbb0 2024-08-31T12:19:08.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.862+0000 7f97a4fdf640 1 -- 172.21.15.19:0/696326421 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f97a0153da0 con 0x7f97a00ccbb0 2024-08-31T12:19:08.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.863+0000 7f978effd640 1 -- 172.21.15.19:0/696326421 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f9794087e20 con 0x7f97a00ccbb0 2024-08-31T12:19:08.864 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:19:08.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.865+0000 7f97a4fdf640 1 -- 172.21.15.19:0/696326421 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9780077a30 msgr2=0x7f9780079f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:08.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.865+0000 7f97a4fdf640 1 --2- 172.21.15.19:0/696326421 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9780077a30 0x7f9780079f40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9790006fd0 tx=0x7f9790006d60 comp rx=0 tx=0).stop 2024-08-31T12:19:08.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.865+0000 7f97a4fdf640 1 -- 172.21.15.19:0/696326421 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f97a00ccbb0 msgr2=0x7f97a00cf970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:08.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.865+0000 7f97a4fdf640 1 --2- 172.21.15.19:0/696326421 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f97a00ccbb0 0x7f97a00cf970 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9794000c00 tx=0x7f97940039e0 comp rx=0 tx=0).stop 2024-08-31T12:19:08.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.865+0000 7f97a4fdf640 1 -- 172.21.15.19:0/696326421 shutdown_connections 2024-08-31T12:19:08.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.865+0000 7f97a4fdf640 1 --2- 172.21.15.19:0/696326421 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9780077a30 0x7f9780079f40 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:08.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.865+0000 7f97a4fdf640 1 --2- 172.21.15.19:0/696326421 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f97a015b660 0x7f97a015a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:08.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.865+0000 7f97a4fdf640 1 --2- 172.21.15.19:0/696326421 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f97a00ccbb0 0x7f97a00cf970 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:08.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.865+0000 7f97a4fdf640 1 --2- 172.21.15.19:0/696326421 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f97a00cb280 0x7f97a00ca4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:08.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.865+0000 7f97a4fdf640 1 -- 172.21.15.19:0/696326421 >> 172.21.15.19:0/696326421 conn(0x7f97a0074810 msgr2=0x7f97a00cc070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:19:08.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.866+0000 7f97a4fdf640 1 -- 172.21.15.19:0/696326421 shutdown_connections 2024-08-31T12:19:08.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:08.866+0000 7f97a4fdf640 1 -- 172.21.15.19:0/696326421 wait complete. 2024-08-31T12:19:11.880 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:19:12.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:11.998+0000 7f6ec8719640 1 -- 172.21.15.19:0/4125930441 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ec00c8050 msgr2=0x7f6ec00caad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:12.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:11.998+0000 7f6ec8719640 1 --2- 172.21.15.19:0/4125930441 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ec00c8050 0x7f6ec00caad0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6ec0073b40 tx=0x7f6ebc02a580 comp rx=0 tx=0).stop 2024-08-31T12:19:12.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:11.999+0000 7f6ec8719640 1 -- 172.21.15.19:0/4125930441 shutdown_connections 2024-08-31T12:19:12.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:11.999+0000 7f6ec8719640 1 --2- 172.21.15.19:0/4125930441 >> v2:172.21.15.43:3300/0 conn(0x7f6ec006e1e0 0x7f6ec0151f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:12.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:11.999+0000 7f6ec8719640 1 --2- 172.21.15.19:0/4125930441 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6ec006ddd0 0x7f6ec01519f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:12.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:11.999+0000 7f6ec8719640 1 --2- 172.21.15.19:0/4125930441 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ec00c8050 0x7f6ec00caad0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:12.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:11.999+0000 7f6ec8719640 1 -- 172.21.15.19:0/4125930441 >> 172.21.15.19:0/4125930441 conn(0x7f6ec00cdcb0 msgr2=0x7f6ec00ca150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:19:12.001 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:11.999+0000 7f6ec8719640 1 -- 172.21.15.19:0/4125930441 shutdown_connections 2024-08-31T12:19:12.001 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:11.999+0000 7f6ec8719640 1 -- 172.21.15.19:0/4125930441 wait complete. 2024-08-31T12:19:12.001 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.001+0000 7f6ec8719640 1 Processor -- start 2024-08-31T12:19:12.001 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.001+0000 7f6ec8719640 1 -- start start 2024-08-31T12:19:12.004 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.002+0000 7f6ec8719640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6ec006ddd0 0x7f6ec0153c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:12.005 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.002+0000 7f6ec8719640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ec006e1e0 0x7f6ec01509f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:12.005 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.002+0000 7f6ec8719640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6ec00c8050 0x7f6ec0145f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:12.005 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.002+0000 7f6ec8719640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec00c7a90 con 0x7f6ec006e1e0 2024-08-31T12:19:12.005 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.002+0000 7f6ec8719640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec00c7910 con 0x7f6ec00c8050 2024-08-31T12:19:12.005 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.002+0000 7f6ec8719640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f6ec00c68c0 con 0x7f6ec006ddd0 2024-08-31T12:19:12.005 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.002+0000 7f6ec6cb6640 1 -- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6ec00c8050 msgr2=0x7f6ec0145f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3300/0 2024-08-31T12:19:12.005 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.002+0000 7f6ec5cb4640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ec006e1e0 0x7f6ec01509f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:12.005 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.002+0000 7f6ec6cb6640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6ec00c8050 0x7f6ec0145f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:19:12.005 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.002+0000 7f6ec5cb4640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ec006e1e0 0x7f6ec01509f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:44512/0 (socket says 172.21.15.19:44512) 2024-08-31T12:19:12.005 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.002+0000 7f6ec5cb4640 1 -- 172.21.15.19:0/1450140097 learned_addr learned my addr 172.21.15.19:0/1450140097 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:19:12.006 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.002+0000 7f6ec64b5640 1 --2- 172.21.15.19:0/1450140097 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6ec006ddd0 0x7f6ec0153c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:12.006 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.003+0000 7f6ec5cb4640 1 -- 172.21.15.19:0/1450140097 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6ec006ddd0 msgr2=0x7f6ec0153c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:12.006 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.003+0000 7f6ec5cb4640 1 --2- 172.21.15.19:0/1450140097 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6ec006ddd0 0x7f6ec0153c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:12.006 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.003+0000 7f6ec5cb4640 1 -- 172.21.15.19:0/1450140097 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6ec00c8050 msgr2=0x7f6ec0145f10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:19:12.006 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.003+0000 7f6ec5cb4640 1 --2- 172.21.15.19:0/1450140097 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6ec00c8050 0x7f6ec0145f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:12.006 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.003+0000 7f6ec5cb4640 1 -- 172.21.15.19:0/1450140097 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ec01541b0 con 0x7f6ec006e1e0 2024-08-31T12:19:12.006 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.003+0000 7f6ec5cb4640 1 --2- 172.21.15.19:0/1450140097 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ec006e1e0 0x7f6ec01509f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6eb000d780 tx=0x7f6eb000dc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:12.006 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.003+0000 7f6eb6ffd640 1 -- 172.21.15.19:0/1450140097 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb0017070 con 0x7f6ec006e1e0 2024-08-31T12:19:12.006 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.003+0000 7f6eb6ffd640 1 -- 172.21.15.19:0/1450140097 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6eb000de90 con 0x7f6ec006e1e0 2024-08-31T12:19:12.006 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.003+0000 7f6ec8719640 1 -- 172.21.15.19:0/1450140097 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ec0154810 con 0x7f6ec006e1e0 2024-08-31T12:19:12.006 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.003+0000 7f6eb6ffd640 1 -- 172.21.15.19:0/1450140097 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb00099d0 con 0x7f6ec006e1e0 2024-08-31T12:19:12.006 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.003+0000 7f6ec8719640 1 -- 172.21.15.19:0/1450140097 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ec0152c90 con 0x7f6ec006e1e0 2024-08-31T12:19:12.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.004+0000 7f6ec8719640 1 -- 172.21.15.19:0/1450140097 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ec00caad0 con 0x7f6ec006e1e0 2024-08-31T12:19:12.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.005+0000 7f6eb6ffd640 1 -- 172.21.15.19:0/1450140097 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f6eb0009b70 con 0x7f6ec006e1e0 2024-08-31T12:19:12.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.006+0000 7f6eb6ffd640 1 --2- 172.21.15.19:0/1450140097 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6ea4077a30 0x7f6ea4079f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:12.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.006+0000 7f6eb6ffd640 1 -- 172.21.15.19:0/1450140097 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f6eb0099b50 con 0x7f6ec006e1e0 2024-08-31T12:19:12.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.006+0000 7f6ec64b5640 1 --2- 172.21.15.19:0/1450140097 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6ea4077a30 0x7f6ea4079f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:12.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.007+0000 7f6ec64b5640 1 --2- 172.21.15.19:0/1450140097 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6ea4077a30 0x7f6ea4079f40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc033000 tx=0x7f6ebc004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:12.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.013+0000 7f6eb6ffd640 1 -- 172.21.15.19:0/1450140097 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f6eb0020020 con 0x7f6ec006e1e0 2024-08-31T12:19:12.293 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.292+0000 7f6ec8719640 1 -- 172.21.15.19:0/1450140097 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6ec014eb00 con 0x7f6ec006e1e0 2024-08-31T12:19:12.294 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.293+0000 7f6eb6ffd640 1 -- 172.21.15.19:0/1450140097 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f6eb0062730 con 0x7f6ec006e1e0 2024-08-31T12:19:12.294 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:19:12.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.295+0000 7f6ec8719640 1 -- 172.21.15.19:0/1450140097 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6ea4077a30 msgr2=0x7f6ea4079f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:12.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.295+0000 7f6ec8719640 1 --2- 172.21.15.19:0/1450140097 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6ea4077a30 0x7f6ea4079f40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc033000 tx=0x7f6ebc004730 comp rx=0 tx=0).stop 2024-08-31T12:19:12.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.295+0000 7f6ec8719640 1 -- 172.21.15.19:0/1450140097 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ec006e1e0 msgr2=0x7f6ec01509f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:12.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.295+0000 7f6ec8719640 1 --2- 172.21.15.19:0/1450140097 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ec006e1e0 0x7f6ec01509f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6eb000d780 tx=0x7f6eb000dc40 comp rx=0 tx=0).stop 2024-08-31T12:19:12.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.295+0000 7f6ec8719640 1 -- 172.21.15.19:0/1450140097 shutdown_connections 2024-08-31T12:19:12.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.295+0000 7f6ec8719640 1 --2- 172.21.15.19:0/1450140097 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6ea4077a30 0x7f6ea4079f40 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:12.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.295+0000 7f6ec8719640 1 --2- 172.21.15.19:0/1450140097 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6ec00c8050 0x7f6ec0145f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:12.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.295+0000 7f6ec8719640 1 --2- 172.21.15.19:0/1450140097 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ec006e1e0 0x7f6ec01509f0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:12.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.295+0000 7f6ec8719640 1 --2- 172.21.15.19:0/1450140097 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6ec006ddd0 0x7f6ec0153c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:12.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.295+0000 7f6ec8719640 1 -- 172.21.15.19:0/1450140097 >> 172.21.15.19:0/1450140097 conn(0x7f6ec00cdcb0 msgr2=0x7f6ec00710d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:19:12.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.295+0000 7f6ec8719640 1 -- 172.21.15.19:0/1450140097 shutdown_connections 2024-08-31T12:19:12.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:12.295+0000 7f6ec8719640 1 -- 172.21.15.19:0/1450140097 wait complete. 2024-08-31T12:19:15.308 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:19:15.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.429+0000 7f652b007640 1 -- 172.21.15.19:0/1553976757 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f65240c7ac0 msgr2=0x7f6524152040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:15.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.429+0000 7f652b007640 1 --2- 172.21.15.19:0/1553976757 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f65240c7ac0 0x7f6524152040 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6520009820 tx=0x7f652002a530 comp rx=0 tx=0).stop 2024-08-31T12:19:15.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.429+0000 7f652b007640 1 -- 172.21.15.19:0/1553976757 shutdown_connections 2024-08-31T12:19:15.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.429+0000 7f652b007640 1 --2- 172.21.15.19:0/1553976757 >> v2:172.21.15.43:3300/0 conn(0x7f65240b28d0 0x7f65240c1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:15.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.429+0000 7f652b007640 1 --2- 172.21.15.19:0/1553976757 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f65240c7ac0 0x7f6524152040 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:15.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.429+0000 7f652b007640 1 -- 172.21.15.19:0/1553976757 >> 172.21.15.19:0/1553976757 conn(0x7f652406cae0 msgr2=0x7f652406cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:19:15.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.429+0000 7f652b007640 1 -- 172.21.15.19:0/1553976757 shutdown_connections 2024-08-31T12:19:15.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.429+0000 7f652b007640 1 -- 172.21.15.19:0/1553976757 wait complete. 2024-08-31T12:19:15.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.431+0000 7f652b007640 1 Processor -- start 2024-08-31T12:19:15.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.431+0000 7f652b007640 1 -- start start 2024-08-31T12:19:15.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.433+0000 7f652b007640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f65240b28d0 0x7f6524073c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:15.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.433+0000 7f652b007640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f65240c2400 0x7f6524142770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:15.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.433+0000 7f652b007640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f65240c7ac0 0x7f652414c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:15.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.433+0000 7f652b007640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f652406da00 con 0x7f65240c2400 2024-08-31T12:19:15.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.433+0000 7f652b007640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6524068fd0 con 0x7f65240b28d0 2024-08-31T12:19:15.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.433+0000 7f652b007640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f65240c3e90 con 0x7f65240c7ac0 2024-08-31T12:19:15.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.433+0000 7f6528da3640 1 -- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f65240b28d0 msgr2=0x7f6524073c50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3300/0 2024-08-31T12:19:15.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.433+0000 7f6528da3640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f65240b28d0 0x7f6524073c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:19:15.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.433+0000 7f651bfff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f65240c2400 0x7f6524142770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:15.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.433+0000 7f651bfff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f65240c2400 0x7f6524142770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:44522/0 (socket says 172.21.15.19:44522) 2024-08-31T12:19:15.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.433+0000 7f651bfff640 1 -- 172.21.15.19:0/3485249845 learned_addr learned my addr 172.21.15.19:0/3485249845 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:19:15.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.433+0000 7f65295a4640 1 --2- 172.21.15.19:0/3485249845 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f65240c7ac0 0x7f652414c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:15.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.433+0000 7f651bfff640 1 -- 172.21.15.19:0/3485249845 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f65240c7ac0 msgr2=0x7f652414c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:15.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.433+0000 7f651bfff640 1 --2- 172.21.15.19:0/3485249845 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f65240c7ac0 0x7f652414c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:15.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.433+0000 7f651bfff640 1 -- 172.21.15.19:0/3485249845 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f65240b28d0 msgr2=0x7f6524073c50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:19:15.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.433+0000 7f651bfff640 1 --2- 172.21.15.19:0/3485249845 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f65240b28d0 0x7f6524073c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:15.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.433+0000 7f651bfff640 1 -- 172.21.15.19:0/3485249845 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6524073960 con 0x7f65240c2400 2024-08-31T12:19:15.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.434+0000 7f651bfff640 1 --2- 172.21.15.19:0/3485249845 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f65240c2400 0x7f6524142770 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f651400b520 tx=0x7f651400b9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:15.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.434+0000 7f65197fa640 1 -- 172.21.15.19:0/3485249845 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6514014070 con 0x7f65240c2400 2024-08-31T12:19:15.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.434+0000 7f652b007640 1 -- 172.21.15.19:0/3485249845 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6524074190 con 0x7f65240c2400 2024-08-31T12:19:15.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.434+0000 7f65197fa640 1 -- 172.21.15.19:0/3485249845 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6514010980 con 0x7f65240c2400 2024-08-31T12:19:15.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.434+0000 7f65197fa640 1 -- 172.21.15.19:0/3485249845 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6514010db0 con 0x7f65240c2400 2024-08-31T12:19:15.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.434+0000 7f652b007640 1 -- 172.21.15.19:0/3485249845 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6524073680 con 0x7f65240c2400 2024-08-31T12:19:15.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.435+0000 7f652b007640 1 -- 172.21.15.19:0/3485249845 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65240c1ec0 con 0x7f65240c2400 2024-08-31T12:19:15.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.436+0000 7f65197fa640 1 -- 172.21.15.19:0/3485249845 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f6514017070 con 0x7f65240c2400 2024-08-31T12:19:15.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.436+0000 7f65197fa640 1 --2- 172.21.15.19:0/3485249845 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6510077a00 0x7f6510079f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:15.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.436+0000 7f65197fa640 1 -- 172.21.15.19:0/3485249845 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f65140995c0 con 0x7f65240c2400 2024-08-31T12:19:15.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.437+0000 7f6528da3640 1 --2- 172.21.15.19:0/3485249845 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6510077a00 0x7f6510079f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:15.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.437+0000 7f6528da3640 1 --2- 172.21.15.19:0/3485249845 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6510077a00 0x7f6510079f10 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6520034000 tx=0x7f6520004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:15.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.442+0000 7f65197fa640 1 -- 172.21.15.19:0/3485249845 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f6514062230 con 0x7f65240c2400 2024-08-31T12:19:15.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.717+0000 7f652b007640 1 -- 172.21.15.19:0/3485249845 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f652414e300 con 0x7f65240c2400 2024-08-31T12:19:15.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.718+0000 7f65197fa640 1 -- 172.21.15.19:0/3485249845 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f6514061b30 con 0x7f65240c2400 2024-08-31T12:19:15.719 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:19:15.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.720+0000 7f652b007640 1 -- 172.21.15.19:0/3485249845 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6510077a00 msgr2=0x7f6510079f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:15.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.720+0000 7f652b007640 1 --2- 172.21.15.19:0/3485249845 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6510077a00 0x7f6510079f10 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6520034000 tx=0x7f6520004730 comp rx=0 tx=0).stop 2024-08-31T12:19:15.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.720+0000 7f652b007640 1 -- 172.21.15.19:0/3485249845 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f65240c2400 msgr2=0x7f6524142770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:15.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.720+0000 7f652b007640 1 --2- 172.21.15.19:0/3485249845 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f65240c2400 0x7f6524142770 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f651400b520 tx=0x7f651400b9e0 comp rx=0 tx=0).stop 2024-08-31T12:19:15.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.720+0000 7f652b007640 1 -- 172.21.15.19:0/3485249845 shutdown_connections 2024-08-31T12:19:15.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.720+0000 7f652b007640 1 --2- 172.21.15.19:0/3485249845 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6510077a00 0x7f6510079f10 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:15.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.720+0000 7f652b007640 1 --2- 172.21.15.19:0/3485249845 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f65240c7ac0 0x7f652414c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:15.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.720+0000 7f652b007640 1 --2- 172.21.15.19:0/3485249845 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f65240c2400 0x7f6524142770 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:15.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.720+0000 7f652b007640 1 --2- 172.21.15.19:0/3485249845 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f65240b28d0 0x7f6524073c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:15.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.720+0000 7f652b007640 1 -- 172.21.15.19:0/3485249845 >> 172.21.15.19:0/3485249845 conn(0x7f652406cae0 msgr2=0x7f6524145cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:19:15.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.720+0000 7f652b007640 1 -- 172.21.15.19:0/3485249845 shutdown_connections 2024-08-31T12:19:15.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:15.720+0000 7f652b007640 1 -- 172.21.15.19:0/3485249845 wait complete. 2024-08-31T12:19:18.733 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:19:23.269 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.268+0000 7f718872b640 1 -- 172.21.15.19:0/4219263732 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7164007330 msgr2=0x7f7164009820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:23.270 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.268+0000 7f718872b640 1 --2- 172.21.15.19:0/4219263732 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7164007330 0x7f7164009820 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7170005f10 tx=0x7f717002c4e0 comp rx=0 tx=0).stop 2024-08-31T12:19:23.270 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.268+0000 7f7186cc8640 1 -- 172.21.15.19:0/4219263732 reap_dead start 2024-08-31T12:19:23.270 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.268+0000 7f718872b640 1 -- 172.21.15.19:0/4219263732 shutdown_connections 2024-08-31T12:19:23.270 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.268+0000 7f718872b640 1 -- 172.21.15.19:0/4219263732 >> 172.21.15.19:0/4219263732 conn(0x7f71800cefb0 msgr2=0x7f71800cb340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:19:23.270 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.268+0000 7f718872b640 1 -- 172.21.15.19:0/4219263732 shutdown_connections 2024-08-31T12:19:23.270 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.268+0000 7f718872b640 1 -- 172.21.15.19:0/4219263732 wait complete. 2024-08-31T12:19:23.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.270+0000 7f718872b640 1 Processor -- start 2024-08-31T12:19:23.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.270+0000 7f718872b640 1 -- start start 2024-08-31T12:19:23.274 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.271+0000 7f718872b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7164007330 0x7f718014c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:23.274 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.271+0000 7f718872b640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f718014afe0 0x7f71800cddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:23.274 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.271+0000 7f718872b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7180146f00 0x7f7180147310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:23.274 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.271+0000 7f718872b640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f71800c98d0 con 0x7f7180146f00 2024-08-31T12:19:23.274 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.271+0000 7f718872b640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f71800714f0 con 0x7f718014afe0 2024-08-31T12:19:23.274 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.271+0000 7f718872b640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f718015d3a0 con 0x7f7164007330 2024-08-31T12:19:23.274 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.271+0000 7f7185cc6640 1 -- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f718014afe0 msgr2=0x7f71800cddc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3300/0 2024-08-31T12:19:23.274 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.271+0000 7f7185cc6640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f718014afe0 0x7f71800cddc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:19:23.274 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.272+0000 7f7186cc8640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7180146f00 0x7f7180147310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:23.274 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.272+0000 7f7186cc8640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7180146f00 0x7f7180147310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:49288/0 (socket says 172.21.15.19:49288) 2024-08-31T12:19:23.275 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.272+0000 7f7186cc8640 1 -- 172.21.15.19:0/2956789407 learned_addr learned my addr 172.21.15.19:0/2956789407 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:19:23.275 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.272+0000 7f71864c7640 1 --2- 172.21.15.19:0/2956789407 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7164007330 0x7f718014c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:23.275 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.272+0000 7f7186cc8640 1 -- 172.21.15.19:0/2956789407 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7164007330 msgr2=0x7f718014c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:23.275 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.272+0000 7f7186cc8640 1 --2- 172.21.15.19:0/2956789407 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7164007330 0x7f718014c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:23.275 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.272+0000 7f7186cc8640 1 -- 172.21.15.19:0/2956789407 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f718014afe0 msgr2=0x7f71800cddc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:19:23.275 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.272+0000 7f7186cc8640 1 --2- 172.21.15.19:0/2956789407 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f718014afe0 0x7f71800cddc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:23.275 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.272+0000 7f7186cc8640 1 -- 172.21.15.19:0/2956789407 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f718014a880 con 0x7f7180146f00 2024-08-31T12:19:23.275 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.272+0000 7f7186cc8640 1 --2- 172.21.15.19:0/2956789407 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7180146f00 0x7f7180147310 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7164007300 tx=0x7f717800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:23.275 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.272+0000 7f7176ffd640 1 -- 172.21.15.19:0/2956789407 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7178017070 con 0x7f7180146f00 2024-08-31T12:19:23.275 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.272+0000 7f7176ffd640 1 -- 172.21.15.19:0/2956789407 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7178005e30 con 0x7f7180146f00 2024-08-31T12:19:23.275 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.272+0000 7f7176ffd640 1 -- 172.21.15.19:0/2956789407 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f717800a880 con 0x7f7180146f00 2024-08-31T12:19:23.275 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.272+0000 7f718872b640 1 -- 172.21.15.19:0/2956789407 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f718014ab30 con 0x7f7180146f00 2024-08-31T12:19:23.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.272+0000 7f718872b640 1 -- 172.21.15.19:0/2956789407 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71801f3800 con 0x7f7180146f00 2024-08-31T12:19:23.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.274+0000 7f718872b640 1 -- 172.21.15.19:0/2956789407 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71800ca160 con 0x7f7180146f00 2024-08-31T12:19:23.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.274+0000 7f7176ffd640 1 -- 172.21.15.19:0/2956789407 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f717800aa20 con 0x7f7180146f00 2024-08-31T12:19:23.277 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.275+0000 7f7176ffd640 1 --2- 172.21.15.19:0/2956789407 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f71640787f0 0x7f716407ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:23.277 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.275+0000 7f7176ffd640 1 -- 172.21.15.19:0/2956789407 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f71780987a0 con 0x7f7180146f00 2024-08-31T12:19:23.277 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.275+0000 7f71864c7640 1 --2- 172.21.15.19:0/2956789407 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f71640787f0 0x7f716407ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:23.277 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.276+0000 7f71864c7640 1 --2- 172.21.15.19:0/2956789407 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f71640787f0 0x7f716407ad00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f717c0012f0 tx=0x7f717c009920 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:23.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.280+0000 7f7176ffd640 1 -- 172.21.15.19:0/2956789407 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f7178060ed0 con 0x7f7180146f00 2024-08-31T12:19:23.591 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.591+0000 7f718872b640 1 -- 172.21.15.19:0/2956789407 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f718015ce80 con 0x7f7180146f00 2024-08-31T12:19:23.592 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.591+0000 7f7176ffd640 1 -- 172.21.15.19:0/2956789407 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7f7178065690 con 0x7f7180146f00 2024-08-31T12:19:23.592 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":30,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:19:23.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.593+0000 7f718872b640 1 -- 172.21.15.19:0/2956789407 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f71640787f0 msgr2=0x7f716407ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:23.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.593+0000 7f718872b640 1 --2- 172.21.15.19:0/2956789407 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f71640787f0 0x7f716407ad00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f717c0012f0 tx=0x7f717c009920 comp rx=0 tx=0).stop 2024-08-31T12:19:23.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.593+0000 7f718872b640 1 -- 172.21.15.19:0/2956789407 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7180146f00 msgr2=0x7f7180147310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:23.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.593+0000 7f718872b640 1 --2- 172.21.15.19:0/2956789407 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7180146f00 0x7f7180147310 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7164007300 tx=0x7f717800cb90 comp rx=0 tx=0).stop 2024-08-31T12:19:23.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.594+0000 7f718872b640 1 -- 172.21.15.19:0/2956789407 shutdown_connections 2024-08-31T12:19:23.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.594+0000 7f718872b640 1 --2- 172.21.15.19:0/2956789407 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f71640787f0 0x7f716407ad00 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:23.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.594+0000 7f718872b640 1 --2- 172.21.15.19:0/2956789407 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7180146f00 0x7f7180147310 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:23.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.594+0000 7f718872b640 1 --2- 172.21.15.19:0/2956789407 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f718014afe0 0x7f71800cddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:23.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.594+0000 7f718872b640 1 --2- 172.21.15.19:0/2956789407 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7164007330 0x7f718014c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:23.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.594+0000 7f718872b640 1 -- 172.21.15.19:0/2956789407 >> 172.21.15.19:0/2956789407 conn(0x7f71800cefb0 msgr2=0x7f71800c49d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:19:23.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.594+0000 7f718872b640 1 -- 172.21.15.19:0/2956789407 shutdown_connections 2024-08-31T12:19:23.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.594+0000 7f718872b640 1 -- 172.21.15.19:0/2956789407 wait complete. 2024-08-31T12:19:23.607 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-08-31T12:19:23.608 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-08-31T12:19:23.608 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-31T12:19:23.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.730+0000 7ff1dd027640 1 -- 172.21.15.19:0/3243587491 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff1d81532f0 msgr2=0x7ff1d815c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:23.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.730+0000 7ff1dd027640 1 --2- 172.21.15.19:0/3243587491 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff1d81532f0 0x7ff1d815c4b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff1d80caf20 tx=0x7ff1cc02a5d0 comp rx=0 tx=0).stop 2024-08-31T12:19:23.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.730+0000 7ff1dd027640 1 -- 172.21.15.19:0/3243587491 shutdown_connections 2024-08-31T12:19:23.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.730+0000 7ff1dd027640 1 --2- 172.21.15.19:0/3243587491 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff1d81532f0 0x7ff1d815c4b0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:23.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.730+0000 7ff1dd027640 1 -- 172.21.15.19:0/3243587491 >> 172.21.15.19:0/3243587491 conn(0x7ff1d80cefb0 msgr2=0x7ff1d80cb340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:19:23.732 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.730+0000 7ff1dd027640 1 -- 172.21.15.19:0/3243587491 shutdown_connections 2024-08-31T12:19:23.732 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.730+0000 7ff1dd027640 1 -- 172.21.15.19:0/3243587491 wait complete. 2024-08-31T12:19:23.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.732+0000 7ff1dd027640 1 Processor -- start 2024-08-31T12:19:23.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.732+0000 7ff1dd027640 1 -- start start 2024-08-31T12:19:23.736 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.733+0000 7ff1dd027640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff1d80c9880 0x7ff1d8158410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:23.736 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.733+0000 7ff1dd027640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff1d80ca7f0 0x7ff1d815bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:23.736 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.733+0000 7ff1dd027640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1d81532f0 0x7ff1d81550b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:23.737 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.733+0000 7ff1dd027640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d80c3860 con 0x7ff1d80ca7f0 2024-08-31T12:19:23.737 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.733+0000 7ff1dd027640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d815d330 con 0x7ff1d81532f0 2024-08-31T12:19:23.737 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.733+0000 7ff1dd027640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7ff1d8158d10 con 0x7ff1d80c9880 2024-08-31T12:19:23.737 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.733+0000 7ff1d659c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff1d80ca7f0 0x7ff1d815bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:23.737 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.733+0000 7ff1d759e640 1 -- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1d81532f0 msgr2=0x7ff1d81550b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3300/0 2024-08-31T12:19:23.737 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.733+0000 7ff1d759e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1d81532f0 0x7ff1d81550b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:19:23.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.733+0000 7ff1d659c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff1d80ca7f0 0x7ff1d815bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:49294/0 (socket says 172.21.15.19:49294) 2024-08-31T12:19:23.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.733+0000 7ff1d659c640 1 -- 172.21.15.19:0/343579910 learned_addr learned my addr 172.21.15.19:0/343579910 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:19:23.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.734+0000 7ff1d6d9d640 1 --2- 172.21.15.19:0/343579910 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff1d80c9880 0x7ff1d8158410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:23.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.734+0000 7ff1d659c640 1 -- 172.21.15.19:0/343579910 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff1d80c9880 msgr2=0x7ff1d8158410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:23.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.734+0000 7ff1d659c640 1 --2- 172.21.15.19:0/343579910 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff1d80c9880 0x7ff1d8158410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:23.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.734+0000 7ff1d659c640 1 -- 172.21.15.19:0/343579910 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1d81532f0 msgr2=0x7ff1d81550b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:19:23.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.734+0000 7ff1d659c640 1 --2- 172.21.15.19:0/343579910 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1d81532f0 0x7ff1d81550b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:23.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.734+0000 7ff1d659c640 1 -- 172.21.15.19:0/343579910 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1d81ef570 con 0x7ff1d80ca7f0 2024-08-31T12:19:23.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.734+0000 7ff1d659c640 1 --2- 172.21.15.19:0/343579910 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff1d80ca7f0 0x7ff1d815bd40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc0099c0 tx=0x7ff1cc0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:23.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.734+0000 7ff1bf7fe640 1 -- 172.21.15.19:0/343579910 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1cc03e030 con 0x7ff1d80ca7f0 2024-08-31T12:19:23.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.734+0000 7ff1bf7fe640 1 -- 172.21.15.19:0/343579910 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff1cc003db0 con 0x7ff1d80ca7f0 2024-08-31T12:19:23.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.734+0000 7ff1dd027640 1 -- 172.21.15.19:0/343579910 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1d81ef830 con 0x7ff1d80ca7f0 2024-08-31T12:19:23.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.734+0000 7ff1bf7fe640 1 -- 172.21.15.19:0/343579910 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1cc035040 con 0x7ff1d80ca7f0 2024-08-31T12:19:23.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.734+0000 7ff1dd027640 1 -- 172.21.15.19:0/343579910 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1d81efdf0 con 0x7ff1d80ca7f0 2024-08-31T12:19:23.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.735+0000 7ff1dd027640 1 --2- 172.21.15.19:0/343579910 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff1d81effc0 0x7ff1d81555f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:23.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.735+0000 7ff1dd027640 1 -- 172.21.15.19:0/343579910 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d80c39e0 con 0x7ff1d81effc0 2024-08-31T12:19:23.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.736+0000 7ff1d6d9d640 1 --2- 172.21.15.19:0/343579910 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff1d81effc0 0x7ff1d81555f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:23.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.736+0000 7ff1d6d9d640 1 -- 172.21.15.19:0/343579910 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff1d81568e0 con 0x7ff1d81effc0 2024-08-31T12:19:23.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.736+0000 7ff1d6d9d640 1 --2- 172.21.15.19:0/343579910 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff1d81effc0 0x7ff1d81555f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff1c8001050 tx=0x7ff1c800ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:23.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.736+0000 7ff1bf7fe640 1 -- 172.21.15.19:0/343579910 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c8017070 con 0x7ff1d81effc0 2024-08-31T12:19:23.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.737+0000 7ff1bf7fe640 1 -- 172.21.15.19:0/343579910 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7ff1cc041030 con 0x7ff1d80ca7f0 2024-08-31T12:19:23.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.737+0000 7ff1bf7fe640 1 --2- 172.21.15.19:0/343579910 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff1c4077980 0x7ff1c4079e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:23.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.737+0000 7ff1bf7fe640 1 -- 172.21.15.19:0/343579910 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7ff1cc0b9ba0 con 0x7ff1d80ca7f0 2024-08-31T12:19:23.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.737+0000 7ff1d759e640 1 --2- 172.21.15.19:0/343579910 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff1c4077980 0x7ff1c4079e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:23.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.737+0000 7ff1bf7fe640 1 -- 172.21.15.19:0/343579910 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff1c800ee50 con 0x7ff1d81effc0 2024-08-31T12:19:23.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.737+0000 7ff1bf7fe640 1 -- 172.21.15.19:0/343579910 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff1d81effc0 msgr2=0x7ff1d81555f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:23.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.737+0000 7ff1bf7fe640 1 --2- 172.21.15.19:0/343579910 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff1d81effc0 0x7ff1d81555f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff1c8001050 tx=0x7ff1c800ec00 comp rx=0 tx=0).stop 2024-08-31T12:19:23.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.737+0000 7ff1bf7fe640 1 -- 172.21.15.19:0/343579910 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff1d81effc0 msgr2=0x7ff1d81555f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:19:23.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.737+0000 7ff1bf7fe640 1 --2- 172.21.15.19:0/343579910 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff1d81effc0 0x7ff1d81555f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:23.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.738+0000 7ff1d759e640 1 --2- 172.21.15.19:0/343579910 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff1c4077980 0x7ff1c4079e90 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8071300 tx=0x7ff1d0007d10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:23.748 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.747+0000 7ff1dd027640 1 --2- 172.21.15.19:0/343579910 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff1d8149020 0x7ff1d8149430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:23.748 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.747+0000 7ff1dd027640 1 -- 172.21.15.19:0/343579910 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d80ca180 con 0x7ff1d8149020 2024-08-31T12:19:23.748 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.747+0000 7ff1d6d9d640 1 --2- 172.21.15.19:0/343579910 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff1d8149020 0x7ff1d8149430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:23.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.748+0000 7ff1d6d9d640 1 -- 172.21.15.19:0/343579910 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff1d814f310 con 0x7ff1d8149020 2024-08-31T12:19:23.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.748+0000 7ff1d6d9d640 1 --2- 172.21.15.19:0/343579910 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff1d8149020 0x7ff1d8149430 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff1c8001050 tx=0x7ff1c8004800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:23.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.748+0000 7ff1bf7fe640 1 -- 172.21.15.19:0/343579910 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c80065c0 con 0x7ff1d8149020 2024-08-31T12:19:23.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.748+0000 7ff1bf7fe640 1 -- 172.21.15.19:0/343579910 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5033 (secure 0 0 0) 0x7ff1c80065c0 con 0x7ff1d8149020 2024-08-31T12:19:23.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.748+0000 7ff1bf7fe640 1 -- 172.21.15.19:0/343579910 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff1d8149020 msgr2=0x7ff1d8149430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:23.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.748+0000 7ff1bf7fe640 1 --2- 172.21.15.19:0/343579910 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff1d8149020 0x7ff1d8149430 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff1c8001050 tx=0x7ff1c8004800 comp rx=0 tx=0).stop 2024-08-31T12:19:23.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.748+0000 7ff1bf7fe640 1 -- 172.21.15.19:0/343579910 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff1d8149020 msgr2=0x7ff1d8149430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:19:23.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.748+0000 7ff1bf7fe640 1 --2- 172.21.15.19:0/343579910 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff1d8149020 0x7ff1d8149430 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:23.751 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:19:23.751 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:19:23.751 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:19:23.751 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "leader", 2024-08-31T12:19:23.751 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 30, 2024-08-31T12:19:23.751 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 143739, 2024-08-31T12:19:23.751 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:19:23.751 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:19:23.751 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:19:23.751 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:23.751 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 1, 2024-08-31T12:19:23.751 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:19:23.752 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:19:23.752 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:19:23.752 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:19:23.752 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:19:23.752 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:19:23.752 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:19:23.752 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:19:23.752 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:19:23.752 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:19:23.752 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:19:23.752 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:19:23.752 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:19:23.753 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:19:23.753 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:23.753 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:19:23.753 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:19:23.753 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:19:23.753 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:19:23.753 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:19:23.753 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:19:23.753 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:19:23.753 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:19:23.753 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:19:23.753 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:19:23.753 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:19:23.754 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:19:23.754 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:19:23.754 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:23.754 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:23.754 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:19:23.754 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:19:23.754 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:19:23.754 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:19:23.754 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:19:23.754 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:19:23.754 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:19:23.754 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:19:23.755 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:19:23.755 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:19:23.755 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:19:23.755 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:19:23.755 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:19:23.755 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:19:23.755 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:19:23.755 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:19:23.755 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:19:23.755 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:19:23.756 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:19:23.756 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:19:23.756 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:19:23.756 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:19:23.756 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:19:23.756 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:19:23.756 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:19:23.756 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:19:23.756 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:19:23.756 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:19:23.757 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:23.757 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:19:23.757 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:23.757 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:19:23.757 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:23.757 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:19:23.757 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:19:23.757 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:19:23.757 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:19:23.757 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:23.757 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:19:23.758 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:19:23.758 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:23.758 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:23.758 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:23.758 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:19:23.758 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:19:23.758 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:23.758 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:23.758 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:23.758 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:23.758 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:19:23.758 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:19:23.758 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:19:23.759 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:19:23.759 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:19:23.759 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:23.759 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:23.759 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:19:23.759 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:19:23.759 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:19:23.759 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:19:23.759 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:23.759 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:19:23.759 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:19:23.759 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:23.759 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:23.759 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:23.760 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:19:23.760 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:19:23.760 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:23.760 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:23.760 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:23.760 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:23.760 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:19:23.760 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:19:23.760 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:19:23.760 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:19:23.760 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:19:23.760 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:23.761 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:23.761 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:19:23.761 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:19:23.761 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:19:23.761 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:19:23.761 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:23.761 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:19:23.761 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:19:23.761 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:23.761 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:23.761 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:23.762 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:19:23.762 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:19:23.762 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:23.762 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:23.762 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:23.762 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:23.762 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:19:23.762 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:19:23.762 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:19:23.762 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:19:23.762 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:19:23.763 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:23.763 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:23.763 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:23.763 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:19:23.763 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:19:23.763 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:23.763 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:19:23.763 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:19:23.763 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:19:23.763 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:23.763 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:23.763 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:19:23.763 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:23.763 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:19:23.764 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:19:23.764 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 5 2024-08-31T12:19:23.764 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:23.764 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:23.764 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:19:23.764 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:23.764 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:19:23.764 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:19:23.764 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 8 2024-08-31T12:19:23.765 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:23.765 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:23.765 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:19:23.765 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:23.765 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:19:23.765 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:19:23.765 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 6 2024-08-31T12:19:23.765 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:23.765 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:23.765 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:19:23.766 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:23.766 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:19:23.766 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:19:23.766 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:19:23.766 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:23.766 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:23.766 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:23.766 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:19:23.766 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:19:23.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.749+0000 7ff1dd027640 1 -- 172.21.15.19:0/343579910 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff1c4077980 msgr2=0x7ff1c4079e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:23.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.749+0000 7ff1dd027640 1 --2- 172.21.15.19:0/343579910 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff1c4077980 0x7ff1c4079e90 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8071300 tx=0x7ff1d0007d10 comp rx=0 tx=0).stop 2024-08-31T12:19:23.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.749+0000 7ff1dd027640 1 -- 172.21.15.19:0/343579910 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff1d80ca7f0 msgr2=0x7ff1d815bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:23.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.749+0000 7ff1dd027640 1 --2- 172.21.15.19:0/343579910 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff1d80ca7f0 0x7ff1d815bd40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc0099c0 tx=0x7ff1cc0038a0 comp rx=0 tx=0).stop 2024-08-31T12:19:23.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.749+0000 7ff1d759e640 1 -- 172.21.15.19:0/343579910 reap_dead start 2024-08-31T12:19:23.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.749+0000 7ff1d759e640 1 -- 172.21.15.19:0/343579910 reap_dead start 2024-08-31T12:19:23.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.749+0000 7ff1dd027640 1 -- 172.21.15.19:0/343579910 shutdown_connections 2024-08-31T12:19:23.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.749+0000 7ff1dd027640 1 -- 172.21.15.19:0/343579910 >> 172.21.15.19:0/343579910 conn(0x7ff1d80cefb0 msgr2=0x7ff1d80cb320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:19:23.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.749+0000 7ff1dd027640 1 -- 172.21.15.19:0/343579910 shutdown_connections 2024-08-31T12:19:23.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.749+0000 7ff1dd027640 1 -- 172.21.15.19:0/343579910 wait complete. 2024-08-31T12:19:23.769 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-08-31T12:19:23.769 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-08-31T12:19:23.769 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-31T12:19:23.892 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.890+0000 7f851f3ad640 1 -- 172.21.15.19:0/3325175519 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8518153190 msgr2=0x7f8518159820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:23.892 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.890+0000 7f851f3ad640 1 --2- 172.21.15.19:0/3325175519 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8518153190 0x7f8518159820 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f851815d2b0 tx=0x7f851402a5f0 comp rx=0 tx=0).stop 2024-08-31T12:19:23.892 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.890+0000 7f851f3ad640 1 -- 172.21.15.19:0/3325175519 shutdown_connections 2024-08-31T12:19:23.892 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.890+0000 7f851f3ad640 1 --2- 172.21.15.19:0/3325175519 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8518153190 0x7f8518159820 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:23.892 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.891+0000 7f851f3ad640 1 -- 172.21.15.19:0/3325175519 >> 172.21.15.19:0/3325175519 conn(0x7f85180728b0 msgr2=0x7f8518072cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:19:23.892 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.891+0000 7f851f3ad640 1 -- 172.21.15.19:0/3325175519 shutdown_connections 2024-08-31T12:19:23.892 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.891+0000 7f851f3ad640 1 -- 172.21.15.19:0/3325175519 wait complete. 2024-08-31T12:19:23.893 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.893+0000 7f851f3ad640 1 Processor -- start 2024-08-31T12:19:23.893 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.893+0000 7f851f3ad640 1 -- start start 2024-08-31T12:19:23.897 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.894+0000 7f851f3ad640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f85180c9030 0x7f85181effc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:23.897 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.894+0000 7f851f3ad640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f85180c9440 0x7f8518158180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:23.897 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.894+0000 7f851f3ad640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8518153190 0x7f85180cea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:23.897 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.894+0000 7f851f3ad640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f85180c4db0 con 0x7f8518153190 2024-08-31T12:19:23.897 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.894+0000 7f851f3ad640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8518159d60 con 0x7f85180c9030 2024-08-31T12:19:23.897 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.894+0000 7f851f3ad640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f85181535a0 con 0x7f85180c9440 2024-08-31T12:19:23.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.894+0000 7f851d149640 1 -- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f85180c9030 msgr2=0x7f85181effc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3300/0 2024-08-31T12:19:23.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.894+0000 7f851d149640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f85180c9030 0x7f85181effc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:19:23.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.894+0000 7f851d94a640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8518153190 0x7f85180cea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:23.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.894+0000 7f851d94a640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8518153190 0x7f85180cea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:49330/0 (socket says 172.21.15.19:49330) 2024-08-31T12:19:23.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.894+0000 7f851d94a640 1 -- 172.21.15.19:0/2808406947 learned_addr learned my addr 172.21.15.19:0/2808406947 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:19:23.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.894+0000 7f851c948640 1 --2- 172.21.15.19:0/2808406947 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f85180c9440 0x7f8518158180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:23.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.895+0000 7f851d94a640 1 -- 172.21.15.19:0/2808406947 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f85180c9440 msgr2=0x7f8518158180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:23.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.895+0000 7f851d94a640 1 --2- 172.21.15.19:0/2808406947 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f85180c9440 0x7f8518158180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:23.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.895+0000 7f851d94a640 1 -- 172.21.15.19:0/2808406947 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f85180c9030 msgr2=0x7f85181effc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:19:23.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.895+0000 7f851d94a640 1 --2- 172.21.15.19:0/2808406947 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f85180c9030 0x7f85181effc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:23.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.895+0000 7f851d94a640 1 -- 172.21.15.19:0/2808406947 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f851815a980 con 0x7f8518153190 2024-08-31T12:19:23.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.895+0000 7f851d94a640 1 --2- 172.21.15.19:0/2808406947 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8518153190 0x7f85180cea70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8510002230 tx=0x7f851000cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:23.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.895+0000 7f850dffb640 1 -- 172.21.15.19:0/2808406947 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8510003b70 con 0x7f8518153190 2024-08-31T12:19:23.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.895+0000 7f850dffb640 1 -- 172.21.15.19:0/2808406947 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8510003d10 con 0x7f8518153190 2024-08-31T12:19:23.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.895+0000 7f851f3ad640 1 -- 172.21.15.19:0/2808406947 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85181524a0 con 0x7f8518153190 2024-08-31T12:19:23.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.895+0000 7f850dffb640 1 -- 172.21.15.19:0/2808406947 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8510005330 con 0x7f8518153190 2024-08-31T12:19:23.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.895+0000 7f851f3ad640 1 -- 172.21.15.19:0/2808406947 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f851815a0a0 con 0x7f8518153190 2024-08-31T12:19:23.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.896+0000 7f851f3ad640 1 --2- 172.21.15.19:0/2808406947 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f85180cefb0 0x7f8518150e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:23.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.896+0000 7f851f3ad640 1 -- 172.21.15.19:0/2808406947 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f851815a270 con 0x7f85180cefb0 2024-08-31T12:19:23.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.897+0000 7f851d149640 1 --2- 172.21.15.19:0/2808406947 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f85180cefb0 0x7f8518150e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:23.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.897+0000 7f851d149640 1 -- 172.21.15.19:0/2808406947 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f85181f0500 con 0x7f85180cefb0 2024-08-31T12:19:23.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.897+0000 7f851d149640 1 --2- 172.21.15.19:0/2808406947 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f85180cefb0 0x7f8518150e50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f85180c5700 tx=0x7f8514005750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:23.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.897+0000 7f850dffb640 1 -- 172.21.15.19:0/2808406947 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8514035040 con 0x7f85180cefb0 2024-08-31T12:19:23.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.897+0000 7f850dffb640 1 -- 172.21.15.19:0/2808406947 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f85100056e0 con 0x7f8518153190 2024-08-31T12:19:23.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.898+0000 7f850dffb640 1 --2- 172.21.15.19:0/2808406947 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f84fc077950 0x7f84fc079e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:23.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.898+0000 7f850dffb640 1 -- 172.21.15.19:0/2808406947 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f8510099550 con 0x7f8518153190 2024-08-31T12:19:23.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.898+0000 7f851c948640 1 --2- 172.21.15.19:0/2808406947 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f84fc077950 0x7f84fc079e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:23.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.898+0000 7f850dffb640 1 -- 172.21.15.19:0/2808406947 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8514005030 con 0x7f85180cefb0 2024-08-31T12:19:23.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.898+0000 7f850dffb640 1 -- 172.21.15.19:0/2808406947 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f85180cefb0 msgr2=0x7f8518150e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:23.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.898+0000 7f850dffb640 1 --2- 172.21.15.19:0/2808406947 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f85180cefb0 0x7f8518150e50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f85180c5700 tx=0x7f8514005750 comp rx=0 tx=0).stop 2024-08-31T12:19:23.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.898+0000 7f850dffb640 1 -- 172.21.15.19:0/2808406947 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f85180cefb0 msgr2=0x7f8518150e50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:19:23.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.898+0000 7f850dffb640 1 --2- 172.21.15.19:0/2808406947 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f85180cefb0 0x7f8518150e50 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:23.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.899+0000 7f851c948640 1 --2- 172.21.15.19:0/2808406947 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f84fc077950 0x7f84fc079e60 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f85181e9e00 tx=0x7f8508009600 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:23.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.905+0000 7f851f3ad640 1 --2- 172.21.15.19:0/2808406947 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f851814d660 0x7f851814da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:23.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.905+0000 7f851f3ad640 1 -- 172.21.15.19:0/2808406947 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f8518159820 con 0x7f851814d660 2024-08-31T12:19:23.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.905+0000 7f851d149640 1 --2- 172.21.15.19:0/2808406947 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f851814d660 0x7f851814da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:23.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.905+0000 7f851d149640 1 -- 172.21.15.19:0/2808406947 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f85180cf3c0 con 0x7f851814d660 2024-08-31T12:19:23.907 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.905+0000 7f851d149640 1 --2- 172.21.15.19:0/2808406947 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f851814d660 0x7f851814da70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8514005fd0 tx=0x7f851402a5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:23.907 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.906+0000 7f850dffb640 1 -- 172.21.15.19:0/2808406947 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85140037f0 con 0x7f851814d660 2024-08-31T12:19:23.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.906+0000 7f850dffb640 1 -- 172.21.15.19:0/2808406947 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4696 (secure 0 0 0) 0x7f8514043290 con 0x7f851814d660 2024-08-31T12:19:23.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.906+0000 7f850dffb640 1 -- 172.21.15.19:0/2808406947 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f851814d660 msgr2=0x7f851814da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:23.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.906+0000 7f850dffb640 1 --2- 172.21.15.19:0/2808406947 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f851814d660 0x7f851814da70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8514005fd0 tx=0x7f851402a5f0 comp rx=0 tx=0).stop 2024-08-31T12:19:23.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.906+0000 7f850dffb640 1 -- 172.21.15.19:0/2808406947 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f851814d660 msgr2=0x7f851814da70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:19:23.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.906+0000 7f850dffb640 1 --2- 172.21.15.19:0/2808406947 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f851814d660 0x7f851814da70 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:23.908 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:19:23.909 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:19:23.909 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:19:23.909 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:19:23.909 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 30, 2024-08-31T12:19:23.909 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 92027, 2024-08-31T12:19:23.909 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:19:23.910 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:19:23.910 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:19:23.910 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:23.910 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 1, 2024-08-31T12:19:23.910 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:19:23.910 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:19:23.910 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:19:23.910 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:19:23.910 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:19:23.910 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:19:23.910 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:19:23.910 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:19:23.910 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:19:23.910 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:19:23.911 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:19:23.911 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:19:23.911 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:19:23.911 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:19:23.911 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:23.911 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:19:23.911 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:19:23.911 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:19:23.911 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:19:23.911 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:19:23.911 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:19:23.911 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:19:23.911 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:19:23.912 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:19:23.912 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:19:23.912 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:19:23.912 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:19:23.912 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:19:23.912 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:23.912 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:23.912 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:19:23.912 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:19:23.912 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:19:23.912 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:19:23.913 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:19:23.913 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:19:23.913 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:19:23.913 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:19:23.913 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:19:23.913 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:19:23.913 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:19:23.913 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:19:23.913 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:19:23.913 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:19:23.913 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:19:23.913 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:19:23.913 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:19:23.914 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:19:23.914 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:19:23.914 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:19:23.914 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:19:23.914 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:19:23.914 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:19:23.914 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:19:23.914 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:19:23.914 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:19:23.914 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:19:23.914 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:19:23.914 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:23.914 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:19:23.914 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:23.915 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:19:23.915 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:23.915 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:19:23.915 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:19:23.915 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:19:23.915 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:19:23.915 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:23.915 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:19:23.915 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:19:23.915 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:23.915 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:23.915 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:23.916 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:19:23.916 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:19:23.916 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:23.916 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:23.916 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:23.916 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:23.916 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:19:23.916 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:19:23.916 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:19:23.916 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:19:23.916 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:19:23.917 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:23.917 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:23.917 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:19:23.917 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:19:23.917 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:19:23.917 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:19:23.917 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:23.917 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:19:23.917 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:19:23.917 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:23.917 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:23.917 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:23.918 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:19:23.918 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:19:23.918 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:23.918 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:23.918 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:23.918 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:23.918 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:19:23.918 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:19:23.918 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:19:23.918 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:19:23.918 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:19:23.919 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:23.919 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:23.919 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:19:23.919 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:19:23.919 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:19:23.919 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:19:23.919 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:23.919 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:19:23.919 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:19:23.919 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:23.919 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:23.919 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:23.920 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:19:23.920 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:19:23.920 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:23.920 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:23.920 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:23.920 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:23.920 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:19:23.920 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:19:23.920 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:19:23.920 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:19:23.920 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:19:23.920 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:23.920 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:23.921 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:23.921 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:19:23.921 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:19:23.921 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:23.921 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:19:23.921 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:19:23.921 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:19:23.921 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:23.921 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:23.921 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:19:23.921 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:23.921 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:19:23.922 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:19:23.922 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:19:23.922 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:23.922 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:23.922 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:19:23.922 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:23.922 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:19:23.922 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:19:23.922 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:19:23.922 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:23.922 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:23.922 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:23.922 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:19:23.923 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:19:23.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.907+0000 7f851f3ad640 1 -- 172.21.15.19:0/2808406947 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f84fc077950 msgr2=0x7f84fc079e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:23.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.907+0000 7f851f3ad640 1 --2- 172.21.15.19:0/2808406947 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f84fc077950 0x7f84fc079e60 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f85181e9e00 tx=0x7f8508009600 comp rx=0 tx=0).stop 2024-08-31T12:19:23.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.907+0000 7f851f3ad640 1 -- 172.21.15.19:0/2808406947 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8518153190 msgr2=0x7f85180cea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:23.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.907+0000 7f851f3ad640 1 --2- 172.21.15.19:0/2808406947 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8518153190 0x7f85180cea70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8510002230 tx=0x7f851000cb90 comp rx=0 tx=0).stop 2024-08-31T12:19:23.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.907+0000 7f851d94a640 1 -- 172.21.15.19:0/2808406947 reap_dead start 2024-08-31T12:19:23.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.907+0000 7f851f3ad640 1 -- 172.21.15.19:0/2808406947 shutdown_connections 2024-08-31T12:19:23.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.907+0000 7f851f3ad640 1 --2- 172.21.15.19:0/2808406947 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8518153190 0x7f85180cea70 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:23.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.907+0000 7f851f3ad640 1 -- 172.21.15.19:0/2808406947 >> 172.21.15.19:0/2808406947 conn(0x7f85180728b0 msgr2=0x7f8518156a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:19:23.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.907+0000 7f851f3ad640 1 -- 172.21.15.19:0/2808406947 shutdown_connections 2024-08-31T12:19:23.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:23.907+0000 7f851f3ad640 1 -- 172.21.15.19:0/2808406947 wait complete. 2024-08-31T12:19:23.925 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-08-31T12:19:23.925 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-08-31T12:19:26.061 DEBUG:teuthology.orchestra.run.smithi019:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:19:26.064 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:19:43.926 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-08-31T12:19:43.926 INFO:tasks.ceph.mon.b:Restarting daemon 2024-08-31T12:19:43.927 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-08-31T12:19:43.930 INFO:tasks.ceph.mon.b:Started 2024-08-31T12:19:43.930 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-08-31T12:19:43.930 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:19:44.053 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.052+0000 7fb44b277640 1 -- 172.21.15.19:0/2087421442 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb4440caad0 msgr2=0x7fb4441519f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:44.053 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.052+0000 7fb44b277640 1 --2- 172.21.15.19:0/2087421442 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb4440caad0 0x7fb4441519f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb4440c4950 tx=0x7fb43402a5f0 comp rx=0 tx=0).stop 2024-08-31T12:19:44.053 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.052+0000 7fb44b277640 1 -- 172.21.15.19:0/2087421442 shutdown_connections 2024-08-31T12:19:44.054 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.052+0000 7fb44b277640 1 --2- 172.21.15.19:0/2087421442 >> v2:172.21.15.43:3300/0 conn(0x7fb444151f30 0x7fb444158170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:44.054 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.052+0000 7fb44b277640 1 --2- 172.21.15.19:0/2087421442 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb4440caad0 0x7fb4441519f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:44.054 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.052+0000 7fb44b277640 1 -- 172.21.15.19:0/2087421442 >> 172.21.15.19:0/2087421442 conn(0x7fb4440cdcb0 msgr2=0x7fb4440ca150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:19:44.054 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.052+0000 7fb44b277640 1 -- 172.21.15.19:0/2087421442 shutdown_connections 2024-08-31T12:19:44.054 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.052+0000 7fb44b277640 1 -- 172.21.15.19:0/2087421442 wait complete. 2024-08-31T12:19:44.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.054+0000 7fb44b277640 1 Processor -- start 2024-08-31T12:19:44.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.054+0000 7fb44b277640 1 -- start start 2024-08-31T12:19:44.057 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.055+0000 7fb44b277640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb4440c8050 0x7fb44414db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:44.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.055+0000 7fb44b277640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb4440caad0 0x7fb4440cce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:44.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.055+0000 7fb44b277640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb444151f30 0x7fb44414cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:44.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.055+0000 7fb44b277640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fb4440c5870 con 0x7fb4440caad0 2024-08-31T12:19:44.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.055+0000 7fb44b277640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb4440c5590 con 0x7fb444151f30 2024-08-31T12:19:44.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.055+0000 7fb44b277640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fb4440c7910 con 0x7fb4440c8050 2024-08-31T12:19:44.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.055+0000 7fb448812640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb4440caad0 0x7fb4440cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:44.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.055+0000 7fb448812640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb4440caad0 0x7fb4440cce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:33664/0 (socket says 172.21.15.19:33664) 2024-08-31T12:19:44.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.055+0000 7fb448812640 1 -- 172.21.15.19:0/4235728746 learned_addr learned my addr 172.21.15.19:0/4235728746 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:19:44.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.055+0000 7fb449013640 1 --2- 172.21.15.19:0/4235728746 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb4440c8050 0x7fb44414db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:44.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.055+0000 7fb449814640 1 --2- 172.21.15.19:0/4235728746 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb444151f30 0x7fb44414cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:44.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.056+0000 7fb449814640 1 -- 172.21.15.19:0/4235728746 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb444151f30 msgr2=0x7fb44414cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-08-31T12:19:44.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.056+0000 7fb449814640 1 -- 172.21.15.19:0/4235728746 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb444151f30 msgr2=0x7fb44414cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-31T12:19:44.059 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.056+0000 7fb449814640 1 --2- 172.21.15.19:0/4235728746 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb444151f30 0x7fb44414cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-31T12:19:44.059 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.056+0000 7fb449814640 1 --2- 172.21.15.19:0/4235728746 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb444151f30 0x7fb44414cc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:19:44.059 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.056+0000 7fb448812640 1 -- 172.21.15.19:0/4235728746 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb4440c8050 msgr2=0x7fb44414db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:44.059 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.056+0000 7fb448812640 1 --2- 172.21.15.19:0/4235728746 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb4440c8050 0x7fb44414db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:44.059 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.056+0000 7fb448812640 1 -- 172.21.15.19:0/4235728746 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb444151f30 msgr2=0x7fb44414cc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:19:44.059 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.056+0000 7fb448812640 1 --2- 172.21.15.19:0/4235728746 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb444151f30 0x7fb44414cc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:44.059 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.056+0000 7fb448812640 1 -- 172.21.15.19:0/4235728746 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4440cb450 con 0x7fb4440caad0 2024-08-31T12:19:44.059 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.056+0000 7fb448812640 1 --2- 172.21.15.19:0/4235728746 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb4440caad0 0x7fb4440cce70 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb434009850 tx=0x7fb434003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:44.059 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.056+0000 7fb439ffb640 1 -- 172.21.15.19:0/4235728746 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb43403e030 con 0x7fb4440caad0 2024-08-31T12:19:44.059 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.056+0000 7fb439ffb640 1 -- 172.21.15.19:0/4235728746 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb434003e00 con 0x7fb4440caad0 2024-08-31T12:19:44.059 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.056+0000 7fb439ffb640 1 -- 172.21.15.19:0/4235728746 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb434035040 con 0x7fb4440caad0 2024-08-31T12:19:44.059 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.056+0000 7fb44b277640 1 -- 172.21.15.19:0/4235728746 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4440cb9f0 con 0x7fb4440caad0 2024-08-31T12:19:44.059 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.056+0000 7fb44b277640 1 -- 172.21.15.19:0/4235728746 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb44414c430 con 0x7fb4440caad0 2024-08-31T12:19:44.060 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.057+0000 7fb44b277640 1 -- 172.21.15.19:0/4235728746 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb44414e850 con 0x7fb4440caad0 2024-08-31T12:19:44.061 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.058+0000 7fb439ffb640 1 -- 172.21.15.19:0/4235728746 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fb434041030 con 0x7fb4440caad0 2024-08-31T12:19:44.061 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.059+0000 7fb439ffb640 1 --2- 172.21.15.19:0/4235728746 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb428077a80 0x7fb428079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:44.061 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.059+0000 7fb439ffb640 1 -- 172.21.15.19:0/4235728746 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fb4340b9e70 con 0x7fb4440caad0 2024-08-31T12:19:44.061 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.059+0000 7fb449013640 1 --2- 172.21.15.19:0/4235728746 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb428077a80 0x7fb428079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:44.061 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.059+0000 7fb449013640 1 --2- 172.21.15.19:0/4235728746 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb428077a80 0x7fb428079f90 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb440001720 tx=0x7fb440009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:44.064 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.063+0000 7fb439ffb640 1 -- 172.21.15.19:0/4235728746 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fb44414e850 con 0x7fb4440caad0 2024-08-31T12:19:44.381 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.380+0000 7fb44b277640 1 -- 172.21.15.19:0/4235728746 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb4440c68c0 con 0x7fb4440caad0 2024-08-31T12:19:44.381 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.380+0000 7fb439ffb640 1 -- 172.21.15.19:0/4235728746 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7fb434082480 con 0x7fb4440caad0 2024-08-31T12:19:44.381 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":30,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:19:44.384 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.383+0000 7fb44b277640 1 -- 172.21.15.19:0/4235728746 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb428077a80 msgr2=0x7fb428079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:44.384 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.383+0000 7fb44b277640 1 --2- 172.21.15.19:0/4235728746 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb428077a80 0x7fb428079f90 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb440001720 tx=0x7fb440009040 comp rx=0 tx=0).stop 2024-08-31T12:19:44.384 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.383+0000 7fb44b277640 1 -- 172.21.15.19:0/4235728746 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb4440caad0 msgr2=0x7fb4440cce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:44.384 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.383+0000 7fb44b277640 1 --2- 172.21.15.19:0/4235728746 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb4440caad0 0x7fb4440cce70 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb434009850 tx=0x7fb434003910 comp rx=0 tx=0).stop 2024-08-31T12:19:44.384 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.383+0000 7fb44b277640 1 -- 172.21.15.19:0/4235728746 shutdown_connections 2024-08-31T12:19:44.384 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.383+0000 7fb44b277640 1 --2- 172.21.15.19:0/4235728746 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb428077a80 0x7fb428079f90 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:44.384 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.383+0000 7fb44b277640 1 --2- 172.21.15.19:0/4235728746 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb444151f30 0x7fb44414cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:44.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.383+0000 7fb44b277640 1 --2- 172.21.15.19:0/4235728746 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb4440caad0 0x7fb4440cce70 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:44.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.383+0000 7fb44b277640 1 --2- 172.21.15.19:0/4235728746 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb4440c8050 0x7fb44414db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:44.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.383+0000 7fb44b277640 1 -- 172.21.15.19:0/4235728746 >> 172.21.15.19:0/4235728746 conn(0x7fb4440cdcb0 msgr2=0x7fb44414e120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:19:44.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.383+0000 7fb44b277640 1 -- 172.21.15.19:0/4235728746 shutdown_connections 2024-08-31T12:19:44.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:44.383+0000 7fb44b277640 1 -- 172.21.15.19:0/4235728746 wait complete. 2024-08-31T12:19:47.396 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:19:47.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.527+0000 7f6ae3ba1640 1 -- 172.21.15.19:0/454573942 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6adc0caa20 msgr2=0x7f6ac8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:47.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.527+0000 7f6ae3ba1640 1 --2- 172.21.15.19:0/454573942 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6adc0caa20 0x7f6ac8004800 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6adc0cc4e0 tx=0x7f6ac8005830 comp rx=0 tx=0).stop 2024-08-31T12:19:47.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.527+0000 7f6ae213e640 1 -- 172.21.15.19:0/454573942 reap_dead start 2024-08-31T12:19:47.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.527+0000 7f6ae3ba1640 1 -- 172.21.15.19:0/454573942 shutdown_connections 2024-08-31T12:19:47.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.527+0000 7f6ae3ba1640 1 -- 172.21.15.19:0/454573942 >> 172.21.15.19:0/454573942 conn(0x7f6adc072c80 msgr2=0x7f6adc073090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:19:47.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.527+0000 7f6ae3ba1640 1 -- 172.21.15.19:0/454573942 shutdown_connections 2024-08-31T12:19:47.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.527+0000 7f6ae3ba1640 1 -- 172.21.15.19:0/454573942 wait complete. 2024-08-31T12:19:47.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.528+0000 7f6ae3ba1640 1 Processor -- start 2024-08-31T12:19:47.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.528+0000 7f6ae3ba1640 1 -- start start 2024-08-31T12:19:47.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.530+0000 7f6ae3ba1640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6adc0caa20 0x7f6adc1f4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:47.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.530+0000 7f6ae3ba1640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6adc14cd30 0x7f6adc1f18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:47.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.530+0000 7f6ae3ba1640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6adc151610 0x7f6adc14e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:47.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.530+0000 7f6ae3ba1640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f6adc0721e0 con 0x7f6adc151610 2024-08-31T12:19:47.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.530+0000 7f6ae3ba1640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6adc0cae30 con 0x7f6adc14cd30 2024-08-31T12:19:47.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.530+0000 7f6ae3ba1640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f6adc072360 con 0x7f6adc0caa20 2024-08-31T12:19:47.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.530+0000 7f6ae213e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6adc151610 0x7f6adc14e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:47.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.530+0000 7f6ae213e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6adc151610 0x7f6adc14e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:33686/0 (socket says 172.21.15.19:33686) 2024-08-31T12:19:47.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.530+0000 7f6ae213e640 1 -- 172.21.15.19:0/2899796845 learned_addr learned my addr 172.21.15.19:0/2899796845 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:19:47.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.530+0000 7f6ae193d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6adc0caa20 0x7f6adc1f4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:47.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.530+0000 7f6ae113c640 1 --2- 172.21.15.19:0/2899796845 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6adc14cd30 0x7f6adc1f18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:47.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.530+0000 7f6ae113c640 1 -- 172.21.15.19:0/2899796845 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6adc14cd30 msgr2=0x7f6adc1f18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-08-31T12:19:47.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.530+0000 7f6ae113c640 1 -- 172.21.15.19:0/2899796845 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6adc14cd30 msgr2=0x7f6adc1f18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-31T12:19:47.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.530+0000 7f6ae113c640 1 --2- 172.21.15.19:0/2899796845 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6adc14cd30 0x7f6adc1f18c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-31T12:19:47.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.530+0000 7f6ae113c640 1 --2- 172.21.15.19:0/2899796845 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6adc14cd30 0x7f6adc1f18c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:19:47.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.530+0000 7f6ae213e640 1 -- 172.21.15.19:0/2899796845 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6adc0caa20 msgr2=0x7f6adc1f4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:47.534 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.530+0000 7f6ae213e640 1 --2- 172.21.15.19:0/2899796845 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6adc0caa20 0x7f6adc1f4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:47.534 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.530+0000 7f6ae213e640 1 -- 172.21.15.19:0/2899796845 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6adc14cd30 msgr2=0x7f6adc1f18c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:19:47.534 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.530+0000 7f6ae213e640 1 --2- 172.21.15.19:0/2899796845 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6adc14cd30 0x7f6adc1f18c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:47.534 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.530+0000 7f6ae213e640 1 -- 172.21.15.19:0/2899796845 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6adc1f51b0 con 0x7f6adc151610 2024-08-31T12:19:47.534 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.530+0000 7f6ae213e640 1 --2- 172.21.15.19:0/2899796845 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6adc151610 0x7f6adc14e6e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6ad4005fd0 tx=0x7f6ad400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:47.534 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.531+0000 7f6ad27fc640 1 -- 172.21.15.19:0/2899796845 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad4009ad0 con 0x7f6adc151610 2024-08-31T12:19:47.534 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.531+0000 7f6ad27fc640 1 -- 172.21.15.19:0/2899796845 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6ad4003d80 con 0x7f6adc151610 2024-08-31T12:19:47.534 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.531+0000 7f6ae3ba1640 1 -- 172.21.15.19:0/2899796845 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6adc1f53c0 con 0x7f6adc151610 2024-08-31T12:19:47.534 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.531+0000 7f6ad27fc640 1 -- 172.21.15.19:0/2899796845 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad4004290 con 0x7f6adc151610 2024-08-31T12:19:47.534 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.531+0000 7f6ae3ba1640 1 -- 172.21.15.19:0/2899796845 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6adc06f640 con 0x7f6adc151610 2024-08-31T12:19:47.535 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.532+0000 7f6ae3ba1640 1 -- 172.21.15.19:0/2899796845 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6adc070ad0 con 0x7f6adc151610 2024-08-31T12:19:47.535 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.533+0000 7f6ad27fc640 1 -- 172.21.15.19:0/2899796845 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f6ad4004430 con 0x7f6adc151610 2024-08-31T12:19:47.535 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.533+0000 7f6ad27fc640 1 --2- 172.21.15.19:0/2899796845 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6ac0077a50 0x7f6ac0079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:47.535 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.534+0000 7f6ad27fc640 1 -- 172.21.15.19:0/2899796845 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f6ad409a720 con 0x7f6adc151610 2024-08-31T12:19:47.535 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.534+0000 7f6ae193d640 1 --2- 172.21.15.19:0/2899796845 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6ac0077a50 0x7f6ac0079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:47.535 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.534+0000 7f6ae193d640 1 --2- 172.21.15.19:0/2899796845 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6ac0077a50 0x7f6ac0079f60 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8001220 tx=0x7f6ad8006720 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:47.540 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.540+0000 7f6ad27fc640 1 -- 172.21.15.19:0/2899796845 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f6ad4063380 con 0x7f6adc151610 2024-08-31T12:19:47.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.835+0000 7f6ae3ba1640 1 -- 172.21.15.19:0/2899796845 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6adc0ca110 con 0x7f6adc151610 2024-08-31T12:19:47.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.836+0000 7f6ad27fc640 1 -- 172.21.15.19:0/2899796845 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f6ad400fe60 con 0x7f6adc151610 2024-08-31T12:19:47.837 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":30,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":25,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:19:47.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.838+0000 7f6ae3ba1640 1 -- 172.21.15.19:0/2899796845 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6ac0077a50 msgr2=0x7f6ac0079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:47.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.838+0000 7f6ae3ba1640 1 --2- 172.21.15.19:0/2899796845 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6ac0077a50 0x7f6ac0079f60 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8001220 tx=0x7f6ad8006720 comp rx=0 tx=0).stop 2024-08-31T12:19:47.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.838+0000 7f6ae3ba1640 1 -- 172.21.15.19:0/2899796845 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6adc151610 msgr2=0x7f6adc14e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:47.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.838+0000 7f6ae3ba1640 1 --2- 172.21.15.19:0/2899796845 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6adc151610 0x7f6adc14e6e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6ad4005fd0 tx=0x7f6ad400f790 comp rx=0 tx=0).stop 2024-08-31T12:19:47.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.838+0000 7f6ae3ba1640 1 -- 172.21.15.19:0/2899796845 shutdown_connections 2024-08-31T12:19:47.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.838+0000 7f6ae3ba1640 1 --2- 172.21.15.19:0/2899796845 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6ac0077a50 0x7f6ac0079f60 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:47.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.838+0000 7f6ae3ba1640 1 --2- 172.21.15.19:0/2899796845 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6adc151610 0x7f6adc14e6e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:47.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.838+0000 7f6ae3ba1640 1 --2- 172.21.15.19:0/2899796845 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6adc14cd30 0x7f6adc1f18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:47.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.838+0000 7f6ae3ba1640 1 --2- 172.21.15.19:0/2899796845 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6adc0caa20 0x7f6adc1f4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:47.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.838+0000 7f6ae3ba1640 1 -- 172.21.15.19:0/2899796845 >> 172.21.15.19:0/2899796845 conn(0x7f6adc072c80 msgr2=0x7f6adc070ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:19:47.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.839+0000 7f6ae3ba1640 1 -- 172.21.15.19:0/2899796845 shutdown_connections 2024-08-31T12:19:47.841 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:47.839+0000 7f6ae3ba1640 1 -- 172.21.15.19:0/2899796845 wait complete. 2024-08-31T12:19:50.852 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:19:50.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:50.973+0000 7f5b99533640 1 -- 172.21.15.19:0/1913990905 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5b940caa20 msgr2=0x7f5b94070ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:50.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:50.973+0000 7f5b99533640 1 --2- 172.21.15.19:0/1913990905 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5b940caa20 0x7f5b94070ad0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5b88005fd0 tx=0x7f5b8802a670 comp rx=0 tx=0).stop 2024-08-31T12:19:50.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:50.973+0000 7f5b99533640 1 -- 172.21.15.19:0/1913990905 shutdown_connections 2024-08-31T12:19:50.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:50.973+0000 7f5b99533640 1 --2- 172.21.15.19:0/1913990905 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5b940caa20 0x7f5b94070ad0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:50.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:50.973+0000 7f5b99533640 1 --2- 172.21.15.19:0/1913990905 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5b94153230 0x7f5b94071b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:50.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:50.973+0000 7f5b99533640 1 -- 172.21.15.19:0/1913990905 >> 172.21.15.19:0/1913990905 conn(0x7f5b94072c80 msgr2=0x7f5b94073090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:19:50.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:50.973+0000 7f5b99533640 1 -- 172.21.15.19:0/1913990905 shutdown_connections 2024-08-31T12:19:50.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:50.973+0000 7f5b99533640 1 -- 172.21.15.19:0/1913990905 wait complete. 2024-08-31T12:19:50.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:50.975+0000 7f5b99533640 1 Processor -- start 2024-08-31T12:19:50.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:50.975+0000 7f5b99533640 1 -- start start 2024-08-31T12:19:50.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:50.977+0000 7f5b99533640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5b94071010 0x7f5b94157910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:50.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:50.977+0000 7f5b99533640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5b940caa20 0x7f5b941e5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:50.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:50.977+0000 7f5b99533640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5b94153230 0x7f5b940b2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:50.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:50.977+0000 7f5b99533640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f5b940721e0 con 0x7f5b94071010 2024-08-31T12:19:50.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:50.977+0000 7f5b99533640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5b940cae30 con 0x7f5b940caa20 2024-08-31T12:19:50.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:50.977+0000 7f5b99533640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f5b94072360 con 0x7f5b94153230 2024-08-31T12:19:50.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:50.977+0000 7f5b92ffd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5b94071010 0x7f5b94157910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:50.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:50.977+0000 7f5b92ffd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5b94071010 0x7f5b94157910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:41844/0 (socket says 172.21.15.19:41844) 2024-08-31T12:19:50.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:50.977+0000 7f5b92ffd640 1 -- 172.21.15.19:0/347142845 learned_addr learned my addr 172.21.15.19:0/347142845 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:19:50.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:50.977+0000 7f5b937fe640 1 --2- 172.21.15.19:0/347142845 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5b94153230 0x7f5b940b2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:50.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:50.977+0000 7f5b927fc640 1 --2- 172.21.15.19:0/347142845 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5b940caa20 0x7f5b941e5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:50.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:50.977+0000 7f5b92ffd640 1 -- 172.21.15.19:0/347142845 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5b94153230 msgr2=0x7f5b940b2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:50.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:50.977+0000 7f5b92ffd640 1 --2- 172.21.15.19:0/347142845 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5b94153230 0x7f5b940b2400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:50.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:50.977+0000 7f5b92ffd640 1 -- 172.21.15.19:0/347142845 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5b940caa20 msgr2=0x7f5b941e5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:50.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:50.977+0000 7f5b92ffd640 1 --2- 172.21.15.19:0/347142845 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5b940caa20 0x7f5b941e5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:50.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:50.977+0000 7f5b92ffd640 1 -- 172.21.15.19:0/347142845 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b940b3cc0 con 0x7f5b94071010 2024-08-31T12:19:50.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:50.977+0000 7f5b937fe640 1 --2- 172.21.15.19:0/347142845 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5b94153230 0x7f5b940b2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-31T12:19:50.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:50.977+0000 7f5b927fc640 1 --2- 172.21.15.19:0/347142845 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5b940caa20 0x7f5b941e5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-31T12:19:50.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:50.977+0000 7f5b92ffd640 1 --2- 172.21.15.19:0/347142845 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5b94071010 0x7f5b94157910 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5b8400d780 tx=0x7f5b8400dc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:50.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:50.978+0000 7f5b7b7fe640 1 -- 172.21.15.19:0/347142845 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b84017070 con 0x7f5b94071010 2024-08-31T12:19:50.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:50.978+0000 7f5b7b7fe640 1 -- 172.21.15.19:0/347142845 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5b8400de90 con 0x7f5b94071010 2024-08-31T12:19:50.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:50.978+0000 7f5b7b7fe640 1 -- 172.21.15.19:0/347142845 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b840099d0 con 0x7f5b94071010 2024-08-31T12:19:50.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:50.978+0000 7f5b99533640 1 -- 172.21.15.19:0/347142845 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b94157630 con 0x7f5b94071010 2024-08-31T12:19:50.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:50.978+0000 7f5b99533640 1 -- 172.21.15.19:0/347142845 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b94156c20 con 0x7f5b94071010 2024-08-31T12:19:50.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:50.979+0000 7f5b99533640 1 -- 172.21.15.19:0/347142845 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b940cecd0 con 0x7f5b94071010 2024-08-31T12:19:50.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:50.980+0000 7f5b7b7fe640 1 -- 172.21.15.19:0/347142845 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f5b84009b70 con 0x7f5b94071010 2024-08-31T12:19:50.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:50.980+0000 7f5b7b7fe640 1 --2- 172.21.15.19:0/347142845 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5b80077a50 0x7f5b80079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:50.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:50.980+0000 7f5b7b7fe640 1 -- 172.21.15.19:0/347142845 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f5b84099bb0 con 0x7f5b94071010 2024-08-31T12:19:50.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:50.981+0000 7f5b927fc640 1 --2- 172.21.15.19:0/347142845 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5b80077a50 0x7f5b80079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:50.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:50.981+0000 7f5b927fc640 1 --2- 172.21.15.19:0/347142845 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5b80077a50 0x7f5b80079f60 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5b88033000 tx=0x7f5b88004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:50.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:50.985+0000 7f5b7b7fe640 1 -- 172.21.15.19:0/347142845 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f5b84020020 con 0x7f5b94071010 2024-08-31T12:19:51.278 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.276+0000 7f5b99533640 1 -- 172.21.15.19:0/347142845 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5b940ca110 con 0x7f5b94071010 2024-08-31T12:19:51.278 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.277+0000 7f5b7b7fe640 1 -- 172.21.15.19:0/347142845 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f5b84062810 con 0x7f5b94071010 2024-08-31T12:19:51.279 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:19:51.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.279+0000 7f5b99533640 1 -- 172.21.15.19:0/347142845 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5b80077a50 msgr2=0x7f5b80079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:51.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.279+0000 7f5b99533640 1 --2- 172.21.15.19:0/347142845 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5b80077a50 0x7f5b80079f60 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5b88033000 tx=0x7f5b88004730 comp rx=0 tx=0).stop 2024-08-31T12:19:51.282 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.279+0000 7f5b99533640 1 -- 172.21.15.19:0/347142845 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5b94071010 msgr2=0x7f5b94157910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:51.282 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.279+0000 7f5b99533640 1 --2- 172.21.15.19:0/347142845 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5b94071010 0x7f5b94157910 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5b8400d780 tx=0x7f5b8400dc40 comp rx=0 tx=0).stop 2024-08-31T12:19:51.282 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.279+0000 7f5b99533640 1 -- 172.21.15.19:0/347142845 shutdown_connections 2024-08-31T12:19:51.282 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.279+0000 7f5b99533640 1 --2- 172.21.15.19:0/347142845 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5b80077a50 0x7f5b80079f60 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:51.282 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.279+0000 7f5b99533640 1 --2- 172.21.15.19:0/347142845 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5b94153230 0x7f5b940b2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:51.282 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.279+0000 7f5b99533640 1 --2- 172.21.15.19:0/347142845 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5b940caa20 0x7f5b941e5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:51.282 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.279+0000 7f5b99533640 1 --2- 172.21.15.19:0/347142845 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5b94071010 0x7f5b94157910 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:51.282 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.279+0000 7f5b99533640 1 -- 172.21.15.19:0/347142845 >> 172.21.15.19:0/347142845 conn(0x7f5b94072c80 msgr2=0x7f5b940c4670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:19:51.282 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.279+0000 7f5b99533640 1 -- 172.21.15.19:0/347142845 shutdown_connections 2024-08-31T12:19:51.282 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.279+0000 7f5b99533640 1 -- 172.21.15.19:0/347142845 wait complete. 2024-08-31T12:19:51.293 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-08-31T12:19:51.293 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-08-31T12:19:51.293 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-31T12:19:51.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.415+0000 7f13719ad640 1 -- 172.21.15.19:0/2648810766 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f136c0c7ac0 msgr2=0x7f136c14e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:51.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.415+0000 7f13719ad640 1 --2- 172.21.15.19:0/2648810766 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f136c0c7ac0 0x7f136c14e720 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f135c0098e0 tx=0x7f135c02a5d0 comp rx=0 tx=0).stop 2024-08-31T12:19:51.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.416+0000 7f13719ad640 1 -- 172.21.15.19:0/2648810766 shutdown_connections 2024-08-31T12:19:51.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.416+0000 7f13719ad640 1 --2- 172.21.15.19:0/2648810766 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f136c0c85a0 0x7f136c0c89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:51.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.416+0000 7f13719ad640 1 --2- 172.21.15.19:0/2648810766 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f136c0c7ac0 0x7f136c14e720 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:51.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.416+0000 7f13719ad640 1 -- 172.21.15.19:0/2648810766 >> 172.21.15.19:0/2648810766 conn(0x7f136c06cae0 msgr2=0x7f136c06cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:19:51.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.416+0000 7f13719ad640 1 -- 172.21.15.19:0/2648810766 shutdown_connections 2024-08-31T12:19:51.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.416+0000 7f13719ad640 1 -- 172.21.15.19:0/2648810766 wait complete. 2024-08-31T12:19:51.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.417+0000 7f13719ad640 1 Processor -- start 2024-08-31T12:19:51.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.417+0000 7f13719ad640 1 -- start start 2024-08-31T12:19:51.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.419+0000 7f13719ad640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f136c0c7ac0 0x7f136c1f1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:51.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.419+0000 7f13719ad640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f136c0c85a0 0x7f136c0cbf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:51.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.419+0000 7f13719ad640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f136c1587f0 0x7f136c0c4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:51.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.419+0000 7f13719ad640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f136c152380 con 0x7f136c0c7ac0 2024-08-31T12:19:51.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.419+0000 7f13719ad640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f136c0c8ef0 con 0x7f136c0c85a0 2024-08-31T12:19:51.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.419+0000 7f13719ad640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f136c0c3e90 con 0x7f136c1587f0 2024-08-31T12:19:51.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.419+0000 7f136affd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f136c0c7ac0 0x7f136c1f1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:51.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.419+0000 7f136affd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f136c0c7ac0 0x7f136c1f1a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:41872/0 (socket says 172.21.15.19:41872) 2024-08-31T12:19:51.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.419+0000 7f136affd640 1 -- 172.21.15.19:0/3221356795 learned_addr learned my addr 172.21.15.19:0/3221356795 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:19:51.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.419+0000 7f136b7fe640 1 --2- 172.21.15.19:0/3221356795 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f136c1587f0 0x7f136c0c4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:51.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.419+0000 7f1363fff640 1 --2- 172.21.15.19:0/3221356795 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f136c0c85a0 0x7f136c0cbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:51.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.420+0000 7f136affd640 1 -- 172.21.15.19:0/3221356795 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f136c1587f0 msgr2=0x7f136c0c4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:51.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.420+0000 7f136affd640 1 --2- 172.21.15.19:0/3221356795 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f136c1587f0 0x7f136c0c4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:51.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.420+0000 7f136affd640 1 -- 172.21.15.19:0/3221356795 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f136c0c85a0 msgr2=0x7f136c0cbf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:51.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.420+0000 7f136affd640 1 --2- 172.21.15.19:0/3221356795 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f136c0c85a0 0x7f136c0cbf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:51.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.420+0000 7f136affd640 1 -- 172.21.15.19:0/3221356795 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f136c0c6ee0 con 0x7f136c0c7ac0 2024-08-31T12:19:51.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.420+0000 7f136b7fe640 1 --2- 172.21.15.19:0/3221356795 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f136c1587f0 0x7f136c0c4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:19:51.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.420+0000 7f136affd640 1 --2- 172.21.15.19:0/3221356795 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f136c0c7ac0 0x7f136c1f1a10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f135c009a00 tx=0x7f135c0316b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:51.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.420+0000 7f1368ff9640 1 -- 172.21.15.19:0/3221356795 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f135c03f070 con 0x7f136c0c7ac0 2024-08-31T12:19:51.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.420+0000 7f1368ff9640 1 -- 172.21.15.19:0/3221356795 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f135c031950 con 0x7f136c0c7ac0 2024-08-31T12:19:51.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.420+0000 7f1368ff9640 1 -- 172.21.15.19:0/3221356795 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f135c031da0 con 0x7f136c0c7ac0 2024-08-31T12:19:51.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.420+0000 7f13719ad640 1 -- 172.21.15.19:0/3221356795 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f136c142ca0 con 0x7f136c0c7ac0 2024-08-31T12:19:51.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.420+0000 7f13719ad640 1 -- 172.21.15.19:0/3221356795 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f136c1f21d0 con 0x7f136c0c7ac0 2024-08-31T12:19:51.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.421+0000 7f13719ad640 1 --2- 172.21.15.19:0/3221356795 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f136c1e1060 0x7f136c141ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:51.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.421+0000 7f13719ad640 1 -- 172.21.15.19:0/3221356795 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f136c142b20 con 0x7f136c1e1060 2024-08-31T12:19:51.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.421+0000 7f1363fff640 1 --2- 172.21.15.19:0/3221356795 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f136c1e1060 0x7f136c141ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:51.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.422+0000 7f1363fff640 1 -- 172.21.15.19:0/3221356795 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f136c1e3100 con 0x7f136c1e1060 2024-08-31T12:19:51.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.422+0000 7f1363fff640 1 --2- 172.21.15.19:0/3221356795 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f136c1e1060 0x7f136c141ae0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f136c075080 tx=0x7f135400f9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:51.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.422+0000 7f1368ff9640 1 -- 172.21.15.19:0/3221356795 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13540052e0 con 0x7f136c1e1060 2024-08-31T12:19:51.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.422+0000 7f1368ff9640 1 -- 172.21.15.19:0/3221356795 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f135c042070 con 0x7f136c0c7ac0 2024-08-31T12:19:51.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.423+0000 7f1368ff9640 1 --2- 172.21.15.19:0/3221356795 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f134c077950 0x7f134c079e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:51.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.423+0000 7f1368ff9640 1 -- 172.21.15.19:0/3221356795 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f135c0ba110 con 0x7f136c0c7ac0 2024-08-31T12:19:51.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.423+0000 7f1368ff9640 1 -- 172.21.15.19:0/3221356795 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1354005480 con 0x7f136c1e1060 2024-08-31T12:19:51.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.423+0000 7f1368ff9640 1 -- 172.21.15.19:0/3221356795 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f136c1e1060 msgr2=0x7f136c141ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:51.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.423+0000 7f1368ff9640 1 --2- 172.21.15.19:0/3221356795 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f136c1e1060 0x7f136c141ae0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f136c075080 tx=0x7f135400f9d0 comp rx=0 tx=0).stop 2024-08-31T12:19:51.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.423+0000 7f1368ff9640 1 -- 172.21.15.19:0/3221356795 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f136c1e1060 msgr2=0x7f136c141ae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:19:51.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.423+0000 7f1368ff9640 1 --2- 172.21.15.19:0/3221356795 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f136c1e1060 0x7f136c141ae0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:51.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.423+0000 7f136b7fe640 1 --2- 172.21.15.19:0/3221356795 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f134c077950 0x7f134c079e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:51.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.424+0000 7f136b7fe640 1 --2- 172.21.15.19:0/3221356795 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f134c077950 0x7f134c079e60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f136c143660 tx=0x7f1364009500 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:51.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.433+0000 7f13719ad640 1 --2- 172.21.15.19:0/3221356795 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f136c142020 0x7f136c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:51.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.433+0000 7f13719ad640 1 -- 172.21.15.19:0/3221356795 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f136c0c6c90 con 0x7f136c142020 2024-08-31T12:19:51.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.433+0000 7f1363fff640 1 --2- 172.21.15.19:0/3221356795 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f136c142020 0x7f136c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:51.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.433+0000 7f1363fff640 1 -- 172.21.15.19:0/3221356795 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f136c06af80 con 0x7f136c142020 2024-08-31T12:19:51.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.433+0000 7f1363fff640 1 --2- 172.21.15.19:0/3221356795 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f136c142020 0x7f136c06f1a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f136c075080 tx=0x7f1354000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:51.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.434+0000 7f1368ff9640 1 -- 172.21.15.19:0/3221356795 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1354000f70 con 0x7f136c142020 2024-08-31T12:19:51.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.434+0000 7f1368ff9640 1 -- 172.21.15.19:0/3221356795 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5044 (secure 0 0 0) 0x7f1354000f70 con 0x7f136c142020 2024-08-31T12:19:51.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.434+0000 7f1368ff9640 1 -- 172.21.15.19:0/3221356795 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f136c142020 msgr2=0x7f136c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:51.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.434+0000 7f1368ff9640 1 --2- 172.21.15.19:0/3221356795 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f136c142020 0x7f136c06f1a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f136c075080 tx=0x7f1354000ed0 comp rx=0 tx=0).stop 2024-08-31T12:19:51.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.434+0000 7f1368ff9640 1 -- 172.21.15.19:0/3221356795 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f136c142020 msgr2=0x7f136c06f1a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:19:51.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.434+0000 7f1368ff9640 1 --2- 172.21.15.19:0/3221356795 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f136c142020 0x7f136c06f1a0 secure :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f136c075080 tx=0x7f1354000ed0 comp rx=0 tx=0).stop 2024-08-31T12:19:51.437 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:19:51.437 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:19:51.437 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:19:51.437 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "leader", 2024-08-31T12:19:51.437 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 32, 2024-08-31T12:19:51.437 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 171426, 2024-08-31T12:19:51.437 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:19:51.437 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:19:51.437 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:19:51.438 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:19:51.438 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:51.438 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 2, 2024-08-31T12:19:51.438 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:19:51.438 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:19:51.438 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:19:51.438 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:19:51.438 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:19:51.438 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:19:51.438 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:19:51.438 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:19:51.438 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:19:51.438 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:19:51.438 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:19:51.439 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:19:51.439 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:19:51.439 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:19:51.439 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:51.439 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:19:51.439 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:19:51.439 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:19:51.439 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:19:51.439 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:19:51.439 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:19:51.439 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:19:51.439 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:19:51.439 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:19:51.439 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:19:51.440 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:19:51.440 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:19:51.440 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:19:51.440 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:51.440 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:51.440 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:19:51.440 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:19:51.440 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:19:51.440 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:19:51.440 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:19:51.440 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:19:51.440 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:19:51.440 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:19:51.441 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:19:51.441 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:19:51.441 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:19:51.441 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:19:51.441 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:19:51.441 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:19:51.441 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:19:51.441 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:19:51.441 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:19:51.441 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:19:51.441 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:19:51.441 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:19:51.441 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:19:51.441 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:19:51.442 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:19:51.442 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:19:51.442 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:19:51.442 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:19:51.442 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:19:51.442 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:19:51.442 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:51.442 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:19:51.442 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:51.442 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:19:51.443 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:51.443 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:19:51.443 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:19:51.443 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:19:51.443 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:19:51.443 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:51.443 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:19:51.443 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:19:51.443 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:51.443 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:51.443 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:51.443 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:19:51.443 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:19:51.444 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:51.444 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:51.444 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:51.444 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:51.444 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:19:51.444 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:19:51.444 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:19:51.444 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:19:51.444 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:19:51.444 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:51.444 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:51.444 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:19:51.445 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:19:51.445 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:19:51.445 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:19:51.445 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:51.445 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:19:51.445 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:19:51.445 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:51.445 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:51.445 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:51.445 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:19:51.445 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:19:51.445 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:51.445 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:51.446 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:51.446 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:51.446 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:19:51.446 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:19:51.446 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:19:51.446 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:19:51.446 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:19:51.446 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:51.446 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:51.446 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:19:51.446 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:19:51.446 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:19:51.446 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:19:51.447 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:51.447 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:19:51.447 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:19:51.447 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:51.447 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:51.447 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:51.447 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:19:51.447 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:19:51.447 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:51.447 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:51.447 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:51.447 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:51.448 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:19:51.448 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:19:51.448 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:19:51.448 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:19:51.448 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:19:51.448 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:51.448 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:51.448 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:51.448 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:19:51.448 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:19:51.448 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:51.448 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:19:51.449 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:19:51.449 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:19:51.449 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:51.449 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:51.449 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:19:51.449 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:51.449 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:19:51.449 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:19:51.449 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 5 2024-08-31T12:19:51.449 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:51.450 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:51.450 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:19:51.450 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:51.450 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:19:51.450 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:19:51.450 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 8 2024-08-31T12:19:51.450 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:51.450 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:51.450 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:19:51.450 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:51.450 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:19:51.450 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:19:51.450 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 6 2024-08-31T12:19:51.451 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:51.451 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:51.451 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:19:51.451 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:51.451 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:19:51.451 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:19:51.451 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:19:51.451 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:51.451 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:51.451 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:51.451 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:19:51.451 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:19:51.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.434+0000 7f13719ad640 1 -- 172.21.15.19:0/3221356795 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f134c077950 msgr2=0x7f134c079e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:51.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.434+0000 7f13719ad640 1 --2- 172.21.15.19:0/3221356795 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f134c077950 0x7f134c079e60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f136c143660 tx=0x7f1364009500 comp rx=0 tx=0).stop 2024-08-31T12:19:51.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.434+0000 7f13719ad640 1 -- 172.21.15.19:0/3221356795 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f136c0c7ac0 msgr2=0x7f136c1f1a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:51.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.434+0000 7f13719ad640 1 --2- 172.21.15.19:0/3221356795 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f136c0c7ac0 0x7f136c1f1a10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f135c009a00 tx=0x7f135c0316b0 comp rx=0 tx=0).stop 2024-08-31T12:19:51.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.434+0000 7f136b7fe640 1 -- 172.21.15.19:0/3221356795 reap_dead start 2024-08-31T12:19:51.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.434+0000 7f136b7fe640 1 -- 172.21.15.19:0/3221356795 reap_dead start 2024-08-31T12:19:51.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.434+0000 7f13719ad640 1 -- 172.21.15.19:0/3221356795 shutdown_connections 2024-08-31T12:19:51.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.434+0000 7f13719ad640 1 -- 172.21.15.19:0/3221356795 >> 172.21.15.19:0/3221356795 conn(0x7f136c06cae0 msgr2=0x7f136c073330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:19:51.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.434+0000 7f13719ad640 1 -- 172.21.15.19:0/3221356795 shutdown_connections 2024-08-31T12:19:51.454 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.434+0000 7f13719ad640 1 -- 172.21.15.19:0/3221356795 wait complete. 2024-08-31T12:19:51.454 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:19:51.454 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-08-31T12:19:51.454 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-31T12:19:51.584 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.582+0000 7f7f28c7e640 1 -- 172.21.15.19:0/2470596100 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7f240cb2f0 msgr2=0x7f7f240cc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:51.584 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.582+0000 7f7f28c7e640 1 --2- 172.21.15.19:0/2470596100 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7f240cb2f0 0x7f7f240cc4d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c005fd0 tx=0x7f7f1c02ab20 comp rx=0 tx=0).stop 2024-08-31T12:19:51.584 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.582+0000 7f7f28c7e640 1 -- 172.21.15.19:0/2470596100 shutdown_connections 2024-08-31T12:19:51.584 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.582+0000 7f7f28c7e640 1 --2- 172.21.15.19:0/2470596100 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7f240cb2f0 0x7f7f240cc4d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:51.584 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.582+0000 7f7f28c7e640 1 --2- 172.21.15.19:0/2470596100 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7f240c5c20 0x7f7f240cbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:51.584 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.582+0000 7f7f28c7e640 1 -- 172.21.15.19:0/2470596100 >> 172.21.15.19:0/2470596100 conn(0x7f7f24074a90 msgr2=0x7f7f24068aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:19:51.584 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.582+0000 7f7f28c7e640 1 -- 172.21.15.19:0/2470596100 shutdown_connections 2024-08-31T12:19:51.584 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.582+0000 7f7f28c7e640 1 -- 172.21.15.19:0/2470596100 wait complete. 2024-08-31T12:19:51.586 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.584+0000 7f7f28c7e640 1 Processor -- start 2024-08-31T12:19:51.586 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.584+0000 7f7f28c7e640 1 -- start start 2024-08-31T12:19:51.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.585+0000 7f7f28c7e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7f240c5c20 0x7f7f2415a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:51.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.585+0000 7f7f28c7e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7f240cb2f0 0x7f7f241f5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:51.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.585+0000 7f7f28c7e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f24155520 0x7f7f24155f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:51.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.585+0000 7f7f28c7e640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f7f240d1140 con 0x7f7f240cb2f0 2024-08-31T12:19:51.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.585+0000 7f7f28c7e640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7f240c6750 con 0x7f7f24155520 2024-08-31T12:19:51.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.585+0000 7f7f28c7e640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f7f2415b730 con 0x7f7f240c5c20 2024-08-31T12:19:51.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.585+0000 7f7f1bfff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7f240cb2f0 0x7f7f241f5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:51.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.586+0000 7f7f1bfff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7f240cb2f0 0x7f7f241f5030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:41888/0 (socket says 172.21.15.19:41888) 2024-08-31T12:19:51.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.586+0000 7f7f1bfff640 1 -- 172.21.15.19:0/3360967655 learned_addr learned my addr 172.21.15.19:0/3360967655 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:19:51.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.586+0000 7f7f2259c640 1 --2- 172.21.15.19:0/3360967655 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7f240c5c20 0x7f7f2415a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:51.590 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.586+0000 7f7f22d9d640 1 --2- 172.21.15.19:0/3360967655 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f24155520 0x7f7f24155f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:51.590 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.586+0000 7f7f1bfff640 1 -- 172.21.15.19:0/3360967655 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7f240c5c20 msgr2=0x7f7f2415a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:51.590 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.586+0000 7f7f1bfff640 1 --2- 172.21.15.19:0/3360967655 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7f240c5c20 0x7f7f2415a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:51.590 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.586+0000 7f7f1bfff640 1 -- 172.21.15.19:0/3360967655 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f24155520 msgr2=0x7f7f24155f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:51.590 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.586+0000 7f7f1bfff640 1 --2- 172.21.15.19:0/3360967655 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f24155520 0x7f7f24155f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:51.590 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.586+0000 7f7f1bfff640 1 -- 172.21.15.19:0/3360967655 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f2415a800 con 0x7f7f240cb2f0 2024-08-31T12:19:51.590 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.586+0000 7f7f1bfff640 1 --2- 172.21.15.19:0/3360967655 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7f240cb2f0 0x7f7f241f5030 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c00e7c0 tx=0x7f7f0c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:51.590 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.586+0000 7f7f1affd640 1 -- 172.21.15.19:0/3360967655 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c00ccc0 con 0x7f7f240cb2f0 2024-08-31T12:19:51.590 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.586+0000 7f7f1affd640 1 -- 172.21.15.19:0/3360967655 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7f0c010430 con 0x7f7f240cb2f0 2024-08-31T12:19:51.590 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.586+0000 7f7f1affd640 1 -- 172.21.15.19:0/3360967655 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c010860 con 0x7f7f240cb2f0 2024-08-31T12:19:51.590 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.586+0000 7f7f28c7e640 1 -- 172.21.15.19:0/3360967655 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f241f58c0 con 0x7f7f240cb2f0 2024-08-31T12:19:51.591 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.586+0000 7f7f28c7e640 1 -- 172.21.15.19:0/3360967655 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f241e62b0 con 0x7f7f240cb2f0 2024-08-31T12:19:51.592 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.587+0000 7f7f28c7e640 1 --2- 172.21.15.19:0/3360967655 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f241564a0 0x7f7f2414f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:51.592 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.587+0000 7f7f28c7e640 1 -- 172.21.15.19:0/3360967655 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7f24152c40 con 0x7f7f241564a0 2024-08-31T12:19:51.592 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.588+0000 7f7f2259c640 1 --2- 172.21.15.19:0/3360967655 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f241564a0 0x7f7f2414f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:51.592 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.588+0000 7f7f2259c640 1 -- 172.21.15.19:0/3360967655 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7f240cb800 con 0x7f7f241564a0 2024-08-31T12:19:51.592 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.588+0000 7f7f2259c640 1 --2- 172.21.15.19:0/3360967655 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f241564a0 0x7f7f2414f830 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7f1400f570 tx=0x7f7f1400fa30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:51.592 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.588+0000 7f7f1affd640 1 -- 172.21.15.19:0/3360967655 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f140052e0 con 0x7f7f241564a0 2024-08-31T12:19:51.593 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.588+0000 7f7f1affd640 1 -- 172.21.15.19:0/3360967655 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f7f0c010a00 con 0x7f7f240cb2f0 2024-08-31T12:19:51.593 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.589+0000 7f7f1affd640 1 --2- 172.21.15.19:0/3360967655 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7f04077980 0x7f7f04079e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:51.593 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.589+0000 7f7f1affd640 1 -- 172.21.15.19:0/3360967655 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f7f0c099ff0 con 0x7f7f240cb2f0 2024-08-31T12:19:51.593 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.589+0000 7f7f22d9d640 1 --2- 172.21.15.19:0/3360967655 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7f04077980 0x7f7f04079e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:51.593 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.589+0000 7f7f1affd640 1 -- 172.21.15.19:0/3360967655 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7f14018070 con 0x7f7f241564a0 2024-08-31T12:19:51.593 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.589+0000 7f7f1affd640 1 -- 172.21.15.19:0/3360967655 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f241564a0 msgr2=0x7f7f2414f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:51.593 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.589+0000 7f7f1affd640 1 --2- 172.21.15.19:0/3360967655 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f241564a0 0x7f7f2414f830 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7f1400f570 tx=0x7f7f1400fa30 comp rx=0 tx=0).stop 2024-08-31T12:19:51.593 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.589+0000 7f7f1affd640 1 -- 172.21.15.19:0/3360967655 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f241564a0 msgr2=0x7f7f2414f830 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:19:51.593 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.589+0000 7f7f1affd640 1 --2- 172.21.15.19:0/3360967655 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f241564a0 0x7f7f2414f830 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:51.593 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.590+0000 7f7f22d9d640 1 --2- 172.21.15.19:0/3360967655 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7f04077980 0x7f7f04079e90 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c035000 tx=0x7f7f1c038040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:51.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.596+0000 7f7f28c7e640 1 --2- 172.21.15.19:0/3360967655 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f2414fd70 0x7f7f241f17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:51.598 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.596+0000 7f7f28c7e640 1 -- 172.21.15.19:0/3360967655 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7f240b3bb0 con 0x7f7f2414fd70 2024-08-31T12:19:51.598 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.596+0000 7f7f2259c640 1 --2- 172.21.15.19:0/3360967655 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f2414fd70 0x7f7f241f17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:51.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.596+0000 7f7f2259c640 1 -- 172.21.15.19:0/3360967655 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7f240cbd70 con 0x7f7f2414fd70 2024-08-31T12:19:51.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.596+0000 7f7f2259c640 1 --2- 172.21.15.19:0/3360967655 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f2414fd70 0x7f7f241f17c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7f1400f570 tx=0x7f7f14000ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:51.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.597+0000 7f7f1affd640 1 -- 172.21.15.19:0/3360967655 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f14005480 con 0x7f7f2414fd70 2024-08-31T12:19:51.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.597+0000 7f7f1affd640 1 -- 172.21.15.19:0/3360967655 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4539 (secure 0 0 0) 0x7f7f14005480 con 0x7f7f2414fd70 2024-08-31T12:19:51.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.597+0000 7f7f1affd640 1 -- 172.21.15.19:0/3360967655 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f2414fd70 msgr2=0x7f7f241f17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:51.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.597+0000 7f7f1affd640 1 --2- 172.21.15.19:0/3360967655 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f2414fd70 0x7f7f241f17c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7f1400f570 tx=0x7f7f14000ed0 comp rx=0 tx=0).stop 2024-08-31T12:19:51.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.597+0000 7f7f1affd640 1 -- 172.21.15.19:0/3360967655 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f2414fd70 msgr2=0x7f7f241f17c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:19:51.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.597+0000 7f7f1affd640 1 --2- 172.21.15.19:0/3360967655 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f2414fd70 0x7f7f241f17c0 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7f1400f570 tx=0x7f7f14000ed0 comp rx=0 tx=0).stop 2024-08-31T12:19:51.600 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:19:51.600 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:19:51.600 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:19:51.600 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:19:51.600 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 32, 2024-08-31T12:19:51.600 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 7565, 2024-08-31T12:19:51.600 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:19:51.600 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:19:51.600 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:19:51.600 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:19:51.601 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:51.601 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 2, 2024-08-31T12:19:51.601 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:19:51.601 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:19:51.601 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:19:51.601 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:19:51.601 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:19:51.601 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:19:51.601 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:19:51.601 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:19:51.601 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:19:51.601 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:19:51.601 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:19:51.602 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:19:51.602 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:19:51.602 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:19:51.602 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:51.602 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:19:51.602 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:19:51.602 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:19:51.602 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:19:51.602 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:19:51.602 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:19:51.602 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:19:51.602 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:19:51.602 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:19:51.602 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:19:51.603 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:19:51.603 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:19:51.603 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:19:51.603 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:51.603 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:51.603 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:19:51.603 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:19:51.603 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:19:51.603 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:19:51.603 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:19:51.603 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:19:51.603 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:19:51.603 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:19:51.604 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:19:51.604 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:19:51.604 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:19:51.604 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:19:51.604 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:19:51.604 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:19:51.604 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:19:51.604 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:19:51.604 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:19:51.604 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:19:51.604 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:19:51.605 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:19:51.605 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:19:51.605 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:19:51.605 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:19:51.605 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:19:51.605 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:19:51.605 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:19:51.605 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:19:51.605 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:19:51.605 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:51.605 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:19:51.605 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:51.605 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:19:51.606 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:51.606 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:19:51.606 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:19:51.606 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:19:51.606 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:19:51.606 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:51.606 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:19:51.606 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:19:51.606 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:51.606 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:51.606 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:51.606 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:19:51.606 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:19:51.607 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:51.607 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:51.607 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:51.607 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:51.607 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:19:51.607 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:19:51.607 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:19:51.607 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:19:51.607 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:19:51.607 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:51.607 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:51.607 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:19:51.608 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:19:51.608 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:19:51.608 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:19:51.608 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:51.608 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:19:51.608 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:19:51.608 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:51.608 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:51.608 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:51.608 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:19:51.608 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:19:51.608 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:51.608 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:51.609 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:51.609 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:51.609 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:19:51.609 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:19:51.609 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:19:51.609 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:19:51.610 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:19:51.610 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:51.610 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:51.610 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:19:51.610 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:19:51.610 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:19:51.610 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:19:51.610 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:51.610 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:19:51.610 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:19:51.610 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:51.610 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:51.611 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:51.611 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:19:51.611 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:19:51.611 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:51.611 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:51.611 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:51.611 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:51.611 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:19:51.611 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:19:51.611 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:19:51.611 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:19:51.611 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:19:51.611 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:51.612 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:51.612 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:51.612 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:19:51.612 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:19:51.612 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:51.612 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:19:51.612 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:19:51.612 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:19:51.612 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:51.612 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:51.612 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:19:51.612 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:51.612 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:19:51.612 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:19:51.613 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:19:51.613 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:51.613 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:51.613 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:51.613 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:19:51.613 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:19:51.615 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.597+0000 7f7f28c7e640 1 -- 172.21.15.19:0/3360967655 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7f04077980 msgr2=0x7f7f04079e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:51.615 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.597+0000 7f7f28c7e640 1 --2- 172.21.15.19:0/3360967655 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7f04077980 0x7f7f04079e90 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c035000 tx=0x7f7f1c038040 comp rx=0 tx=0).stop 2024-08-31T12:19:51.615 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.597+0000 7f7f28c7e640 1 -- 172.21.15.19:0/3360967655 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7f240cb2f0 msgr2=0x7f7f241f5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:51.615 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.597+0000 7f7f28c7e640 1 --2- 172.21.15.19:0/3360967655 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7f240cb2f0 0x7f7f241f5030 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c00e7c0 tx=0x7f7f0c00ec80 comp rx=0 tx=0).stop 2024-08-31T12:19:51.615 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.597+0000 7f7f22d9d640 1 -- 172.21.15.19:0/3360967655 reap_dead start 2024-08-31T12:19:51.615 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.597+0000 7f7f28c7e640 1 -- 172.21.15.19:0/3360967655 shutdown_connections 2024-08-31T12:19:51.615 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.597+0000 7f7f28c7e640 1 --2- 172.21.15.19:0/3360967655 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7f240cb2f0 0x7f7f241f5030 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:51.615 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.597+0000 7f7f28c7e640 1 -- 172.21.15.19:0/3360967655 >> 172.21.15.19:0/3360967655 conn(0x7f7f24074a90 msgr2=0x7f7f240b5400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:19:51.615 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.597+0000 7f7f28c7e640 1 -- 172.21.15.19:0/3360967655 shutdown_connections 2024-08-31T12:19:51.615 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.597+0000 7f7f28c7e640 1 -- 172.21.15.19:0/3360967655 wait complete. 2024-08-31T12:19:51.616 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:19:51.616 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-08-31T12:19:51.616 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-31T12:19:51.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.736+0000 7feefb9cc640 1 -- 172.21.15.19:0/1075176916 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7feef40c9030 msgr2=0x7feef40cb040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:51.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.736+0000 7feefb9cc640 1 --2- 172.21.15.19:0/1075176916 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7feef40c9030 0x7feef40cb040 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7feee4005fd0 tx=0x7feee402a5c0 comp rx=0 tx=0).stop 2024-08-31T12:19:51.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.736+0000 7feefb9cc640 1 -- 172.21.15.19:0/1075176916 shutdown_connections 2024-08-31T12:19:51.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.736+0000 7feefb9cc640 1 --2- 172.21.15.19:0/1075176916 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7feef40c9030 0x7feef40cb040 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:51.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.736+0000 7feefb9cc640 1 --2- 172.21.15.19:0/1075176916 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feef4153190 0x7feef4159820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:51.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.736+0000 7feefb9cc640 1 -- 172.21.15.19:0/1075176916 >> 172.21.15.19:0/1075176916 conn(0x7feef40728b0 msgr2=0x7feef4072cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:19:51.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.736+0000 7feefb9cc640 1 -- 172.21.15.19:0/1075176916 shutdown_connections 2024-08-31T12:19:51.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.736+0000 7feefb9cc640 1 -- 172.21.15.19:0/1075176916 wait complete. 2024-08-31T12:19:51.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.738+0000 7feefb9cc640 1 Processor -- start 2024-08-31T12:19:51.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.738+0000 7feefb9cc640 1 -- start start 2024-08-31T12:19:51.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.739+0000 7feefb9cc640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7feef40c9030 0x7feef406f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:51.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.739+0000 7feefb9cc640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feef40c9440 0x7feef415b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:51.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.739+0000 7feefb9cc640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feef4153190 0x7feef4140e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:51.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.739+0000 7feefb9cc640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7feef41535a0 con 0x7feef40c9030 2024-08-31T12:19:51.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.739+0000 7feefb9cc640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7feef40c4db0 con 0x7feef40c9440 2024-08-31T12:19:51.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.739+0000 7feefb9cc640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7feef415cdc0 con 0x7feef4153190 2024-08-31T12:19:51.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.739+0000 7feef9768640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7feef40c9030 0x7feef406f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:51.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.739+0000 7feef9768640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7feef40c9030 0x7feef406f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:41918/0 (socket says 172.21.15.19:41918) 2024-08-31T12:19:51.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.739+0000 7feef9768640 1 -- 172.21.15.19:0/3118935816 learned_addr learned my addr 172.21.15.19:0/3118935816 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:19:51.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.739+0000 7feef8f67640 1 --2- 172.21.15.19:0/3118935816 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feef40c9440 0x7feef415b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:51.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.739+0000 7feef9f69640 1 --2- 172.21.15.19:0/3118935816 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feef4153190 0x7feef4140e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:51.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.739+0000 7feef9768640 1 -- 172.21.15.19:0/3118935816 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feef4153190 msgr2=0x7feef4140e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:51.744 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.739+0000 7feef9768640 1 --2- 172.21.15.19:0/3118935816 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feef4153190 0x7feef4140e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:51.744 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.739+0000 7feef9768640 1 -- 172.21.15.19:0/3118935816 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feef40c9440 msgr2=0x7feef415b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:51.744 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.739+0000 7feef9768640 1 --2- 172.21.15.19:0/3118935816 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feef40c9440 0x7feef415b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:51.744 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.739+0000 7feef9768640 1 -- 172.21.15.19:0/3118935816 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feef40ae960 con 0x7feef40c9030 2024-08-31T12:19:51.744 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.740+0000 7feef9768640 1 --2- 172.21.15.19:0/3118935816 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7feef40c9030 0x7feef406f870 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7feef000d6d0 tx=0x7feef000db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:51.744 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.740+0000 7feee9ffb640 1 -- 172.21.15.19:0/3118935816 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef0017070 con 0x7feef40c9030 2024-08-31T12:19:51.744 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.740+0000 7feee9ffb640 1 -- 172.21.15.19:0/3118935816 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feef0005330 con 0x7feef40c9030 2024-08-31T12:19:51.744 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.740+0000 7feefb9cc640 1 -- 172.21.15.19:0/3118935816 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feef414d660 con 0x7feef40c9030 2024-08-31T12:19:51.744 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.740+0000 7feee9ffb640 1 -- 172.21.15.19:0/3118935816 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef0009820 con 0x7feef40c9030 2024-08-31T12:19:51.744 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.740+0000 7feefb9cc640 1 -- 172.21.15.19:0/3118935816 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feef40aebd0 con 0x7feef40c9030 2024-08-31T12:19:51.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.741+0000 7feefb9cc640 1 --2- 172.21.15.19:0/3118935816 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feef406e600 0x7feef4141340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:51.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.741+0000 7feefb9cc640 1 -- 172.21.15.19:0/3118935816 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7feef4159820 con 0x7feef406e600 2024-08-31T12:19:51.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.741+0000 7feef8f67640 1 --2- 172.21.15.19:0/3118935816 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feef406e600 0x7feef4141340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:51.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.742+0000 7feef8f67640 1 -- 172.21.15.19:0/3118935816 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feef40c9a20 con 0x7feef406e600 2024-08-31T12:19:51.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.742+0000 7feef8f67640 1 --2- 172.21.15.19:0/3118935816 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feef406e600 0x7feef4141340 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7feef4074d30 tx=0x7feee40035f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:51.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.742+0000 7feee9ffb640 1 -- 172.21.15.19:0/3118935816 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee4040070 con 0x7feef406e600 2024-08-31T12:19:51.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.742+0000 7feee9ffb640 1 -- 172.21.15.19:0/3118935816 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7feef00099c0 con 0x7feef40c9030 2024-08-31T12:19:51.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.743+0000 7feee9ffb640 1 --2- 172.21.15.19:0/3118935816 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7feed8077950 0x7feed8079e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:51.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.743+0000 7feee9ffb640 1 -- 172.21.15.19:0/3118935816 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7feef0099700 con 0x7feef40c9030 2024-08-31T12:19:51.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.743+0000 7feee9ffb640 1 -- 172.21.15.19:0/3118935816 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7feee4003c80 con 0x7feef406e600 2024-08-31T12:19:51.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.743+0000 7feee9ffb640 1 -- 172.21.15.19:0/3118935816 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feef406e600 msgr2=0x7feef4141340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:51.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.743+0000 7feee9ffb640 1 --2- 172.21.15.19:0/3118935816 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feef406e600 0x7feef4141340 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7feef4074d30 tx=0x7feee40035f0 comp rx=0 tx=0).stop 2024-08-31T12:19:51.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.743+0000 7feee9ffb640 1 -- 172.21.15.19:0/3118935816 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feef406e600 msgr2=0x7feef4141340 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:19:51.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.743+0000 7feee9ffb640 1 --2- 172.21.15.19:0/3118935816 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feef406e600 0x7feef4141340 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:51.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.743+0000 7feef9f69640 1 --2- 172.21.15.19:0/3118935816 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7feed8077950 0x7feed8079e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:51.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.744+0000 7feef9f69640 1 --2- 172.21.15.19:0/3118935816 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7feed8077950 0x7feed8079e60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7feef40b1d50 tx=0x7feeec00d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:51.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.751+0000 7feefb9cc640 1 --2- 172.21.15.19:0/3118935816 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feef414d830 0x7feef41f4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:51.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.751+0000 7feefb9cc640 1 -- 172.21.15.19:0/3118935816 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7feef40c4530 con 0x7feef414d830 2024-08-31T12:19:51.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.751+0000 7feef8f67640 1 --2- 172.21.15.19:0/3118935816 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feef414d830 0x7feef41f4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:51.754 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.752+0000 7feef8f67640 1 -- 172.21.15.19:0/3118935816 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7feef40c4250 con 0x7feef414d830 2024-08-31T12:19:51.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.752+0000 7feef8f67640 1 --2- 172.21.15.19:0/3118935816 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feef414d830 0x7feef41f4f80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7feee4009850 tx=0x7feee4009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:51.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.752+0000 7feee9ffb640 1 -- 172.21.15.19:0/3118935816 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee4035040 con 0x7feef414d830 2024-08-31T12:19:51.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.752+0000 7feee9ffb640 1 -- 172.21.15.19:0/3118935816 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4708 (secure 0 0 0) 0x7feee4044b90 con 0x7feef414d830 2024-08-31T12:19:51.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.753+0000 7feee9ffb640 1 -- 172.21.15.19:0/3118935816 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feef414d830 msgr2=0x7feef41f4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:51.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.753+0000 7feee9ffb640 1 --2- 172.21.15.19:0/3118935816 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feef414d830 0x7feef41f4f80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7feee4009850 tx=0x7feee4009260 comp rx=0 tx=0).stop 2024-08-31T12:19:51.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.753+0000 7feee9ffb640 1 -- 172.21.15.19:0/3118935816 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feef414d830 msgr2=0x7feef41f4f80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:19:51.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.753+0000 7feee9ffb640 1 --2- 172.21.15.19:0/3118935816 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feef414d830 0x7feef41f4f80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:51.755 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:19:51.755 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:19:51.755 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:19:51.756 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:19:51.756 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 32, 2024-08-31T12:19:51.756 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 119875, 2024-08-31T12:19:51.756 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:19:51.756 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:19:51.756 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:19:51.756 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:19:51.756 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:51.756 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 2, 2024-08-31T12:19:51.756 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:19:51.756 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:19:51.757 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:19:51.757 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:19:51.757 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:19:51.757 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:19:51.757 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:19:51.757 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:19:51.757 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:19:51.757 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:19:51.757 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:19:51.757 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:19:51.757 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:19:51.757 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:19:51.757 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:51.758 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:19:51.758 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:19:51.758 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:19:51.758 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:19:51.758 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:19:51.758 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:19:51.758 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:19:51.758 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:19:51.758 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:19:51.758 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:19:51.758 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:19:51.758 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:19:51.758 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:19:51.758 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:51.758 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:51.758 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:19:51.759 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:19:51.759 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:19:51.759 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:19:51.759 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:19:51.759 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:19:51.759 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:19:51.759 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:19:51.759 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:19:51.759 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:19:51.759 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:19:51.759 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:19:51.759 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:19:51.759 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:19:51.760 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:19:51.760 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:19:51.760 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:19:51.760 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:19:51.760 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:19:51.760 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:19:51.760 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:19:51.760 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:19:51.760 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:19:51.761 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:19:51.761 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:19:51.761 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:19:51.761 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:19:51.761 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:19:51.761 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:51.761 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:19:51.761 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:51.761 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:19:51.761 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:51.761 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:19:51.761 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:19:51.762 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:19:51.762 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:19:51.762 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:51.762 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:19:51.762 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:19:51.762 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:51.762 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:51.762 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:51.762 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:19:51.762 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:19:51.762 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:51.763 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:51.763 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:51.763 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:51.763 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:19:51.763 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:19:51.763 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:19:51.763 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:19:51.763 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:19:51.763 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:51.763 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:51.763 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:19:51.763 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:19:51.764 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:19:51.764 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:19:51.764 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:51.764 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:19:51.764 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:19:51.764 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:51.764 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:51.764 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:51.764 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:19:51.764 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:19:51.764 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:51.764 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:51.765 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:51.765 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:51.765 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:19:51.765 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:19:51.765 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:19:51.765 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:19:51.765 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:19:51.765 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:51.765 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:51.765 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:19:51.766 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:19:51.766 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:19:51.766 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:19:51.766 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:51.766 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:19:51.766 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:19:51.766 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:51.766 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:51.766 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:51.766 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:19:51.766 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:19:51.766 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:19:51.766 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:51.766 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:51.767 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:51.767 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:19:51.767 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:19:51.767 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:19:51.767 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:19:51.767 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:19:51.767 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:51.767 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:51.767 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:51.767 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:19:51.767 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:19:51.767 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:51.767 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:19:51.767 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:19:51.768 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:19:51.768 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:51.768 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:51.768 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:19:51.768 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:51.768 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:19:51.768 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:19:51.768 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:19:51.768 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:51.768 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:19:51.768 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:19:51.768 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:19:51.768 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:19:51.768 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:19:51.769 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:19:51.769 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:19:51.769 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:19:51.769 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:19:51.769 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:19:51.769 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:19:51.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.753+0000 7feefb9cc640 1 -- 172.21.15.19:0/3118935816 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7feed8077950 msgr2=0x7feed8079e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:51.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.753+0000 7feefb9cc640 1 --2- 172.21.15.19:0/3118935816 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7feed8077950 0x7feed8079e60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7feef40b1d50 tx=0x7feeec00d040 comp rx=0 tx=0).stop 2024-08-31T12:19:51.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.753+0000 7feefb9cc640 1 -- 172.21.15.19:0/3118935816 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7feef40c9030 msgr2=0x7feef406f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:51.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.753+0000 7feefb9cc640 1 --2- 172.21.15.19:0/3118935816 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7feef40c9030 0x7feef406f870 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7feef000d6d0 tx=0x7feef000db90 comp rx=0 tx=0).stop 2024-08-31T12:19:51.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.753+0000 7feef9f69640 1 -- 172.21.15.19:0/3118935816 reap_dead start 2024-08-31T12:19:51.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.753+0000 7feef9f69640 1 -- 172.21.15.19:0/3118935816 reap_dead start 2024-08-31T12:19:51.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.753+0000 7feefb9cc640 1 -- 172.21.15.19:0/3118935816 shutdown_connections 2024-08-31T12:19:51.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.753+0000 7feefb9cc640 1 -- 172.21.15.19:0/3118935816 >> 172.21.15.19:0/3118935816 conn(0x7feef40728b0 msgr2=0x7feef4158010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:19:51.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.753+0000 7feefb9cc640 1 -- 172.21.15.19:0/3118935816 shutdown_connections 2024-08-31T12:19:51.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.753+0000 7feefb9cc640 1 -- 172.21.15.19:0/3118935816 wait complete. 2024-08-31T12:19:51.771 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:19:51.771 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-08-31T12:19:51.771 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-08-31T12:19:51.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.900+0000 7f0df339d640 1 -- 172.21.15.19:0/2327636356 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0dec14e530 msgr2=0x7f0dec157b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:51.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.900+0000 7f0df339d640 1 --2- 172.21.15.19:0/2327636356 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0dec14e530 0x7f0dec157b30 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0dec06eea0 tx=0x7f0ddc02a5f0 comp rx=0 tx=0).stop 2024-08-31T12:19:51.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.900+0000 7f0df339d640 1 -- 172.21.15.19:0/2327636356 shutdown_connections 2024-08-31T12:19:51.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.900+0000 7f0df339d640 1 --2- 172.21.15.19:0/2327636356 >> v2:172.21.15.43:3300/0 conn(0x7f0dec158070 0x7f0dec06dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:51.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.900+0000 7f0df339d640 1 --2- 172.21.15.19:0/2327636356 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0dec14e530 0x7f0dec157b30 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:51.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.900+0000 7f0df339d640 1 --2- 172.21.15.19:0/2327636356 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0dec0cb070 0x7f0dec0c4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:51.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.900+0000 7f0df339d640 1 -- 172.21.15.19:0/2327636356 >> 172.21.15.19:0/2327636356 conn(0x7f0dec0718e0 msgr2=0x7f0dec071d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:19:51.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.900+0000 7f0df339d640 1 -- 172.21.15.19:0/2327636356 shutdown_connections 2024-08-31T12:19:51.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.900+0000 7f0df339d640 1 -- 172.21.15.19:0/2327636356 wait complete. 2024-08-31T12:19:51.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.902+0000 7f0df339d640 1 Processor -- start 2024-08-31T12:19:51.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.902+0000 7f0df339d640 1 -- start start 2024-08-31T12:19:51.907 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.903+0000 7f0df339d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0dec0cb070 0x7f0dec0c8730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:51.907 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.903+0000 7f0df339d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0dec14e530 0x7f0dec1f2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:51.907 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.903+0000 7f0df339d640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0dec158070 0x7f0dec1573a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:51.907 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.903+0000 7f0df339d640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f0dec0c6d00 con 0x7f0dec14e530 2024-08-31T12:19:51.907 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.903+0000 7f0df339d640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0dec0c6a20 con 0x7f0dec158070 2024-08-31T12:19:51.907 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.903+0000 7f0df339d640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f0dec14e940 con 0x7f0dec0cb070 2024-08-31T12:19:51.907 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.903+0000 7f0df1b9a640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0dec14e530 0x7f0dec1f2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:51.907 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.903+0000 7f0df1b9a640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0dec14e530 0x7f0dec1f2140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:41938/0 (socket says 172.21.15.19:41938) 2024-08-31T12:19:51.907 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.903+0000 7f0df1b9a640 1 -- 172.21.15.19:0/1629029695 learned_addr learned my addr 172.21.15.19:0/1629029695 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:19:51.907 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.903+0000 7f0df239b640 1 --2- 172.21.15.19:0/1629029695 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0dec0cb070 0x7f0dec0c8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:51.907 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.903+0000 7f0df2b9c640 1 --2- 172.21.15.19:0/1629029695 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0dec158070 0x7f0dec1573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:51.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.904+0000 7f0df1b9a640 1 -- 172.21.15.19:0/1629029695 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0dec0cb070 msgr2=0x7f0dec0c8730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:51.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.904+0000 7f0df1b9a640 1 --2- 172.21.15.19:0/1629029695 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0dec0cb070 0x7f0dec0c8730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:51.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.904+0000 7f0df1b9a640 1 -- 172.21.15.19:0/1629029695 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0dec158070 msgr2=0x7f0dec1573a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:51.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.904+0000 7f0df1b9a640 1 --2- 172.21.15.19:0/1629029695 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0dec158070 0x7f0dec1573a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:51.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.904+0000 7f0df1b9a640 1 -- 172.21.15.19:0/1629029695 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dec1f2680 con 0x7f0dec14e530 2024-08-31T12:19:51.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.904+0000 7f0df1b9a640 1 --2- 172.21.15.19:0/1629029695 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0dec14e530 0x7f0dec1f2140 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc034040 tx=0x7f0ddc0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:51.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.904+0000 7f0de2ffd640 1 -- 172.21.15.19:0/1629029695 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ddc03e030 con 0x7f0dec14e530 2024-08-31T12:19:51.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.904+0000 7f0de2ffd640 1 -- 172.21.15.19:0/1629029695 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0ddc003ca0 con 0x7f0dec14e530 2024-08-31T12:19:51.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.904+0000 7f0df339d640 1 -- 172.21.15.19:0/1629029695 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dec0c9480 con 0x7f0dec14e530 2024-08-31T12:19:51.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.904+0000 7f0de2ffd640 1 -- 172.21.15.19:0/1629029695 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ddc035040 con 0x7f0dec14e530 2024-08-31T12:19:51.909 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.904+0000 7f0df339d640 1 -- 172.21.15.19:0/1629029695 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dec1f1940 con 0x7f0dec14e530 2024-08-31T12:19:51.909 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.906+0000 7f0df339d640 1 -- 172.21.15.19:0/1629029695 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dec1f1730 con 0x7f0dec14e530 2024-08-31T12:19:51.909 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.906+0000 7f0de2ffd640 1 -- 172.21.15.19:0/1629029695 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f0ddc041030 con 0x7f0dec14e530 2024-08-31T12:19:51.910 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.907+0000 7f0de2ffd640 1 --2- 172.21.15.19:0/1629029695 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0dd0077a80 0x7f0dd0079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:19:51.910 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.907+0000 7f0de2ffd640 1 -- 172.21.15.19:0/1629029695 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f0ddc0b9d50 con 0x7f0dec14e530 2024-08-31T12:19:51.910 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.907+0000 7f0df239b640 1 --2- 172.21.15.19:0/1629029695 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0dd0077a80 0x7f0dd0079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:19:51.910 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.908+0000 7f0df239b640 1 --2- 172.21.15.19:0/1629029695 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0dd0077a80 0x7f0dd0079f90 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0de8009b30 tx=0x7f0de8005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:19:51.914 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:51.913+0000 7f0de2ffd640 1 -- 172.21.15.19:0/1629029695 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f0ddc0bd030 con 0x7f0dec14e530 2024-08-31T12:19:52.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:52.215+0000 7f0df339d640 1 -- 172.21.15.19:0/1629029695 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f0dec070e80 con 0x7f0dec14e530 2024-08-31T12:19:52.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:52.216+0000 7f0de2ffd640 1 -- 172.21.15.19:0/1629029695 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f0ddc0823f0 con 0x7f0dec14e530 2024-08-31T12:19:52.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:52.218+0000 7f0df339d640 1 -- 172.21.15.19:0/1629029695 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0dd0077a80 msgr2=0x7f0dd0079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:52.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:52.218+0000 7f0df339d640 1 --2- 172.21.15.19:0/1629029695 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0dd0077a80 0x7f0dd0079f90 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0de8009b30 tx=0x7f0de8005f90 comp rx=0 tx=0).stop 2024-08-31T12:19:52.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:52.218+0000 7f0df339d640 1 -- 172.21.15.19:0/1629029695 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0dec14e530 msgr2=0x7f0dec1f2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:19:52.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:52.218+0000 7f0df339d640 1 --2- 172.21.15.19:0/1629029695 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0dec14e530 0x7f0dec1f2140 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc034040 tx=0x7f0ddc0038a0 comp rx=0 tx=0).stop 2024-08-31T12:19:52.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:52.219+0000 7f0df339d640 1 -- 172.21.15.19:0/1629029695 shutdown_connections 2024-08-31T12:19:52.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:52.219+0000 7f0df339d640 1 --2- 172.21.15.19:0/1629029695 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0dd0077a80 0x7f0dd0079f90 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:52.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:52.219+0000 7f0df339d640 1 --2- 172.21.15.19:0/1629029695 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0dec158070 0x7f0dec1573a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:52.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:52.219+0000 7f0df339d640 1 --2- 172.21.15.19:0/1629029695 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0dec14e530 0x7f0dec1f2140 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:52.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:52.219+0000 7f0df339d640 1 --2- 172.21.15.19:0/1629029695 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0dec0cb070 0x7f0dec0c8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:19:52.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:52.219+0000 7f0df339d640 1 -- 172.21.15.19:0/1629029695 >> 172.21.15.19:0/1629029695 conn(0x7f0dec0718e0 msgr2=0x7f0dec140f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:19:52.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:52.219+0000 7f0df339d640 1 -- 172.21.15.19:0/1629029695 shutdown_connections 2024-08-31T12:19:52.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:19:52.219+0000 7f0df339d640 1 -- 172.21.15.19:0/1629029695 wait complete. 2024-08-31T12:19:52.233 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2024-08-31T12:19:56.325 DEBUG:teuthology.orchestra.run.smithi019:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:19:56.327 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:20:02.233 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-08-31T12:20:02.234 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:20:02.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.355+0000 7f6110a84640 1 -- 172.21.15.19:0/1591661368 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f610c0cb280 msgr2=0x7f610c15b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:02.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.355+0000 7f6110a84640 1 --2- 172.21.15.19:0/1591661368 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f610c0cb280 0x7f610c15b120 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f60f4005fd0 tx=0x7f60f402a710 comp rx=0 tx=0).stop 2024-08-31T12:20:02.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.355+0000 7f6110a84640 1 -- 172.21.15.19:0/1591661368 shutdown_connections 2024-08-31T12:20:02.358 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.355+0000 7f6110a84640 1 --2- 172.21.15.19:0/1591661368 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f610c0cb280 0x7f610c15b120 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:02.358 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.355+0000 7f6110a84640 1 -- 172.21.15.19:0/1591661368 >> 172.21.15.19:0/1591661368 conn(0x7f610c074810 msgr2=0x7f610c068aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:02.358 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.355+0000 7f6110a84640 1 -- 172.21.15.19:0/1591661368 shutdown_connections 2024-08-31T12:20:02.358 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.355+0000 7f6110a84640 1 -- 172.21.15.19:0/1591661368 wait complete. 2024-08-31T12:20:02.359 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.357+0000 7f6110a84640 1 Processor -- start 2024-08-31T12:20:02.359 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.357+0000 7f6110a84640 1 -- start start 2024-08-31T12:20:02.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.359+0000 7f6110a84640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f610c0cb280 0x7f610c0c7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:02.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.359+0000 7f6110a84640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f610c0ccbb0 0x7f610c0d0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:02.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.359+0000 7f6110a84640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f610c15b660 0x7f610c0c8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:02.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.359+0000 7f6110a84640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f610c15ba70 con 0x7f610c0ccbb0 2024-08-31T12:20:02.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.359+0000 7f6110a84640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f610c15f050 con 0x7f610c0cb280 2024-08-31T12:20:02.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.359+0000 7f6110a84640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f610c0cc780 con 0x7f610c15b660 2024-08-31T12:20:02.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.359+0000 7f6109d9b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f610c0ccbb0 0x7f610c0d0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:02.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.359+0000 7f6109d9b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f610c0ccbb0 0x7f610c0d0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:48042/0 (socket says 172.21.15.19:48042) 2024-08-31T12:20:02.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.359+0000 7f6109d9b640 1 -- 172.21.15.19:0/1282824415 learned_addr learned my addr 172.21.15.19:0/1282824415 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:20:02.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.359+0000 7f610a59c640 1 --2- 172.21.15.19:0/1282824415 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f610c0cb280 0x7f610c0c7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:02.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.359+0000 7f610ad9d640 1 --2- 172.21.15.19:0/1282824415 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f610c15b660 0x7f610c0c8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:02.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.359+0000 7f6109d9b640 1 -- 172.21.15.19:0/1282824415 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f610c15b660 msgr2=0x7f610c0c8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:02.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.359+0000 7f6109d9b640 1 --2- 172.21.15.19:0/1282824415 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f610c15b660 0x7f610c0c8bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:02.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.359+0000 7f6109d9b640 1 -- 172.21.15.19:0/1282824415 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f610c0cb280 msgr2=0x7f610c0c7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:02.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.359+0000 7f6109d9b640 1 --2- 172.21.15.19:0/1282824415 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f610c0cb280 0x7f610c0c7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:02.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.359+0000 7f6109d9b640 1 -- 172.21.15.19:0/1282824415 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f610c15b440 con 0x7f610c0ccbb0 2024-08-31T12:20:02.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.359+0000 7f6109d9b640 1 --2- 172.21.15.19:0/1282824415 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f610c0ccbb0 0x7f610c0d0d50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f60f4006010 tx=0x7f60f40039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:02.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.360+0000 7f6102ffd640 1 -- 172.21.15.19:0/1282824415 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f403e030 con 0x7f610c0ccbb0 2024-08-31T12:20:02.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.360+0000 7f6102ffd640 1 -- 172.21.15.19:0/1282824415 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f60f40047a0 con 0x7f610c0ccbb0 2024-08-31T12:20:02.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.360+0000 7f6110a84640 1 -- 172.21.15.19:0/1282824415 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f610c0d0650 con 0x7f610c0ccbb0 2024-08-31T12:20:02.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.360+0000 7f6102ffd640 1 -- 172.21.15.19:0/1282824415 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f4035040 con 0x7f610c0ccbb0 2024-08-31T12:20:02.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.360+0000 7f6110a84640 1 -- 172.21.15.19:0/1282824415 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f610c1521d0 con 0x7f610c0ccbb0 2024-08-31T12:20:02.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.361+0000 7f6110a84640 1 -- 172.21.15.19:0/1282824415 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f610c155660 con 0x7f610c0ccbb0 2024-08-31T12:20:02.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.362+0000 7f6102ffd640 1 -- 172.21.15.19:0/1282824415 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f60f4041030 con 0x7f610c0ccbb0 2024-08-31T12:20:02.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.362+0000 7f6102ffd640 1 --2- 172.21.15.19:0/1282824415 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f60ec077a80 0x7f60ec079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:02.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.363+0000 7f6102ffd640 1 -- 172.21.15.19:0/1282824415 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f60f40b9e70 con 0x7f610c0ccbb0 2024-08-31T12:20:02.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.363+0000 7f610a59c640 1 --2- 172.21.15.19:0/1282824415 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f60ec077a80 0x7f60ec079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:02.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.363+0000 7f610a59c640 1 --2- 172.21.15.19:0/1282824415 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f60ec077a80 0x7f60ec079f90 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f60fc001e20 tx=0x7f60fc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:02.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.367+0000 7f6102ffd640 1 -- 172.21.15.19:0/1282824415 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f60f4082520 con 0x7f610c0ccbb0 2024-08-31T12:20:02.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.671+0000 7f6110a84640 1 -- 172.21.15.19:0/1282824415 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f610c0cbe50 con 0x7f610c0ccbb0 2024-08-31T12:20:02.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.671+0000 7f6102ffd640 1 -- 172.21.15.19:0/1282824415 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f60f4086ce0 con 0x7f610c0ccbb0 2024-08-31T12:20:02.674 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:20:02.676 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.674+0000 7f6110a84640 1 -- 172.21.15.19:0/1282824415 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f60ec077a80 msgr2=0x7f60ec079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:02.676 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.674+0000 7f6110a84640 1 --2- 172.21.15.19:0/1282824415 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f60ec077a80 0x7f60ec079f90 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f60fc001e20 tx=0x7f60fc009040 comp rx=0 tx=0).stop 2024-08-31T12:20:02.676 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.674+0000 7f6110a84640 1 -- 172.21.15.19:0/1282824415 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f610c0ccbb0 msgr2=0x7f610c0d0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:02.676 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.674+0000 7f6110a84640 1 --2- 172.21.15.19:0/1282824415 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f610c0ccbb0 0x7f610c0d0d50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f60f4006010 tx=0x7f60f40039e0 comp rx=0 tx=0).stop 2024-08-31T12:20:02.676 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.674+0000 7f6110a84640 1 -- 172.21.15.19:0/1282824415 shutdown_connections 2024-08-31T12:20:02.677 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.674+0000 7f6110a84640 1 --2- 172.21.15.19:0/1282824415 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f60ec077a80 0x7f60ec079f90 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:02.677 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.674+0000 7f6110a84640 1 --2- 172.21.15.19:0/1282824415 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f610c15b660 0x7f610c0c8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:02.677 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.674+0000 7f6110a84640 1 --2- 172.21.15.19:0/1282824415 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f610c0ccbb0 0x7f610c0d0d50 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:02.677 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.674+0000 7f6110a84640 1 --2- 172.21.15.19:0/1282824415 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f610c0cb280 0x7f610c0c7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:02.677 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.674+0000 7f6110a84640 1 -- 172.21.15.19:0/1282824415 >> 172.21.15.19:0/1282824415 conn(0x7f610c074810 msgr2=0x7f610c0cc070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:02.677 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.674+0000 7f6110a84640 1 -- 172.21.15.19:0/1282824415 shutdown_connections 2024-08-31T12:20:02.677 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.674+0000 7f6110a84640 1 -- 172.21.15.19:0/1282824415 wait complete. 2024-08-31T12:20:02.689 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-08-31T12:20:02.689 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-08-31T12:20:02.689 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-08-31T12:20:02.690 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-31T12:20:02.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.811+0000 7fb67dd8b640 1 -- 172.21.15.19:0/329285670 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb678155300 msgr2=0x7fb67806d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:02.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.811+0000 7fb67dd8b640 1 --2- 172.21.15.19:0/329285670 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb678155300 0x7fb67806d000 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb67815b0f0 tx=0x7fb67002ab40 comp rx=0 tx=0).stop 2024-08-31T12:20:02.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.811+0000 7fb67dd8b640 1 -- 172.21.15.19:0/329285670 shutdown_connections 2024-08-31T12:20:02.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.811+0000 7fb67dd8b640 1 --2- 172.21.15.19:0/329285670 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb678155300 0x7fb67806d000 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:02.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.811+0000 7fb67dd8b640 1 --2- 172.21.15.19:0/329285670 >> v2:172.21.15.43:3300/0 conn(0x7fb67806e0e0 0x7fb6780c9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:02.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.811+0000 7fb67dd8b640 1 -- 172.21.15.19:0/329285670 >> 172.21.15.19:0/329285670 conn(0x7fb6780690d0 msgr2=0x7fb6780d0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:02.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.811+0000 7fb67dd8b640 1 -- 172.21.15.19:0/329285670 shutdown_connections 2024-08-31T12:20:02.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.811+0000 7fb67dd8b640 1 -- 172.21.15.19:0/329285670 wait complete. 2024-08-31T12:20:02.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.813+0000 7fb67dd8b640 1 Processor -- start 2024-08-31T12:20:02.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.813+0000 7fb67dd8b640 1 -- start start 2024-08-31T12:20:02.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.815+0000 7fb67dd8b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb67806e0e0 0x7fb6780c5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:02.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.815+0000 7fb67dd8b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb6780c8a40 0x7fb6781f19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:02.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.815+0000 7fb67dd8b640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb678155300 0x7fb6780c2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:02.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.815+0000 7fb67dd8b640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fb6780c8330 con 0x7fb6780c8a40 2024-08-31T12:20:02.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.815+0000 7fb67dd8b640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb67815b250 con 0x7fb678155300 2024-08-31T12:20:02.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.815+0000 7fb67dd8b640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fb6780c8610 con 0x7fb67806e0e0 2024-08-31T12:20:02.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.815+0000 7fb676ffd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb6780c8a40 0x7fb6781f19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:02.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.815+0000 7fb676ffd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb6780c8a40 0x7fb6781f19e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:48068/0 (socket says 172.21.15.19:48068) 2024-08-31T12:20:02.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.815+0000 7fb676ffd640 1 -- 172.21.15.19:0/2616717784 learned_addr learned my addr 172.21.15.19:0/2616717784 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:20:02.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.815+0000 7fb6777fe640 1 --2- 172.21.15.19:0/2616717784 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb67806e0e0 0x7fb6780c5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:02.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.815+0000 7fb677fff640 1 --2- 172.21.15.19:0/2616717784 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb678155300 0x7fb6780c2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:02.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.815+0000 7fb676ffd640 1 -- 172.21.15.19:0/2616717784 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb67806e0e0 msgr2=0x7fb6780c5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:02.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.815+0000 7fb676ffd640 1 --2- 172.21.15.19:0/2616717784 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb67806e0e0 0x7fb6780c5360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:02.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.815+0000 7fb676ffd640 1 -- 172.21.15.19:0/2616717784 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb678155300 msgr2=0x7fb6780c2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:02.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.815+0000 7fb676ffd640 1 --2- 172.21.15.19:0/2616717784 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb678155300 0x7fb6780c2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:02.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.815+0000 7fb676ffd640 1 -- 172.21.15.19:0/2616717784 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6780c58a0 con 0x7fb6780c8a40 2024-08-31T12:20:02.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.815+0000 7fb676ffd640 1 --2- 172.21.15.19:0/2616717784 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb6780c8a40 0x7fb6781f19e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb66c00e7c0 tx=0x7fb66c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:02.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.815+0000 7fb65ffff640 1 -- 172.21.15.19:0/2616717784 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb66c00ccc0 con 0x7fb6780c8a40 2024-08-31T12:20:02.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.816+0000 7fb65ffff640 1 -- 172.21.15.19:0/2616717784 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb66c010430 con 0x7fb6780c8a40 2024-08-31T12:20:02.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.816+0000 7fb67dd8b640 1 -- 172.21.15.19:0/2616717784 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6781f2370 con 0x7fb6780c8a40 2024-08-31T12:20:02.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.816+0000 7fb65ffff640 1 -- 172.21.15.19:0/2616717784 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb66c010860 con 0x7fb6780c8a40 2024-08-31T12:20:02.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.816+0000 7fb67dd8b640 1 -- 172.21.15.19:0/2616717784 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6781e60f0 con 0x7fb6780c8a40 2024-08-31T12:20:02.820 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.817+0000 7fb67dd8b640 1 --2- 172.21.15.19:0/2616717784 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb6780c2850 0x7fb6780c5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:02.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.817+0000 7fb67dd8b640 1 -- 172.21.15.19:0/2616717784 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fb6780c9850 con 0x7fb6780c2850 2024-08-31T12:20:02.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.817+0000 7fb6777fe640 1 --2- 172.21.15.19:0/2616717784 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb6780c2850 0x7fb6780c5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:02.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.817+0000 7fb6777fe640 1 -- 172.21.15.19:0/2616717784 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb6780ccf20 con 0x7fb6780c2850 2024-08-31T12:20:02.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.817+0000 7fb6777fe640 1 --2- 172.21.15.19:0/2616717784 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb6780c2850 0x7fb6780c5d00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb66800f570 tx=0x7fb66800fa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:02.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.818+0000 7fb65ffff640 1 -- 172.21.15.19:0/2616717784 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6680052e0 con 0x7fb6780c2850 2024-08-31T12:20:02.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.818+0000 7fb65ffff640 1 -- 172.21.15.19:0/2616717784 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fb66c010a00 con 0x7fb6780c8a40 2024-08-31T12:20:02.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.818+0000 7fb65ffff640 1 --2- 172.21.15.19:0/2616717784 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb664077950 0x7fb664079e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:02.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.818+0000 7fb65ffff640 1 -- 172.21.15.19:0/2616717784 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fb66c09a040 con 0x7fb6780c8a40 2024-08-31T12:20:02.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.819+0000 7fb65ffff640 1 -- 172.21.15.19:0/2616717784 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb668018070 con 0x7fb6780c2850 2024-08-31T12:20:02.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.819+0000 7fb65ffff640 1 -- 172.21.15.19:0/2616717784 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb6780c2850 msgr2=0x7fb6780c5d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:02.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.819+0000 7fb65ffff640 1 --2- 172.21.15.19:0/2616717784 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb6780c2850 0x7fb6780c5d00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb66800f570 tx=0x7fb66800fa30 comp rx=0 tx=0).stop 2024-08-31T12:20:02.822 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.819+0000 7fb65ffff640 1 -- 172.21.15.19:0/2616717784 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb6780c2850 msgr2=0x7fb6780c5d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:20:02.822 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.819+0000 7fb677fff640 1 --2- 172.21.15.19:0/2616717784 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb664077950 0x7fb664079e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:02.822 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.819+0000 7fb65ffff640 1 --2- 172.21.15.19:0/2616717784 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb6780c2850 0x7fb6780c5d00 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:02.822 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.819+0000 7fb677fff640 1 --2- 172.21.15.19:0/2616717784 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb664077950 0x7fb664079e60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb670035000 tx=0x7fb670038040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:02.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.825+0000 7fb67dd8b640 1 --2- 172.21.15.19:0/2616717784 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb6780c6240 0x7fb6780c6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:02.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.825+0000 7fb67dd8b640 1 -- 172.21.15.19:0/2616717784 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fb6781e6f00 con 0x7fb6780c6240 2024-08-31T12:20:02.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.825+0000 7fb6777fe640 1 --2- 172.21.15.19:0/2616717784 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb6780c6240 0x7fb6780c6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:02.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.825+0000 7fb6777fe640 1 -- 172.21.15.19:0/2616717784 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb6781e6c20 con 0x7fb6780c6240 2024-08-31T12:20:02.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.826+0000 7fb6777fe640 1 --2- 172.21.15.19:0/2616717784 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb6780c6240 0x7fb6780c6650 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb66800f570 tx=0x7fb668000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:02.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.826+0000 7fb65ffff640 1 -- 172.21.15.19:0/2616717784 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb668005480 con 0x7fb6780c6240 2024-08-31T12:20:02.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.826+0000 7fb65ffff640 1 -- 172.21.15.19:0/2616717784 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5045 (secure 0 0 0) 0x7fb668005480 con 0x7fb6780c6240 2024-08-31T12:20:02.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.826+0000 7fb65ffff640 1 -- 172.21.15.19:0/2616717784 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb6780c6240 msgr2=0x7fb6780c6650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:02.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.826+0000 7fb65ffff640 1 --2- 172.21.15.19:0/2616717784 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb6780c6240 0x7fb6780c6650 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb66800f570 tx=0x7fb668000ed0 comp rx=0 tx=0).stop 2024-08-31T12:20:02.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.826+0000 7fb65ffff640 1 -- 172.21.15.19:0/2616717784 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb6780c6240 msgr2=0x7fb6780c6650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:20:02.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.826+0000 7fb65ffff640 1 --2- 172.21.15.19:0/2616717784 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb6780c6240 0x7fb6780c6650 secure :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb66800f570 tx=0x7fb668000ed0 comp rx=0 tx=0).stop 2024-08-31T12:20:02.829 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:20:02.829 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:20:02.829 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:20:02.829 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "leader", 2024-08-31T12:20:02.829 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 32, 2024-08-31T12:20:02.829 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 182818, 2024-08-31T12:20:02.829 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:20:02.829 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:20:02.829 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:20:02.830 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:20:02.830 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:02.830 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 13, 2024-08-31T12:20:02.830 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:20:02.830 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:20:02.830 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:20:02.830 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:20:02.830 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:20:02.830 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:20:02.830 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:20:02.830 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:20:02.830 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:20:02.830 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:20:02.830 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:20:02.831 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:20:02.831 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:20:02.831 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:20:02.831 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:02.831 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:20:02.831 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:20:02.831 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:20:02.831 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:20:02.831 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:20:02.831 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:20:02.831 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:20:02.831 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:20:02.831 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:20:02.832 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:20:02.832 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:20:02.832 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:20:02.832 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:20:02.832 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:02.832 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:02.832 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:20:02.832 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:20:02.832 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:20:02.832 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:20:02.832 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:20:02.832 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:20:02.832 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:20:02.833 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:20:02.833 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:20:02.833 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:20:02.833 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:20:02.833 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:20:02.833 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:20:02.833 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:20:02.833 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:20:02.833 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:20:02.833 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:20:02.833 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:20:02.834 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:20:02.834 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:20:02.834 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:20:02.834 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:20:02.834 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:20:02.834 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:20:02.834 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:20:02.834 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:20:02.834 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:20:02.834 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:20:02.834 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:02.834 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:20:02.834 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:02.835 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:20:02.835 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:02.835 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:20:02.835 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:20:02.835 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:20:02.835 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:20:02.835 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:02.835 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:20:02.835 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:20:02.835 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:02.835 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:02.835 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:02.835 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:20:02.835 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:20:02.836 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:02.836 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:02.836 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:02.836 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:02.836 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:20:02.836 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:20:02.836 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:20:02.836 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:20:02.836 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:20:02.836 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:02.836 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:02.836 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:20:02.837 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:20:02.837 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:20:02.837 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:20:02.837 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:02.837 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:20:02.837 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:20:02.837 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:02.837 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:02.837 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:02.837 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:20:02.837 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:20:02.837 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:02.837 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:02.837 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:02.838 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:02.838 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:20:02.838 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:20:02.838 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:20:02.838 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:20:02.838 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:20:02.838 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:02.838 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:02.838 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:20:02.838 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:20:02.838 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:20:02.838 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:20:02.838 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:02.838 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:20:02.839 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:20:02.839 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:02.839 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:02.839 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:02.839 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:20:02.839 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:20:02.839 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:02.839 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:02.839 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:02.839 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:02.839 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:20:02.839 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:20:02.839 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:20:02.840 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:20:02.840 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:20:02.840 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:02.840 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:02.840 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:02.840 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:20:02.840 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:20:02.840 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:02.840 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:20:02.840 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:20:02.840 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:20:02.840 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:02.840 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:02.840 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:20:02.841 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:02.841 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:20:02.841 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:20:02.841 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 5 2024-08-31T12:20:02.841 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:02.841 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:02.841 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:20:02.841 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:02.841 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:20:02.841 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:20:02.841 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 8 2024-08-31T12:20:02.841 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:02.841 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:02.841 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:20:02.842 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:02.842 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:20:02.842 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:20:02.842 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 6 2024-08-31T12:20:02.842 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:02.842 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:02.842 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:20:02.842 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:02.842 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:20:02.842 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:20:02.842 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:20:02.842 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:02.842 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:02.843 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:02.843 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:20:02.843 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:20:02.844 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.826+0000 7fb67dd8b640 1 -- 172.21.15.19:0/2616717784 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb664077950 msgr2=0x7fb664079e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:02.844 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.826+0000 7fb67dd8b640 1 --2- 172.21.15.19:0/2616717784 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb664077950 0x7fb664079e60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb670035000 tx=0x7fb670038040 comp rx=0 tx=0).stop 2024-08-31T12:20:02.844 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.826+0000 7fb67dd8b640 1 -- 172.21.15.19:0/2616717784 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb6780c8a40 msgr2=0x7fb6781f19e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:02.844 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.826+0000 7fb67dd8b640 1 --2- 172.21.15.19:0/2616717784 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb6780c8a40 0x7fb6781f19e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb66c00e7c0 tx=0x7fb66c00ec80 comp rx=0 tx=0).stop 2024-08-31T12:20:02.844 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.826+0000 7fb677fff640 1 -- 172.21.15.19:0/2616717784 reap_dead start 2024-08-31T12:20:02.844 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.826+0000 7fb67dd8b640 1 -- 172.21.15.19:0/2616717784 shutdown_connections 2024-08-31T12:20:02.844 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.826+0000 7fb67dd8b640 1 --2- 172.21.15.19:0/2616717784 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb6780c8a40 0x7fb6781f19e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:02.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.826+0000 7fb67dd8b640 1 -- 172.21.15.19:0/2616717784 >> 172.21.15.19:0/2616717784 conn(0x7fb6780690d0 msgr2=0x7fb6780c3170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:02.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.827+0000 7fb67dd8b640 1 -- 172.21.15.19:0/2616717784 shutdown_connections 2024-08-31T12:20:02.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.827+0000 7fb67dd8b640 1 -- 172.21.15.19:0/2616717784 wait complete. 2024-08-31T12:20:02.845 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:20:02.845 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-08-31T12:20:02.845 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-31T12:20:02.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.972+0000 7ff0ce807640 1 -- 172.21.15.19:0/1096762379 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff0b0003a50 msgr2=0x7ff0b0003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:02.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.972+0000 7ff0ce807640 1 --2- 172.21.15.19:0/1096762379 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff0b0003a50 0x7ff0b0003ec0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff0b8006fd0 tx=0x7ff0b0005830 comp rx=0 tx=0).stop 2024-08-31T12:20:02.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.972+0000 7ff0ce807640 1 -- 172.21.15.19:0/1096762379 shutdown_connections 2024-08-31T12:20:02.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.972+0000 7ff0ce807640 1 --2- 172.21.15.19:0/1096762379 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff0b0003a50 0x7ff0b0003ec0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:02.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.972+0000 7ff0ce807640 1 -- 172.21.15.19:0/1096762379 >> 172.21.15.19:0/1096762379 conn(0x7ff0c80ca670 msgr2=0x7ff0c80cd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:02.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.972+0000 7ff0ce807640 1 -- 172.21.15.19:0/1096762379 shutdown_connections 2024-08-31T12:20:02.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.972+0000 7ff0ce807640 1 -- 172.21.15.19:0/1096762379 wait complete. 2024-08-31T12:20:02.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.975+0000 7ff0ce807640 1 Processor -- start 2024-08-31T12:20:02.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.975+0000 7ff0ce807640 1 -- start start 2024-08-31T12:20:02.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.976+0000 7ff0ce807640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff0b0003a50 0x7ff0c80c8750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:02.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.976+0000 7ff0ce807640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff0c8159520 0x7ff0c8159930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:02.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.976+0000 7ff0ce807640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff0c8143c90 0x7ff0c81440a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:02.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.976+0000 7ff0ce807640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c80c7650 con 0x7ff0c8159520 2024-08-31T12:20:02.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.976+0000 7ff0ce807640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c80ce140 con 0x7ff0c8143c90 2024-08-31T12:20:02.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.976+0000 7ff0ce807640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7ff0c80cb8c0 con 0x7ff0b0003a50 2024-08-31T12:20:02.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.976+0000 7ff0c77fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff0c8159520 0x7ff0c8159930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:02.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.976+0000 7ff0c77fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff0c8159520 0x7ff0c8159930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:48116/0 (socket says 172.21.15.19:48116) 2024-08-31T12:20:02.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.976+0000 7ff0c77fe640 1 -- 172.21.15.19:0/476857800 learned_addr learned my addr 172.21.15.19:0/476857800 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:20:02.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.976+0000 7ff0ccda4640 1 --2- 172.21.15.19:0/476857800 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff0c8143c90 0x7ff0c81440a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:02.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.976+0000 7ff0c7fff640 1 --2- 172.21.15.19:0/476857800 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff0b0003a50 0x7ff0c80c8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:02.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.977+0000 7ff0ccda4640 1 -- 172.21.15.19:0/476857800 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff0b0003a50 msgr2=0x7ff0c80c8750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:02.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.977+0000 7ff0ccda4640 1 --2- 172.21.15.19:0/476857800 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff0b0003a50 0x7ff0c80c8750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:02.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.977+0000 7ff0ccda4640 1 -- 172.21.15.19:0/476857800 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff0c8159520 msgr2=0x7ff0c8159930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:02.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.977+0000 7ff0ccda4640 1 --2- 172.21.15.19:0/476857800 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff0c8159520 0x7ff0c8159930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:02.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.977+0000 7ff0ccda4640 1 -- 172.21.15.19:0/476857800 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0c81f2730 con 0x7ff0c8143c90 2024-08-31T12:20:02.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.977+0000 7ff0c77fe640 1 --2- 172.21.15.19:0/476857800 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff0c8159520 0x7ff0c8159930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:20:02.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.977+0000 7ff0ccda4640 1 --2- 172.21.15.19:0/476857800 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff0c8143c90 0x7ff0c81440a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8157e10 tx=0x7ff0c0006430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:02.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.977+0000 7ff0c4ff9640 1 -- 172.21.15.19:0/476857800 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c0015070 con 0x7ff0c8143c90 2024-08-31T12:20:02.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.977+0000 7ff0c4ff9640 1 -- 172.21.15.19:0/476857800 <== mon.1 v2:172.21.15.43:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff0c0011840 con 0x7ff0c8143c90 2024-08-31T12:20:02.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.977+0000 7ff0c4ff9640 1 -- 172.21.15.19:0/476857800 <== mon.1 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c0011c90 con 0x7ff0c8143c90 2024-08-31T12:20:02.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.977+0000 7ff0ce807640 1 -- 172.21.15.19:0/476857800 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0c80c8c90 con 0x7ff0c8143c90 2024-08-31T12:20:02.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.977+0000 7ff0ce807640 1 -- 172.21.15.19:0/476857800 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0c81f2450 con 0x7ff0c8143c90 2024-08-31T12:20:02.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.978+0000 7ff0ce807640 1 --2- 172.21.15.19:0/476857800 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff0c80c7e90 0x7ff0c81f4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:02.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.978+0000 7ff0ce807640 1 -- 172.21.15.19:0/476857800 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c81f2200 con 0x7ff0c80c7e90 2024-08-31T12:20:02.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.978+0000 7ff0c7fff640 1 --2- 172.21.15.19:0/476857800 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff0c80c7e90 0x7ff0c81f4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:02.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.979+0000 7ff0c7fff640 1 -- 172.21.15.19:0/476857800 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff0c8157c30 con 0x7ff0c80c7e90 2024-08-31T12:20:02.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.979+0000 7ff0c7fff640 1 --2- 172.21.15.19:0/476857800 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff0c80c7e90 0x7ff0c81f4d70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff0b8006fa0 tx=0x7ff0b8006290 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:02.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.979+0000 7ff0c4ff9640 1 -- 172.21.15.19:0/476857800 <== mon.1 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7ff0c0018070 con 0x7ff0c8143c90 2024-08-31T12:20:02.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.980+0000 7ff0c4ff9640 1 --2- 172.21.15.19:0/476857800 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff0b4077a80 0x7ff0b4079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:02.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.980+0000 7ff0c4ff9640 1 -- 172.21.15.19:0/476857800 <== mon.1 v2:172.21.15.43:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7ff0c009a1b0 con 0x7ff0c8143c90 2024-08-31T12:20:02.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.980+0000 7ff0c4ff9640 1 -- 172.21.15.19:0/476857800 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b8046070 con 0x7ff0c80c7e90 2024-08-31T12:20:02.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.980+0000 7ff0c77fe640 1 --2- 172.21.15.19:0/476857800 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff0b4077a80 0x7ff0b4079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:02.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.980+0000 7ff0c4ff9640 1 -- 172.21.15.19:0/476857800 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff0b80037f0 con 0x7ff0c80c7e90 2024-08-31T12:20:02.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.980+0000 7ff0c4ff9640 1 -- 172.21.15.19:0/476857800 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff0c80c7e90 msgr2=0x7ff0c81f4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:02.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.980+0000 7ff0c4ff9640 1 --2- 172.21.15.19:0/476857800 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff0c80c7e90 0x7ff0c81f4d70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff0b8006fa0 tx=0x7ff0b8006290 comp rx=0 tx=0).stop 2024-08-31T12:20:02.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.980+0000 7ff0c4ff9640 1 -- 172.21.15.19:0/476857800 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff0c80c7e90 msgr2=0x7ff0c81f4d70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:20:02.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.980+0000 7ff0c4ff9640 1 --2- 172.21.15.19:0/476857800 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff0c80c7e90 0x7ff0c81f4d70 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:02.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.980+0000 7ff0c77fe640 1 --2- 172.21.15.19:0/476857800 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff0b4077a80 0x7ff0b4079f90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8156720 tx=0x7ff0bc005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:02.990 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.988+0000 7ff0ce807640 1 --2- 172.21.15.19:0/476857800 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff0c81f52b0 0x7ff0c81f56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:02.990 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.988+0000 7ff0ce807640 1 -- 172.21.15.19:0/476857800 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c81f5c00 con 0x7ff0c81f52b0 2024-08-31T12:20:02.990 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.989+0000 7ff0c7fff640 1 --2- 172.21.15.19:0/476857800 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff0c81f52b0 0x7ff0c81f56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:02.991 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.989+0000 7ff0c7fff640 1 -- 172.21.15.19:0/476857800 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff0c806e760 con 0x7ff0c81f52b0 2024-08-31T12:20:02.991 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.989+0000 7ff0c7fff640 1 --2- 172.21.15.19:0/476857800 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff0c81f52b0 0x7ff0c81f56c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff0b8007010 tx=0x7ff0b8006720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:02.991 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.989+0000 7ff0c4ff9640 1 -- 172.21.15.19:0/476857800 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b8046040 con 0x7ff0c81f52b0 2024-08-31T12:20:02.992 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.990+0000 7ff0c4ff9640 1 -- 172.21.15.19:0/476857800 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4541 (secure 0 0 0) 0x7ff0b8036450 con 0x7ff0c81f52b0 2024-08-31T12:20:02.992 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.990+0000 7ff0c4ff9640 1 -- 172.21.15.19:0/476857800 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff0c81f52b0 msgr2=0x7ff0c81f56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:02.992 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.990+0000 7ff0c4ff9640 1 --2- 172.21.15.19:0/476857800 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff0c81f52b0 0x7ff0c81f56c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff0b8007010 tx=0x7ff0b8006720 comp rx=0 tx=0).stop 2024-08-31T12:20:02.992 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.990+0000 7ff0c4ff9640 1 -- 172.21.15.19:0/476857800 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff0c81f52b0 msgr2=0x7ff0c81f56c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:20:02.992 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.990+0000 7ff0c4ff9640 1 --2- 172.21.15.19:0/476857800 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff0c81f52b0 0x7ff0c81f56c0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:02.992 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:20:02.993 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:20:02.993 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:20:02.993 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:20:02.993 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 32, 2024-08-31T12:20:02.993 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 18958, 2024-08-31T12:20:02.993 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:20:02.993 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:20:02.993 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:20:02.993 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:20:02.993 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:02.993 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 13, 2024-08-31T12:20:02.993 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:20:02.993 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:20:02.994 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:20:02.994 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:20:02.994 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:20:02.994 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:20:02.994 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:20:02.994 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:20:02.994 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:20:02.994 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:20:02.994 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:20:02.994 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:20:02.994 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:20:02.994 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:20:02.994 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:02.994 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:20:02.995 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:20:02.995 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:20:02.995 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:20:02.995 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:20:02.995 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:20:02.995 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:20:02.995 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:20:02.995 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:20:02.995 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:20:02.995 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:20:02.995 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:20:02.995 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:20:02.996 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:02.996 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:02.996 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:20:02.996 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:20:02.996 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:20:02.996 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:20:02.996 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:20:02.996 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:20:02.996 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:20:02.996 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:20:02.996 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:20:02.996 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:20:02.996 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:20:02.997 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:20:02.997 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:20:02.997 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:20:02.997 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:20:02.997 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:20:02.997 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:20:02.997 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:20:02.997 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:20:02.997 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:20:02.997 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:20:02.997 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:20:02.997 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:20:02.997 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:20:02.998 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:20:02.998 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:20:02.998 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:20:02.998 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:20:02.998 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:02.998 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:20:02.998 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:02.998 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:20:02.998 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:02.998 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:20:02.998 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:20:02.998 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:20:02.998 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:20:02.998 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:02.999 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:20:02.999 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:20:02.999 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:02.999 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:02.999 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:02.999 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:20:02.999 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:20:02.999 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:02.999 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:02.999 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:02.999 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:02.999 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:20:02.999 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:20:03.000 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:20:03.000 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:20:03.000 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:20:03.000 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:03.000 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:03.000 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:20:03.000 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:20:03.000 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:20:03.000 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:20:03.000 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:03.000 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:20:03.000 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:20:03.000 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:03.001 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:03.001 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:03.001 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:20:03.001 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:20:03.001 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:03.001 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:03.001 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:03.001 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:03.001 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:20:03.001 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:20:03.001 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:20:03.001 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:20:03.002 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:20:03.002 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:03.002 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:03.002 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:20:03.002 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:20:03.002 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:20:03.002 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:20:03.002 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:03.002 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:20:03.002 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:20:03.003 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:03.003 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:03.003 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:03.003 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:20:03.003 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:20:03.003 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:03.003 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:03.003 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:03.003 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:03.003 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:20:03.003 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:20:03.003 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:20:03.004 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:20:03.004 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:20:03.004 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:03.004 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:03.004 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:03.004 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:20:03.004 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:20:03.004 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:03.004 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:20:03.004 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:20:03.004 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:20:03.004 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:03.005 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:03.005 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:20:03.005 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:03.005 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:20:03.005 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:20:03.005 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:20:03.005 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:03.005 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:03.005 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:03.005 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:20:03.005 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:20:03.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.990+0000 7ff0ce807640 1 -- 172.21.15.19:0/476857800 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff0b4077a80 msgr2=0x7ff0b4079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:03.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.990+0000 7ff0ce807640 1 --2- 172.21.15.19:0/476857800 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff0b4077a80 0x7ff0b4079f90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8156720 tx=0x7ff0bc005f90 comp rx=0 tx=0).stop 2024-08-31T12:20:03.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.990+0000 7ff0ce807640 1 -- 172.21.15.19:0/476857800 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff0c8143c90 msgr2=0x7ff0c81440a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:03.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.990+0000 7ff0ccda4640 1 -- 172.21.15.19:0/476857800 reap_dead start 2024-08-31T12:20:03.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.990+0000 7ff0ce807640 1 --2- 172.21.15.19:0/476857800 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff0c8143c90 0x7ff0c81440a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8157e10 tx=0x7ff0c0006430 comp rx=0 tx=0).stop 2024-08-31T12:20:03.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.990+0000 7ff0ce807640 1 -- 172.21.15.19:0/476857800 shutdown_connections 2024-08-31T12:20:03.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.990+0000 7ff0ce807640 1 --2- 172.21.15.19:0/476857800 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff0c8143c90 0x7ff0c81440a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:03.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.990+0000 7ff0ce807640 1 -- 172.21.15.19:0/476857800 >> 172.21.15.19:0/476857800 conn(0x7ff0c80ca670 msgr2=0x7ff0c80cc1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:03.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.990+0000 7ff0ce807640 1 -- 172.21.15.19:0/476857800 shutdown_connections 2024-08-31T12:20:03.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:02.990+0000 7ff0ce807640 1 -- 172.21.15.19:0/476857800 wait complete. 2024-08-31T12:20:03.010 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:20:03.010 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-08-31T12:20:03.010 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-31T12:20:03.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.128+0000 7f0797de6640 1 -- 172.21.15.19:0/1614089446 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f07900ca7f0 msgr2=0x7f0790071bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:03.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.128+0000 7f0797de6640 1 --2- 172.21.15.19:0/1614089446 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f07900ca7f0 0x7f0790071bd0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f07900cbf90 tx=0x7f078c02a5f0 comp rx=0 tx=0).stop 2024-08-31T12:20:03.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.128+0000 7f0797de6640 1 -- 172.21.15.19:0/1614089446 shutdown_connections 2024-08-31T12:20:03.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.128+0000 7f0797de6640 1 --2- 172.21.15.19:0/1614089446 >> v2:172.21.15.43:3300/0 conn(0x7f07901532f0 0x7f079015c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:03.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.128+0000 7f0797de6640 1 --2- 172.21.15.19:0/1614089446 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f07900ca7f0 0x7f0790071bd0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:03.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.128+0000 7f0797de6640 1 -- 172.21.15.19:0/1614089446 >> 172.21.15.19:0/1614089446 conn(0x7f07900cefb0 msgr2=0x7f07900cb340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:03.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.128+0000 7f0797de6640 1 -- 172.21.15.19:0/1614089446 shutdown_connections 2024-08-31T12:20:03.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.128+0000 7f0797de6640 1 -- 172.21.15.19:0/1614089446 wait complete. 2024-08-31T12:20:03.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.130+0000 7f0797de6640 1 Processor -- start 2024-08-31T12:20:03.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.130+0000 7f0797de6640 1 -- start start 2024-08-31T12:20:03.135 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.132+0000 7f0797de6640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f07900c9880 0x7f07901efd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:03.135 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.132+0000 7f0797de6640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f07900ca7f0 0x7f079014c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:03.135 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.132+0000 7f0797de6640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f07901532f0 0x7f079014b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:03.135 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.132+0000 7f0797de6640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f079015d330 con 0x7f07900c9880 2024-08-31T12:20:03.135 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.132+0000 7f0797de6640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0790072110 con 0x7f07900ca7f0 2024-08-31T12:20:03.135 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.132+0000 7f0797de6640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f07900c3860 con 0x7f07901532f0 2024-08-31T12:20:03.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.132+0000 7f0795b82640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f07900c9880 0x7f07901efd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:03.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.132+0000 7f0795b82640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f07900c9880 0x7f07901efd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:48132/0 (socket says 172.21.15.19:48132) 2024-08-31T12:20:03.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.132+0000 7f0795b82640 1 -- 172.21.15.19:0/1255792916 learned_addr learned my addr 172.21.15.19:0/1255792916 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:20:03.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.132+0000 7f0795381640 1 --2- 172.21.15.19:0/1255792916 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f07900ca7f0 0x7f079014c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:03.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.132+0000 7f0796383640 1 --2- 172.21.15.19:0/1255792916 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f07901532f0 0x7f079014b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:03.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.132+0000 7f0795b82640 1 -- 172.21.15.19:0/1255792916 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f07901532f0 msgr2=0x7f079014b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:03.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.132+0000 7f0795b82640 1 --2- 172.21.15.19:0/1255792916 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f07901532f0 0x7f079014b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:03.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.132+0000 7f0795b82640 1 -- 172.21.15.19:0/1255792916 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f07900ca7f0 msgr2=0x7f079014c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:03.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.132+0000 7f0795b82640 1 --2- 172.21.15.19:0/1255792916 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f07900ca7f0 0x7f079014c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:03.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.132+0000 7f0795b82640 1 -- 172.21.15.19:0/1255792916 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07901561c0 con 0x7f07900c9880 2024-08-31T12:20:03.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.132+0000 7f0795b82640 1 --2- 172.21.15.19:0/1255792916 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f07900c9880 0x7f07901efd80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f078c034040 tx=0x7f078c003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:03.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.133+0000 7f07867fc640 1 -- 172.21.15.19:0/1255792916 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078c03e030 con 0x7f07900c9880 2024-08-31T12:20:03.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.133+0000 7f07867fc640 1 -- 172.21.15.19:0/1255792916 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f078c003d10 con 0x7f07900c9880 2024-08-31T12:20:03.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.133+0000 7f0797de6640 1 -- 172.21.15.19:0/1255792916 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0790156460 con 0x7f07900c9880 2024-08-31T12:20:03.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.133+0000 7f07867fc640 1 -- 172.21.15.19:0/1255792916 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078c035040 con 0x7f07900c9880 2024-08-31T12:20:03.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.133+0000 7f0797de6640 1 -- 172.21.15.19:0/1255792916 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f079014d2a0 con 0x7f07900c9880 2024-08-31T12:20:03.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.134+0000 7f0797de6640 1 --2- 172.21.15.19:0/1255792916 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f07901538c0 0x7f079014b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:03.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.134+0000 7f0797de6640 1 -- 172.21.15.19:0/1255792916 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f07900b4500 con 0x7f07901538c0 2024-08-31T12:20:03.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.134+0000 7f0795381640 1 --2- 172.21.15.19:0/1255792916 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f07901538c0 0x7f079014b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:03.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.134+0000 7f0795381640 1 -- 172.21.15.19:0/1255792916 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f07901568e0 con 0x7f07901538c0 2024-08-31T12:20:03.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.135+0000 7f0795381640 1 --2- 172.21.15.19:0/1255792916 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f07901538c0 0x7f079014b940 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f07901515c0 tx=0x7f0780011be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:03.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.135+0000 7f07867fc640 1 -- 172.21.15.19:0/1255792916 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078001f070 con 0x7f07901538c0 2024-08-31T12:20:03.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.135+0000 7f07867fc640 1 -- 172.21.15.19:0/1255792916 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f078c041030 con 0x7f07900c9880 2024-08-31T12:20:03.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.135+0000 7f07867fc640 1 --2- 172.21.15.19:0/1255792916 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0774077980 0x7f0774079e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:03.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.136+0000 7f07867fc640 1 -- 172.21.15.19:0/1255792916 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f078c0b9db0 con 0x7f07900c9880 2024-08-31T12:20:03.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.136+0000 7f0796383640 1 --2- 172.21.15.19:0/1255792916 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0774077980 0x7f0774079e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:03.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.136+0000 7f07867fc640 1 -- 172.21.15.19:0/1255792916 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f078000fbb0 con 0x7f07901538c0 2024-08-31T12:20:03.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.136+0000 7f07867fc640 1 -- 172.21.15.19:0/1255792916 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f07901538c0 msgr2=0x7f079014b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:03.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.136+0000 7f07867fc640 1 --2- 172.21.15.19:0/1255792916 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f07901538c0 0x7f079014b940 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f07901515c0 tx=0x7f0780011be0 comp rx=0 tx=0).stop 2024-08-31T12:20:03.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.136+0000 7f07867fc640 1 -- 172.21.15.19:0/1255792916 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f07901538c0 msgr2=0x7f079014b940 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:20:03.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.136+0000 7f07867fc640 1 --2- 172.21.15.19:0/1255792916 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f07901538c0 0x7f079014b940 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:03.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.136+0000 7f0796383640 1 --2- 172.21.15.19:0/1255792916 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0774077980 0x7f0774079e90 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0788007fd0 tx=0x7f0788009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:03.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.143+0000 7f0797de6640 1 --2- 172.21.15.19:0/1255792916 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f07901ef4a0 0x7f07900cd970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:03.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.143+0000 7f0797de6640 1 -- 172.21.15.19:0/1255792916 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f07900cb450 con 0x7f07901ef4a0 2024-08-31T12:20:03.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.143+0000 7f0795381640 1 --2- 172.21.15.19:0/1255792916 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f07901ef4a0 0x7f07900cd970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:03.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.143+0000 7f0795381640 1 -- 172.21.15.19:0/1255792916 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f07901e98d0 con 0x7f07901ef4a0 2024-08-31T12:20:03.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.143+0000 7f0795381640 1 --2- 172.21.15.19:0/1255792916 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f07901ef4a0 0x7f07900cd970 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f07901515c0 tx=0x7f07800046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:03.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.144+0000 7f07867fc640 1 -- 172.21.15.19:0/1255792916 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0780012c60 con 0x7f07901ef4a0 2024-08-31T12:20:03.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.144+0000 7f07867fc640 1 -- 172.21.15.19:0/1255792916 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4709 (secure 0 0 0) 0x7f0780012c60 con 0x7f07901ef4a0 2024-08-31T12:20:03.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.144+0000 7f07867fc640 1 -- 172.21.15.19:0/1255792916 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f07901ef4a0 msgr2=0x7f07900cd970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:03.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.144+0000 7f07867fc640 1 --2- 172.21.15.19:0/1255792916 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f07901ef4a0 0x7f07900cd970 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f07901515c0 tx=0x7f07800046c0 comp rx=0 tx=0).stop 2024-08-31T12:20:03.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.144+0000 7f07867fc640 1 -- 172.21.15.19:0/1255792916 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f07901ef4a0 msgr2=0x7f07900cd970 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:20:03.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.144+0000 7f07867fc640 1 --2- 172.21.15.19:0/1255792916 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f07901ef4a0 0x7f07900cd970 secure :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f07901515c0 tx=0x7f07800046c0 comp rx=0 tx=0).stop 2024-08-31T12:20:03.147 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:20:03.147 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:20:03.147 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:20:03.147 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:20:03.147 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 32, 2024-08-31T12:20:03.147 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 131266, 2024-08-31T12:20:03.147 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:20:03.147 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:20:03.147 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:20:03.147 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:20:03.147 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:03.148 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 14, 2024-08-31T12:20:03.148 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:20:03.148 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:20:03.148 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:20:03.148 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:20:03.148 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:20:03.148 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:20:03.148 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:20:03.148 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:20:03.148 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:20:03.148 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:20:03.148 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:20:03.148 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:20:03.148 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:20:03.149 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:20:03.149 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:03.149 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:20:03.149 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:20:03.149 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:20:03.149 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:20:03.149 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:20:03.149 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:20:03.149 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:20:03.149 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:20:03.149 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:20:03.149 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:20:03.150 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:20:03.150 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:20:03.150 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:20:03.150 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:03.150 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:03.150 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:20:03.150 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:20:03.150 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:20:03.150 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:20:03.150 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:20:03.150 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:20:03.150 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:20:03.150 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:20:03.151 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:20:03.151 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:20:03.151 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:20:03.151 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:20:03.151 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:20:03.151 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:20:03.151 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:20:03.151 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:20:03.151 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:20:03.151 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:20:03.151 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:20:03.151 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:20:03.151 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:20:03.151 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:20:03.152 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:20:03.152 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:20:03.152 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:20:03.152 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:20:03.152 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:20:03.152 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:20:03.152 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:03.152 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:20:03.152 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:03.152 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:20:03.152 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:03.152 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:20:03.152 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:20:03.152 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:20:03.153 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:20:03.153 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:03.153 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:20:03.153 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:20:03.153 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:03.153 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:03.153 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:03.153 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:20:03.153 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:20:03.153 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:03.153 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:03.153 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:03.153 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:03.153 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:20:03.154 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:20:03.154 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:20:03.154 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:20:03.154 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:20:03.154 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:03.154 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:03.154 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:20:03.154 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:20:03.154 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:20:03.154 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:20:03.154 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:03.154 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:20:03.154 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:20:03.155 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:03.155 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:03.155 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:03.155 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:20:03.155 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:20:03.155 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:03.155 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:03.155 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:03.155 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:03.155 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:20:03.155 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:20:03.155 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:20:03.155 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:20:03.155 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:20:03.156 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:03.156 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:03.156 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:20:03.156 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:20:03.156 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:20:03.156 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:20:03.156 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:03.156 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:20:03.156 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:20:03.156 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:03.156 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:03.156 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:03.156 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:20:03.157 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:20:03.157 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:03.157 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:03.157 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:03.157 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:03.157 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:20:03.157 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:20:03.157 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:20:03.157 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:20:03.157 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:20:03.157 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:03.157 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:03.157 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:03.157 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:20:03.158 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:20:03.158 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:03.158 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:20:03.158 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:20:03.158 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:20:03.158 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:03.158 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:03.158 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:20:03.158 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:03.158 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:20:03.158 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:20:03.158 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:20:03.158 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:03.158 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:03.158 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:20:03.159 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:03.159 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:20:03.159 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:20:03.159 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:20:03.159 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:03.159 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:03.159 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:03.159 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:20:03.159 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:20:03.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.144+0000 7f0797de6640 1 -- 172.21.15.19:0/1255792916 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0774077980 msgr2=0x7f0774079e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:03.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.144+0000 7f0797de6640 1 --2- 172.21.15.19:0/1255792916 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0774077980 0x7f0774079e90 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0788007fd0 tx=0x7f0788009040 comp rx=0 tx=0).stop 2024-08-31T12:20:03.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.144+0000 7f0797de6640 1 -- 172.21.15.19:0/1255792916 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f07900c9880 msgr2=0x7f07901efd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:03.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.144+0000 7f0797de6640 1 --2- 172.21.15.19:0/1255792916 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f07900c9880 0x7f07901efd80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f078c034040 tx=0x7f078c003820 comp rx=0 tx=0).stop 2024-08-31T12:20:03.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.144+0000 7f0796383640 1 -- 172.21.15.19:0/1255792916 reap_dead start 2024-08-31T12:20:03.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.144+0000 7f0797de6640 1 -- 172.21.15.19:0/1255792916 shutdown_connections 2024-08-31T12:20:03.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.144+0000 7f0797de6640 1 --2- 172.21.15.19:0/1255792916 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f07900c9880 0x7f07901efd80 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:03.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.144+0000 7f0797de6640 1 -- 172.21.15.19:0/1255792916 >> 172.21.15.19:0/1255792916 conn(0x7f07900cefb0 msgr2=0x7f079015c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:03.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.144+0000 7f0797de6640 1 -- 172.21.15.19:0/1255792916 shutdown_connections 2024-08-31T12:20:03.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.144+0000 7f0797de6640 1 -- 172.21.15.19:0/1255792916 wait complete. 2024-08-31T12:20:03.161 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:20:03.162 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-08-31T12:20:03.162 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-08-31T12:20:03.162 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-08-31T12:20:03.162 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-08-31T12:20:03.162 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-08-31T12:20:03.162 INFO:teuthology.orchestra.run:waiting for 300 2024-08-31T12:20:03.174 INFO:tasks.ceph.mon.c:Stopped 2024-08-31T12:20:03.174 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-08-31T12:20:03.174 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:20:03.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.298+0000 7f80836bd640 1 -- 172.21.15.19:0/1973923488 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8068003fa0 msgr2=0x7f8068004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:03.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.298+0000 7f80836bd640 1 --2- 172.21.15.19:0/1973923488 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8068003fa0 0x7f8068004800 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f806c009b50 tx=0x7f8068005830 comp rx=0 tx=0).stop 2024-08-31T12:20:03.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.298+0000 7f8081c5a640 1 -- 172.21.15.19:0/1973923488 reap_dead start 2024-08-31T12:20:03.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.298+0000 7f80836bd640 1 -- 172.21.15.19:0/1973923488 shutdown_connections 2024-08-31T12:20:03.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.298+0000 7f80836bd640 1 -- 172.21.15.19:0/1973923488 >> 172.21.15.19:0/1973923488 conn(0x7f807c072c80 msgr2=0x7f807c073090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:03.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.298+0000 7f80836bd640 1 -- 172.21.15.19:0/1973923488 shutdown_connections 2024-08-31T12:20:03.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.299+0000 7f80836bd640 1 -- 172.21.15.19:0/1973923488 wait complete. 2024-08-31T12:20:03.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.300+0000 7f80836bd640 1 Processor -- start 2024-08-31T12:20:03.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.300+0000 7f80836bd640 1 -- start start 2024-08-31T12:20:03.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.301+0000 7f80836bd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f807c0b1160 0x7f807c15bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:03.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.301+0000 7f80836bd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f807c1f1550 0x7f807c1f1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:03.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.301+0000 7f80836bd640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f807c0ae8c0 0x7f807c0aecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:03.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.301+0000 7f80836bd640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f807c0721e0 con 0x7f807c1f1550 2024-08-31T12:20:03.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.301+0000 7f80836bd640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f807c0cae30 con 0x7f807c0ae8c0 2024-08-31T12:20:03.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.301+0000 7f80836bd640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f807c072360 con 0x7f807c0b1160 2024-08-31T12:20:03.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.302+0000 7f8080c58640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f807c1f1550 0x7f807c1f1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:03.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.302+0000 7f8081459640 1 -- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f807c0b1160 msgr2=0x7f807c15bbe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3301/0 2024-08-31T12:20:03.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.302+0000 7f8080c58640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f807c1f1550 0x7f807c1f1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:48162/0 (socket says 172.21.15.19:48162) 2024-08-31T12:20:03.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.302+0000 7f8080c58640 1 -- 172.21.15.19:0/3494757264 learned_addr learned my addr 172.21.15.19:0/3494757264 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:20:03.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.302+0000 7f8081459640 1 --2- 172.21.15.19:0/3494757264 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f807c0b1160 0x7f807c15bbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:20:03.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.302+0000 7f8081c5a640 1 --2- 172.21.15.19:0/3494757264 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f807c0ae8c0 0x7f807c0aecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:03.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.302+0000 7f8080c58640 1 -- 172.21.15.19:0/3494757264 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f807c0b1160 msgr2=0x7f807c15bbe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:20:03.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.302+0000 7f8080c58640 1 --2- 172.21.15.19:0/3494757264 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f807c0b1160 0x7f807c15bbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:03.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.302+0000 7f8080c58640 1 -- 172.21.15.19:0/3494757264 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f807c0ae8c0 msgr2=0x7f807c0aecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:03.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.302+0000 7f8080c58640 1 --2- 172.21.15.19:0/3494757264 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f807c0ae8c0 0x7f807c0aecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:03.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.302+0000 7f8080c58640 1 -- 172.21.15.19:0/3494757264 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f807c0af5b0 con 0x7f807c1f1550 2024-08-31T12:20:03.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.302+0000 7f8080c58640 1 --2- 172.21.15.19:0/3494757264 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f807c1f1550 0x7f807c1f1960 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f806c02b4f0 tx=0x7f806c033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:03.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.302+0000 7f8071ffb640 1 -- 172.21.15.19:0/3494757264 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f806c038040 con 0x7f807c1f1550 2024-08-31T12:20:03.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.302+0000 7f8071ffb640 1 -- 172.21.15.19:0/3494757264 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f806c004020 con 0x7f807c1f1550 2024-08-31T12:20:03.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.302+0000 7f80836bd640 1 -- 172.21.15.19:0/3494757264 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f807c14f4b0 con 0x7f807c1f1550 2024-08-31T12:20:03.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.302+0000 7f8071ffb640 1 -- 172.21.15.19:0/3494757264 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f806c004470 con 0x7f807c1f1550 2024-08-31T12:20:03.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.303+0000 7f80836bd640 1 -- 172.21.15.19:0/3494757264 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f807c06f820 con 0x7f807c1f1550 2024-08-31T12:20:03.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.304+0000 7f80836bd640 1 -- 172.21.15.19:0/3494757264 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f807c070ad0 con 0x7f807c1f1550 2024-08-31T12:20:03.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.304+0000 7f8071ffb640 1 -- 172.21.15.19:0/3494757264 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f806c045020 con 0x7f807c1f1550 2024-08-31T12:20:03.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.305+0000 7f8071ffb640 1 --2- 172.21.15.19:0/3494757264 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8060077a30 0x7f8060079f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:03.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.305+0000 7f8071ffb640 1 -- 172.21.15.19:0/3494757264 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f806c0b9f50 con 0x7f807c1f1550 2024-08-31T12:20:03.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.305+0000 7f8081459640 1 --2- 172.21.15.19:0/3494757264 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8060077a30 0x7f8060079f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:03.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.306+0000 7f8081459640 1 --2- 172.21.15.19:0/3494757264 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8060077a30 0x7f8060079f40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8078006fd0 tx=0x7f8078006d60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:03.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.311+0000 7f8071ffb640 1 -- 172.21.15.19:0/3494757264 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f806c082600 con 0x7f807c1f1550 2024-08-31T12:20:03.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.600+0000 7f80836bd640 1 -- 172.21.15.19:0/3494757264 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f807c0c9650 con 0x7f807c1f1550 2024-08-31T12:20:03.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.600+0000 7f8071ffb640 1 -- 172.21.15.19:0/3494757264 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f806c08b100 con 0x7f807c1f1550 2024-08-31T12:20:03.602 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:20:03.605 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.603+0000 7f80836bd640 1 -- 172.21.15.19:0/3494757264 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8060077a30 msgr2=0x7f8060079f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:03.605 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.603+0000 7f80836bd640 1 --2- 172.21.15.19:0/3494757264 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8060077a30 0x7f8060079f40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8078006fd0 tx=0x7f8078006d60 comp rx=0 tx=0).stop 2024-08-31T12:20:03.606 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.603+0000 7f80836bd640 1 -- 172.21.15.19:0/3494757264 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f807c1f1550 msgr2=0x7f807c1f1960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:03.606 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.603+0000 7f80836bd640 1 --2- 172.21.15.19:0/3494757264 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f807c1f1550 0x7f807c1f1960 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f806c02b4f0 tx=0x7f806c033670 comp rx=0 tx=0).stop 2024-08-31T12:20:03.606 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.603+0000 7f80836bd640 1 -- 172.21.15.19:0/3494757264 shutdown_connections 2024-08-31T12:20:03.606 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.603+0000 7f80836bd640 1 --2- 172.21.15.19:0/3494757264 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8060077a30 0x7f8060079f40 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:03.606 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.603+0000 7f80836bd640 1 --2- 172.21.15.19:0/3494757264 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f807c0ae8c0 0x7f807c0aecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:03.606 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.603+0000 7f80836bd640 1 --2- 172.21.15.19:0/3494757264 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f807c1f1550 0x7f807c1f1960 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:03.606 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.603+0000 7f80836bd640 1 --2- 172.21.15.19:0/3494757264 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f807c0b1160 0x7f807c15bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:03.606 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.603+0000 7f80836bd640 1 -- 172.21.15.19:0/3494757264 >> 172.21.15.19:0/3494757264 conn(0x7f807c072c80 msgr2=0x7f807c14df00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:03.606 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.603+0000 7f80836bd640 1 -- 172.21.15.19:0/3494757264 shutdown_connections 2024-08-31T12:20:03.606 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:03.603+0000 7f80836bd640 1 -- 172.21.15.19:0/3494757264 wait complete. 2024-08-31T12:20:06.616 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:20:06.736 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:06.734+0000 7f5e388a4640 1 -- 172.21.15.19:0/59152128 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5e1c004d40 msgr2=0x7f5e1c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:06.736 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:06.734+0000 7f5e388a4640 1 --2- 172.21.15.19:0/59152128 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5e1c004d40 0x7f5e1c005190 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c00c650 tx=0x7f5e1c005830 comp rx=0 tx=0).stop 2024-08-31T12:20:06.736 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:06.734+0000 7f5e32d9d640 1 -- 172.21.15.19:0/59152128 reap_dead start 2024-08-31T12:20:06.736 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:06.734+0000 7f5e388a4640 1 -- 172.21.15.19:0/59152128 shutdown_connections 2024-08-31T12:20:06.736 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:06.734+0000 7f5e388a4640 1 -- 172.21.15.19:0/59152128 >> 172.21.15.19:0/59152128 conn(0x7f5e340cd7d0 msgr2=0x7f5e340cdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:06.737 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:06.734+0000 7f5e388a4640 1 -- 172.21.15.19:0/59152128 shutdown_connections 2024-08-31T12:20:06.737 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:06.734+0000 7f5e388a4640 1 -- 172.21.15.19:0/59152128 wait complete. 2024-08-31T12:20:06.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:06.736+0000 7f5e388a4640 1 Processor -- start 2024-08-31T12:20:06.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:06.736+0000 7f5e388a4640 1 -- start start 2024-08-31T12:20:06.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:06.737+0000 7f5e388a4640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5e1c004d40 0x7f5e34152600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:06.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:06.737+0000 7f5e388a4640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5e341e3f80 0x7f5e34139150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:06.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:06.737+0000 7f5e388a4640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5e34137ab0 0x7f5e34137ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:06.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:06.737+0000 7f5e388a4640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f5e340c5360 con 0x7f5e34137ab0 2024-08-31T12:20:06.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:06.737+0000 7f5e388a4640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5e340c6a70 con 0x7f5e1c004d40 2024-08-31T12:20:06.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:06.737+0000 7f5e388a4640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f5e34074c70 con 0x7f5e341e3f80 2024-08-31T12:20:06.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:06.737+0000 7f5e31d9b640 1 -- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5e341e3f80 msgr2=0x7f5e34139150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3301/0 2024-08-31T12:20:06.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:06.737+0000 7f5e31d9b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5e341e3f80 0x7f5e34139150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:20:06.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:06.737+0000 7f5e32d9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5e34137ab0 0x7f5e34137ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:06.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:06.737+0000 7f5e32d9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5e34137ab0 0x7f5e34137ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:48174/0 (socket says 172.21.15.19:48174) 2024-08-31T12:20:06.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:06.737+0000 7f5e32d9d640 1 -- 172.21.15.19:0/434500190 learned_addr learned my addr 172.21.15.19:0/434500190 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:20:06.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:06.738+0000 7f5e3259c640 1 --2- 172.21.15.19:0/434500190 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5e1c004d40 0x7f5e34152600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:06.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:06.738+0000 7f5e32d9d640 1 -- 172.21.15.19:0/434500190 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5e341e3f80 msgr2=0x7f5e34139150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:20:06.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:06.738+0000 7f5e32d9d640 1 --2- 172.21.15.19:0/434500190 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5e341e3f80 0x7f5e34139150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:06.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:06.738+0000 7f5e32d9d640 1 -- 172.21.15.19:0/434500190 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5e1c004d40 msgr2=0x7f5e34152600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:06.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:06.738+0000 7f5e32d9d640 1 --2- 172.21.15.19:0/434500190 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5e1c004d40 0x7f5e34152600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:06.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:06.738+0000 7f5e32d9d640 1 -- 172.21.15.19:0/434500190 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e34152d10 con 0x7f5e34137ab0 2024-08-31T12:20:06.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:06.738+0000 7f5e32d9d640 1 --2- 172.21.15.19:0/434500190 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5e34137ab0 0x7f5e34137ec0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c00c770 tx=0x7f5e2c001aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:06.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:06.738+0000 7f5e22ffd640 1 -- 172.21.15.19:0/434500190 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e2c00c2a0 con 0x7f5e34137ab0 2024-08-31T12:20:06.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:06.738+0000 7f5e22ffd640 1 -- 172.21.15.19:0/434500190 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5e2c02b9f0 con 0x7f5e34137ab0 2024-08-31T12:20:06.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:06.738+0000 7f5e388a4640 1 -- 172.21.15.19:0/434500190 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e3413c040 con 0x7f5e34137ab0 2024-08-31T12:20:06.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:06.738+0000 7f5e22ffd640 1 -- 172.21.15.19:0/434500190 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e2c02be40 con 0x7f5e34137ab0 2024-08-31T12:20:06.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:06.738+0000 7f5e388a4640 1 -- 172.21.15.19:0/434500190 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e3413a130 con 0x7f5e34137ab0 2024-08-31T12:20:06.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:06.740+0000 7f5e388a4640 1 -- 172.21.15.19:0/434500190 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e34152b40 con 0x7f5e34137ab0 2024-08-31T12:20:06.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:06.740+0000 7f5e22ffd640 1 -- 172.21.15.19:0/434500190 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f5e2c03c030 con 0x7f5e34137ab0 2024-08-31T12:20:06.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:06.741+0000 7f5e22ffd640 1 --2- 172.21.15.19:0/434500190 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5e14077a00 0x7f5e14079f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:06.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:06.741+0000 7f5e22ffd640 1 -- 172.21.15.19:0/434500190 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f5e2c0b8d50 con 0x7f5e34137ab0 2024-08-31T12:20:06.744 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:06.741+0000 7f5e3259c640 1 --2- 172.21.15.19:0/434500190 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5e14077a00 0x7f5e14079f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:06.744 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:06.742+0000 7f5e3259c640 1 --2- 172.21.15.19:0/434500190 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5e14077a00 0x7f5e14079f10 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5e24009530 tx=0x7f5e24005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:06.748 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:06.747+0000 7f5e22ffd640 1 -- 172.21.15.19:0/434500190 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f5e2c0bc030 con 0x7f5e34137ab0 2024-08-31T12:20:07.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:07.054+0000 7f5e388a4640 1 -- 172.21.15.19:0/434500190 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5e34156590 con 0x7f5e34137ab0 2024-08-31T12:20:07.057 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:07.055+0000 7f5e22ffd640 1 -- 172.21.15.19:0/434500190 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f5e34156590 con 0x7f5e34137ab0 2024-08-31T12:20:07.057 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":18,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:20:07.059 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:07.057+0000 7f5e388a4640 1 -- 172.21.15.19:0/434500190 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5e14077a00 msgr2=0x7f5e14079f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:07.059 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:07.057+0000 7f5e388a4640 1 --2- 172.21.15.19:0/434500190 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5e14077a00 0x7f5e14079f10 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5e24009530 tx=0x7f5e24005f90 comp rx=0 tx=0).stop 2024-08-31T12:20:07.060 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:07.057+0000 7f5e388a4640 1 -- 172.21.15.19:0/434500190 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5e34137ab0 msgr2=0x7f5e34137ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:07.060 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:07.057+0000 7f5e388a4640 1 --2- 172.21.15.19:0/434500190 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5e34137ab0 0x7f5e34137ec0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c00c770 tx=0x7f5e2c001aa0 comp rx=0 tx=0).stop 2024-08-31T12:20:07.060 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:07.057+0000 7f5e388a4640 1 -- 172.21.15.19:0/434500190 shutdown_connections 2024-08-31T12:20:07.060 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:07.057+0000 7f5e388a4640 1 --2- 172.21.15.19:0/434500190 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5e14077a00 0x7f5e14079f10 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:07.060 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:07.057+0000 7f5e388a4640 1 --2- 172.21.15.19:0/434500190 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5e34137ab0 0x7f5e34137ec0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:07.060 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:07.057+0000 7f5e388a4640 1 --2- 172.21.15.19:0/434500190 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5e341e3f80 0x7f5e34139150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:07.060 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:07.057+0000 7f5e388a4640 1 --2- 172.21.15.19:0/434500190 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5e1c004d40 0x7f5e34152600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:07.060 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:07.057+0000 7f5e388a4640 1 -- 172.21.15.19:0/434500190 >> 172.21.15.19:0/434500190 conn(0x7f5e340cd7d0 msgr2=0x7f5e34155d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:07.060 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:07.057+0000 7f5e388a4640 1 -- 172.21.15.19:0/434500190 shutdown_connections 2024-08-31T12:20:07.060 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:07.057+0000 7f5e388a4640 1 -- 172.21.15.19:0/434500190 wait complete. 2024-08-31T12:20:10.072 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:20:10.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.191+0000 7f0103d19640 1 -- 172.21.15.19:0/1680089325 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f00fc153230 msgr2=0x7f00fc071b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:10.194 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.191+0000 7f0103d19640 1 --2- 172.21.15.19:0/1680089325 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f00fc153230 0x7f00fc071b40 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f00f8005fd0 tx=0x7f00f802a710 comp rx=0 tx=0).stop 2024-08-31T12:20:10.194 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.191+0000 7f0103d19640 1 -- 172.21.15.19:0/1680089325 shutdown_connections 2024-08-31T12:20:10.194 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.191+0000 7f0103d19640 1 --2- 172.21.15.19:0/1680089325 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f00fc153230 0x7f00fc071b40 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:10.194 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.191+0000 7f0103d19640 1 -- 172.21.15.19:0/1680089325 >> 172.21.15.19:0/1680089325 conn(0x7f00fc072c80 msgr2=0x7f00fc073090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:10.194 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.191+0000 7f0103d19640 1 -- 172.21.15.19:0/1680089325 shutdown_connections 2024-08-31T12:20:10.194 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.191+0000 7f0103d19640 1 -- 172.21.15.19:0/1680089325 wait complete. 2024-08-31T12:20:10.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.193+0000 7f0103d19640 1 Processor -- start 2024-08-31T12:20:10.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.193+0000 7f0103d19640 1 -- start start 2024-08-31T12:20:10.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.195+0000 7f0103d19640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f00fc0caa20 0x7f00fc158190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:10.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.195+0000 7f0103d19640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f00fc153230 0x7f00fc0af5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:10.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.195+0000 7f0103d19640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f00fc159470 0x7f00fc0afb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:10.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.195+0000 7f0103d19640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f00fc0721e0 con 0x7f00fc153230 2024-08-31T12:20:10.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.195+0000 7f0103d19640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f00fc0cae30 con 0x7f00fc159470 2024-08-31T12:20:10.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.195+0000 7f0103d19640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f00fc072360 con 0x7f00fc0caa20 2024-08-31T12:20:10.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.195+0000 7f0101ab5640 1 -- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f00fc0caa20 msgr2=0x7f00fc158190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3301/0 2024-08-31T12:20:10.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.195+0000 7f01012b4640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f00fc153230 0x7f00fc0af5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:10.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.195+0000 7f0101ab5640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f00fc0caa20 0x7f00fc158190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:20:10.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.195+0000 7f01012b4640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f00fc153230 0x7f00fc0af5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:45562/0 (socket says 172.21.15.19:45562) 2024-08-31T12:20:10.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.195+0000 7f01012b4640 1 -- 172.21.15.19:0/3201943210 learned_addr learned my addr 172.21.15.19:0/3201943210 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:20:10.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.195+0000 7f01022b6640 1 --2- 172.21.15.19:0/3201943210 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f00fc159470 0x7f00fc0afb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:10.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.195+0000 7f01012b4640 1 -- 172.21.15.19:0/3201943210 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f00fc0caa20 msgr2=0x7f00fc158190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:20:10.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.195+0000 7f01012b4640 1 --2- 172.21.15.19:0/3201943210 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f00fc0caa20 0x7f00fc158190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:10.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.195+0000 7f01012b4640 1 -- 172.21.15.19:0/3201943210 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f00fc159470 msgr2=0x7f00fc0afb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:10.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.195+0000 7f01012b4640 1 --2- 172.21.15.19:0/3201943210 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f00fc159470 0x7f00fc0afb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:10.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.195+0000 7f01012b4640 1 -- 172.21.15.19:0/3201943210 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00fc0aedb0 con 0x7f00fc153230 2024-08-31T12:20:10.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.195+0000 7f01012b4640 1 --2- 172.21.15.19:0/3201943210 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f00fc153230 0x7f00fc0af5f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f00ec00c400 tx=0x7f00ec00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:10.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.195+0000 7f00f27fc640 1 -- 172.21.15.19:0/3201943210 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00ec017070 con 0x7f00fc153230 2024-08-31T12:20:10.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.196+0000 7f00f27fc640 1 -- 172.21.15.19:0/3201943210 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f00ec0053c0 con 0x7f00fc153230 2024-08-31T12:20:10.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.196+0000 7f00f27fc640 1 -- 172.21.15.19:0/3201943210 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00ec003850 con 0x7f00fc153230 2024-08-31T12:20:10.200 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.196+0000 7f0103d19640 1 -- 172.21.15.19:0/3201943210 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00fc0aead0 con 0x7f00fc153230 2024-08-31T12:20:10.200 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.196+0000 7f0103d19640 1 -- 172.21.15.19:0/3201943210 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00fc1f2470 con 0x7f00fc153230 2024-08-31T12:20:10.200 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.197+0000 7f0103d19640 1 -- 172.21.15.19:0/3201943210 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00fc071b40 con 0x7f00fc153230 2024-08-31T12:20:10.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.198+0000 7f00f27fc640 1 -- 172.21.15.19:0/3201943210 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f00ec01a070 con 0x7f00fc153230 2024-08-31T12:20:10.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.198+0000 7f00f27fc640 1 --2- 172.21.15.19:0/3201943210 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f00e0077a00 0x7f00e0079f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:10.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.199+0000 7f00f27fc640 1 -- 172.21.15.19:0/3201943210 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f00ec0989a0 con 0x7f00fc153230 2024-08-31T12:20:10.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.199+0000 7f0101ab5640 1 --2- 172.21.15.19:0/3201943210 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f00e0077a00 0x7f00e0079f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:10.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.199+0000 7f0101ab5640 1 --2- 172.21.15.19:0/3201943210 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f00e0077a00 0x7f00e0079f10 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f00f8033000 tx=0x7f00f8004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:10.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.202+0000 7f00f27fc640 1 -- 172.21.15.19:0/3201943210 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f00ec021080 con 0x7f00fc153230 2024-08-31T12:20:10.509 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.507+0000 7f0103d19640 1 -- 172.21.15.19:0/3201943210 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f00fc0ca110 con 0x7f00fc153230 2024-08-31T12:20:10.510 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.508+0000 7f00f27fc640 1 -- 172.21.15.19:0/3201943210 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f00ec0611b0 con 0x7f00fc153230 2024-08-31T12:20:10.510 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:20:10.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.510+0000 7f0103d19640 1 -- 172.21.15.19:0/3201943210 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f00e0077a00 msgr2=0x7f00e0079f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:10.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.510+0000 7f0103d19640 1 --2- 172.21.15.19:0/3201943210 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f00e0077a00 0x7f00e0079f10 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f00f8033000 tx=0x7f00f8004730 comp rx=0 tx=0).stop 2024-08-31T12:20:10.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.510+0000 7f0103d19640 1 -- 172.21.15.19:0/3201943210 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f00fc153230 msgr2=0x7f00fc0af5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:10.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.510+0000 7f0103d19640 1 --2- 172.21.15.19:0/3201943210 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f00fc153230 0x7f00fc0af5f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f00ec00c400 tx=0x7f00ec00c8c0 comp rx=0 tx=0).stop 2024-08-31T12:20:10.513 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.510+0000 7f0103d19640 1 -- 172.21.15.19:0/3201943210 shutdown_connections 2024-08-31T12:20:10.513 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.510+0000 7f0103d19640 1 --2- 172.21.15.19:0/3201943210 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f00e0077a00 0x7f00e0079f10 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:10.513 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.510+0000 7f0103d19640 1 --2- 172.21.15.19:0/3201943210 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f00fc159470 0x7f00fc0afb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:10.513 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.510+0000 7f0103d19640 1 --2- 172.21.15.19:0/3201943210 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f00fc153230 0x7f00fc0af5f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:10.513 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.510+0000 7f0103d19640 1 --2- 172.21.15.19:0/3201943210 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f00fc0caa20 0x7f00fc158190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:10.513 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.510+0000 7f0103d19640 1 -- 172.21.15.19:0/3201943210 >> 172.21.15.19:0/3201943210 conn(0x7f00fc072c80 msgr2=0x7f00fc149320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:10.513 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.510+0000 7f0103d19640 1 -- 172.21.15.19:0/3201943210 shutdown_connections 2024-08-31T12:20:10.513 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:10.510+0000 7f0103d19640 1 -- 172.21.15.19:0/3201943210 wait complete. 2024-08-31T12:20:13.524 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:20:17.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:17.716+0000 7f9828780640 1 -- 172.21.15.19:0/955113303 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f980c005350 msgr2=0x7f980c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:17.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:17.716+0000 7f9828780640 1 --2- 172.21.15.19:0/955113303 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f980c005350 0x7f980c00b9c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f98200ce0c0 tx=0x7f980c005830 comp rx=0 tx=0).stop 2024-08-31T12:20:17.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:17.716+0000 7f9828780640 1 -- 172.21.15.19:0/955113303 shutdown_connections 2024-08-31T12:20:17.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:17.716+0000 7f9828780640 1 --2- 172.21.15.19:0/955113303 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f980c005350 0x7f980c00b9c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:17.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:17.716+0000 7f9828780640 1 --2- 172.21.15.19:0/955113303 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f980c004800 0x7f980c004c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:17.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:17.716+0000 7f9828780640 1 --2- 172.21.15.19:0/955113303 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f980c0039b0 0x7f980c003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:17.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:17.716+0000 7f9828780640 1 -- 172.21.15.19:0/955113303 >> 172.21.15.19:0/955113303 conn(0x7f9820074810 msgr2=0x7f9820068aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:17.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:17.716+0000 7f9828780640 1 -- 172.21.15.19:0/955113303 shutdown_connections 2024-08-31T12:20:17.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:17.716+0000 7f9828780640 1 -- 172.21.15.19:0/955113303 wait complete. 2024-08-31T12:20:17.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:17.718+0000 7f9828780640 1 Processor -- start 2024-08-31T12:20:17.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:17.718+0000 7f9828780640 1 -- start start 2024-08-31T12:20:17.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:17.720+0000 7f9828780640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f980c0039b0 0x7f982014cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:17.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:17.720+0000 7f9828780640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f980c004800 0x7f982014b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:17.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:17.720+0000 7f9828780640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f980c005350 0x7f98201532f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:17.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:17.720+0000 7f9828780640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f982015ba70 con 0x7f980c004800 2024-08-31T12:20:17.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:17.720+0000 7f9828780640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f982015f050 con 0x7f980c005350 2024-08-31T12:20:17.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:17.720+0000 7f9828780640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f9820155660 con 0x7f980c0039b0 2024-08-31T12:20:17.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:17.720+0000 7f9825d1b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f980c004800 0x7f982014b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:17.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:17.720+0000 7f982651c640 1 -- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f980c0039b0 msgr2=0x7f982014cc80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3301/0 2024-08-31T12:20:17.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:17.720+0000 7f9825d1b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f980c004800 0x7f982014b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:45602/0 (socket says 172.21.15.19:45602) 2024-08-31T12:20:17.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:17.720+0000 7f9825d1b640 1 -- 172.21.15.19:0/1100468261 learned_addr learned my addr 172.21.15.19:0/1100468261 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:20:17.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:17.720+0000 7f982651c640 1 --2- 172.21.15.19:0/1100468261 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f980c0039b0 0x7f982014cc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:20:17.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:17.720+0000 7f9826d1d640 1 --2- 172.21.15.19:0/1100468261 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f980c005350 0x7f98201532f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:17.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:17.720+0000 7f9825d1b640 1 -- 172.21.15.19:0/1100468261 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f980c0039b0 msgr2=0x7f982014cc80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:20:17.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:17.720+0000 7f9825d1b640 1 --2- 172.21.15.19:0/1100468261 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f980c0039b0 0x7f982014cc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:17.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:17.720+0000 7f9825d1b640 1 -- 172.21.15.19:0/1100468261 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f980c005350 msgr2=0x7f98201532f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:17.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:17.720+0000 7f9825d1b640 1 --2- 172.21.15.19:0/1100468261 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f980c005350 0x7f98201532f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:17.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:17.720+0000 7f9825d1b640 1 -- 172.21.15.19:0/1100468261 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f982015d790 con 0x7f980c004800 2024-08-31T12:20:17.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:17.720+0000 7f9825d1b640 1 --2- 172.21.15.19:0/1100468261 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f980c004800 0x7f982014b3c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9810007b20 tx=0x7f9810001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:17.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:17.721+0000 7f9816ffd640 1 -- 172.21.15.19:0/1100468261 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f981001b070 con 0x7f980c004800 2024-08-31T12:20:17.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:17.721+0000 7f9816ffd640 1 -- 172.21.15.19:0/1100468261 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9810013b40 con 0x7f980c004800 2024-08-31T12:20:17.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:17.721+0000 7f9816ffd640 1 -- 172.21.15.19:0/1100468261 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9810017430 con 0x7f980c004800 2024-08-31T12:20:17.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:17.721+0000 7f9828780640 1 -- 172.21.15.19:0/1100468261 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9820151010 con 0x7f980c004800 2024-08-31T12:20:17.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:17.721+0000 7f9828780640 1 -- 172.21.15.19:0/1100468261 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9820150d20 con 0x7f980c004800 2024-08-31T12:20:17.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:17.722+0000 7f9828780640 1 -- 172.21.15.19:0/1100468261 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f982006e1a0 con 0x7f980c004800 2024-08-31T12:20:17.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:17.723+0000 7f9816ffd640 1 -- 172.21.15.19:0/1100468261 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f981001e070 con 0x7f980c004800 2024-08-31T12:20:17.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:17.723+0000 7f9816ffd640 1 --2- 172.21.15.19:0/1100468261 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9804078340 0x7f980407a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:17.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:17.724+0000 7f9816ffd640 1 -- 172.21.15.19:0/1100468261 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f9810099f00 con 0x7f980c004800 2024-08-31T12:20:17.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:17.724+0000 7f982651c640 1 --2- 172.21.15.19:0/1100468261 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9804078340 0x7f980407a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:17.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:17.724+0000 7f982651c640 1 --2- 172.21.15.19:0/1100468261 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9804078340 0x7f980407a850 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f981c001050 tx=0x7f981c0015e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:17.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:17.728+0000 7f9816ffd640 1 -- 172.21.15.19:0/1100468261 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f9810025030 con 0x7f980c004800 2024-08-31T12:20:18.048 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.046+0000 7f9828780640 1 -- 172.21.15.19:0/1100468261 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9820158b40 con 0x7f980c004800 2024-08-31T12:20:18.048 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.046+0000 7f9816ffd640 1 -- 172.21.15.19:0/1100468261 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7f9810062bf0 con 0x7f980c004800 2024-08-31T12:20:18.048 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":36,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:20:18.051 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.049+0000 7f9828780640 1 -- 172.21.15.19:0/1100468261 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9804078340 msgr2=0x7f980407a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:18.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.049+0000 7f9828780640 1 --2- 172.21.15.19:0/1100468261 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9804078340 0x7f980407a850 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f981c001050 tx=0x7f981c0015e0 comp rx=0 tx=0).stop 2024-08-31T12:20:18.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.049+0000 7f9828780640 1 -- 172.21.15.19:0/1100468261 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f980c004800 msgr2=0x7f982014b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:18.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.049+0000 7f9828780640 1 --2- 172.21.15.19:0/1100468261 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f980c004800 0x7f982014b3c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9810007b20 tx=0x7f9810001230 comp rx=0 tx=0).stop 2024-08-31T12:20:18.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.049+0000 7f9828780640 1 -- 172.21.15.19:0/1100468261 shutdown_connections 2024-08-31T12:20:18.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.049+0000 7f9828780640 1 --2- 172.21.15.19:0/1100468261 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9804078340 0x7f980407a850 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:18.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.049+0000 7f9828780640 1 --2- 172.21.15.19:0/1100468261 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f980c005350 0x7f98201532f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:18.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.049+0000 7f9828780640 1 --2- 172.21.15.19:0/1100468261 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f980c004800 0x7f982014b3c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:18.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.049+0000 7f9828780640 1 --2- 172.21.15.19:0/1100468261 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f980c0039b0 0x7f982014cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:18.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.049+0000 7f9828780640 1 -- 172.21.15.19:0/1100468261 >> 172.21.15.19:0/1100468261 conn(0x7f9820074810 msgr2=0x7f9820068aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:18.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.049+0000 7f9828780640 1 -- 172.21.15.19:0/1100468261 shutdown_connections 2024-08-31T12:20:18.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.049+0000 7f9828780640 1 -- 172.21.15.19:0/1100468261 wait complete. 2024-08-31T12:20:18.063 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-08-31T12:20:18.064 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-08-31T12:20:18.064 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-31T12:20:18.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.197+0000 7fd356fda640 1 -- 172.21.15.19:0/920150453 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd350157920 msgr2=0x7fd350157d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:18.200 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.197+0000 7fd356fda640 1 --2- 172.21.15.19:0/920150453 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd350157920 0x7fd350157d30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd35006f220 tx=0x7fd34802a8b0 comp rx=0 tx=0).stop 2024-08-31T12:20:18.200 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.197+0000 7fd356fda640 1 -- 172.21.15.19:0/920150453 shutdown_connections 2024-08-31T12:20:18.200 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.197+0000 7fd356fda640 1 --2- 172.21.15.19:0/920150453 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd350157920 0x7fd350157d30 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:18.200 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.197+0000 7fd356fda640 1 --2- 172.21.15.19:0/920150453 >> v2:172.21.15.43:3300/0 conn(0x7fd3500c3640 0x7fd3500c3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:18.200 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.197+0000 7fd356fda640 1 -- 172.21.15.19:0/920150453 >> 172.21.15.19:0/920150453 conn(0x7fd35006cae0 msgr2=0x7fd35006cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:18.200 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.197+0000 7fd356fda640 1 -- 172.21.15.19:0/920150453 shutdown_connections 2024-08-31T12:20:18.200 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.197+0000 7fd356fda640 1 -- 172.21.15.19:0/920150453 wait complete. 2024-08-31T12:20:18.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.199+0000 7fd356fda640 1 Processor -- start 2024-08-31T12:20:18.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.199+0000 7fd356fda640 1 -- start start 2024-08-31T12:20:18.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.200+0000 7fd356fda640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd350070310 0x7fd3501560c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:18.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.200+0000 7fd356fda640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd3500c3640 0x7fd350145560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:18.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.200+0000 7fd356fda640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd350157920 0x7fd350145aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:18.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.200+0000 7fd356fda640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fd35006f5d0 con 0x7fd3500c3640 2024-08-31T12:20:18.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.200+0000 7fd356fda640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd35006f2f0 con 0x7fd350070310 2024-08-31T12:20:18.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.200+0000 7fd356fda640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fd35006f8b0 con 0x7fd350157920 2024-08-31T12:20:18.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.201+0000 7fd355577640 1 -- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd350157920 msgr2=0x7fd350145aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3301/0 2024-08-31T12:20:18.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.201+0000 7fd34ffff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd3500c3640 0x7fd350145560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:18.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.201+0000 7fd34ffff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd3500c3640 0x7fd350145560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:51998/0 (socket says 172.21.15.19:51998) 2024-08-31T12:20:18.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.201+0000 7fd34ffff640 1 -- 172.21.15.19:0/4285862259 learned_addr learned my addr 172.21.15.19:0/4285862259 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:20:18.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.201+0000 7fd355577640 1 --2- 172.21.15.19:0/4285862259 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd350157920 0x7fd350145aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:20:18.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.201+0000 7fd354d76640 1 --2- 172.21.15.19:0/4285862259 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd350070310 0x7fd3501560c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:18.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.201+0000 7fd34ffff640 1 -- 172.21.15.19:0/4285862259 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd350157920 msgr2=0x7fd350145aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:20:18.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.201+0000 7fd34ffff640 1 --2- 172.21.15.19:0/4285862259 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd350157920 0x7fd350145aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:18.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.201+0000 7fd34ffff640 1 -- 172.21.15.19:0/4285862259 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd350070310 msgr2=0x7fd3501560c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:18.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.201+0000 7fd34ffff640 1 --2- 172.21.15.19:0/4285862259 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd350070310 0x7fd3501560c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:18.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.201+0000 7fd34ffff640 1 -- 172.21.15.19:0/4285862259 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd350156600 con 0x7fd3500c3640 2024-08-31T12:20:18.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.201+0000 7fd34ffff640 1 --2- 172.21.15.19:0/4285862259 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd3500c3640 0x7fd350145560 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd33c00e790 tx=0x7fd33c00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:18.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.201+0000 7fd34d7fa640 1 -- 172.21.15.19:0/4285862259 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd33c00eea0 con 0x7fd3500c3640 2024-08-31T12:20:18.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.201+0000 7fd34d7fa640 1 -- 172.21.15.19:0/4285862259 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd33c00ccc0 con 0x7fd3500c3640 2024-08-31T12:20:18.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.201+0000 7fd356fda640 1 -- 172.21.15.19:0/4285862259 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd35014a530 con 0x7fd3500c3640 2024-08-31T12:20:18.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.201+0000 7fd34d7fa640 1 -- 172.21.15.19:0/4285862259 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd33c005280 con 0x7fd3500c3640 2024-08-31T12:20:18.206 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.201+0000 7fd356fda640 1 -- 172.21.15.19:0/4285862259 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3501558c0 con 0x7fd3500c3640 2024-08-31T12:20:18.207 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.202+0000 7fd356fda640 1 --2- 172.21.15.19:0/4285862259 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd350155ac0 0x7fd350146470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:18.207 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.203+0000 7fd356fda640 1 -- 172.21.15.19:0/4285862259 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fd350156ad0 con 0x7fd350155ac0 2024-08-31T12:20:18.207 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.203+0000 7fd354d76640 1 --2- 172.21.15.19:0/4285862259 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd350155ac0 0x7fd350146470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:18.207 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.203+0000 7fd354d76640 1 -- 172.21.15.19:0/4285862259 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd350145fe0 con 0x7fd350155ac0 2024-08-31T12:20:18.207 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.203+0000 7fd354d76640 1 --2- 172.21.15.19:0/4285862259 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd350155ac0 0x7fd350146470 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd34400f570 tx=0x7fd34400fa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:18.208 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.203+0000 7fd34d7fa640 1 -- 172.21.15.19:0/4285862259 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3440052e0 con 0x7fd350155ac0 2024-08-31T12:20:18.208 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.204+0000 7fd34d7fa640 1 -- 172.21.15.19:0/4285862259 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fd33c005420 con 0x7fd3500c3640 2024-08-31T12:20:18.208 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.204+0000 7fd34d7fa640 1 --2- 172.21.15.19:0/4285862259 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd334077950 0x7fd334079e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:18.208 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.204+0000 7fd34d7fa640 1 -- 172.21.15.19:0/4285862259 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fd33c099e80 con 0x7fd3500c3640 2024-08-31T12:20:18.208 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.205+0000 7fd355577640 1 --2- 172.21.15.19:0/4285862259 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd334077950 0x7fd334079e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:18.208 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.205+0000 7fd34d7fa640 1 -- 172.21.15.19:0/4285862259 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd344018070 con 0x7fd350155ac0 2024-08-31T12:20:18.208 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.205+0000 7fd34d7fa640 1 -- 172.21.15.19:0/4285862259 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd350155ac0 msgr2=0x7fd350146470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:18.208 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.205+0000 7fd34d7fa640 1 --2- 172.21.15.19:0/4285862259 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd350155ac0 0x7fd350146470 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd34400f570 tx=0x7fd34400fa30 comp rx=0 tx=0).stop 2024-08-31T12:20:18.208 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.205+0000 7fd34d7fa640 1 -- 172.21.15.19:0/4285862259 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd350155ac0 msgr2=0x7fd350146470 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:20:18.208 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.205+0000 7fd34d7fa640 1 --2- 172.21.15.19:0/4285862259 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd350155ac0 0x7fd350146470 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:18.208 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.205+0000 7fd355577640 1 --2- 172.21.15.19:0/4285862259 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd334077950 0x7fd334079e60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd348034000 tx=0x7fd348004ab0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:18.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.213+0000 7fd356fda640 1 --2- 172.21.15.19:0/4285862259 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd3500732e0 0x7fd3501469b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:18.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.213+0000 7fd356fda640 1 -- 172.21.15.19:0/4285862259 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fd350149fd0 con 0x7fd3500732e0 2024-08-31T12:20:18.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.213+0000 7fd354d76640 1 --2- 172.21.15.19:0/4285862259 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd3500732e0 0x7fd3501469b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:18.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.213+0000 7fd354d76640 1 -- 172.21.15.19:0/4285862259 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd3500cc210 con 0x7fd3500732e0 2024-08-31T12:20:18.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.214+0000 7fd354d76640 1 --2- 172.21.15.19:0/4285862259 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd3500732e0 0x7fd3501469b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd344017000 tx=0x7fd344007ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:18.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.214+0000 7fd34d7fa640 1 -- 172.21.15.19:0/4285862259 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd344027660 con 0x7fd3500732e0 2024-08-31T12:20:18.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.214+0000 7fd34d7fa640 1 -- 172.21.15.19:0/4285862259 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5033 (secure 0 0 0) 0x7fd344005480 con 0x7fd3500732e0 2024-08-31T12:20:18.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.214+0000 7fd34d7fa640 1 -- 172.21.15.19:0/4285862259 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd3500732e0 msgr2=0x7fd3501469b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:18.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.214+0000 7fd34d7fa640 1 --2- 172.21.15.19:0/4285862259 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd3500732e0 0x7fd3501469b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd344017000 tx=0x7fd344007ba0 comp rx=0 tx=0).stop 2024-08-31T12:20:18.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.214+0000 7fd34d7fa640 1 -- 172.21.15.19:0/4285862259 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd3500732e0 msgr2=0x7fd3501469b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:20:18.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.214+0000 7fd34d7fa640 1 --2- 172.21.15.19:0/4285862259 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd3500732e0 0x7fd3501469b0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:18.217 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:20:18.217 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:20:18.217 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:20:18.217 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "leader", 2024-08-31T12:20:18.217 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 36, 2024-08-31T12:20:18.217 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 198206, 2024-08-31T12:20:18.218 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:20:18.218 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:20:18.218 INFO:teuthology.orchestra.run.smithi019.stdout: 1 2024-08-31T12:20:18.218 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:18.218 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 0, 2024-08-31T12:20:18.218 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:20:18.218 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:20:18.218 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:20:18.218 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:20:18.218 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:20:18.218 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:20:18.218 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:20:18.218 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:20:18.219 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:20:18.219 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:20:18.219 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:20:18.219 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:20:18.219 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:20:18.219 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:20:18.219 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:18.219 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:20:18.219 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:20:18.219 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:20:18.219 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:20:18.219 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:20:18.219 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:20:18.220 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:20:18.220 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:20:18.220 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:20:18.220 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:20:18.220 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:20:18.220 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:20:18.220 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:20:18.220 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:18.220 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:18.220 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:20:18.220 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:20:18.220 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:20:18.220 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:20:18.221 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:20:18.221 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:20:18.221 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:20:18.221 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:20:18.221 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:20:18.221 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:20:18.221 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:20:18.221 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:20:18.221 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:20:18.221 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:20:18.221 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:20:18.221 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:20:18.221 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:20:18.222 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:20:18.222 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:20:18.222 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:20:18.222 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:20:18.222 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:20:18.222 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:20:18.222 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:20:18.222 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:20:18.222 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:20:18.222 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:20:18.222 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:20:18.222 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:18.222 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:20:18.222 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:18.223 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:20:18.223 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:18.223 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:20:18.223 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:20:18.223 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:20:18.223 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:20:18.223 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:18.223 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:20:18.223 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:20:18.223 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:18.223 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:18.223 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:18.223 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:20:18.224 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:20:18.224 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:18.224 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:18.224 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:18.224 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:18.224 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:20:18.224 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:20:18.224 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:20:18.224 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:20:18.224 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:20:18.224 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:18.224 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:18.224 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:20:18.225 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:20:18.225 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:20:18.225 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:20:18.225 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:18.225 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:20:18.225 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:20:18.225 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:18.225 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:18.225 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:18.225 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:20:18.225 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:20:18.225 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:18.225 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:18.226 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:18.226 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:18.226 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:20:18.226 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:20:18.226 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:20:18.226 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:20:18.226 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:20:18.226 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:18.226 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:18.226 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:20:18.226 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:20:18.226 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:20:18.226 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:20:18.226 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:18.227 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:20:18.227 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:20:18.227 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:18.227 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:18.227 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:18.227 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:20:18.227 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:20:18.227 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:18.227 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:18.227 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:18.227 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:18.227 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:20:18.227 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:20:18.227 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:20:18.228 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:20:18.228 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:20:18.228 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:18.228 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:18.228 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:18.228 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:20:18.228 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:20:18.228 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:18.228 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:20:18.228 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:20:18.228 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:20:18.228 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:18.228 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:18.229 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:20:18.229 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:18.229 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:20:18.229 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:20:18.229 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 6 2024-08-31T12:20:18.229 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:18.229 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:18.229 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:20:18.229 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:18.229 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:20:18.229 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:20:18.229 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 8 2024-08-31T12:20:18.229 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:18.229 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:18.229 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:20:18.229 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:18.230 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:20:18.230 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:20:18.230 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 6 2024-08-31T12:20:18.230 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:18.230 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:18.230 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:20:18.230 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:18.230 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:20:18.230 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:20:18.230 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:20:18.230 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:18.230 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:18.230 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:18.230 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:20:18.231 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:20:18.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.214+0000 7fd356fda640 1 -- 172.21.15.19:0/4285862259 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd334077950 msgr2=0x7fd334079e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:18.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.215+0000 7fd356fda640 1 --2- 172.21.15.19:0/4285862259 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd334077950 0x7fd334079e60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd348034000 tx=0x7fd348004ab0 comp rx=0 tx=0).stop 2024-08-31T12:20:18.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.215+0000 7fd356fda640 1 -- 172.21.15.19:0/4285862259 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd3500c3640 msgr2=0x7fd350145560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:18.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.215+0000 7fd356fda640 1 --2- 172.21.15.19:0/4285862259 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd3500c3640 0x7fd350145560 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd33c00e790 tx=0x7fd33c00ec50 comp rx=0 tx=0).stop 2024-08-31T12:20:18.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.215+0000 7fd355577640 1 -- 172.21.15.19:0/4285862259 reap_dead start 2024-08-31T12:20:18.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.215+0000 7fd355577640 1 -- 172.21.15.19:0/4285862259 reap_dead start 2024-08-31T12:20:18.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.215+0000 7fd356fda640 1 -- 172.21.15.19:0/4285862259 shutdown_connections 2024-08-31T12:20:18.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.215+0000 7fd356fda640 1 -- 172.21.15.19:0/4285862259 >> 172.21.15.19:0/4285862259 conn(0x7fd35006cae0 msgr2=0x7fd35014ac50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:18.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.215+0000 7fd356fda640 1 -- 172.21.15.19:0/4285862259 shutdown_connections 2024-08-31T12:20:18.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.215+0000 7fd356fda640 1 -- 172.21.15.19:0/4285862259 wait complete. 2024-08-31T12:20:18.233 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-08-31T12:20:18.233 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-08-31T12:20:18.233 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-31T12:20:18.355 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.353+0000 7f16a54b4640 1 -- 172.21.15.19:0/3040857726 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1688004d40 msgr2=0x7f1688005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:18.355 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.353+0000 7f16a54b4640 1 --2- 172.21.15.19:0/3040857726 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1688004d40 0x7f1688005190 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f16a0068ed0 tx=0x7f1688005830 comp rx=0 tx=0).stop 2024-08-31T12:20:18.355 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.353+0000 7f169f7fe640 1 -- 172.21.15.19:0/3040857726 reap_dead start 2024-08-31T12:20:18.356 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.353+0000 7f16a54b4640 1 -- 172.21.15.19:0/3040857726 shutdown_connections 2024-08-31T12:20:18.356 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.353+0000 7f16a54b4640 1 -- 172.21.15.19:0/3040857726 >> 172.21.15.19:0/3040857726 conn(0x7f16a006cae0 msgr2=0x7f16a006cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:18.356 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.353+0000 7f16a54b4640 1 -- 172.21.15.19:0/3040857726 shutdown_connections 2024-08-31T12:20:18.356 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.353+0000 7f16a54b4640 1 -- 172.21.15.19:0/3040857726 wait complete. 2024-08-31T12:20:18.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.355+0000 7f16a54b4640 1 Processor -- start 2024-08-31T12:20:18.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.355+0000 7f16a54b4640 1 -- start start 2024-08-31T12:20:18.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.357+0000 7f16a54b4640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f16a00b2f60 0x7f16a014e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:18.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.357+0000 7f16a54b4640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16a00cc7b0 0x7f16a01efa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:18.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.357+0000 7f16a54b4640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f16a01eff60 0x7f16a01447f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:18.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.357+0000 7f16a54b4640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f16a0068f90 con 0x7f16a00b2f60 2024-08-31T12:20:18.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.357+0000 7f16a54b4640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f16a0158cb0 con 0x7f16a00cc7b0 2024-08-31T12:20:18.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.357+0000 7f16a54b4640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f16a0069110 con 0x7f16a01eff60 2024-08-31T12:20:18.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.357+0000 7f169effd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f16a00b2f60 0x7f16a014e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:18.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.357+0000 7f169effd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f16a00b2f60 0x7f16a014e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:52018/0 (socket says 172.21.15.19:52018) 2024-08-31T12:20:18.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.357+0000 7f169effd640 1 -- 172.21.15.19:0/1117110736 learned_addr learned my addr 172.21.15.19:0/1117110736 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:20:18.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.357+0000 7f169f7fe640 1 -- 172.21.15.19:0/1117110736 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f16a01eff60 msgr2=0x7f16a01447f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3301/0 2024-08-31T12:20:18.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.357+0000 7f169f7fe640 1 --2- 172.21.15.19:0/1117110736 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f16a01eff60 0x7f16a01447f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:20:18.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.357+0000 7f169e7fc640 1 --2- 172.21.15.19:0/1117110736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16a00cc7b0 0x7f16a01efa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:18.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.357+0000 7f169effd640 1 -- 172.21.15.19:0/1117110736 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f16a01eff60 msgr2=0x7f16a01447f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:20:18.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.357+0000 7f169effd640 1 --2- 172.21.15.19:0/1117110736 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f16a01eff60 0x7f16a01447f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:18.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.357+0000 7f169effd640 1 -- 172.21.15.19:0/1117110736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16a00cc7b0 msgr2=0x7f16a01efa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:18.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.357+0000 7f169effd640 1 --2- 172.21.15.19:0/1117110736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16a00cc7b0 0x7f16a01efa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:18.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.357+0000 7f169effd640 1 -- 172.21.15.19:0/1117110736 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16a01e33b0 con 0x7f16a00b2f60 2024-08-31T12:20:18.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.357+0000 7f169effd640 1 --2- 172.21.15.19:0/1117110736 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f16a00b2f60 0x7f16a014e5f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f169000e770 tx=0x7f169000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:18.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.357+0000 7f16877fe640 1 -- 172.21.15.19:0/1117110736 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f169000ccc0 con 0x7f16a00b2f60 2024-08-31T12:20:18.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.358+0000 7f16877fe640 1 -- 172.21.15.19:0/1117110736 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f169000ee80 con 0x7f16a00b2f60 2024-08-31T12:20:18.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.358+0000 7f16877fe640 1 -- 172.21.15.19:0/1117110736 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1690010690 con 0x7f16a00b2f60 2024-08-31T12:20:18.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.358+0000 7f16a54b4640 1 -- 172.21.15.19:0/1117110736 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16a01f0370 con 0x7f16a00b2f60 2024-08-31T12:20:18.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.358+0000 7f16a54b4640 1 -- 172.21.15.19:0/1117110736 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16a006f590 con 0x7f16a00b2f60 2024-08-31T12:20:18.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.359+0000 7f16a54b4640 1 --2- 172.21.15.19:0/1117110736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16a0144d30 0x7f16a002d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:18.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.359+0000 7f16a54b4640 1 -- 172.21.15.19:0/1117110736 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f16a014e1e0 con 0x7f16a0144d30 2024-08-31T12:20:18.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.359+0000 7f169e7fc640 1 --2- 172.21.15.19:0/1117110736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16a0144d30 0x7f16a002d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:18.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.359+0000 7f169e7fc640 1 -- 172.21.15.19:0/1117110736 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f16a01e3620 con 0x7f16a0144d30 2024-08-31T12:20:18.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.360+0000 7f16877fe640 1 -- 172.21.15.19:0/1117110736 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f1690004920 con 0x7f16a00b2f60 2024-08-31T12:20:18.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.360+0000 7f169e7fc640 1 --2- 172.21.15.19:0/1117110736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16a0144d30 0x7f16a002d2b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f16a014efc0 tx=0x7f169400ff70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:18.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.360+0000 7f16877fe640 1 --2- 172.21.15.19:0/1117110736 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f168c077a00 0x7f168c079f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:18.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.360+0000 7f16877fe640 1 -- 172.21.15.19:0/1117110736 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f1690099e80 con 0x7f16a00b2f60 2024-08-31T12:20:18.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.360+0000 7f16877fe640 1 -- 172.21.15.19:0/1117110736 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1694019070 con 0x7f16a0144d30 2024-08-31T12:20:18.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.360+0000 7f169f7fe640 1 --2- 172.21.15.19:0/1117110736 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f168c077a00 0x7f168c079f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:18.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.360+0000 7f16877fe640 1 -- 172.21.15.19:0/1117110736 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1694005320 con 0x7f16a0144d30 2024-08-31T12:20:18.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.360+0000 7f16877fe640 1 -- 172.21.15.19:0/1117110736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16a0144d30 msgr2=0x7f16a002d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:18.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.360+0000 7f16877fe640 1 --2- 172.21.15.19:0/1117110736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16a0144d30 0x7f16a002d2b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f16a014efc0 tx=0x7f169400ff70 comp rx=0 tx=0).stop 2024-08-31T12:20:18.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.360+0000 7f16877fe640 1 -- 172.21.15.19:0/1117110736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16a0144d30 msgr2=0x7f16a002d2b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:20:18.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.360+0000 7f16877fe640 1 --2- 172.21.15.19:0/1117110736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16a0144d30 0x7f16a002d2b0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:18.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.361+0000 7f169f7fe640 1 --2- 172.21.15.19:0/1117110736 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f168c077a00 0x7f168c079f10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1698035000 tx=0x7f1698004a40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:18.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.368+0000 7f16a54b4640 1 --2- 172.21.15.19:0/1117110736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16a002d7f0 0x7f16a002dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:18.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.368+0000 7f16a54b4640 1 -- 172.21.15.19:0/1117110736 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f16a00c6fa0 con 0x7f16a002d7f0 2024-08-31T12:20:18.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.368+0000 7f169e7fc640 1 --2- 172.21.15.19:0/1117110736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16a002d7f0 0x7f16a002dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:18.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.368+0000 7f169e7fc640 1 -- 172.21.15.19:0/1117110736 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f16a00c7a30 con 0x7f16a002d7f0 2024-08-31T12:20:18.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.369+0000 7f169e7fc640 1 --2- 172.21.15.19:0/1117110736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16a002d7f0 0x7f16a002dc00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f16a014bf50 tx=0x7f1694003610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:18.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.369+0000 7f16877fe640 1 -- 172.21.15.19:0/1117110736 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1694019040 con 0x7f16a002d7f0 2024-08-31T12:20:18.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.369+0000 7f16877fe640 1 -- 172.21.15.19:0/1117110736 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4529 (secure 0 0 0) 0x7f1694007460 con 0x7f16a002d7f0 2024-08-31T12:20:18.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.369+0000 7f16877fe640 1 -- 172.21.15.19:0/1117110736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16a002d7f0 msgr2=0x7f16a002dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:18.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.369+0000 7f16877fe640 1 --2- 172.21.15.19:0/1117110736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16a002d7f0 0x7f16a002dc00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f16a014bf50 tx=0x7f1694003610 comp rx=0 tx=0).stop 2024-08-31T12:20:18.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.369+0000 7f16877fe640 1 -- 172.21.15.19:0/1117110736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16a002d7f0 msgr2=0x7f16a002dc00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:20:18.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.369+0000 7f16877fe640 1 --2- 172.21.15.19:0/1117110736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16a002d7f0 0x7f16a002dc00 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:18.372 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:20:18.372 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:20:18.372 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:20:18.372 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:20:18.372 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 36, 2024-08-31T12:20:18.372 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 34338, 2024-08-31T12:20:18.372 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:20:18.373 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:20:18.373 INFO:teuthology.orchestra.run.smithi019.stdout: 1 2024-08-31T12:20:18.373 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:18.373 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 0, 2024-08-31T12:20:18.373 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:20:18.373 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:20:18.373 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:20:18.373 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:20:18.373 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:20:18.373 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:20:18.373 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:20:18.373 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:20:18.373 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:20:18.374 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:20:18.374 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:20:18.374 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:20:18.374 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:20:18.374 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:20:18.374 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:18.374 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:20:18.374 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:20:18.374 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:20:18.374 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:20:18.374 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:20:18.374 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:20:18.374 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:20:18.374 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:20:18.375 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:20:18.375 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:20:18.375 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:20:18.375 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:20:18.375 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:20:18.375 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:18.375 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:18.375 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:20:18.375 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:20:18.375 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:20:18.375 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:20:18.375 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:20:18.375 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:20:18.375 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:20:18.376 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:20:18.376 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:20:18.376 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:20:18.376 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:20:18.376 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:20:18.376 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:20:18.376 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:20:18.376 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:20:18.376 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:20:18.376 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:20:18.376 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:20:18.376 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:20:18.376 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:20:18.377 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:20:18.377 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:20:18.377 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:20:18.377 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:20:18.377 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:20:18.377 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:20:18.377 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:20:18.377 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:20:18.377 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:18.377 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:20:18.377 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:18.377 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:20:18.377 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:18.378 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:20:18.378 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:20:18.378 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:20:18.378 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:20:18.378 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:18.378 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:20:18.378 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:20:18.378 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:18.378 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:18.378 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:18.378 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:20:18.378 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:20:18.378 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:18.378 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:18.379 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:18.379 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:18.379 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:20:18.379 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:20:18.379 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:20:18.379 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:20:18.379 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:20:18.379 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:18.379 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:18.379 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:20:18.379 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:20:18.379 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:20:18.379 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:20:18.379 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:18.380 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:20:18.380 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:20:18.380 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:18.380 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:18.380 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:18.380 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:20:18.380 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:20:18.380 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:18.380 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:18.380 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:18.380 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:18.380 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:20:18.380 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:20:18.380 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:20:18.381 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:20:18.381 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:20:18.381 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:18.381 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:18.381 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:20:18.381 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:20:18.381 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:20:18.381 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:20:18.381 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:18.381 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:20:18.381 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:20:18.381 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:18.381 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:18.382 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:18.382 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:20:18.382 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:20:18.382 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:18.382 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:18.382 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:18.382 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:18.382 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:20:18.382 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:20:18.382 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:20:18.382 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:20:18.382 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:20:18.382 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:18.383 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:18.383 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:18.383 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:20:18.383 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:20:18.383 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:18.383 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:20:18.383 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:20:18.383 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:20:18.383 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:18.383 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:18.383 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:20:18.383 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:18.384 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:20:18.384 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:20:18.384 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:20:18.384 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:18.384 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:18.384 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:18.384 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:20:18.384 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:20:18.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.370+0000 7f16a54b4640 1 -- 172.21.15.19:0/1117110736 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f168c077a00 msgr2=0x7f168c079f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:18.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.370+0000 7f16a54b4640 1 --2- 172.21.15.19:0/1117110736 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f168c077a00 0x7f168c079f10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1698035000 tx=0x7f1698004a40 comp rx=0 tx=0).stop 2024-08-31T12:20:18.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.370+0000 7f16a54b4640 1 -- 172.21.15.19:0/1117110736 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f16a00b2f60 msgr2=0x7f16a014e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:18.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.370+0000 7f16a54b4640 1 --2- 172.21.15.19:0/1117110736 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f16a00b2f60 0x7f16a014e5f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f169000e770 tx=0x7f169000ec30 comp rx=0 tx=0).stop 2024-08-31T12:20:18.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.370+0000 7f169f7fe640 1 -- 172.21.15.19:0/1117110736 reap_dead start 2024-08-31T12:20:18.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.370+0000 7f169f7fe640 1 -- 172.21.15.19:0/1117110736 reap_dead start 2024-08-31T12:20:18.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.370+0000 7f16a54b4640 1 -- 172.21.15.19:0/1117110736 shutdown_connections 2024-08-31T12:20:18.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.370+0000 7f16a54b4640 1 -- 172.21.15.19:0/1117110736 >> 172.21.15.19:0/1117110736 conn(0x7f16a006cae0 msgr2=0x7f16a006cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:18.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.370+0000 7f16a54b4640 1 -- 172.21.15.19:0/1117110736 shutdown_connections 2024-08-31T12:20:18.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:18.370+0000 7f16a54b4640 1 -- 172.21.15.19:0/1117110736 wait complete. 2024-08-31T12:20:18.386 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-08-31T12:20:18.386 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-08-31T12:20:26.405 DEBUG:teuthology.orchestra.run.smithi019:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:20:26.408 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:20:38.387 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-08-31T12:20:38.387 INFO:tasks.ceph.mon.c:Restarting daemon 2024-08-31T12:20:38.387 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-08-31T12:20:38.391 INFO:tasks.ceph.mon.c:Started 2024-08-31T12:20:38.391 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-08-31T12:20:38.391 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:20:38.522 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.521+0000 7f8811639640 1 -- 172.21.15.19:0/2081580174 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f880c0c7ac0 msgr2=0x7f880c152040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:38.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.521+0000 7f8811639640 1 --2- 172.21.15.19:0/2081580174 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f880c0c7ac0 0x7f880c152040 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f880c071e00 tx=0x7f87fc02a540 comp rx=0 tx=0).stop 2024-08-31T12:20:38.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.521+0000 7f8811639640 1 -- 172.21.15.19:0/2081580174 shutdown_connections 2024-08-31T12:20:38.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.521+0000 7f8811639640 1 --2- 172.21.15.19:0/2081580174 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f880c0c2400 0x7f880c0c89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:38.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.521+0000 7f8811639640 1 --2- 172.21.15.19:0/2081580174 >> v2:172.21.15.43:3300/0 conn(0x7f880c0b28d0 0x7f880c0c1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:38.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.521+0000 7f8811639640 1 --2- 172.21.15.19:0/2081580174 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f880c0c7ac0 0x7f880c152040 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:38.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.521+0000 7f8811639640 1 -- 172.21.15.19:0/2081580174 >> 172.21.15.19:0/2081580174 conn(0x7f880c06cae0 msgr2=0x7f880c06cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:38.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.521+0000 7f8811639640 1 -- 172.21.15.19:0/2081580174 shutdown_connections 2024-08-31T12:20:38.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.521+0000 7f8811639640 1 -- 172.21.15.19:0/2081580174 wait complete. 2024-08-31T12:20:38.524 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.523+0000 7f8811639640 1 Processor -- start 2024-08-31T12:20:38.524 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.523+0000 7f8811639640 1 -- start start 2024-08-31T12:20:38.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.525+0000 7f8811639640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f880c0b28d0 0x7f880c073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:38.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.525+0000 7f8811639640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f880c0c2400 0x7f880c14c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:38.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.525+0000 7f8811639640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f880c0c7ac0 0x7f880c146bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:38.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.525+0000 7f8811639640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f880c06da00 con 0x7f880c0c2400 2024-08-31T12:20:38.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.525+0000 7f8811639640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f880c068fd0 con 0x7f880c0c7ac0 2024-08-31T12:20:38.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.525+0000 7f8811639640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f880c0c3e90 con 0x7f880c0b28d0 2024-08-31T12:20:38.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.525+0000 7f8803fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f880c0c2400 0x7f880c14c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:38.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.525+0000 7f8803fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f880c0c2400 0x7f880c14c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:53802/0 (socket says 172.21.15.19:53802) 2024-08-31T12:20:38.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.525+0000 7f8803fff640 1 -- 172.21.15.19:0/1537562856 learned_addr learned my addr 172.21.15.19:0/1537562856 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:20:38.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.525+0000 7f880b7fe640 1 --2- 172.21.15.19:0/1537562856 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f880c0c7ac0 0x7f880c146bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:38.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.525+0000 7f880affd640 1 --2- 172.21.15.19:0/1537562856 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f880c0b28d0 0x7f880c073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:38.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.525+0000 7f880affd640 1 -- 172.21.15.19:0/1537562856 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f880c0b28d0 msgr2=0x7f880c073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-08-31T12:20:38.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.525+0000 7f880affd640 1 -- 172.21.15.19:0/1537562856 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f880c0b28d0 msgr2=0x7f880c073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-31T12:20:38.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.525+0000 7f880affd640 1 --2- 172.21.15.19:0/1537562856 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f880c0b28d0 0x7f880c073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-31T12:20:38.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.525+0000 7f880affd640 1 --2- 172.21.15.19:0/1537562856 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f880c0b28d0 0x7f880c073330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:20:38.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.525+0000 7f8803fff640 1 -- 172.21.15.19:0/1537562856 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f880c0b28d0 msgr2=0x7f880c073330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:20:38.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.525+0000 7f8803fff640 1 --2- 172.21.15.19:0/1537562856 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f880c0b28d0 0x7f880c073330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:38.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.525+0000 7f8803fff640 1 -- 172.21.15.19:0/1537562856 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f880c0c7ac0 msgr2=0x7f880c146bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:38.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.525+0000 7f8803fff640 1 --2- 172.21.15.19:0/1537562856 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f880c0c7ac0 0x7f880c146bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:38.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.525+0000 7f8803fff640 1 -- 172.21.15.19:0/1537562856 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f880c070060 con 0x7f880c0c2400 2024-08-31T12:20:38.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.526+0000 7f8803fff640 1 --2- 172.21.15.19:0/1537562856 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f880c0c2400 0x7f880c14c680 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f87f400ea20 tx=0x7f87f400eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:38.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.526+0000 7f88037fe640 1 -- 172.21.15.19:0/1537562856 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87f400cd80 con 0x7f880c0c2400 2024-08-31T12:20:38.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.526+0000 7f88037fe640 1 -- 172.21.15.19:0/1537562856 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f87f4004020 con 0x7f880c0c2400 2024-08-31T12:20:38.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.526+0000 7f88037fe640 1 -- 172.21.15.19:0/1537562856 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87f4004470 con 0x7f880c0c2400 2024-08-31T12:20:38.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.526+0000 7f8811639640 1 -- 172.21.15.19:0/1537562856 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f880c06fd30 con 0x7f880c0c2400 2024-08-31T12:20:38.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.526+0000 7f8811639640 1 -- 172.21.15.19:0/1537562856 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f880c142d10 con 0x7f880c0c2400 2024-08-31T12:20:38.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.527+0000 7f8811639640 1 -- 172.21.15.19:0/1537562856 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f880c0c1ec0 con 0x7f880c0c2400 2024-08-31T12:20:38.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.528+0000 7f88037fe640 1 -- 172.21.15.19:0/1537562856 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f87f40035f0 con 0x7f880c0c2400 2024-08-31T12:20:38.531 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.528+0000 7f88037fe640 1 --2- 172.21.15.19:0/1537562856 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f87ec077a50 0x7f87ec079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:38.531 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.528+0000 7f88037fe640 1 -- 172.21.15.19:0/1537562856 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f87f4099f80 con 0x7f880c0c2400 2024-08-31T12:20:38.531 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.529+0000 7f880affd640 1 --2- 172.21.15.19:0/1537562856 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f87ec077a50 0x7f87ec079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:38.531 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.529+0000 7f880affd640 1 --2- 172.21.15.19:0/1537562856 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f87ec077a50 0x7f87ec079f60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f87fc034000 tx=0x7f87fc004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:38.535 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.534+0000 7f88037fe640 1 -- 172.21.15.19:0/1537562856 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f87f4062c70 con 0x7f880c0c2400 2024-08-31T12:20:38.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.820+0000 7f8811639640 1 -- 172.21.15.19:0/1537562856 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f880c15c250 con 0x7f880c0c2400 2024-08-31T12:20:38.822 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.821+0000 7f88037fe640 1 -- 172.21.15.19:0/1537562856 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f87f4062570 con 0x7f880c0c2400 2024-08-31T12:20:38.822 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":36,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:20:38.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.823+0000 7f8811639640 1 -- 172.21.15.19:0/1537562856 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f87ec077a50 msgr2=0x7f87ec079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:38.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.823+0000 7f8811639640 1 --2- 172.21.15.19:0/1537562856 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f87ec077a50 0x7f87ec079f60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f87fc034000 tx=0x7f87fc004730 comp rx=0 tx=0).stop 2024-08-31T12:20:38.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.823+0000 7f8811639640 1 -- 172.21.15.19:0/1537562856 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f880c0c2400 msgr2=0x7f880c14c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:38.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.823+0000 7f8811639640 1 --2- 172.21.15.19:0/1537562856 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f880c0c2400 0x7f880c14c680 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f87f400ea20 tx=0x7f87f400eee0 comp rx=0 tx=0).stop 2024-08-31T12:20:38.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.823+0000 7f8811639640 1 -- 172.21.15.19:0/1537562856 shutdown_connections 2024-08-31T12:20:38.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.823+0000 7f8811639640 1 --2- 172.21.15.19:0/1537562856 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f87ec077a50 0x7f87ec079f60 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:38.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.823+0000 7f8811639640 1 --2- 172.21.15.19:0/1537562856 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f880c0c7ac0 0x7f880c146bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:38.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.823+0000 7f8811639640 1 --2- 172.21.15.19:0/1537562856 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f880c0c2400 0x7f880c14c680 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:38.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.823+0000 7f8811639640 1 --2- 172.21.15.19:0/1537562856 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f880c0b28d0 0x7f880c073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:38.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.823+0000 7f8811639640 1 -- 172.21.15.19:0/1537562856 >> 172.21.15.19:0/1537562856 conn(0x7f880c06cae0 msgr2=0x7f880c156590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:38.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.823+0000 7f8811639640 1 -- 172.21.15.19:0/1537562856 shutdown_connections 2024-08-31T12:20:38.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:38.824+0000 7f8811639640 1 -- 172.21.15.19:0/1537562856 wait complete. 2024-08-31T12:20:41.836 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:20:41.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:41.969+0000 7f422b625640 1 -- 172.21.15.19:0/1026777842 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4210004800 msgr2=0x7f4210004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:41.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:41.969+0000 7f422b625640 1 --2- 172.21.15.19:0/1026777842 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4210004800 0x7f4210004c10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4214001740 tx=0x7f4210005830 comp rx=0 tx=0).stop 2024-08-31T12:20:41.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:41.969+0000 7f422b625640 1 -- 172.21.15.19:0/1026777842 shutdown_connections 2024-08-31T12:20:41.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:41.969+0000 7f422b625640 1 --2- 172.21.15.19:0/1026777842 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4210004800 0x7f4210004c10 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:41.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:41.969+0000 7f422b625640 1 -- 172.21.15.19:0/1026777842 >> 172.21.15.19:0/1026777842 conn(0x7f42240cefb0 msgr2=0x7f42240cb340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:41.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:41.969+0000 7f422b625640 1 -- 172.21.15.19:0/1026777842 shutdown_connections 2024-08-31T12:20:41.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:41.969+0000 7f422b625640 1 -- 172.21.15.19:0/1026777842 wait complete. 2024-08-31T12:20:41.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:41.971+0000 7f422b625640 1 Processor -- start 2024-08-31T12:20:41.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:41.971+0000 7f422b625640 1 -- start start 2024-08-31T12:20:41.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:41.973+0000 7f422b625640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4210004800 0x7f42241eecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:41.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:41.973+0000 7f422b625640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f422414e620 0x7f42240727b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:41.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:41.973+0000 7f422b625640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42241471c0 0x7f42241475d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:41.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:41.973+0000 7f422b625640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f42240714f0 con 0x7f4210004800 2024-08-31T12:20:41.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:41.973+0000 7f422b625640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f42240c9bb0 con 0x7f42241471c0 2024-08-31T12:20:41.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:41.973+0000 7f422b625640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f422415d3a0 con 0x7f422414e620 2024-08-31T12:20:41.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:41.973+0000 7f42293c1640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4210004800 0x7f42241eecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:41.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:41.973+0000 7f42293c1640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4210004800 0x7f42241eecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:53816/0 (socket says 172.21.15.19:53816) 2024-08-31T12:20:41.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:41.973+0000 7f42293c1640 1 -- 172.21.15.19:0/3898697254 learned_addr learned my addr 172.21.15.19:0/3898697254 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:20:41.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:41.973+0000 7f4228bc0640 1 --2- 172.21.15.19:0/3898697254 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f422414e620 0x7f42240727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:41.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:41.973+0000 7f4229bc2640 1 --2- 172.21.15.19:0/3898697254 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42241471c0 0x7f42241475d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:41.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:41.973+0000 7f4228bc0640 1 -- 172.21.15.19:0/3898697254 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f422414e620 msgr2=0x7f42240727b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-08-31T12:20:41.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:41.973+0000 7f4228bc0640 1 -- 172.21.15.19:0/3898697254 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f422414e620 msgr2=0x7f42240727b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-31T12:20:41.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:41.973+0000 7f4228bc0640 1 --2- 172.21.15.19:0/3898697254 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f422414e620 0x7f42240727b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-31T12:20:41.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:41.973+0000 7f4228bc0640 1 --2- 172.21.15.19:0/3898697254 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f422414e620 0x7f42240727b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:20:41.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:41.973+0000 7f42293c1640 1 -- 172.21.15.19:0/3898697254 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f422414e620 msgr2=0x7f42240727b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:20:41.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:41.973+0000 7f42293c1640 1 --2- 172.21.15.19:0/3898697254 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f422414e620 0x7f42240727b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:41.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:41.973+0000 7f42293c1640 1 -- 172.21.15.19:0/3898697254 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42241471c0 msgr2=0x7f42241475d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:41.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:41.973+0000 7f42293c1640 1 --2- 172.21.15.19:0/3898697254 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42241471c0 0x7f42241475d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:41.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:41.973+0000 7f42293c1640 1 -- 172.21.15.19:0/3898697254 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4224155c60 con 0x7f4210004800 2024-08-31T12:20:41.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:41.973+0000 7f42293c1640 1 --2- 172.21.15.19:0/3898697254 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4210004800 0x7f42241eecf0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4220011b10 tx=0x7f4220009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:41.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:41.974+0000 7f4219ffb640 1 -- 172.21.15.19:0/3898697254 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f422000fce0 con 0x7f4210004800 2024-08-31T12:20:41.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:41.974+0000 7f4219ffb640 1 -- 172.21.15.19:0/3898697254 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f422000fe80 con 0x7f4210004800 2024-08-31T12:20:41.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:41.974+0000 7f422b625640 1 -- 172.21.15.19:0/3898697254 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4224155f00 con 0x7f4210004800 2024-08-31T12:20:41.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:41.974+0000 7f4219ffb640 1 -- 172.21.15.19:0/3898697254 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4220004280 con 0x7f4210004800 2024-08-31T12:20:41.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:41.974+0000 7f422b625640 1 -- 172.21.15.19:0/3898697254 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42241564c0 con 0x7f4210004800 2024-08-31T12:20:41.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:41.975+0000 7f422b625640 1 -- 172.21.15.19:0/3898697254 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42240c9ec0 con 0x7f4210004800 2024-08-31T12:20:41.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:41.976+0000 7f4219ffb640 1 -- 172.21.15.19:0/3898697254 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f42200035f0 con 0x7f4210004800 2024-08-31T12:20:41.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:41.976+0000 7f4219ffb640 1 --2- 172.21.15.19:0/3898697254 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4208077a80 0x7f4208079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:41.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:41.976+0000 7f4219ffb640 1 -- 172.21.15.19:0/3898697254 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f422009a0f0 con 0x7f4210004800 2024-08-31T12:20:41.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:41.977+0000 7f4228bc0640 1 --2- 172.21.15.19:0/3898697254 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4208077a80 0x7f4208079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:41.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:41.977+0000 7f4228bc0640 1 --2- 172.21.15.19:0/3898697254 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4208077a80 0x7f4208079f90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4214001710 tx=0x7f4214001270 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:41.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:41.981+0000 7f4219ffb640 1 -- 172.21.15.19:0/3898697254 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f4220062790 con 0x7f4210004800 2024-08-31T12:20:42.278 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.276+0000 7f422b625640 1 -- 172.21.15.19:0/3898697254 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f42240ca160 con 0x7f4210004800 2024-08-31T12:20:42.278 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.277+0000 7f4219ffb640 1 -- 172.21.15.19:0/3898697254 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f4220066f50 con 0x7f4210004800 2024-08-31T12:20:42.278 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:20:42.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.279+0000 7f422b625640 1 -- 172.21.15.19:0/3898697254 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4208077a80 msgr2=0x7f4208079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:42.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.279+0000 7f422b625640 1 --2- 172.21.15.19:0/3898697254 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4208077a80 0x7f4208079f90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4214001710 tx=0x7f4214001270 comp rx=0 tx=0).stop 2024-08-31T12:20:42.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.279+0000 7f422b625640 1 -- 172.21.15.19:0/3898697254 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4210004800 msgr2=0x7f42241eecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:42.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.279+0000 7f422b625640 1 --2- 172.21.15.19:0/3898697254 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4210004800 0x7f42241eecf0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4220011b10 tx=0x7f4220009f10 comp rx=0 tx=0).stop 2024-08-31T12:20:42.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.279+0000 7f422b625640 1 -- 172.21.15.19:0/3898697254 shutdown_connections 2024-08-31T12:20:42.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.279+0000 7f422b625640 1 --2- 172.21.15.19:0/3898697254 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4208077a80 0x7f4208079f90 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:42.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.279+0000 7f422b625640 1 --2- 172.21.15.19:0/3898697254 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42241471c0 0x7f42241475d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:42.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.279+0000 7f422b625640 1 --2- 172.21.15.19:0/3898697254 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f422414e620 0x7f42240727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:42.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.279+0000 7f422b625640 1 --2- 172.21.15.19:0/3898697254 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4210004800 0x7f42241eecf0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:42.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.279+0000 7f422b625640 1 -- 172.21.15.19:0/3898697254 >> 172.21.15.19:0/3898697254 conn(0x7f42240cefb0 msgr2=0x7f42240718f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:42.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.279+0000 7f422b625640 1 -- 172.21.15.19:0/3898697254 shutdown_connections 2024-08-31T12:20:42.282 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.279+0000 7f422b625640 1 -- 172.21.15.19:0/3898697254 wait complete. 2024-08-31T12:20:42.292 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-08-31T12:20:42.292 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-08-31T12:20:42.292 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-31T12:20:42.416 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.415+0000 7fd0718a4640 1 -- 172.21.15.19:0/121703019 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd054003a00 msgr2=0x7fd054003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:42.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.415+0000 7fd0718a4640 1 --2- 172.21.15.19:0/121703019 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd054003a00 0x7fd054003e70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd05c00c290 tx=0x7fd054005830 comp rx=0 tx=0).stop 2024-08-31T12:20:42.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.415+0000 7fd06b7fe640 1 -- 172.21.15.19:0/121703019 reap_dead start 2024-08-31T12:20:42.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.415+0000 7fd0718a4640 1 -- 172.21.15.19:0/121703019 shutdown_connections 2024-08-31T12:20:42.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.415+0000 7fd0718a4640 1 -- 172.21.15.19:0/121703019 >> 172.21.15.19:0/121703019 conn(0x7fd06c06cae0 msgr2=0x7fd06c06cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:42.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.416+0000 7fd0718a4640 1 -- 172.21.15.19:0/121703019 shutdown_connections 2024-08-31T12:20:42.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.416+0000 7fd0718a4640 1 -- 172.21.15.19:0/121703019 wait complete. 2024-08-31T12:20:42.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.417+0000 7fd0718a4640 1 Processor -- start 2024-08-31T12:20:42.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.418+0000 7fd0718a4640 1 -- start start 2024-08-31T12:20:42.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.419+0000 7fd0718a4640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd054003a00 0x7fd06c1e2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:42.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.419+0000 7fd0718a4640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd06c151310 0x7fd06c1e3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:42.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.419+0000 7fd0718a4640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd06c143690 0x7fd06c143aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:42.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.419+0000 7fd0718a4640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fd06c06f0a0 con 0x7fd054003a00 2024-08-31T12:20:42.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.419+0000 7fd0718a4640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd06c070260 con 0x7fd06c143690 2024-08-31T12:20:42.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.419+0000 7fd0718a4640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fd06c153400 con 0x7fd06c151310 2024-08-31T12:20:42.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.419+0000 7fd06affd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd054003a00 0x7fd06c1e2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:42.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.419+0000 7fd06affd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd054003a00 0x7fd06c1e2690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:53828/0 (socket says 172.21.15.19:53828) 2024-08-31T12:20:42.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.419+0000 7fd06affd640 1 -- 172.21.15.19:0/2192038182 learned_addr learned my addr 172.21.15.19:0/2192038182 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:20:42.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.419+0000 7fd06b7fe640 1 --2- 172.21.15.19:0/2192038182 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd06c143690 0x7fd06c143aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:42.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.419+0000 7fd06a7fc640 1 --2- 172.21.15.19:0/2192038182 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd06c151310 0x7fd06c1e3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:42.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.419+0000 7fd06a7fc640 1 -- 172.21.15.19:0/2192038182 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd06c151310 msgr2=0x7fd06c1e3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-08-31T12:20:42.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.419+0000 7fd06a7fc640 1 -- 172.21.15.19:0/2192038182 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd06c151310 msgr2=0x7fd06c1e3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-31T12:20:42.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.419+0000 7fd06a7fc640 1 --2- 172.21.15.19:0/2192038182 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd06c151310 0x7fd06c1e3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-31T12:20:42.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.419+0000 7fd06affd640 1 -- 172.21.15.19:0/2192038182 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd06c151310 msgr2=0x7fd06c1e3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:42.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.419+0000 7fd06a7fc640 1 --2- 172.21.15.19:0/2192038182 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd06c151310 0x7fd06c1e3640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:20:42.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.419+0000 7fd06affd640 1 --2- 172.21.15.19:0/2192038182 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd06c151310 0x7fd06c1e3640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:42.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.419+0000 7fd06affd640 1 -- 172.21.15.19:0/2192038182 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd06c143690 msgr2=0x7fd06c143aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:42.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.419+0000 7fd06affd640 1 --2- 172.21.15.19:0/2192038182 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd06c143690 0x7fd06c143aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:42.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.419+0000 7fd06affd640 1 -- 172.21.15.19:0/2192038182 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd06c1e2bd0 con 0x7fd054003a00 2024-08-31T12:20:42.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.420+0000 7fd06affd640 1 --2- 172.21.15.19:0/2192038182 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd054003a00 0x7fd06c1e2690 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fd05c030040 tx=0x7fd05c006380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:42.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.420+0000 7fd053fff640 1 -- 172.21.15.19:0/2192038182 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05c031050 con 0x7fd054003a00 2024-08-31T12:20:42.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.420+0000 7fd053fff640 1 -- 172.21.15.19:0/2192038182 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd05c003af0 con 0x7fd054003a00 2024-08-31T12:20:42.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.420+0000 7fd0718a4640 1 -- 172.21.15.19:0/2192038182 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd06c0cc410 con 0x7fd054003a00 2024-08-31T12:20:42.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.420+0000 7fd053fff640 1 -- 172.21.15.19:0/2192038182 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05c006570 con 0x7fd054003a00 2024-08-31T12:20:42.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.420+0000 7fd0718a4640 1 -- 172.21.15.19:0/2192038182 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd06c150780 con 0x7fd054003a00 2024-08-31T12:20:42.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.421+0000 7fd0718a4640 1 --2- 172.21.15.19:0/2192038182 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd06c152020 0x7fd06c1f23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:42.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.421+0000 7fd0718a4640 1 -- 172.21.15.19:0/2192038182 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fd06c144420 con 0x7fd06c152020 2024-08-31T12:20:42.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.421+0000 7fd06a7fc640 1 --2- 172.21.15.19:0/2192038182 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd06c152020 0x7fd06c1f23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:42.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.422+0000 7fd06a7fc640 1 -- 172.21.15.19:0/2192038182 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd06c0c8e50 con 0x7fd06c152020 2024-08-31T12:20:42.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.422+0000 7fd06a7fc640 1 --2- 172.21.15.19:0/2192038182 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd06c152020 0x7fd06c1f23f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd06c0c9700 tx=0x7fd060012e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:42.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.422+0000 7fd053fff640 1 -- 172.21.15.19:0/2192038182 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd060004220 con 0x7fd06c152020 2024-08-31T12:20:42.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.422+0000 7fd053fff640 1 -- 172.21.15.19:0/2192038182 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fd05c02d450 con 0x7fd054003a00 2024-08-31T12:20:42.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.423+0000 7fd053fff640 1 --2- 172.21.15.19:0/2192038182 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd058077950 0x7fd058079e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:42.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.423+0000 7fd053fff640 1 -- 172.21.15.19:0/2192038182 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fd05c0b9f80 con 0x7fd054003a00 2024-08-31T12:20:42.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.423+0000 7fd053fff640 1 -- 172.21.15.19:0/2192038182 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd0600043c0 con 0x7fd06c152020 2024-08-31T12:20:42.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.423+0000 7fd053fff640 1 -- 172.21.15.19:0/2192038182 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd06c152020 msgr2=0x7fd06c1f23f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:42.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.423+0000 7fd053fff640 1 --2- 172.21.15.19:0/2192038182 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd06c152020 0x7fd06c1f23f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd06c0c9700 tx=0x7fd060012e80 comp rx=0 tx=0).stop 2024-08-31T12:20:42.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.423+0000 7fd053fff640 1 -- 172.21.15.19:0/2192038182 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd06c152020 msgr2=0x7fd06c1f23f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:20:42.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.423+0000 7fd053fff640 1 --2- 172.21.15.19:0/2192038182 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd06c152020 0x7fd06c1f23f0 secure :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd06c0c9700 tx=0x7fd060012e80 comp rx=0 tx=0).stop 2024-08-31T12:20:42.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.423+0000 7fd06b7fe640 1 --2- 172.21.15.19:0/2192038182 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd058077950 0x7fd058079e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:42.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.424+0000 7fd06b7fe640 1 --2- 172.21.15.19:0/2192038182 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd058077950 0x7fd058079e60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd06c159f90 tx=0x7fd064009500 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:42.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.429+0000 7fd0718a4640 1 --2- 172.21.15.19:0/2192038182 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd06c1f2930 0x7fd06c1f2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:42.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.429+0000 7fd0718a4640 1 -- 172.21.15.19:0/2192038182 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fd06c1f3280 con 0x7fd06c1f2930 2024-08-31T12:20:42.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.429+0000 7fd06a7fc640 1 --2- 172.21.15.19:0/2192038182 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd06c1f2930 0x7fd06c1f2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:42.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.430+0000 7fd06a7fc640 1 -- 172.21.15.19:0/2192038182 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd06c0cbbc0 con 0x7fd06c1f2930 2024-08-31T12:20:42.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.430+0000 7fd06a7fc640 1 --2- 172.21.15.19:0/2192038182 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd06c1f2930 0x7fd06c1f2d40 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd06001d000 tx=0x7fd060004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:42.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.430+0000 7fd053fff640 1 -- 172.21.15.19:0/2192038182 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd060023950 con 0x7fd06c1f2930 2024-08-31T12:20:42.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.430+0000 7fd053fff640 1 -- 172.21.15.19:0/2192038182 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5044 (secure 0 0 0) 0x7fd0600035f0 con 0x7fd06c1f2930 2024-08-31T12:20:42.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.430+0000 7fd053fff640 1 -- 172.21.15.19:0/2192038182 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd06c1f2930 msgr2=0x7fd06c1f2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:42.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.430+0000 7fd053fff640 1 --2- 172.21.15.19:0/2192038182 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd06c1f2930 0x7fd06c1f2d40 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd06001d000 tx=0x7fd060004730 comp rx=0 tx=0).stop 2024-08-31T12:20:42.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.430+0000 7fd053fff640 1 -- 172.21.15.19:0/2192038182 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd06c1f2930 msgr2=0x7fd06c1f2d40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:20:42.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.430+0000 7fd053fff640 1 --2- 172.21.15.19:0/2192038182 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd06c1f2930 0x7fd06c1f2d40 secure :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd06001d000 tx=0x7fd060004730 comp rx=0 tx=0).stop 2024-08-31T12:20:42.432 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:20:42.432 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:20:42.432 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:20:42.432 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "leader", 2024-08-31T12:20:42.432 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 38, 2024-08-31T12:20:42.432 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 222421, 2024-08-31T12:20:42.432 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:20:42.433 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:20:42.433 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:20:42.433 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:20:42.433 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:42.433 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 0, 2024-08-31T12:20:42.433 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:20:42.433 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:20:42.433 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:20:42.433 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:20:42.433 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:20:42.433 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:20:42.433 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:20:42.433 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:20:42.434 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:20:42.434 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:20:42.434 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:20:42.434 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:20:42.434 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:20:42.434 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:20:42.434 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:42.434 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:20:42.434 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:20:42.434 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:20:42.434 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:20:42.434 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:20:42.434 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:20:42.435 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:20:42.435 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:20:42.435 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:20:42.435 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:20:42.435 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:20:42.435 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:20:42.435 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:20:42.435 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:42.435 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:42.435 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:20:42.435 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:20:42.435 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:20:42.435 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:20:42.435 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:20:42.436 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:20:42.436 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:20:42.436 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:20:42.436 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:20:42.436 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:20:42.436 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:20:42.436 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:20:42.436 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:20:42.436 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:20:42.436 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:20:42.436 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:20:42.436 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:20:42.436 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:20:42.437 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:20:42.437 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:20:42.437 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:20:42.437 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:20:42.437 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:20:42.437 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:20:42.437 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:20:42.437 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:20:42.437 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:20:42.437 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:20:42.437 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:42.437 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:20:42.437 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:42.437 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:20:42.437 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:42.438 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:20:42.438 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:20:42.438 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:20:42.438 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:20:42.438 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:42.438 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:20:42.438 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:20:42.438 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:42.438 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:42.438 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:42.438 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:20:42.438 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:20:42.438 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:42.438 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:42.439 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:42.439 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:42.439 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:20:42.439 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:20:42.439 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:20:42.439 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:20:42.439 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:20:42.439 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:42.439 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:42.439 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:20:42.439 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:20:42.439 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:20:42.439 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:20:42.439 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:42.439 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:20:42.440 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:20:42.440 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:42.440 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:42.440 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:42.440 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:20:42.440 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:20:42.440 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:42.440 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:42.440 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:42.440 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:42.440 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:20:42.440 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:20:42.440 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:20:42.441 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:20:42.441 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:20:42.441 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:42.441 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:42.441 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:20:42.441 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:20:42.441 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:20:42.441 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:20:42.441 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:42.441 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:20:42.441 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:20:42.441 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:42.442 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:42.442 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:42.442 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:20:42.442 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:20:42.442 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:42.442 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:42.442 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:42.442 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:42.442 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:20:42.442 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:20:42.442 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:20:42.442 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:20:42.442 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:20:42.443 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:42.443 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:42.443 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:42.443 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:20:42.443 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:20:42.443 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:42.443 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:20:42.443 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:20:42.443 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:20:42.443 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:42.443 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:42.443 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:20:42.443 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:42.444 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:20:42.444 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:20:42.444 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 6 2024-08-31T12:20:42.444 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:42.444 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:42.444 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:20:42.444 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:42.444 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:20:42.444 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:20:42.444 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 8 2024-08-31T12:20:42.444 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:42.444 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:42.444 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:20:42.445 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:42.445 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:20:42.445 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:20:42.445 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 6 2024-08-31T12:20:42.445 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:42.445 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:42.445 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:20:42.445 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:42.445 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:20:42.445 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:20:42.445 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:20:42.445 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:42.445 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:42.445 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:42.446 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:20:42.446 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:20:42.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.431+0000 7fd0718a4640 1 -- 172.21.15.19:0/2192038182 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd058077950 msgr2=0x7fd058079e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:42.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.431+0000 7fd0718a4640 1 --2- 172.21.15.19:0/2192038182 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd058077950 0x7fd058079e60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd06c159f90 tx=0x7fd064009500 comp rx=0 tx=0).stop 2024-08-31T12:20:42.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.431+0000 7fd0718a4640 1 -- 172.21.15.19:0/2192038182 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd054003a00 msgr2=0x7fd06c1e2690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:42.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.431+0000 7fd0718a4640 1 --2- 172.21.15.19:0/2192038182 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd054003a00 0x7fd06c1e2690 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fd05c030040 tx=0x7fd05c006380 comp rx=0 tx=0).stop 2024-08-31T12:20:42.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.431+0000 7fd06b7fe640 1 -- 172.21.15.19:0/2192038182 reap_dead start 2024-08-31T12:20:42.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.431+0000 7fd06b7fe640 1 -- 172.21.15.19:0/2192038182 reap_dead start 2024-08-31T12:20:42.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.431+0000 7fd0718a4640 1 -- 172.21.15.19:0/2192038182 shutdown_connections 2024-08-31T12:20:42.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.431+0000 7fd0718a4640 1 -- 172.21.15.19:0/2192038182 >> 172.21.15.19:0/2192038182 conn(0x7fd06c06cae0 msgr2=0x7fd06c0c6c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:42.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.431+0000 7fd0718a4640 1 -- 172.21.15.19:0/2192038182 shutdown_connections 2024-08-31T12:20:42.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.431+0000 7fd0718a4640 1 -- 172.21.15.19:0/2192038182 wait complete. 2024-08-31T12:20:42.448 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:20:42.448 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-08-31T12:20:42.448 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-31T12:20:42.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.576+0000 7f5a05550640 1 -- 172.21.15.19:0/4159595996 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5a001579a0 msgr2=0x7f5a0014e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:42.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.576+0000 7f5a05550640 1 --2- 172.21.15.19:0/4159595996 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5a001579a0 0x7f5a0014e070 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5a000703e0 tx=0x7f59f802aa30 comp rx=0 tx=0).stop 2024-08-31T12:20:42.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.576+0000 7f5a05550640 1 -- 172.21.15.19:0/4159595996 shutdown_connections 2024-08-31T12:20:42.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.576+0000 7f5a05550640 1 --2- 172.21.15.19:0/4159595996 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5a001579a0 0x7f5a0014e070 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:42.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.576+0000 7f5a05550640 1 --2- 172.21.15.19:0/4159595996 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a00157590 0x7f5a0014db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:42.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.576+0000 7f5a05550640 1 --2- 172.21.15.19:0/4159595996 >> v2:172.21.15.43:3300/0 conn(0x7f5a000c8220 0x7f5a000cb640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:42.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.576+0000 7f5a05550640 1 -- 172.21.15.19:0/4159595996 >> 172.21.15.19:0/4159595996 conn(0x7f5a000ccff0 msgr2=0x7f5a000cd400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:42.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.576+0000 7f5a05550640 1 -- 172.21.15.19:0/4159595996 shutdown_connections 2024-08-31T12:20:42.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.576+0000 7f5a05550640 1 -- 172.21.15.19:0/4159595996 wait complete. 2024-08-31T12:20:42.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.579+0000 7f5a05550640 1 Processor -- start 2024-08-31T12:20:42.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.579+0000 7f5a05550640 1 -- start start 2024-08-31T12:20:42.583 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.581+0000 7f5a05550640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a000c8220 0x7f5a001500d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:42.583 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.581+0000 7f5a05550640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a00157590 0x7f5a001467c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:42.583 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.581+0000 7f5a05550640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5a001579a0 0x7f5a00142840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:42.583 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.581+0000 7f5a05550640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f5a000cc080 con 0x7f5a001579a0 2024-08-31T12:20:42.584 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.581+0000 7f5a05550640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5a000c8e20 con 0x7f5a00157590 2024-08-31T12:20:42.584 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.581+0000 7f5a05550640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f5a000cac10 con 0x7f5a000c8220 2024-08-31T12:20:42.584 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.581+0000 7f59feffd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a000c8220 0x7f5a001500d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:42.584 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.581+0000 7f59ff7fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5a001579a0 0x7f5a00142840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:42.584 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.581+0000 7f59fe7fc640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a00157590 0x7f5a001467c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:42.584 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.581+0000 7f59ff7fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5a001579a0 0x7f5a00142840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:53864/0 (socket says 172.21.15.19:53864) 2024-08-31T12:20:42.584 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.581+0000 7f59feffd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a000c8220 0x7f5a001500d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.19:56642/0 (socket says 172.21.15.19:56642) 2024-08-31T12:20:42.584 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.581+0000 7f59ff7fe640 1 -- 172.21.15.19:0/1852797874 learned_addr learned my addr 172.21.15.19:0/1852797874 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:20:42.584 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.581+0000 7f59feffd640 1 -- 172.21.15.19:0/1852797874 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a000c8220 msgr2=0x7f5a001500d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-08-31T12:20:42.584 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.581+0000 7f59feffd640 1 -- 172.21.15.19:0/1852797874 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a000c8220 msgr2=0x7f5a001500d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-31T12:20:42.584 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.581+0000 7f59feffd640 1 --2- 172.21.15.19:0/1852797874 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a000c8220 0x7f5a001500d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-31T12:20:42.584 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.581+0000 7f59feffd640 1 --2- 172.21.15.19:0/1852797874 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a000c8220 0x7f5a001500d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:20:42.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.581+0000 7f59ff7fe640 1 -- 172.21.15.19:0/1852797874 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a000c8220 msgr2=0x7f5a001500d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:20:42.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.581+0000 7f59ff7fe640 1 --2- 172.21.15.19:0/1852797874 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a000c8220 0x7f5a001500d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:42.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.581+0000 7f59ff7fe640 1 -- 172.21.15.19:0/1852797874 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a00157590 msgr2=0x7f5a001467c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:42.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.581+0000 7f59ff7fe640 1 --2- 172.21.15.19:0/1852797874 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a00157590 0x7f5a001467c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:42.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.581+0000 7f59ff7fe640 1 -- 172.21.15.19:0/1852797874 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a00154b60 con 0x7f5a001579a0 2024-08-31T12:20:42.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.582+0000 7f59ff7fe640 1 --2- 172.21.15.19:0/1852797874 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5a001579a0 0x7f5a00142840 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f59f8009d60 tx=0x7f59f8003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:42.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.582+0000 7f59e77fe640 1 -- 172.21.15.19:0/1852797874 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f8033050 con 0x7f5a001579a0 2024-08-31T12:20:42.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.582+0000 7f59e77fe640 1 -- 172.21.15.19:0/1852797874 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f59f8004450 con 0x7f5a001579a0 2024-08-31T12:20:42.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.582+0000 7f5a05550640 1 -- 172.21.15.19:0/1852797874 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a00149740 con 0x7f5a001579a0 2024-08-31T12:20:42.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.582+0000 7f59e77fe640 1 -- 172.21.15.19:0/1852797874 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f8038450 con 0x7f5a001579a0 2024-08-31T12:20:42.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.582+0000 7f5a05550640 1 -- 172.21.15.19:0/1852797874 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a00144a00 con 0x7f5a001579a0 2024-08-31T12:20:42.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.583+0000 7f5a05550640 1 --2- 172.21.15.19:0/1852797874 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a00147120 0x7f5a0014f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:42.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.583+0000 7f5a05550640 1 -- 172.21.15.19:0/1852797874 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0006a020 con 0x7f5a00147120 2024-08-31T12:20:42.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.583+0000 7f59feffd640 1 --2- 172.21.15.19:0/1852797874 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a00147120 0x7f5a0014f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:42.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.583+0000 7f59feffd640 1 -- 172.21.15.19:0/1852797874 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5a00070dd0 con 0x7f5a00147120 2024-08-31T12:20:42.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.584+0000 7f59feffd640 1 --2- 172.21.15.19:0/1852797874 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a00147120 0x7f5a0014f910 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f59f000f6d0 tx=0x7f59f000fb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:42.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.584+0000 7f59e77fe640 1 -- 172.21.15.19:0/1852797874 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f000bb10 con 0x7f5a00147120 2024-08-31T12:20:42.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.584+0000 7f59e77fe640 1 -- 172.21.15.19:0/1852797874 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f59f000bcb0 con 0x7f5a00147120 2024-08-31T12:20:42.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.584+0000 7f59e77fe640 1 -- 172.21.15.19:0/1852797874 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a00147120 msgr2=0x7f5a0014f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:42.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.584+0000 7f59e77fe640 1 --2- 172.21.15.19:0/1852797874 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a00147120 0x7f5a0014f910 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f59f000f6d0 tx=0x7f59f000fb90 comp rx=0 tx=0).stop 2024-08-31T12:20:42.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.584+0000 7f59e77fe640 1 -- 172.21.15.19:0/1852797874 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a00147120 msgr2=0x7f5a0014f910 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:20:42.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.584+0000 7f59e77fe640 1 --2- 172.21.15.19:0/1852797874 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a00147120 0x7f5a0014f910 secure :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f59f000f6d0 tx=0x7f59f000fb90 comp rx=0 tx=0).stop 2024-08-31T12:20:42.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.584+0000 7f59e77fe640 1 -- 172.21.15.19:0/1852797874 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f59f8036070 con 0x7f5a001579a0 2024-08-31T12:20:42.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.585+0000 7f59e77fe640 1 --2- 172.21.15.19:0/1852797874 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f59ec077650 0x7f59ec079b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:42.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.585+0000 7f59e77fe640 1 -- 172.21.15.19:0/1852797874 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f59f80bab60 con 0x7f5a001579a0 2024-08-31T12:20:42.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.585+0000 7f59feffd640 1 --2- 172.21.15.19:0/1852797874 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f59ec077650 0x7f59ec079b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:42.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.586+0000 7f59feffd640 1 --2- 172.21.15.19:0/1852797874 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f59ec077650 0x7f59ec079b60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f59f000b730 tx=0x7f59f001d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:42.592 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.591+0000 7f5a05550640 1 --2- 172.21.15.19:0/1852797874 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a00142d80 0x7f5a00148a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:42.592 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.591+0000 7f5a05550640 1 -- 172.21.15.19:0/1852797874 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5a00143190 con 0x7f5a00142d80 2024-08-31T12:20:42.592 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.591+0000 7f59fe7fc640 1 --2- 172.21.15.19:0/1852797874 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a00142d80 0x7f5a00148a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:42.593 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.592+0000 7f59fe7fc640 1 -- 172.21.15.19:0/1852797874 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5a000ccab0 con 0x7f5a00142d80 2024-08-31T12:20:42.593 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.592+0000 7f59fe7fc640 1 --2- 172.21.15.19:0/1852797874 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a00142d80 0x7f5a00148a70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5a000ca380 tx=0x7f59f4008600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:42.593 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.592+0000 7f59e77fe640 1 -- 172.21.15.19:0/1852797874 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f4008c50 con 0x7f5a00142d80 2024-08-31T12:20:42.594 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.592+0000 7f59e77fe640 1 -- 172.21.15.19:0/1852797874 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4540 (secure 0 0 0) 0x7f59f4008c50 con 0x7f5a00142d80 2024-08-31T12:20:42.594 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.592+0000 7f59e77fe640 1 -- 172.21.15.19:0/1852797874 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a00142d80 msgr2=0x7f5a00148a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:42.594 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.592+0000 7f59e77fe640 1 --2- 172.21.15.19:0/1852797874 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a00142d80 0x7f5a00148a70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5a000ca380 tx=0x7f59f4008600 comp rx=0 tx=0).stop 2024-08-31T12:20:42.594 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.592+0000 7f59e77fe640 1 -- 172.21.15.19:0/1852797874 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a00142d80 msgr2=0x7f5a00148a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:20:42.594 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.592+0000 7f59e77fe640 1 --2- 172.21.15.19:0/1852797874 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a00142d80 0x7f5a00148a70 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:42.594 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:20:42.594 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:20:42.594 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:20:42.594 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:20:42.595 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 38, 2024-08-31T12:20:42.595 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 58561, 2024-08-31T12:20:42.595 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:20:42.595 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:20:42.595 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:20:42.595 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:20:42.595 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:42.595 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 1, 2024-08-31T12:20:42.595 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:20:42.595 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:20:42.595 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:20:42.595 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:20:42.595 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:20:42.595 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:20:42.596 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:20:42.596 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:20:42.596 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:20:42.596 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:20:42.596 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:20:42.596 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:20:42.596 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:20:42.596 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:20:42.596 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:42.596 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:20:42.596 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:20:42.596 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:20:42.596 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:20:42.596 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:20:42.596 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:20:42.597 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:20:42.597 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:20:42.597 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:20:42.597 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:20:42.597 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:20:42.597 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:20:42.597 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:20:42.597 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:42.597 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:42.597 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:20:42.597 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:20:42.597 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:20:42.597 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:20:42.598 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:20:42.598 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:20:42.598 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:20:42.598 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:20:42.598 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:20:42.598 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:20:42.598 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:20:42.598 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:20:42.598 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:20:42.598 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:20:42.598 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:20:42.598 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:20:42.598 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:20:42.598 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:20:42.598 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:20:42.599 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:20:42.599 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:20:42.599 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:20:42.599 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:20:42.599 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:20:42.599 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:20:42.599 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:20:42.599 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:20:42.599 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:20:42.599 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:42.599 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:20:42.599 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:42.599 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:20:42.599 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:42.600 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:20:42.600 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:20:42.600 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:20:42.600 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:20:42.600 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:42.600 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:20:42.600 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:20:42.600 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:42.600 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:42.600 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:42.600 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:20:42.600 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:20:42.600 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:42.600 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:42.601 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:42.601 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:42.601 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:20:42.601 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:20:42.601 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:20:42.601 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:20:42.601 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:20:42.601 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:42.601 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:42.601 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:20:42.601 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:20:42.601 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:20:42.601 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:20:42.602 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:42.602 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:20:42.602 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:20:42.602 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:42.602 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:42.602 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:42.602 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:20:42.602 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:20:42.602 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:42.602 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:42.602 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:42.602 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:42.602 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:20:42.602 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:20:42.602 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:20:42.603 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:20:42.603 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:20:42.603 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:42.603 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:42.603 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:20:42.603 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:20:42.603 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:20:42.603 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:20:42.603 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:42.603 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:20:42.603 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:20:42.603 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:42.603 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:42.604 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:42.604 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:20:42.604 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:20:42.604 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:42.604 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:42.604 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:42.604 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:42.604 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:20:42.604 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:20:42.604 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:20:42.604 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:20:42.604 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:20:42.604 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:42.605 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:42.605 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:42.605 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:20:42.605 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:20:42.605 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:42.605 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:20:42.605 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:20:42.605 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:20:42.605 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:42.605 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:42.605 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:20:42.605 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:42.605 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:20:42.605 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:20:42.605 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:20:42.606 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:42.606 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:42.606 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:42.606 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:20:42.606 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:20:42.607 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.593+0000 7f5a05550640 1 -- 172.21.15.19:0/1852797874 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f59ec077650 msgr2=0x7f59ec079b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:42.607 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.593+0000 7f5a05550640 1 --2- 172.21.15.19:0/1852797874 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f59ec077650 0x7f59ec079b60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f59f000b730 tx=0x7f59f001d040 comp rx=0 tx=0).stop 2024-08-31T12:20:42.607 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.593+0000 7f5a05550640 1 -- 172.21.15.19:0/1852797874 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5a001579a0 msgr2=0x7f5a00142840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:42.607 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.593+0000 7f5a05550640 1 --2- 172.21.15.19:0/1852797874 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5a001579a0 0x7f5a00142840 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f59f8009d60 tx=0x7f59f8003f10 comp rx=0 tx=0).stop 2024-08-31T12:20:42.607 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.593+0000 7f59ff7fe640 1 -- 172.21.15.19:0/1852797874 reap_dead start 2024-08-31T12:20:42.607 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.593+0000 7f5a05550640 1 -- 172.21.15.19:0/1852797874 shutdown_connections 2024-08-31T12:20:42.607 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.593+0000 7f5a05550640 1 --2- 172.21.15.19:0/1852797874 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5a001579a0 0x7f5a00142840 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:42.607 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.593+0000 7f5a05550640 1 -- 172.21.15.19:0/1852797874 >> 172.21.15.19:0/1852797874 conn(0x7f5a000ccff0 msgr2=0x7f5a00156560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:42.608 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.593+0000 7f5a05550640 1 -- 172.21.15.19:0/1852797874 shutdown_connections 2024-08-31T12:20:42.608 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.593+0000 7f5a05550640 1 -- 172.21.15.19:0/1852797874 wait complete. 2024-08-31T12:20:42.608 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:20:42.608 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-08-31T12:20:42.608 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-31T12:20:42.732 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.731+0000 7fa26999f640 1 -- 172.21.15.19:0/2227046578 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa264071dd0 msgr2=0x7fa264152dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:42.732 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.731+0000 7fa26999f640 1 --2- 172.21.15.19:0/2227046578 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa264071dd0 0x7fa264152dd0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa264073c20 tx=0x7fa25802a5f0 comp rx=0 tx=0).stop 2024-08-31T12:20:42.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.731+0000 7fa26999f640 1 -- 172.21.15.19:0/2227046578 shutdown_connections 2024-08-31T12:20:42.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.731+0000 7fa26999f640 1 --2- 172.21.15.19:0/2227046578 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa264071dd0 0x7fa264152dd0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:42.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.731+0000 7fa26999f640 1 --2- 172.21.15.19:0/2227046578 >> v2:172.21.15.43:3300/0 conn(0x7fa2640c3ff0 0x7fa2640c8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:42.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.731+0000 7fa26999f640 1 -- 172.21.15.19:0/2227046578 >> 172.21.15.19:0/2227046578 conn(0x7fa2640694c0 msgr2=0x7fa2640728b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:42.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.731+0000 7fa26999f640 1 -- 172.21.15.19:0/2227046578 shutdown_connections 2024-08-31T12:20:42.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.731+0000 7fa26999f640 1 -- 172.21.15.19:0/2227046578 wait complete. 2024-08-31T12:20:42.734 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.733+0000 7fa26999f640 1 Processor -- start 2024-08-31T12:20:42.734 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.733+0000 7fa26999f640 1 -- start start 2024-08-31T12:20:42.737 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.735+0000 7fa26999f640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa264071dd0 0x7fa2641489e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:42.737 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.735+0000 7fa26999f640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa2640c3ff0 0x7fa2640b2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:42.737 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.735+0000 7fa26999f640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa2640ce550 0x7fa2641e6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:42.737 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.735+0000 7fa26999f640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fa264153310 con 0x7fa2640c3ff0 2024-08-31T12:20:42.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.735+0000 7fa26999f640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa264159950 con 0x7fa2640ce550 2024-08-31T12:20:42.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.735+0000 7fa26999f640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fa26415cc80 con 0x7fa264071dd0 2024-08-31T12:20:42.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.735+0000 7fa2627fc640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa2640c3ff0 0x7fa2640b2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:42.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.735+0000 7fa2627fc640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa2640c3ff0 0x7fa2640b2420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:53872/0 (socket says 172.21.15.19:53872) 2024-08-31T12:20:42.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.735+0000 7fa2627fc640 1 -- 172.21.15.19:0/3295446130 learned_addr learned my addr 172.21.15.19:0/3295446130 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:20:42.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.735+0000 7fa2637fe640 1 --2- 172.21.15.19:0/3295446130 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa2640ce550 0x7fa2641e6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:42.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.735+0000 7fa262ffd640 1 --2- 172.21.15.19:0/3295446130 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa264071dd0 0x7fa2641489e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:42.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.735+0000 7fa2627fc640 1 -- 172.21.15.19:0/3295446130 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa264071dd0 msgr2=0x7fa2641489e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:42.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.735+0000 7fa2627fc640 1 --2- 172.21.15.19:0/3295446130 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa264071dd0 0x7fa2641489e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:42.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.735+0000 7fa2627fc640 1 -- 172.21.15.19:0/3295446130 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa2640ce550 msgr2=0x7fa2641e6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:42.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.735+0000 7fa2627fc640 1 --2- 172.21.15.19:0/3295446130 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa2640ce550 0x7fa2641e6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:42.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.735+0000 7fa2627fc640 1 -- 172.21.15.19:0/3295446130 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2640b2f10 con 0x7fa2640c3ff0 2024-08-31T12:20:42.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.735+0000 7fa2627fc640 1 --2- 172.21.15.19:0/3295446130 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa2640c3ff0 0x7fa2640b2420 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa258000c00 tx=0x7fa258003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:42.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.736+0000 7fa24bfff640 1 -- 172.21.15.19:0/3295446130 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa25803e030 con 0x7fa2640c3ff0 2024-08-31T12:20:42.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.736+0000 7fa26999f640 1 -- 172.21.15.19:0/3295446130 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa26406efc0 con 0x7fa2640c3ff0 2024-08-31T12:20:42.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.736+0000 7fa24bfff640 1 -- 172.21.15.19:0/3295446130 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa258003e00 con 0x7fa2640c3ff0 2024-08-31T12:20:42.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.736+0000 7fa24bfff640 1 -- 172.21.15.19:0/3295446130 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa258035040 con 0x7fa2640c3ff0 2024-08-31T12:20:42.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.736+0000 7fa26999f640 1 -- 172.21.15.19:0/3295446130 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2640b3190 con 0x7fa2640c3ff0 2024-08-31T12:20:42.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.737+0000 7fa26999f640 1 --2- 172.21.15.19:0/3295446130 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa2641e6770 0x7fa26406dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:42.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.737+0000 7fa26999f640 1 -- 172.21.15.19:0/3295446130 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fa2641f4f40 con 0x7fa2641e6770 2024-08-31T12:20:42.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.737+0000 7fa262ffd640 1 --2- 172.21.15.19:0/3295446130 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa2641e6770 0x7fa26406dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:42.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.737+0000 7fa262ffd640 1 -- 172.21.15.19:0/3295446130 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa2641e6770 msgr2=0x7fa26406dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 20 2024-08-31T12:20:42.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.737+0000 7fa262ffd640 1 -- 172.21.15.19:0/3295446130 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa2641e6770 msgr2=0x7fa26406dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-31T12:20:42.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.737+0000 7fa262ffd640 1 --2- 172.21.15.19:0/3295446130 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa2641e6770 0x7fa26406dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-31T12:20:42.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.737+0000 7fa262ffd640 1 --2- 172.21.15.19:0/3295446130 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa2641e6770 0x7fa26406dbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:20:42.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.738+0000 7fa24bfff640 1 -- 172.21.15.19:0/3295446130 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fa258041030 con 0x7fa2640c3ff0 2024-08-31T12:20:42.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.738+0000 7fa24bfff640 1 --2- 172.21.15.19:0/3295446130 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa250077a50 0x7fa250079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:42.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.738+0000 7fa24bfff640 1 -- 172.21.15.19:0/3295446130 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fa2580b9fc0 con 0x7fa2640c3ff0 2024-08-31T12:20:42.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.738+0000 7fa2637fe640 1 --2- 172.21.15.19:0/3295446130 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa250077a50 0x7fa250079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:42.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.739+0000 7fa2637fe640 1 --2- 172.21.15.19:0/3295446130 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa250077a50 0x7fa250079f60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa25c007fd0 tx=0x7fa25c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:42.940 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.939+0000 7fa262ffd640 1 --2- 172.21.15.19:0/3295446130 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa2641e6770 0x7fa26406dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:42.941 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.939+0000 7fa262ffd640 1 -- 172.21.15.19:0/3295446130 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa2641e6770 msgr2=0x7fa26406dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 21 2024-08-31T12:20:42.941 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.939+0000 7fa262ffd640 1 -- 172.21.15.19:0/3295446130 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa2641e6770 msgr2=0x7fa26406dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-31T12:20:42.941 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.939+0000 7fa262ffd640 1 --2- 172.21.15.19:0/3295446130 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa2641e6770 0x7fa26406dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-31T12:20:42.941 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:42.939+0000 7fa262ffd640 1 --2- 172.21.15.19:0/3295446130 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa2641e6770 0x7fa26406dbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-31T12:20:43.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:43.340+0000 7fa262ffd640 1 --2- 172.21.15.19:0/3295446130 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa2641e6770 0x7fa26406dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:43.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:43.340+0000 7fa262ffd640 1 -- 172.21.15.19:0/3295446130 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa2641e6770 msgr2=0x7fa26406dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 21 2024-08-31T12:20:43.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:43.340+0000 7fa262ffd640 1 -- 172.21.15.19:0/3295446130 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa2641e6770 msgr2=0x7fa26406dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-31T12:20:43.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:43.340+0000 7fa262ffd640 1 --2- 172.21.15.19:0/3295446130 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa2641e6770 0x7fa26406dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-31T12:20:43.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:43.340+0000 7fa262ffd640 1 --2- 172.21.15.19:0/3295446130 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa2641e6770 0x7fa26406dbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-31T12:20:44.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.141+0000 7fa262ffd640 1 --2- 172.21.15.19:0/3295446130 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa2641e6770 0x7fa26406dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:44.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.141+0000 7fa262ffd640 1 -- 172.21.15.19:0/3295446130 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa264152dd0 con 0x7fa2641e6770 2024-08-31T12:20:44.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.142+0000 7fa262ffd640 1 --2- 172.21.15.19:0/3295446130 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa2641e6770 0x7fa26406dbf0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa2540119e0 tx=0x7fa254011ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:44.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.142+0000 7fa24bfff640 1 -- 172.21.15.19:0/3295446130 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa254021070 con 0x7fa2641e6770 2024-08-31T12:20:44.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.142+0000 7fa24bfff640 1 -- 172.21.15.19:0/3295446130 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa254010c10 con 0x7fa2641e6770 2024-08-31T12:20:44.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.142+0000 7fa24bfff640 1 -- 172.21.15.19:0/3295446130 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa2641e6770 msgr2=0x7fa26406dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:44.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.142+0000 7fa24bfff640 1 --2- 172.21.15.19:0/3295446130 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa2641e6770 0x7fa26406dbf0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa2540119e0 tx=0x7fa254011ea0 comp rx=0 tx=0).stop 2024-08-31T12:20:44.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.142+0000 7fa24bfff640 1 -- 172.21.15.19:0/3295446130 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa2641e6770 msgr2=0x7fa26406dbf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:20:44.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.142+0000 7fa24bfff640 1 --2- 172.21.15.19:0/3295446130 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa2641e6770 0x7fa26406dbf0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:44.151 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.150+0000 7fa26999f640 1 --2- 172.21.15.19:0/3295446130 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa26406e130 0x7fa26406e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:44.151 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.150+0000 7fa26999f640 1 -- 172.21.15.19:0/3295446130 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fa26406ea80 con 0x7fa26406e130 2024-08-31T12:20:44.151 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.151+0000 7fa262ffd640 1 --2- 172.21.15.19:0/3295446130 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa26406e130 0x7fa26406e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:44.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.151+0000 7fa262ffd640 1 -- 172.21.15.19:0/3295446130 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa264152680 con 0x7fa26406e130 2024-08-31T12:20:44.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.151+0000 7fa262ffd640 1 --2- 172.21.15.19:0/3295446130 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa26406e130 0x7fa26406e540 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa25401e000 tx=0x7fa254004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:44.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.151+0000 7fa24bfff640 1 -- 172.21.15.19:0/3295446130 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa254021040 con 0x7fa26406e130 2024-08-31T12:20:44.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.152+0000 7fa24bfff640 1 -- 172.21.15.19:0/3295446130 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4539 (secure 0 0 0) 0x7fa25402e420 con 0x7fa26406e130 2024-08-31T12:20:44.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.152+0000 7fa24bfff640 1 -- 172.21.15.19:0/3295446130 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa26406e130 msgr2=0x7fa26406e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:44.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.152+0000 7fa24bfff640 1 --2- 172.21.15.19:0/3295446130 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa26406e130 0x7fa26406e540 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa25401e000 tx=0x7fa254004730 comp rx=0 tx=0).stop 2024-08-31T12:20:44.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.152+0000 7fa24bfff640 1 -- 172.21.15.19:0/3295446130 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa26406e130 msgr2=0x7fa26406e540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:20:44.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.152+0000 7fa24bfff640 1 --2- 172.21.15.19:0/3295446130 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa26406e130 0x7fa26406e540 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:44.153 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:20:44.154 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:20:44.154 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:20:44.154 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:20:44.154 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 38, 2024-08-31T12:20:44.154 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 5655, 2024-08-31T12:20:44.154 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:20:44.154 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:20:44.154 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:20:44.154 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:20:44.154 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:44.154 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 2, 2024-08-31T12:20:44.154 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:20:44.155 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:20:44.155 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:20:44.155 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:20:44.155 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:20:44.155 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:20:44.155 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:20:44.155 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:20:44.155 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:20:44.155 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:20:44.155 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:20:44.155 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:20:44.155 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:20:44.155 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:20:44.155 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:44.156 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:20:44.156 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:20:44.156 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:20:44.156 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:20:44.156 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:20:44.156 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:20:44.156 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:20:44.156 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:20:44.156 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:20:44.156 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:20:44.156 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:20:44.156 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:20:44.156 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:20:44.156 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:44.157 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:44.157 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:20:44.157 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:20:44.157 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:20:44.157 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:20:44.157 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:20:44.157 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:20:44.157 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:20:44.157 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:20:44.157 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:20:44.157 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:20:44.157 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:20:44.157 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:20:44.157 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:20:44.158 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:20:44.158 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:20:44.158 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:20:44.158 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:20:44.158 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:20:44.158 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:20:44.158 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:20:44.158 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:20:44.158 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:20:44.158 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:20:44.158 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:20:44.158 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:20:44.158 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:20:44.158 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:20:44.159 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:20:44.159 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:44.159 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:20:44.159 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:44.159 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:20:44.159 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:44.159 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:20:44.159 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:20:44.159 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:20:44.159 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:20:44.159 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:44.159 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:20:44.159 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:20:44.159 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:44.160 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:44.160 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:44.160 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:20:44.160 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:20:44.160 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:44.160 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:44.160 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:44.160 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:44.160 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:20:44.160 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:20:44.160 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:20:44.160 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:20:44.160 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:20:44.161 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:44.161 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:44.161 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:20:44.161 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:20:44.161 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:20:44.161 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:20:44.161 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:44.161 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:20:44.161 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:20:44.161 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:44.161 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:44.161 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:44.162 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:20:44.162 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:20:44.162 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:44.162 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:44.162 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:44.162 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:44.162 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:20:44.162 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:20:44.162 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:20:44.162 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:20:44.162 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:20:44.162 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:44.162 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:44.163 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:20:44.163 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:20:44.163 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:20:44.163 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:20:44.163 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:44.163 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:20:44.163 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:20:44.163 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:44.163 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:44.163 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:44.163 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:20:44.163 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:20:44.163 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:44.164 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:44.164 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:44.164 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:44.164 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:20:44.164 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:20:44.164 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:20:44.164 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:20:44.164 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:20:44.164 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:44.164 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:44.164 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:44.164 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:20:44.165 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:20:44.165 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:44.165 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:20:44.165 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:20:44.165 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:20:44.165 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:44.165 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:44.165 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:20:44.165 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:44.165 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:20:44.165 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:20:44.165 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:20:44.165 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:44.165 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:44.166 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:44.166 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:20:44.166 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:20:44.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.152+0000 7fa26999f640 1 -- 172.21.15.19:0/3295446130 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa250077a50 msgr2=0x7fa250079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:44.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.152+0000 7fa26999f640 1 --2- 172.21.15.19:0/3295446130 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa250077a50 0x7fa250079f60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa25c007fd0 tx=0x7fa25c009040 comp rx=0 tx=0).stop 2024-08-31T12:20:44.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.152+0000 7fa26999f640 1 -- 172.21.15.19:0/3295446130 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa2640c3ff0 msgr2=0x7fa2640b2420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:44.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.152+0000 7fa26999f640 1 --2- 172.21.15.19:0/3295446130 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa2640c3ff0 0x7fa2640b2420 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa258000c00 tx=0x7fa258003910 comp rx=0 tx=0).stop 2024-08-31T12:20:44.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.152+0000 7fa2637fe640 1 -- 172.21.15.19:0/3295446130 reap_dead start 2024-08-31T12:20:44.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.152+0000 7fa2637fe640 1 -- 172.21.15.19:0/3295446130 reap_dead start 2024-08-31T12:20:44.168 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.152+0000 7fa26999f640 1 -- 172.21.15.19:0/3295446130 shutdown_connections 2024-08-31T12:20:44.168 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.152+0000 7fa26999f640 1 -- 172.21.15.19:0/3295446130 >> 172.21.15.19:0/3295446130 conn(0x7fa2640694c0 msgr2=0x7fa2640ccc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:44.168 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.152+0000 7fa26999f640 1 -- 172.21.15.19:0/3295446130 shutdown_connections 2024-08-31T12:20:44.168 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.152+0000 7fa26999f640 1 -- 172.21.15.19:0/3295446130 wait complete. 2024-08-31T12:20:44.168 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:20:44.168 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-08-31T12:20:44.168 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-08-31T12:20:44.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.301+0000 7f4947e0a640 1 -- 172.21.15.19:0/887601036 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f49400c9440 msgr2=0x7f4940070bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:44.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.301+0000 7f4947e0a640 1 --2- 172.21.15.19:0/887601036 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f49400c9440 0x7f4940070bf0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4938005fd0 tx=0x7f493802a940 comp rx=0 tx=0).stop 2024-08-31T12:20:44.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.301+0000 7f4947e0a640 1 -- 172.21.15.19:0/887601036 shutdown_connections 2024-08-31T12:20:44.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.301+0000 7f4947e0a640 1 --2- 172.21.15.19:0/887601036 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f49400c9440 0x7f4940070bf0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:44.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.301+0000 7f4947e0a640 1 --2- 172.21.15.19:0/887601036 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4940153190 0x7f4940159820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:44.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.301+0000 7f4947e0a640 1 -- 172.21.15.19:0/887601036 >> 172.21.15.19:0/887601036 conn(0x7f49400728b0 msgr2=0x7f4940072cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:44.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.301+0000 7f4947e0a640 1 -- 172.21.15.19:0/887601036 shutdown_connections 2024-08-31T12:20:44.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.301+0000 7f4947e0a640 1 -- 172.21.15.19:0/887601036 wait complete. 2024-08-31T12:20:44.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.302+0000 7f4947e0a640 1 Processor -- start 2024-08-31T12:20:44.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.302+0000 7f4947e0a640 1 -- start start 2024-08-31T12:20:44.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.304+0000 7f4947e0a640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f49400c9030 0x7f494015a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:44.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.304+0000 7f4947e0a640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49400c9440 0x7f4940156cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:44.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.304+0000 7f4947e0a640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4940153190 0x7f49400ca590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:44.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.304+0000 7f4947e0a640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f49401535a0 con 0x7f49400c9030 2024-08-31T12:20:44.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.304+0000 7f4947e0a640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f49400c4db0 con 0x7f49400c9440 2024-08-31T12:20:44.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.304+0000 7f4947e0a640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f494015cdc0 con 0x7f4940153190 2024-08-31T12:20:44.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.304+0000 7f4945ba6640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f49400c9030 0x7f494015a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:44.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.304+0000 7f4945ba6640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f49400c9030 0x7f494015a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:53884/0 (socket says 172.21.15.19:53884) 2024-08-31T12:20:44.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.304+0000 7f4945ba6640 1 -- 172.21.15.19:0/4218852101 learned_addr learned my addr 172.21.15.19:0/4218852101 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:20:44.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.304+0000 7f49463a7640 1 --2- 172.21.15.19:0/4218852101 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4940153190 0x7f49400ca590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:44.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.304+0000 7f49453a5640 1 --2- 172.21.15.19:0/4218852101 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49400c9440 0x7f4940156cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:44.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.304+0000 7f4945ba6640 1 -- 172.21.15.19:0/4218852101 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4940153190 msgr2=0x7f49400ca590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:44.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.304+0000 7f4945ba6640 1 --2- 172.21.15.19:0/4218852101 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4940153190 0x7f49400ca590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:44.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.304+0000 7f4945ba6640 1 -- 172.21.15.19:0/4218852101 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49400c9440 msgr2=0x7f4940156cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:44.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.304+0000 7f4945ba6640 1 --2- 172.21.15.19:0/4218852101 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49400c9440 0x7f4940156cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:44.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.304+0000 7f4945ba6640 1 -- 172.21.15.19:0/4218852101 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4940157230 con 0x7f49400c9030 2024-08-31T12:20:44.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.305+0000 7f4945ba6640 1 --2- 172.21.15.19:0/4218852101 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f49400c9030 0x7f494015a850 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f493c00e790 tx=0x7f493c00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:44.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.305+0000 7f49367fc640 1 -- 172.21.15.19:0/4218852101 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f493c00eea0 con 0x7f49400c9030 2024-08-31T12:20:44.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.305+0000 7f49367fc640 1 -- 172.21.15.19:0/4218852101 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f493c00cdb0 con 0x7f49400c9030 2024-08-31T12:20:44.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.305+0000 7f49367fc640 1 -- 172.21.15.19:0/4218852101 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f493c010690 con 0x7f49400c9030 2024-08-31T12:20:44.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.305+0000 7f4947e0a640 1 -- 172.21.15.19:0/4218852101 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49400b0280 con 0x7f49400c9030 2024-08-31T12:20:44.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.305+0000 7f4947e0a640 1 -- 172.21.15.19:0/4218852101 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f494015b810 con 0x7f49400c9030 2024-08-31T12:20:44.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.306+0000 7f4947e0a640 1 -- 172.21.15.19:0/4218852101 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4940159820 con 0x7f49400c9030 2024-08-31T12:20:44.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.307+0000 7f49367fc640 1 -- 172.21.15.19:0/4218852101 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f493c010830 con 0x7f49400c9030 2024-08-31T12:20:44.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.308+0000 7f49367fc640 1 --2- 172.21.15.19:0/4218852101 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4924077a80 0x7f4924079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:44.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.308+0000 7f49367fc640 1 -- 172.21.15.19:0/4218852101 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f493c09a0f0 con 0x7f49400c9030 2024-08-31T12:20:44.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.308+0000 7f49453a5640 1 --2- 172.21.15.19:0/4218852101 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4924077a80 0x7f4924079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:44.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.309+0000 7f49453a5640 1 --2- 172.21.15.19:0/4218852101 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4924077a80 0x7f4924079f90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f49401e7610 tx=0x7f4930009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:44.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.314+0000 7f49367fc640 1 -- 172.21.15.19:0/4218852101 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f493c062de0 con 0x7f49400c9030 2024-08-31T12:20:44.604 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.603+0000 7f4947e0a640 1 -- 172.21.15.19:0/4218852101 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f49400c9a20 con 0x7f49400c9030 2024-08-31T12:20:44.604 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.603+0000 7f49367fc640 1 -- 172.21.15.19:0/4218852101 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f493c0626e0 con 0x7f49400c9030 2024-08-31T12:20:44.608 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.607+0000 7f4947e0a640 1 -- 172.21.15.19:0/4218852101 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4924077a80 msgr2=0x7f4924079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:44.608 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.607+0000 7f4947e0a640 1 --2- 172.21.15.19:0/4218852101 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4924077a80 0x7f4924079f90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f49401e7610 tx=0x7f4930009040 comp rx=0 tx=0).stop 2024-08-31T12:20:44.608 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.607+0000 7f4947e0a640 1 -- 172.21.15.19:0/4218852101 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f49400c9030 msgr2=0x7f494015a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:44.608 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.607+0000 7f4947e0a640 1 --2- 172.21.15.19:0/4218852101 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f49400c9030 0x7f494015a850 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f493c00e790 tx=0x7f493c00ec50 comp rx=0 tx=0).stop 2024-08-31T12:20:44.608 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.607+0000 7f4947e0a640 1 -- 172.21.15.19:0/4218852101 shutdown_connections 2024-08-31T12:20:44.609 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.607+0000 7f4947e0a640 1 --2- 172.21.15.19:0/4218852101 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4924077a80 0x7f4924079f90 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:44.609 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.607+0000 7f4947e0a640 1 --2- 172.21.15.19:0/4218852101 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4940153190 0x7f49400ca590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:44.609 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.607+0000 7f4947e0a640 1 --2- 172.21.15.19:0/4218852101 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49400c9440 0x7f4940156cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:44.609 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.607+0000 7f4947e0a640 1 --2- 172.21.15.19:0/4218852101 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f49400c9030 0x7f494015a850 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:44.609 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.607+0000 7f4947e0a640 1 -- 172.21.15.19:0/4218852101 >> 172.21.15.19:0/4218852101 conn(0x7f49400728b0 msgr2=0x7f4940072cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:44.609 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.607+0000 7f4947e0a640 1 -- 172.21.15.19:0/4218852101 shutdown_connections 2024-08-31T12:20:44.609 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:44.607+0000 7f4947e0a640 1 -- 172.21.15.19:0/4218852101 wait complete. 2024-08-31T12:20:44.621 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2024-08-31T12:20:54.622 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-08-31T12:20:54.622 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:20:54.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:54.745+0000 7fb9e6653640 1 -- 172.21.15.19:0/3348375083 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb9c8004210 msgr2=0x7fb9c8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:54.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:54.745+0000 7fb9e6653640 1 --2- 172.21.15.19:0/3348375083 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb9c8004210 0x7fb9c8004800 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb9d4006f60 tx=0x7fb9c8005830 comp rx=0 tx=0).stop 2024-08-31T12:20:54.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:54.745+0000 7fb9e4bf0640 1 -- 172.21.15.19:0/3348375083 reap_dead start 2024-08-31T12:20:54.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:54.745+0000 7fb9e6653640 1 -- 172.21.15.19:0/3348375083 shutdown_connections 2024-08-31T12:20:54.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:54.745+0000 7fb9e6653640 1 -- 172.21.15.19:0/3348375083 >> 172.21.15.19:0/3348375083 conn(0x7fb9e00cefb0 msgr2=0x7fb9e00cb340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:54.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:54.745+0000 7fb9e6653640 1 -- 172.21.15.19:0/3348375083 shutdown_connections 2024-08-31T12:20:54.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:54.746+0000 7fb9e6653640 1 -- 172.21.15.19:0/3348375083 wait complete. 2024-08-31T12:20:54.748 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:54.748+0000 7fb9e6653640 1 Processor -- start 2024-08-31T12:20:54.748 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:54.748+0000 7fb9e6653640 1 -- start start 2024-08-31T12:20:54.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:54.749+0000 7fb9e6653640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb9c8004210 0x7fb9e01ef4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:54.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:54.749+0000 7fb9e6653640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb9e015a060 0x7fb9e0146220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:54.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:54.749+0000 7fb9e6653640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb9e00ce2b0 0x7fb9e0146760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:54.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:54.749+0000 7fb9e6653640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fb9e00714f0 con 0x7fb9c8004210 2024-08-31T12:20:54.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:54.749+0000 7fb9e6653640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb9e00c9bb0 con 0x7fb9e015a060 2024-08-31T12:20:54.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:54.749+0000 7fb9e6653640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fb9e015d3a0 con 0x7fb9e00ce2b0 2024-08-31T12:20:54.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:54.749+0000 7fb9dffff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb9c8004210 0x7fb9e01ef4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:54.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:54.749+0000 7fb9df7fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb9e015a060 0x7fb9e0146220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:54.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:54.749+0000 7fb9dffff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb9c8004210 0x7fb9e01ef4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:51778/0 (socket says 172.21.15.19:51778) 2024-08-31T12:20:54.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:54.749+0000 7fb9dffff640 1 -- 172.21.15.19:0/184589735 learned_addr learned my addr 172.21.15.19:0/184589735 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:20:54.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:54.749+0000 7fb9e4bf0640 1 --2- 172.21.15.19:0/184589735 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb9e00ce2b0 0x7fb9e0146760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:54.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:54.750+0000 7fb9dffff640 1 -- 172.21.15.19:0/184589735 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb9e00ce2b0 msgr2=0x7fb9e0146760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:54.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:54.750+0000 7fb9dffff640 1 --2- 172.21.15.19:0/184589735 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb9e00ce2b0 0x7fb9e0146760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:54.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:54.750+0000 7fb9dffff640 1 -- 172.21.15.19:0/184589735 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb9e015a060 msgr2=0x7fb9e0146220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:54.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:54.750+0000 7fb9dffff640 1 --2- 172.21.15.19:0/184589735 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb9e015a060 0x7fb9e0146220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:54.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:54.750+0000 7fb9dffff640 1 -- 172.21.15.19:0/184589735 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9e01efa10 con 0x7fb9c8004210 2024-08-31T12:20:54.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:54.750+0000 7fb9dffff640 1 --2- 172.21.15.19:0/184589735 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb9c8004210 0x7fb9e01ef4d0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb9d0011b10 tx=0x7fb9d0009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:54.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:54.750+0000 7fb9dcff9640 1 -- 172.21.15.19:0/184589735 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d001e070 con 0x7fb9c8004210 2024-08-31T12:20:54.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:54.750+0000 7fb9dcff9640 1 -- 172.21.15.19:0/184589735 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb9d0004050 con 0x7fb9c8004210 2024-08-31T12:20:54.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:54.750+0000 7fb9e6653640 1 -- 172.21.15.19:0/184589735 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9e014c4a0 con 0x7fb9c8004210 2024-08-31T12:20:54.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:54.750+0000 7fb9dcff9640 1 -- 172.21.15.19:0/184589735 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d000fea0 con 0x7fb9c8004210 2024-08-31T12:20:54.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:54.750+0000 7fb9e6653640 1 -- 172.21.15.19:0/184589735 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9e014ca60 con 0x7fb9c8004210 2024-08-31T12:20:54.754 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:54.751+0000 7fb9e6653640 1 -- 172.21.15.19:0/184589735 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9e00c8f30 con 0x7fb9c8004210 2024-08-31T12:20:54.754 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:54.753+0000 7fb9dcff9640 1 -- 172.21.15.19:0/184589735 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fb9d0021070 con 0x7fb9c8004210 2024-08-31T12:20:54.754 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:54.753+0000 7fb9dcff9640 1 --2- 172.21.15.19:0/184589735 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb9cc077a50 0x7fb9cc079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:54.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:54.753+0000 7fb9dcff9640 1 -- 172.21.15.19:0/184589735 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fb9d009a030 con 0x7fb9c8004210 2024-08-31T12:20:54.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:54.753+0000 7fb9df7fe640 1 --2- 172.21.15.19:0/184589735 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb9cc077a50 0x7fb9cc079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:54.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:54.754+0000 7fb9df7fe640 1 --2- 172.21.15.19:0/184589735 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb9cc077a50 0x7fb9cc079f60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb9d4033000 tx=0x7fb9e00b2470 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:54.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:54.759+0000 7fb9dcff9640 1 -- 172.21.15.19:0/184589735 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fb9d00044a0 con 0x7fb9c8004210 2024-08-31T12:20:55.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.052+0000 7fb9e6653640 1 -- 172.21.15.19:0/184589735 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb9e00ca160 con 0x7fb9c8004210 2024-08-31T12:20:55.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.053+0000 7fb9dcff9640 1 -- 172.21.15.19:0/184589735 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fb9d0062820 con 0x7fb9c8004210 2024-08-31T12:20:55.057 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:20:55.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.056+0000 7fb9e6653640 1 -- 172.21.15.19:0/184589735 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb9cc077a50 msgr2=0x7fb9cc079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:55.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.056+0000 7fb9e6653640 1 --2- 172.21.15.19:0/184589735 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb9cc077a50 0x7fb9cc079f60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb9d4033000 tx=0x7fb9e00b2470 comp rx=0 tx=0).stop 2024-08-31T12:20:55.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.056+0000 7fb9e6653640 1 -- 172.21.15.19:0/184589735 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb9c8004210 msgr2=0x7fb9e01ef4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:55.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.056+0000 7fb9e6653640 1 --2- 172.21.15.19:0/184589735 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb9c8004210 0x7fb9e01ef4d0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb9d0011b10 tx=0x7fb9d0009e90 comp rx=0 tx=0).stop 2024-08-31T12:20:55.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.056+0000 7fb9e6653640 1 -- 172.21.15.19:0/184589735 shutdown_connections 2024-08-31T12:20:55.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.056+0000 7fb9e6653640 1 --2- 172.21.15.19:0/184589735 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb9cc077a50 0x7fb9cc079f60 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:55.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.056+0000 7fb9e6653640 1 --2- 172.21.15.19:0/184589735 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb9e00ce2b0 0x7fb9e0146760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:55.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.056+0000 7fb9e6653640 1 --2- 172.21.15.19:0/184589735 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb9e015a060 0x7fb9e0146220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:55.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.056+0000 7fb9e6653640 1 --2- 172.21.15.19:0/184589735 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb9c8004210 0x7fb9e01ef4d0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:55.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.056+0000 7fb9e6653640 1 -- 172.21.15.19:0/184589735 >> 172.21.15.19:0/184589735 conn(0x7fb9e00cefb0 msgr2=0x7fb9e015c8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:55.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.056+0000 7fb9e6653640 1 -- 172.21.15.19:0/184589735 shutdown_connections 2024-08-31T12:20:55.059 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.056+0000 7fb9e6653640 1 -- 172.21.15.19:0/184589735 wait complete. 2024-08-31T12:20:55.070 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-08-31T12:20:55.070 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-08-31T12:20:55.070 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-08-31T12:20:55.070 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-31T12:20:55.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.197+0000 7f1d23778640 1 -- 172.21.15.19:0/2328627103 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d1c06e0e0 msgr2=0x7f1d1c0c9850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:55.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.197+0000 7f1d23778640 1 --2- 172.21.15.19:0/2328627103 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d1c06e0e0 0x7f1d1c0c9850 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c009830 tx=0x7f1d0c02a640 comp rx=0 tx=0).stop 2024-08-31T12:20:55.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.197+0000 7f1d23778640 1 -- 172.21.15.19:0/2328627103 shutdown_connections 2024-08-31T12:20:55.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.197+0000 7f1d23778640 1 --2- 172.21.15.19:0/2328627103 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d1c06e0e0 0x7f1d1c0c9850 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:55.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.198+0000 7f1d23778640 1 -- 172.21.15.19:0/2328627103 >> 172.21.15.19:0/2328627103 conn(0x7f1d1c0690d0 msgr2=0x7f1d1c0d0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:55.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.198+0000 7f1d23778640 1 -- 172.21.15.19:0/2328627103 shutdown_connections 2024-08-31T12:20:55.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.198+0000 7f1d23778640 1 -- 172.21.15.19:0/2328627103 wait complete. 2024-08-31T12:20:55.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.200+0000 7f1d23778640 1 Processor -- start 2024-08-31T12:20:55.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.200+0000 7f1d23778640 1 -- start start 2024-08-31T12:20:55.203 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.201+0000 7f1d23778640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d1c06e0e0 0x7f1d1c1f1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:55.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.201+0000 7f1d23778640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d1c0c8a40 0x7f1d1c14f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:55.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.201+0000 7f1d23778640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d1c155300 0x7f1d1c1563a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:55.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.201+0000 7f1d23778640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f1d1c15b250 con 0x7f1d1c0c8a40 2024-08-31T12:20:55.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.201+0000 7f1d23778640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1d1c15af70 con 0x7f1d1c06e0e0 2024-08-31T12:20:55.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.201+0000 7f1d23778640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f1d1c0c8610 con 0x7f1d1c155300 2024-08-31T12:20:55.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.201+0000 7f1d20d13640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d1c0c8a40 0x7f1d1c14f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:55.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.201+0000 7f1d20d13640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d1c0c8a40 0x7f1d1c14f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:51792/0 (socket says 172.21.15.19:51792) 2024-08-31T12:20:55.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.201+0000 7f1d20d13640 1 -- 172.21.15.19:0/4130021613 learned_addr learned my addr 172.21.15.19:0/4130021613 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:20:55.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.202+0000 7f1d21514640 1 --2- 172.21.15.19:0/4130021613 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d1c06e0e0 0x7f1d1c1f1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:55.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.202+0000 7f1d21d15640 1 --2- 172.21.15.19:0/4130021613 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d1c155300 0x7f1d1c1563a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:55.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.202+0000 7f1d20d13640 1 -- 172.21.15.19:0/4130021613 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d1c155300 msgr2=0x7f1d1c1563a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:55.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.202+0000 7f1d20d13640 1 --2- 172.21.15.19:0/4130021613 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d1c155300 0x7f1d1c1563a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:55.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.202+0000 7f1d20d13640 1 -- 172.21.15.19:0/4130021613 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d1c06e0e0 msgr2=0x7f1d1c1f1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:55.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.202+0000 7f1d20d13640 1 --2- 172.21.15.19:0/4130021613 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d1c06e0e0 0x7f1d1c1f1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:55.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.202+0000 7f1d20d13640 1 -- 172.21.15.19:0/4130021613 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d1c1f1e40 con 0x7f1d1c0c8a40 2024-08-31T12:20:55.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.202+0000 7f1d20d13640 1 --2- 172.21.15.19:0/4130021613 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d1c0c8a40 0x7f1d1c14f900 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c000c00 tx=0x7f1d0c033df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:55.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.202+0000 7f1d11ffb640 1 -- 172.21.15.19:0/4130021613 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d0c0332e0 con 0x7f1d1c0c8a40 2024-08-31T12:20:55.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.202+0000 7f1d11ffb640 1 -- 172.21.15.19:0/4130021613 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1d0c033570 con 0x7f1d1c0c8a40 2024-08-31T12:20:55.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.202+0000 7f1d23778640 1 -- 172.21.15.19:0/4130021613 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d1c0c4810 con 0x7f1d1c0c8a40 2024-08-31T12:20:55.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.202+0000 7f1d11ffb640 1 -- 172.21.15.19:0/4130021613 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d0c031a10 con 0x7f1d1c0c8a40 2024-08-31T12:20:55.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.203+0000 7f1d23778640 1 -- 172.21.15.19:0/4130021613 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d1c0c4ab0 con 0x7f1d1c0c8a40 2024-08-31T12:20:55.207 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.204+0000 7f1d23778640 1 --2- 172.21.15.19:0/4130021613 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d1c1557f0 0x7f1d1c0c2a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:55.207 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.204+0000 7f1d23778640 1 -- 172.21.15.19:0/4130021613 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f1d1c0c3890 con 0x7f1d1c1557f0 2024-08-31T12:20:55.207 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.204+0000 7f1d21514640 1 --2- 172.21.15.19:0/4130021613 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d1c1557f0 0x7f1d1c0c2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:55.207 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.204+0000 7f1d21514640 1 -- 172.21.15.19:0/4130021613 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1d1c0c9850 con 0x7f1d1c1557f0 2024-08-31T12:20:55.207 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.204+0000 7f1d21514640 1 --2- 172.21.15.19:0/4130021613 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d1c1557f0 0x7f1d1c0c2a90 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1d1800f770 tx=0x7f1d1800fc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:55.207 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.204+0000 7f1d11ffb640 1 -- 172.21.15.19:0/4130021613 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d18018020 con 0x7f1d1c1557f0 2024-08-31T12:20:55.207 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.205+0000 7f1d11ffb640 1 -- 172.21.15.19:0/4130021613 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1d1800d700 con 0x7f1d1c1557f0 2024-08-31T12:20:55.207 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.205+0000 7f1d11ffb640 1 -- 172.21.15.19:0/4130021613 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d1c1557f0 msgr2=0x7f1d1c0c2a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:55.207 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.205+0000 7f1d11ffb640 1 --2- 172.21.15.19:0/4130021613 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d1c1557f0 0x7f1d1c0c2a90 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1d1800f770 tx=0x7f1d1800fc30 comp rx=0 tx=0).stop 2024-08-31T12:20:55.207 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.205+0000 7f1d11ffb640 1 -- 172.21.15.19:0/4130021613 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d1c1557f0 msgr2=0x7f1d1c0c2a90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:20:55.207 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.205+0000 7f1d11ffb640 1 --2- 172.21.15.19:0/4130021613 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d1c1557f0 0x7f1d1c0c2a90 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:55.208 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.205+0000 7f1d11ffb640 1 -- 172.21.15.19:0/4130021613 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f1d0c031bb0 con 0x7f1d1c0c8a40 2024-08-31T12:20:55.208 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.205+0000 7f1d11ffb640 1 --2- 172.21.15.19:0/4130021613 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1d000774c0 0x7f1d000799d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:55.208 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.205+0000 7f1d11ffb640 1 -- 172.21.15.19:0/4130021613 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f1d0c0ba9a0 con 0x7f1d1c0c8a40 2024-08-31T12:20:55.208 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.206+0000 7f1d21514640 1 --2- 172.21.15.19:0/4130021613 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1d000774c0 0x7f1d000799d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:55.208 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.206+0000 7f1d21514640 1 --2- 172.21.15.19:0/4130021613 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1d000774c0 0x7f1d000799d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1d18010010 tx=0x7f1d18021040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:55.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.214+0000 7f1d23778640 1 --2- 172.21.15.19:0/4130021613 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d1c1568e0 0x7f1d1c0c2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:55.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.214+0000 7f1d23778640 1 -- 172.21.15.19:0/4130021613 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f1d1c0c3510 con 0x7f1d1c1568e0 2024-08-31T12:20:55.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.214+0000 7f1d21d15640 1 --2- 172.21.15.19:0/4130021613 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d1c1568e0 0x7f1d1c0c2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:55.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.215+0000 7f1d21d15640 1 -- 172.21.15.19:0/4130021613 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1d1c1e6c90 con 0x7f1d1c1568e0 2024-08-31T12:20:55.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.215+0000 7f1d21d15640 1 --2- 172.21.15.19:0/4130021613 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d1c1568e0 0x7f1d1c0c2fd0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1d140012e0 tx=0x7f1d1400fd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:55.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.215+0000 7f1d11ffb640 1 -- 172.21.15.19:0/4130021613 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d14003b80 con 0x7f1d1c1568e0 2024-08-31T12:20:55.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.215+0000 7f1d11ffb640 1 -- 172.21.15.19:0/4130021613 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5045 (secure 0 0 0) 0x7f1d14003b80 con 0x7f1d1c1568e0 2024-08-31T12:20:55.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.215+0000 7f1d11ffb640 1 -- 172.21.15.19:0/4130021613 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d1c1568e0 msgr2=0x7f1d1c0c2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:55.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.215+0000 7f1d11ffb640 1 --2- 172.21.15.19:0/4130021613 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d1c1568e0 0x7f1d1c0c2fd0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1d140012e0 tx=0x7f1d1400fd40 comp rx=0 tx=0).stop 2024-08-31T12:20:55.217 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:20:55.217 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:20:55.217 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:20:55.217 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "leader", 2024-08-31T12:20:55.217 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 38, 2024-08-31T12:20:55.217 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 235206, 2024-08-31T12:20:55.217 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:20:55.217 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:20:55.217 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:20:55.218 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:20:55.218 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:55.218 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 13, 2024-08-31T12:20:55.218 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:20:55.218 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:20:55.218 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:20:55.218 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:20:55.218 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:20:55.218 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:20:55.218 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:20:55.218 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:20:55.218 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:20:55.218 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:20:55.218 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:20:55.218 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:20:55.219 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:20:55.219 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:20:55.219 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:55.219 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:20:55.219 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:20:55.219 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:20:55.219 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:20:55.219 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:20:55.219 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:20:55.219 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:20:55.219 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:20:55.219 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:20:55.219 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:20:55.220 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:20:55.220 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:20:55.220 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:20:55.220 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:55.220 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:55.220 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:20:55.220 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:20:55.220 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:20:55.221 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:20:55.221 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:20:55.221 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:20:55.221 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:20:55.221 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:20:55.221 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:20:55.221 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:20:55.221 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:20:55.221 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:20:55.221 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:20:55.221 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:20:55.221 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:20:55.221 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:20:55.222 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:20:55.222 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:20:55.222 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:20:55.222 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:20:55.222 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:20:55.222 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:20:55.222 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:20:55.222 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:20:55.222 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:20:55.222 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:20:55.222 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:20:55.222 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:20:55.222 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:55.223 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:20:55.223 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:55.223 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:20:55.223 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:55.223 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:20:55.223 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:20:55.223 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:20:55.223 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:20:55.223 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:55.223 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:20:55.223 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:20:55.223 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:55.223 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:55.223 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:55.224 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:20:55.224 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:20:55.224 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:55.224 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:55.224 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:55.224 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:55.224 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:20:55.224 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:20:55.224 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:20:55.224 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:20:55.224 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:20:55.224 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:55.225 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:55.225 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:20:55.225 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:20:55.225 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:20:55.225 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:20:55.225 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:55.225 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:20:55.225 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:20:55.225 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:55.225 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:55.225 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:55.225 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:20:55.225 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:20:55.226 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:55.226 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:55.226 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:55.226 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:55.226 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:20:55.226 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:20:55.226 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:20:55.226 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:20:55.226 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:20:55.226 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:55.226 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:55.226 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:20:55.226 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:20:55.227 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:20:55.227 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:20:55.227 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:55.227 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:20:55.227 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:20:55.227 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:55.227 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:55.227 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:55.227 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:20:55.227 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:20:55.227 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:55.227 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:55.227 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:55.228 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:55.228 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:20:55.228 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:20:55.228 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:20:55.228 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:20:55.228 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:20:55.228 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:55.228 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:55.228 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:55.228 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:20:55.228 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:20:55.228 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:55.229 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:20:55.229 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:20:55.229 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:20:55.229 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:55.229 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:55.229 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:20:55.229 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:55.229 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:20:55.229 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:20:55.229 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 6 2024-08-31T12:20:55.229 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:55.229 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:55.229 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:20:55.230 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:55.230 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:20:55.230 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:20:55.230 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 8 2024-08-31T12:20:55.230 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:55.230 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:55.230 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:20:55.230 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:55.230 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:20:55.230 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:20:55.230 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 6 2024-08-31T12:20:55.230 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:55.230 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:55.230 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:20:55.230 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:55.231 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:20:55.231 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:20:55.231 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:20:55.231 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:55.231 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:55.231 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:55.231 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:20:55.231 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:20:55.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.215+0000 7f1d11ffb640 1 -- 172.21.15.19:0/4130021613 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d1c1568e0 msgr2=0x7f1d1c0c2fd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:20:55.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.215+0000 7f1d11ffb640 1 --2- 172.21.15.19:0/4130021613 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d1c1568e0 0x7f1d1c0c2fd0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:55.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.216+0000 7f1d23778640 1 -- 172.21.15.19:0/4130021613 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1d000774c0 msgr2=0x7f1d000799d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:55.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.216+0000 7f1d23778640 1 --2- 172.21.15.19:0/4130021613 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1d000774c0 0x7f1d000799d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1d18010010 tx=0x7f1d18021040 comp rx=0 tx=0).stop 2024-08-31T12:20:55.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.216+0000 7f1d23778640 1 -- 172.21.15.19:0/4130021613 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d1c0c8a40 msgr2=0x7f1d1c14f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:55.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.216+0000 7f1d21d15640 1 -- 172.21.15.19:0/4130021613 reap_dead start 2024-08-31T12:20:55.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.216+0000 7f1d23778640 1 --2- 172.21.15.19:0/4130021613 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d1c0c8a40 0x7f1d1c14f900 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c000c00 tx=0x7f1d0c033df0 comp rx=0 tx=0).stop 2024-08-31T12:20:55.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.216+0000 7f1d23778640 1 -- 172.21.15.19:0/4130021613 shutdown_connections 2024-08-31T12:20:55.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.216+0000 7f1d23778640 1 --2- 172.21.15.19:0/4130021613 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d1c0c8a40 0x7f1d1c14f900 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:55.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.216+0000 7f1d23778640 1 -- 172.21.15.19:0/4130021613 >> 172.21.15.19:0/4130021613 conn(0x7f1d1c0690d0 msgr2=0x7f1d1c15a460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:55.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.216+0000 7f1d23778640 1 -- 172.21.15.19:0/4130021613 shutdown_connections 2024-08-31T12:20:55.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.216+0000 7f1d23778640 1 -- 172.21.15.19:0/4130021613 wait complete. 2024-08-31T12:20:55.234 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:20:55.234 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-08-31T12:20:55.234 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-31T12:20:55.356 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.355+0000 7fe66ea38640 1 -- 172.21.15.19:0/1206678734 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe6680c9440 msgr2=0x7fe668070bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:55.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.355+0000 7fe66ea38640 1 --2- 172.21.15.19:0/1206678734 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe6680c9440 0x7fe668070bf0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe660005fd0 tx=0x7fe66002aae0 comp rx=0 tx=0).stop 2024-08-31T12:20:55.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.355+0000 7fe66ea38640 1 -- 172.21.15.19:0/1206678734 shutdown_connections 2024-08-31T12:20:55.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.355+0000 7fe66ea38640 1 --2- 172.21.15.19:0/1206678734 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe6680c9440 0x7fe668070bf0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:55.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.355+0000 7fe66ea38640 1 --2- 172.21.15.19:0/1206678734 >> v2:172.21.15.43:3300/0 conn(0x7fe668153190 0x7fe668159820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:55.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.355+0000 7fe66ea38640 1 -- 172.21.15.19:0/1206678734 >> 172.21.15.19:0/1206678734 conn(0x7fe6680728b0 msgr2=0x7fe668072cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:55.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.355+0000 7fe66ea38640 1 -- 172.21.15.19:0/1206678734 shutdown_connections 2024-08-31T12:20:55.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.355+0000 7fe66ea38640 1 -- 172.21.15.19:0/1206678734 wait complete. 2024-08-31T12:20:55.358 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.357+0000 7fe66ea38640 1 Processor -- start 2024-08-31T12:20:55.358 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.357+0000 7fe66ea38640 1 -- start start 2024-08-31T12:20:55.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.358+0000 7fe66ea38640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe6680c9030 0x7fe6681f5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:55.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.358+0000 7fe66ea38640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe6680c9440 0x7fe668157c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:55.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.358+0000 7fe66ea38640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe668153190 0x7fe668147f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:55.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.358+0000 7fe66ea38640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fe6681535a0 con 0x7fe668153190 2024-08-31T12:20:55.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.358+0000 7fe66ea38640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe6680c4db0 con 0x7fe6680c9440 2024-08-31T12:20:55.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.358+0000 7fe66ea38640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fe66815cdc0 con 0x7fe6680c9030 2024-08-31T12:20:55.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.359+0000 7fe66cfd5640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe668153190 0x7fe668147f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:55.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.359+0000 7fe66cfd5640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe668153190 0x7fe668147f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:51828/0 (socket says 172.21.15.19:51828) 2024-08-31T12:20:55.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.359+0000 7fe66cfd5640 1 -- 172.21.15.19:0/1777736643 learned_addr learned my addr 172.21.15.19:0/1777736643 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:20:55.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.359+0000 7fe667fff640 1 --2- 172.21.15.19:0/1777736643 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe6680c9030 0x7fe6681f5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:55.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.359+0000 7fe6677fe640 1 --2- 172.21.15.19:0/1777736643 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe6680c9440 0x7fe668157c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:55.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.359+0000 7fe66cfd5640 1 -- 172.21.15.19:0/1777736643 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe6680c9030 msgr2=0x7fe6681f5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:55.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.359+0000 7fe66cfd5640 1 --2- 172.21.15.19:0/1777736643 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe6680c9030 0x7fe6681f5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:55.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.359+0000 7fe66cfd5640 1 -- 172.21.15.19:0/1777736643 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe6680c9440 msgr2=0x7fe668157c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:55.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.359+0000 7fe66cfd5640 1 --2- 172.21.15.19:0/1777736643 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe6680c9440 0x7fe668157c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:55.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.359+0000 7fe66cfd5640 1 -- 172.21.15.19:0/1777736643 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6681f57e0 con 0x7fe668153190 2024-08-31T12:20:55.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.359+0000 7fe66cfd5640 1 --2- 172.21.15.19:0/1777736643 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe668153190 0x7fe668147f00 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe660006010 tx=0x7fe660003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:55.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.359+0000 7fe664ff9640 1 -- 172.21.15.19:0/1777736643 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe660044070 con 0x7fe668153190 2024-08-31T12:20:55.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.359+0000 7fe66ea38640 1 -- 172.21.15.19:0/1777736643 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6681f55d0 con 0x7fe668153190 2024-08-31T12:20:55.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.359+0000 7fe664ff9640 1 -- 172.21.15.19:0/1777736643 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe66002e800 con 0x7fe668153190 2024-08-31T12:20:55.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.359+0000 7fe664ff9640 1 -- 172.21.15.19:0/1777736643 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6600044b0 con 0x7fe668153190 2024-08-31T12:20:55.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.359+0000 7fe66ea38640 1 -- 172.21.15.19:0/1777736643 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6681589b0 con 0x7fe668153190 2024-08-31T12:20:55.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.360+0000 7fe66ea38640 1 --2- 172.21.15.19:0/1777736643 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe66806dbf0 0x7fe668148440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:55.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.360+0000 7fe66ea38640 1 -- 172.21.15.19:0/1777736643 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe66806f8c0 con 0x7fe66806dbf0 2024-08-31T12:20:55.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.361+0000 7fe667fff640 1 --2- 172.21.15.19:0/1777736643 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe66806dbf0 0x7fe668148440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:55.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.361+0000 7fe667fff640 1 -- 172.21.15.19:0/1777736643 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe668159820 con 0x7fe66806dbf0 2024-08-31T12:20:55.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.361+0000 7fe667fff640 1 --2- 172.21.15.19:0/1777736643 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe66806dbf0 0x7fe668148440 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe658003ed0 tx=0x7fe65800b9b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:55.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.361+0000 7fe664ff9640 1 -- 172.21.15.19:0/1777736643 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fe660047070 con 0x7fe668153190 2024-08-31T12:20:55.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.362+0000 7fe664ff9640 1 --2- 172.21.15.19:0/1777736643 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe654077a50 0x7fe654079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:55.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.362+0000 7fe664ff9640 1 -- 172.21.15.19:0/1777736643 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fe6600ba660 con 0x7fe668153190 2024-08-31T12:20:55.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.362+0000 7fe664ff9640 1 -- 172.21.15.19:0/1777736643 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe658005220 con 0x7fe66806dbf0 2024-08-31T12:20:55.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.362+0000 7fe6677fe640 1 --2- 172.21.15.19:0/1777736643 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe654077a50 0x7fe654079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:55.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.362+0000 7fe664ff9640 1 -- 172.21.15.19:0/1777736643 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe6580053c0 con 0x7fe66806dbf0 2024-08-31T12:20:55.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.362+0000 7fe664ff9640 1 -- 172.21.15.19:0/1777736643 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe66806dbf0 msgr2=0x7fe668148440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:55.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.362+0000 7fe664ff9640 1 --2- 172.21.15.19:0/1777736643 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe66806dbf0 0x7fe668148440 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe658003ed0 tx=0x7fe65800b9b0 comp rx=0 tx=0).stop 2024-08-31T12:20:55.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.362+0000 7fe664ff9640 1 -- 172.21.15.19:0/1777736643 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe66806dbf0 msgr2=0x7fe668148440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:20:55.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.362+0000 7fe664ff9640 1 --2- 172.21.15.19:0/1777736643 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe66806dbf0 0x7fe668148440 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe658003ed0 tx=0x7fe65800b9b0 comp rx=0 tx=0).stop 2024-08-31T12:20:55.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.363+0000 7fe6677fe640 1 --2- 172.21.15.19:0/1777736643 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe654077a50 0x7fe654079f60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe66814bc10 tx=0x7fe65c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:55.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.369+0000 7fe66ea38640 1 --2- 172.21.15.19:0/1777736643 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe66814c040 0x7fe6681f4890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:55.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.369+0000 7fe66ea38640 1 -- 172.21.15.19:0/1777736643 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe6680c9a20 con 0x7fe66814c040 2024-08-31T12:20:55.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.369+0000 7fe667fff640 1 --2- 172.21.15.19:0/1777736643 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe66814c040 0x7fe6681f4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:55.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.370+0000 7fe667fff640 1 -- 172.21.15.19:0/1777736643 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe6680b2770 con 0x7fe66814c040 2024-08-31T12:20:55.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.370+0000 7fe667fff640 1 --2- 172.21.15.19:0/1777736643 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe66814c040 0x7fe6681f4890 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe658019000 tx=0x7fe6580046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:55.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.370+0000 7fe664ff9640 1 -- 172.21.15.19:0/1777736643 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6580106b0 con 0x7fe66814c040 2024-08-31T12:20:55.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.370+0000 7fe664ff9640 1 -- 172.21.15.19:0/1777736643 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4541 (secure 0 0 0) 0x7fe6580106b0 con 0x7fe66814c040 2024-08-31T12:20:55.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.370+0000 7fe664ff9640 1 -- 172.21.15.19:0/1777736643 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe66814c040 msgr2=0x7fe6681f4890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:55.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.371+0000 7fe664ff9640 1 --2- 172.21.15.19:0/1777736643 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe66814c040 0x7fe6681f4890 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe658019000 tx=0x7fe6580046c0 comp rx=0 tx=0).stop 2024-08-31T12:20:55.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.371+0000 7fe664ff9640 1 -- 172.21.15.19:0/1777736643 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe66814c040 msgr2=0x7fe6681f4890 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:20:55.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.371+0000 7fe664ff9640 1 --2- 172.21.15.19:0/1777736643 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe66814c040 0x7fe6681f4890 secure :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe658019000 tx=0x7fe6580046c0 comp rx=0 tx=0).stop 2024-08-31T12:20:55.372 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:20:55.372 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:20:55.372 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:20:55.372 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:20:55.373 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 38, 2024-08-31T12:20:55.373 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 71339, 2024-08-31T12:20:55.373 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:20:55.373 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:20:55.373 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:20:55.373 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:20:55.373 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:55.373 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 13, 2024-08-31T12:20:55.373 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:20:55.373 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:20:55.373 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:20:55.373 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:20:55.374 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:20:55.374 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:20:55.374 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:20:55.374 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:20:55.374 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:20:55.374 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:20:55.374 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:20:55.374 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:20:55.374 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:20:55.374 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:20:55.374 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:55.374 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:20:55.374 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:20:55.375 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:20:55.375 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:20:55.375 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:20:55.375 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:20:55.375 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:20:55.375 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:20:55.375 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:20:55.375 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:20:55.375 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:20:55.375 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:20:55.375 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:20:55.375 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:55.375 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:55.376 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:20:55.376 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:20:55.376 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:20:55.376 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:20:55.376 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:20:55.376 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:20:55.376 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:20:55.376 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:20:55.376 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:20:55.376 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:20:55.376 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:20:55.376 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:20:55.376 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:20:55.377 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:20:55.377 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:20:55.377 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:20:55.377 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:20:55.377 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:20:55.377 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:20:55.377 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:20:55.377 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:20:55.377 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:20:55.377 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:20:55.377 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:20:55.377 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:20:55.377 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:20:55.378 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:20:55.378 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:20:55.378 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:55.378 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:20:55.378 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:55.378 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:20:55.378 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:55.378 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:20:55.378 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:20:55.378 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:20:55.378 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:20:55.378 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:55.378 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:20:55.379 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:20:55.379 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:55.379 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:55.379 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:55.379 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:20:55.379 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:20:55.379 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:55.379 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:55.379 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:55.379 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:55.379 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:20:55.379 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:20:55.379 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:20:55.380 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:20:55.380 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:20:55.380 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:55.380 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:55.380 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:20:55.380 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:20:55.380 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:20:55.380 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:20:55.380 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:55.380 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:20:55.380 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:20:55.380 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:55.381 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:55.381 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:55.381 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:20:55.381 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:20:55.381 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:55.381 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:55.381 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:55.381 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:55.381 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:20:55.381 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:20:55.381 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:20:55.381 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:20:55.382 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:20:55.382 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:55.382 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:55.382 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:20:55.382 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:20:55.382 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:20:55.382 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:20:55.382 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:55.382 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:20:55.382 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:20:55.382 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:55.382 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:55.382 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:55.383 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:20:55.383 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:20:55.383 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:55.383 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:55.383 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:55.383 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:55.383 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:20:55.383 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:20:55.383 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:20:55.383 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:20:55.383 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:20:55.383 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:55.384 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:55.384 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:55.384 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:20:55.384 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:20:55.384 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:55.384 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:20:55.384 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:20:55.384 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:20:55.384 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:55.384 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:55.384 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:20:55.384 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:55.384 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:20:55.385 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:20:55.385 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:20:55.385 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:55.385 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:55.385 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:55.385 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:20:55.385 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:20:55.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.371+0000 7fe66ea38640 1 -- 172.21.15.19:0/1777736643 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe654077a50 msgr2=0x7fe654079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:55.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.371+0000 7fe66ea38640 1 --2- 172.21.15.19:0/1777736643 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe654077a50 0x7fe654079f60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe66814bc10 tx=0x7fe65c009040 comp rx=0 tx=0).stop 2024-08-31T12:20:55.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.371+0000 7fe66ea38640 1 -- 172.21.15.19:0/1777736643 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe668153190 msgr2=0x7fe668147f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:55.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.371+0000 7fe66ea38640 1 --2- 172.21.15.19:0/1777736643 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe668153190 0x7fe668147f00 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe660006010 tx=0x7fe660003f10 comp rx=0 tx=0).stop 2024-08-31T12:20:55.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.371+0000 7fe66cfd5640 1 -- 172.21.15.19:0/1777736643 reap_dead start 2024-08-31T12:20:55.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.371+0000 7fe66ea38640 1 -- 172.21.15.19:0/1777736643 shutdown_connections 2024-08-31T12:20:55.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.371+0000 7fe66ea38640 1 --2- 172.21.15.19:0/1777736643 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe668153190 0x7fe668147f00 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:55.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.371+0000 7fe66ea38640 1 -- 172.21.15.19:0/1777736643 >> 172.21.15.19:0/1777736643 conn(0x7fe6680728b0 msgr2=0x7fe668150a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:55.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.371+0000 7fe66ea38640 1 -- 172.21.15.19:0/1777736643 shutdown_connections 2024-08-31T12:20:55.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.371+0000 7fe66ea38640 1 -- 172.21.15.19:0/1777736643 wait complete. 2024-08-31T12:20:55.387 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:20:55.387 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-08-31T12:20:55.388 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-31T12:20:55.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.510+0000 7fa4ff91a640 1 -- 172.21.15.19:0/3624602729 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4f8070310 msgr2=0x7fa4f815c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:55.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.510+0000 7fa4ff91a640 1 --2- 172.21.15.19:0/3624602729 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4f8070310 0x7fa4f815c490 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa4e8009830 tx=0x7fa4e802a5c0 comp rx=0 tx=0).stop 2024-08-31T12:20:55.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.510+0000 7fa4ff91a640 1 -- 172.21.15.19:0/3624602729 shutdown_connections 2024-08-31T12:20:55.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.510+0000 7fa4ff91a640 1 --2- 172.21.15.19:0/3624602729 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4f8070310 0x7fa4f815c490 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:55.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.510+0000 7fa4ff91a640 1 -- 172.21.15.19:0/3624602729 >> 172.21.15.19:0/3624602729 conn(0x7fa4f806cae0 msgr2=0x7fa4f806cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:55.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.511+0000 7fa4ff91a640 1 -- 172.21.15.19:0/3624602729 shutdown_connections 2024-08-31T12:20:55.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.511+0000 7fa4ff91a640 1 -- 172.21.15.19:0/3624602729 wait complete. 2024-08-31T12:20:55.513 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.512+0000 7fa4ff91a640 1 Processor -- start 2024-08-31T12:20:55.513 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.512+0000 7fa4ff91a640 1 -- start start 2024-08-31T12:20:55.516 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.514+0000 7fa4ff91a640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa4f806fdc0 0x7fa4f814a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:55.516 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.514+0000 7fa4ff91a640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa4f8070310 0x7fa4f81efdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:55.516 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.514+0000 7fa4ff91a640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4f8152960 0x7fa4f80c5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:55.516 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.514+0000 7fa4ff91a640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fa4f80690a0 con 0x7fa4f8070310 2024-08-31T12:20:55.517 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.514+0000 7fa4ff91a640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa4f8074550 con 0x7fa4f8152960 2024-08-31T12:20:55.517 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.514+0000 7fa4ff91a640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fa4f8074fa0 con 0x7fa4f806fdc0 2024-08-31T12:20:55.517 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.514+0000 7fa4fceb5640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa4f8070310 0x7fa4f81efdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:55.517 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.514+0000 7fa4fceb5640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa4f8070310 0x7fa4f81efdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:51836/0 (socket says 172.21.15.19:51836) 2024-08-31T12:20:55.517 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.514+0000 7fa4fceb5640 1 -- 172.21.15.19:0/102301278 learned_addr learned my addr 172.21.15.19:0/102301278 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:20:55.517 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.514+0000 7fa4fd6b6640 1 --2- 172.21.15.19:0/102301278 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa4f806fdc0 0x7fa4f814a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:55.517 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.514+0000 7fa4fdeb7640 1 --2- 172.21.15.19:0/102301278 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4f8152960 0x7fa4f80c5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:55.517 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.515+0000 7fa4fceb5640 1 -- 172.21.15.19:0/102301278 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa4f806fdc0 msgr2=0x7fa4f814a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:55.517 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.515+0000 7fa4fceb5640 1 --2- 172.21.15.19:0/102301278 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa4f806fdc0 0x7fa4f814a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:55.517 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.515+0000 7fa4fceb5640 1 -- 172.21.15.19:0/102301278 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4f8152960 msgr2=0x7fa4f80c5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:55.517 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.515+0000 7fa4fceb5640 1 --2- 172.21.15.19:0/102301278 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4f8152960 0x7fa4f80c5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:55.517 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.515+0000 7fa4fceb5640 1 -- 172.21.15.19:0/102301278 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4f814c0b0 con 0x7fa4f8070310 2024-08-31T12:20:55.517 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.515+0000 7fa4fceb5640 1 --2- 172.21.15.19:0/102301278 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa4f8070310 0x7fa4f81efdd0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa4e8009950 tx=0x7fa4e8033ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:55.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.515+0000 7fa4edffb640 1 -- 172.21.15.19:0/102301278 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e8031b50 con 0x7fa4f8070310 2024-08-31T12:20:55.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.515+0000 7fa4edffb640 1 -- 172.21.15.19:0/102301278 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa4e8031de0 con 0x7fa4f8070310 2024-08-31T12:20:55.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.515+0000 7fa4ff91a640 1 -- 172.21.15.19:0/102301278 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4f81f0310 con 0x7fa4f8070310 2024-08-31T12:20:55.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.515+0000 7fa4edffb640 1 -- 172.21.15.19:0/102301278 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e8005280 con 0x7fa4f8070310 2024-08-31T12:20:55.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.515+0000 7fa4ff91a640 1 -- 172.21.15.19:0/102301278 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4f8149700 con 0x7fa4f8070310 2024-08-31T12:20:55.519 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.516+0000 7fa4ff91a640 1 --2- 172.21.15.19:0/102301278 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa4f80c5970 0x7fa4f8149940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:55.519 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.516+0000 7fa4ff91a640 1 -- 172.21.15.19:0/102301278 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fa4f80c5f90 con 0x7fa4f80c5970 2024-08-31T12:20:55.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.517+0000 7fa4fd6b6640 1 --2- 172.21.15.19:0/102301278 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa4f80c5970 0x7fa4f8149940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:55.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.517+0000 7fa4fd6b6640 1 -- 172.21.15.19:0/102301278 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa4f80c5d80 con 0x7fa4f80c5970 2024-08-31T12:20:55.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.517+0000 7fa4fd6b6640 1 --2- 172.21.15.19:0/102301278 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa4f80c5970 0x7fa4f8149940 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa4f400f520 tx=0x7fa4f400f9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:55.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.517+0000 7fa4edffb640 1 -- 172.21.15.19:0/102301278 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f40052e0 con 0x7fa4f80c5970 2024-08-31T12:20:55.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.518+0000 7fa4edffb640 1 -- 172.21.15.19:0/102301278 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa4f4018070 con 0x7fa4f80c5970 2024-08-31T12:20:55.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.518+0000 7fa4edffb640 1 -- 172.21.15.19:0/102301278 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa4f80c5970 msgr2=0x7fa4f8149940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:55.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.518+0000 7fa4edffb640 1 --2- 172.21.15.19:0/102301278 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa4f80c5970 0x7fa4f8149940 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa4f400f520 tx=0x7fa4f400f9e0 comp rx=0 tx=0).stop 2024-08-31T12:20:55.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.518+0000 7fa4edffb640 1 -- 172.21.15.19:0/102301278 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa4f80c5970 msgr2=0x7fa4f8149940 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:20:55.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.518+0000 7fa4edffb640 1 --2- 172.21.15.19:0/102301278 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa4f80c5970 0x7fa4f8149940 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa4f400f520 tx=0x7fa4f400f9e0 comp rx=0 tx=0).stop 2024-08-31T12:20:55.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.518+0000 7fa4edffb640 1 -- 172.21.15.19:0/102301278 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fa4e8005420 con 0x7fa4f8070310 2024-08-31T12:20:55.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.518+0000 7fa4edffb640 1 --2- 172.21.15.19:0/102301278 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa4dc077530 0x7fa4dc079a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:55.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.518+0000 7fa4edffb640 1 -- 172.21.15.19:0/102301278 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fa4e80ba840 con 0x7fa4f8070310 2024-08-31T12:20:55.521 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.518+0000 7fa4fd6b6640 1 --2- 172.21.15.19:0/102301278 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa4dc077530 0x7fa4dc079a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:55.521 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.519+0000 7fa4fd6b6640 1 --2- 172.21.15.19:0/102301278 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa4dc077530 0x7fa4dc079a40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa4f400fa20 tx=0x7fa4f4005000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:55.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.526+0000 7fa4ff91a640 1 --2- 172.21.15.19:0/102301278 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa4f8149e80 0x7fa4f80c69d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:55.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.526+0000 7fa4ff91a640 1 -- 172.21.15.19:0/102301278 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fa4f80c63f0 con 0x7fa4f8149e80 2024-08-31T12:20:55.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.526+0000 7fa4fdeb7640 1 --2- 172.21.15.19:0/102301278 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa4f8149e80 0x7fa4f80c69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:55.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.526+0000 7fa4fdeb7640 1 -- 172.21.15.19:0/102301278 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa4f806f260 con 0x7fa4f8149e80 2024-08-31T12:20:55.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.527+0000 7fa4fdeb7640 1 --2- 172.21.15.19:0/102301278 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa4f8149e80 0x7fa4f80c69d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa4f00020f0 tx=0x7fa4f000d520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:55.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.527+0000 7fa4edffb640 1 -- 172.21.15.19:0/102301278 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f0018030 con 0x7fa4f8149e80 2024-08-31T12:20:55.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.527+0000 7fa4edffb640 1 -- 172.21.15.19:0/102301278 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4541 (secure 0 0 0) 0x7fa4f000dcc0 con 0x7fa4f8149e80 2024-08-31T12:20:55.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.527+0000 7fa4edffb640 1 -- 172.21.15.19:0/102301278 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa4f8149e80 msgr2=0x7fa4f80c69d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:55.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.527+0000 7fa4edffb640 1 --2- 172.21.15.19:0/102301278 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa4f8149e80 0x7fa4f80c69d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa4f00020f0 tx=0x7fa4f000d520 comp rx=0 tx=0).stop 2024-08-31T12:20:55.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.527+0000 7fa4edffb640 1 -- 172.21.15.19:0/102301278 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa4f8149e80 msgr2=0x7fa4f80c69d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:20:55.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.527+0000 7fa4edffb640 1 --2- 172.21.15.19:0/102301278 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa4f8149e80 0x7fa4f80c69d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:55.529 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:20:55.529 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:20:55.529 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:20:55.530 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:20:55.530 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 38, 2024-08-31T12:20:55.530 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 17030, 2024-08-31T12:20:55.530 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:20:55.530 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:20:55.530 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:20:55.530 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:20:55.530 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:55.530 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 14, 2024-08-31T12:20:55.530 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:20:55.530 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:20:55.530 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:20:55.530 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:20:55.530 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:20:55.531 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:20:55.531 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:20:55.531 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:20:55.531 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:20:55.531 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:20:55.531 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:20:55.531 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:20:55.531 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:20:55.531 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:20:55.531 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:55.531 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:20:55.531 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:20:55.531 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:20:55.532 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:20:55.532 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:20:55.532 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:20:55.532 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:20:55.532 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:20:55.532 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:20:55.532 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:20:55.532 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:20:55.532 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:20:55.532 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:20:55.532 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:55.533 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:55.533 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:20:55.533 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:20:55.533 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:20:55.533 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:20:55.533 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:20:55.533 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:20:55.533 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:20:55.533 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:20:55.533 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:20:55.533 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:20:55.533 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:20:55.533 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:20:55.534 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:20:55.534 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:20:55.534 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:20:55.534 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:20:55.534 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:20:55.534 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:20:55.534 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:20:55.534 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:20:55.534 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:20:55.534 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:20:55.534 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:20:55.534 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:20:55.535 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:20:55.535 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:20:55.535 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:20:55.535 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:20:55.535 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:55.535 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:20:55.535 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:55.535 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:20:55.535 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:55.535 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:20:55.535 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:20:55.535 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:20:55.535 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:20:55.535 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:55.536 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:20:55.536 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:20:55.536 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:55.536 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:55.536 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:55.536 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:20:55.536 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:20:55.536 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:55.536 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:55.536 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:55.536 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:55.536 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:20:55.536 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:20:55.537 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:20:55.537 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:20:55.537 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:20:55.537 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:55.537 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:55.537 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:20:55.537 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:20:55.537 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:20:55.537 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:20:55.537 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:55.537 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:20:55.537 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:20:55.537 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:55.538 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:55.538 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:55.538 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:20:55.538 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:20:55.538 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:55.538 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:55.538 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:55.538 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:55.538 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:20:55.538 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:20:55.538 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:20:55.538 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:20:55.538 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:20:55.538 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:55.539 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:55.539 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:20:55.539 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:20:55.539 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:20:55.539 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:20:55.539 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:55.539 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:20:55.539 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:20:55.539 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:55.539 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:55.539 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:55.539 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:20:55.539 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:20:55.539 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:20:55.540 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:55.540 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:55.540 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:55.540 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:20:55.540 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:20:55.540 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:20:55.540 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:20:55.540 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:20:55.540 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:55.540 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:55.540 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:55.540 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:20:55.541 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:20:55.541 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:55.541 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:20:55.541 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:20:55.541 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:20:55.541 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:55.541 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:20:55.541 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:20:55.541 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:20:55.541 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:20:55.541 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:20:55.541 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:20:55.542 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:20:55.542 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:20:55.542 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:20:55.542 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:20:55.542 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:20:55.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.527+0000 7fa4ff91a640 1 -- 172.21.15.19:0/102301278 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa4dc077530 msgr2=0x7fa4dc079a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:55.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.527+0000 7fa4ff91a640 1 --2- 172.21.15.19:0/102301278 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa4dc077530 0x7fa4dc079a40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa4f400fa20 tx=0x7fa4f4005000 comp rx=0 tx=0).stop 2024-08-31T12:20:55.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.527+0000 7fa4ff91a640 1 -- 172.21.15.19:0/102301278 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa4f8070310 msgr2=0x7fa4f81efdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:55.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.527+0000 7fa4ff91a640 1 --2- 172.21.15.19:0/102301278 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa4f8070310 0x7fa4f81efdd0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa4e8009950 tx=0x7fa4e8033ba0 comp rx=0 tx=0).stop 2024-08-31T12:20:55.544 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.527+0000 7fa4fdeb7640 1 -- 172.21.15.19:0/102301278 reap_dead start 2024-08-31T12:20:55.544 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.527+0000 7fa4fdeb7640 1 -- 172.21.15.19:0/102301278 reap_dead start 2024-08-31T12:20:55.544 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.528+0000 7fa4ff91a640 1 -- 172.21.15.19:0/102301278 shutdown_connections 2024-08-31T12:20:55.544 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.528+0000 7fa4ff91a640 1 -- 172.21.15.19:0/102301278 >> 172.21.15.19:0/102301278 conn(0x7fa4f806cae0 msgr2=0x7fa4f81489a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:55.544 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.528+0000 7fa4ff91a640 1 -- 172.21.15.19:0/102301278 shutdown_connections 2024-08-31T12:20:55.544 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.528+0000 7fa4ff91a640 1 -- 172.21.15.19:0/102301278 wait complete. 2024-08-31T12:20:55.544 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:20:55.544 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-08-31T12:20:55.545 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-08-31T12:20:55.545 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-08-31T12:20:55.545 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-08-31T12:20:55.545 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-08-31T12:20:55.545 INFO:teuthology.orchestra.run:waiting for 300 2024-08-31T12:20:55.557 INFO:tasks.ceph.mon.b:Stopped 2024-08-31T12:20:55.557 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-08-31T12:20:55.557 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:20:55.681 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.680+0000 7ff4cf362640 1 -- 172.21.15.19:0/3231961675 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff4c80b28d0 msgr2=0x7ff4c80c1ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:55.681 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.680+0000 7ff4cf362640 1 --2- 172.21.15.19:0/3231961675 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff4c80b28d0 0x7ff4c80c1ec0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff4b8005fd0 tx=0x7ff4b802a530 comp rx=0 tx=0).stop 2024-08-31T12:20:55.682 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.680+0000 7ff4cf362640 1 -- 172.21.15.19:0/3231961675 shutdown_connections 2024-08-31T12:20:55.682 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.680+0000 7ff4cf362640 1 --2- 172.21.15.19:0/3231961675 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff4c80b28d0 0x7ff4c80c1ec0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:55.682 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.680+0000 7ff4cf362640 1 --2- 172.21.15.19:0/3231961675 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff4c80c7ac0 0x7ff4c8152040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:55.682 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.680+0000 7ff4cf362640 1 -- 172.21.15.19:0/3231961675 >> 172.21.15.19:0/3231961675 conn(0x7ff4c806cae0 msgr2=0x7ff4c806cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:55.682 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.680+0000 7ff4cf362640 1 -- 172.21.15.19:0/3231961675 shutdown_connections 2024-08-31T12:20:55.682 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.680+0000 7ff4cf362640 1 -- 172.21.15.19:0/3231961675 wait complete. 2024-08-31T12:20:55.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.682+0000 7ff4cf362640 1 Processor -- start 2024-08-31T12:20:55.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.682+0000 7ff4cf362640 1 -- start start 2024-08-31T12:20:55.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.683+0000 7ff4cf362640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff4c80b28d0 0x7ff4c81f32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:55.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.683+0000 7ff4cf362640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff4c80c2400 0x7ff4c81e1360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:55.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.683+0000 7ff4cf362640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff4c80c7ac0 0x7ff4c8140560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:55.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.683+0000 7ff4cf362640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c806da00 con 0x7ff4c80c2400 2024-08-31T12:20:55.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.683+0000 7ff4cf362640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c8068fd0 con 0x7ff4c80c7ac0 2024-08-31T12:20:55.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.683+0000 7ff4cf362640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7ff4c80c3e90 con 0x7ff4c80b28d0 2024-08-31T12:20:55.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.683+0000 7ff4cd8ff640 1 -- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff4c80c7ac0 msgr2=0x7ff4c8140560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3300/0 2024-08-31T12:20:55.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.683+0000 7ff4cc8fd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff4c80c2400 0x7ff4c81e1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:55.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.683+0000 7ff4cd8ff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff4c80c7ac0 0x7ff4c8140560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:20:55.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.683+0000 7ff4cc8fd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff4c80c2400 0x7ff4c81e1360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:51848/0 (socket says 172.21.15.19:51848) 2024-08-31T12:20:55.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.683+0000 7ff4cc8fd640 1 -- 172.21.15.19:0/3207392039 learned_addr learned my addr 172.21.15.19:0/3207392039 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:20:55.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.684+0000 7ff4cd0fe640 1 --2- 172.21.15.19:0/3207392039 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff4c80b28d0 0x7ff4c81f32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:55.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.684+0000 7ff4cc8fd640 1 -- 172.21.15.19:0/3207392039 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff4c80b28d0 msgr2=0x7ff4c81f32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:55.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.684+0000 7ff4cc8fd640 1 --2- 172.21.15.19:0/3207392039 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff4c80b28d0 0x7ff4c81f32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:55.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.684+0000 7ff4cc8fd640 1 -- 172.21.15.19:0/3207392039 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff4c80c7ac0 msgr2=0x7ff4c8140560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:20:55.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.684+0000 7ff4cc8fd640 1 --2- 172.21.15.19:0/3207392039 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff4c80c7ac0 0x7ff4c8140560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:55.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.684+0000 7ff4cc8fd640 1 -- 172.21.15.19:0/3207392039 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4c81f37e0 con 0x7ff4c80c2400 2024-08-31T12:20:55.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.684+0000 7ff4cc8fd640 1 --2- 172.21.15.19:0/3207392039 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff4c80c2400 0x7ff4c81e1360 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff4b8000c00 tx=0x7ff4b80038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:55.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.684+0000 7ff4bdffb640 1 -- 172.21.15.19:0/3207392039 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b803e030 con 0x7ff4c80c2400 2024-08-31T12:20:55.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.684+0000 7ff4bdffb640 1 -- 172.21.15.19:0/3207392039 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff4b8003db0 con 0x7ff4c80c2400 2024-08-31T12:20:55.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.684+0000 7ff4cf362640 1 -- 172.21.15.19:0/3207392039 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4c814fe50 con 0x7ff4c80c2400 2024-08-31T12:20:55.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.684+0000 7ff4bdffb640 1 -- 172.21.15.19:0/3207392039 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b8035040 con 0x7ff4c80c2400 2024-08-31T12:20:55.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.684+0000 7ff4cf362640 1 -- 172.21.15.19:0/3207392039 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4c814ddc0 con 0x7ff4c80c2400 2024-08-31T12:20:55.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.685+0000 7ff4cf362640 1 -- 172.21.15.19:0/3207392039 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4c80c1ec0 con 0x7ff4c80c2400 2024-08-31T12:20:55.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.687+0000 7ff4bdffb640 1 -- 172.21.15.19:0/3207392039 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7ff4b8041030 con 0x7ff4c80c2400 2024-08-31T12:20:55.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.687+0000 7ff4bdffb640 1 --2- 172.21.15.19:0/3207392039 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff4ac077a30 0x7ff4ac079f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:55.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.687+0000 7ff4bdffb640 1 -- 172.21.15.19:0/3207392039 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7ff4b80b9d30 con 0x7ff4c80c2400 2024-08-31T12:20:55.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.687+0000 7ff4cd0fe640 1 --2- 172.21.15.19:0/3207392039 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff4ac077a30 0x7ff4ac079f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:55.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.688+0000 7ff4cd0fe640 1 --2- 172.21.15.19:0/3207392039 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff4ac077a30 0x7ff4ac079f40 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4009c00 tx=0x7ff4c4005f30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:55.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.692+0000 7ff4bdffb640 1 -- 172.21.15.19:0/3207392039 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7ff4b8082350 con 0x7ff4c80c2400 2024-08-31T12:20:55.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.983+0000 7ff4cf362640 1 -- 172.21.15.19:0/3207392039 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff4c81f2be0 con 0x7ff4c80c2400 2024-08-31T12:20:55.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.983+0000 7ff4bdffb640 1 -- 172.21.15.19:0/3207392039 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7ff4b8086b10 con 0x7ff4c80c2400 2024-08-31T12:20:55.984 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:20:55.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.985+0000 7ff4cf362640 1 -- 172.21.15.19:0/3207392039 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff4ac077a30 msgr2=0x7ff4ac079f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:55.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.985+0000 7ff4cf362640 1 --2- 172.21.15.19:0/3207392039 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff4ac077a30 0x7ff4ac079f40 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4009c00 tx=0x7ff4c4005f30 comp rx=0 tx=0).stop 2024-08-31T12:20:55.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.985+0000 7ff4cf362640 1 -- 172.21.15.19:0/3207392039 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff4c80c2400 msgr2=0x7ff4c81e1360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:55.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.986+0000 7ff4cf362640 1 --2- 172.21.15.19:0/3207392039 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff4c80c2400 0x7ff4c81e1360 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff4b8000c00 tx=0x7ff4b80038a0 comp rx=0 tx=0).stop 2024-08-31T12:20:55.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.986+0000 7ff4cf362640 1 -- 172.21.15.19:0/3207392039 shutdown_connections 2024-08-31T12:20:55.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.986+0000 7ff4cf362640 1 --2- 172.21.15.19:0/3207392039 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff4ac077a30 0x7ff4ac079f40 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:55.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.986+0000 7ff4cf362640 1 --2- 172.21.15.19:0/3207392039 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff4c80c7ac0 0x7ff4c8140560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:55.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.986+0000 7ff4cf362640 1 --2- 172.21.15.19:0/3207392039 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff4c80c2400 0x7ff4c81e1360 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:55.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.986+0000 7ff4cf362640 1 --2- 172.21.15.19:0/3207392039 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff4c80b28d0 0x7ff4c81f32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:55.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.986+0000 7ff4cf362640 1 -- 172.21.15.19:0/3207392039 >> 172.21.15.19:0/3207392039 conn(0x7ff4c806cae0 msgr2=0x7ff4c8142940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:55.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.986+0000 7ff4cf362640 1 -- 172.21.15.19:0/3207392039 shutdown_connections 2024-08-31T12:20:55.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:55.986+0000 7ff4cf362640 1 -- 172.21.15.19:0/3207392039 wait complete. 2024-08-31T12:20:56.477 DEBUG:teuthology.orchestra.run.smithi019:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:20:56.479 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:20:58.999 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:20:59.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.124+0000 7f553c7c1640 1 -- 172.21.15.19:0/4284625491 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f553415b660 msgr2=0x7f5534165aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:59.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.124+0000 7f553c7c1640 1 --2- 172.21.15.19:0/4284625491 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f553415b660 0x7f5534165aa0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f552c005fd0 tx=0x7f552c02aa90 comp rx=0 tx=0).stop 2024-08-31T12:20:59.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.124+0000 7f553c7c1640 1 -- 172.21.15.19:0/4284625491 shutdown_connections 2024-08-31T12:20:59.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.124+0000 7f553c7c1640 1 --2- 172.21.15.19:0/4284625491 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f553415b660 0x7f5534165aa0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:59.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.124+0000 7f553c7c1640 1 --2- 172.21.15.19:0/4284625491 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f55340cb280 0x7f553415b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:59.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.124+0000 7f553c7c1640 1 -- 172.21.15.19:0/4284625491 >> 172.21.15.19:0/4284625491 conn(0x7f5534074810 msgr2=0x7f5534068aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:59.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.124+0000 7f553c7c1640 1 -- 172.21.15.19:0/4284625491 shutdown_connections 2024-08-31T12:20:59.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.124+0000 7f553c7c1640 1 -- 172.21.15.19:0/4284625491 wait complete. 2024-08-31T12:20:59.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.126+0000 7f553c7c1640 1 Processor -- start 2024-08-31T12:20:59.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.126+0000 7f553c7c1640 1 -- start start 2024-08-31T12:20:59.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.128+0000 7f553c7c1640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f55340cb280 0x7f5534156320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:59.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.128+0000 7f553c7c1640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f55340ccbb0 0x7f5534159550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:59.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.128+0000 7f553c7c1640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f553415b660 0x7f553415d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:59.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.128+0000 7f553c7c1640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f553415ba70 con 0x7f55340ccbb0 2024-08-31T12:20:59.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.128+0000 7f553c7c1640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f553415f050 con 0x7f553415b660 2024-08-31T12:20:59.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.128+0000 7f553c7c1640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f55340cc780 con 0x7f55340cb280 2024-08-31T12:20:59.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.128+0000 7f5539d5c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f55340ccbb0 0x7f5534159550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:59.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.128+0000 7f553ad5e640 1 -- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f553415b660 msgr2=0x7f553415d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3300/0 2024-08-31T12:20:59.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.128+0000 7f553ad5e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f553415b660 0x7f553415d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:20:59.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.128+0000 7f5539d5c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f55340ccbb0 0x7f5534159550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:35734/0 (socket says 172.21.15.19:35734) 2024-08-31T12:20:59.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.128+0000 7f5539d5c640 1 -- 172.21.15.19:0/741507803 learned_addr learned my addr 172.21.15.19:0/741507803 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:20:59.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.128+0000 7f553a55d640 1 --2- 172.21.15.19:0/741507803 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f55340cb280 0x7f5534156320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:59.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.128+0000 7f5539d5c640 1 -- 172.21.15.19:0/741507803 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f55340cb280 msgr2=0x7f5534156320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:59.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.128+0000 7f5539d5c640 1 --2- 172.21.15.19:0/741507803 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f55340cb280 0x7f5534156320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:59.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.128+0000 7f5539d5c640 1 -- 172.21.15.19:0/741507803 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f553415b660 msgr2=0x7f553415d510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:20:59.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.128+0000 7f5539d5c640 1 --2- 172.21.15.19:0/741507803 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f553415b660 0x7f553415d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:59.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.128+0000 7f5539d5c640 1 -- 172.21.15.19:0/741507803 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5534156860 con 0x7f55340ccbb0 2024-08-31T12:20:59.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.128+0000 7f5539d5c640 1 --2- 172.21.15.19:0/741507803 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f55340ccbb0 0x7f5534159550 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5528009fd0 tx=0x7f552800ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:59.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.129+0000 7f5532ffd640 1 -- 172.21.15.19:0/741507803 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f552800ccc0 con 0x7f55340ccbb0 2024-08-31T12:20:59.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.129+0000 7f5532ffd640 1 -- 172.21.15.19:0/741507803 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f552800ee70 con 0x7f55340ccbb0 2024-08-31T12:20:59.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.129+0000 7f553c7c1640 1 -- 172.21.15.19:0/741507803 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5534159a90 con 0x7f55340ccbb0 2024-08-31T12:20:59.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.129+0000 7f5532ffd640 1 -- 172.21.15.19:0/741507803 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f552800ce60 con 0x7f55340ccbb0 2024-08-31T12:20:59.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.129+0000 7f553c7c1640 1 -- 172.21.15.19:0/741507803 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f553415a7d0 con 0x7f55340ccbb0 2024-08-31T12:20:59.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.130+0000 7f553c7c1640 1 -- 172.21.15.19:0/741507803 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5534155660 con 0x7f55340ccbb0 2024-08-31T12:20:59.133 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.131+0000 7f5532ffd640 1 -- 172.21.15.19:0/741507803 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f5528005260 con 0x7f55340ccbb0 2024-08-31T12:20:59.133 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.131+0000 7f5532ffd640 1 --2- 172.21.15.19:0/741507803 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5520077a30 0x7f5520079f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:20:59.133 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.131+0000 7f5532ffd640 1 -- 172.21.15.19:0/741507803 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f5528099db0 con 0x7f55340ccbb0 2024-08-31T12:20:59.133 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.131+0000 7f553a55d640 1 --2- 172.21.15.19:0/741507803 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5520077a30 0x7f5520079f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:20:59.133 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.132+0000 7f553a55d640 1 --2- 172.21.15.19:0/741507803 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5520077a30 0x7f5520079f40 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5524006ab0 tx=0x7f5524006a40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:20:59.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.137+0000 7f5532ffd640 1 -- 172.21.15.19:0/741507803 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f5528062b30 con 0x7f55340ccbb0 2024-08-31T12:20:59.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.429+0000 7f553c7c1640 1 -- 172.21.15.19:0/741507803 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5534158b40 con 0x7f55340ccbb0 2024-08-31T12:20:59.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.429+0000 7f5532ffd640 1 -- 172.21.15.19:0/741507803 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f5528062430 con 0x7f55340ccbb0 2024-08-31T12:20:59.430 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:20:59.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.431+0000 7f553c7c1640 1 -- 172.21.15.19:0/741507803 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5520077a30 msgr2=0x7f5520079f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:59.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.432+0000 7f553c7c1640 1 --2- 172.21.15.19:0/741507803 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5520077a30 0x7f5520079f40 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5524006ab0 tx=0x7f5524006a40 comp rx=0 tx=0).stop 2024-08-31T12:20:59.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.432+0000 7f553c7c1640 1 -- 172.21.15.19:0/741507803 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f55340ccbb0 msgr2=0x7f5534159550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:20:59.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.432+0000 7f553c7c1640 1 --2- 172.21.15.19:0/741507803 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f55340ccbb0 0x7f5534159550 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5528009fd0 tx=0x7f552800ec20 comp rx=0 tx=0).stop 2024-08-31T12:20:59.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.432+0000 7f553c7c1640 1 -- 172.21.15.19:0/741507803 shutdown_connections 2024-08-31T12:20:59.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.432+0000 7f553c7c1640 1 --2- 172.21.15.19:0/741507803 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5520077a30 0x7f5520079f40 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:59.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.432+0000 7f553c7c1640 1 --2- 172.21.15.19:0/741507803 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f553415b660 0x7f553415d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:59.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.432+0000 7f553c7c1640 1 --2- 172.21.15.19:0/741507803 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f55340ccbb0 0x7f5534159550 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:59.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.432+0000 7f553c7c1640 1 --2- 172.21.15.19:0/741507803 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f55340cb280 0x7f5534156320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:20:59.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.432+0000 7f553c7c1640 1 -- 172.21.15.19:0/741507803 >> 172.21.15.19:0/741507803 conn(0x7f5534074810 msgr2=0x7f55340cc070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:20:59.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.432+0000 7f553c7c1640 1 -- 172.21.15.19:0/741507803 shutdown_connections 2024-08-31T12:20:59.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:20:59.432+0000 7f553c7c1640 1 -- 172.21.15.19:0/741507803 wait complete. 2024-08-31T12:21:02.445 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:21:02.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.574+0000 7f3f865c1640 1 -- 172.21.15.19:0/448931056 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3f800c7dd0 msgr2=0x7f3f801404e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:02.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.574+0000 7f3f865c1640 1 --2- 172.21.15.19:0/448931056 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3f800c7dd0 0x7f3f801404e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3f80073af0 tx=0x7f3f7002a560 comp rx=0 tx=0).stop 2024-08-31T12:21:02.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.574+0000 7f3f865c1640 1 -- 172.21.15.19:0/448931056 shutdown_connections 2024-08-31T12:21:02.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.574+0000 7f3f865c1640 1 --2- 172.21.15.19:0/448931056 >> v2:172.21.15.43:3300/0 conn(0x7f3f800c6f90 0x7f3f800c73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:02.576 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.574+0000 7f3f865c1640 1 --2- 172.21.15.19:0/448931056 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3f800c7dd0 0x7f3f801404e0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:02.576 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.574+0000 7f3f865c1640 1 -- 172.21.15.19:0/448931056 >> 172.21.15.19:0/448931056 conn(0x7f3f800cd7d0 msgr2=0x7f3f800cdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:21:02.576 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.574+0000 7f3f865c1640 1 -- 172.21.15.19:0/448931056 shutdown_connections 2024-08-31T12:21:02.576 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.574+0000 7f3f865c1640 1 -- 172.21.15.19:0/448931056 wait complete. 2024-08-31T12:21:02.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.576+0000 7f3f865c1640 1 Processor -- start 2024-08-31T12:21:02.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.576+0000 7f3f865c1640 1 -- start start 2024-08-31T12:21:02.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.578+0000 7f3f865c1640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3f800719b0 0x7f3f80153070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:02.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.578+0000 7f3f865c1640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3f800c6f90 0x7f3f800cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:02.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.578+0000 7f3f865c1640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3f800c7dd0 0x7f3f801539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:02.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.578+0000 7f3f865c1640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f3f800c5360 con 0x7f3f800719b0 2024-08-31T12:21:02.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.578+0000 7f3f865c1640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3f800c6a70 con 0x7f3f800c6f90 2024-08-31T12:21:02.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.578+0000 7f3f865c1640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f3f80074c70 con 0x7f3f800c7dd0 2024-08-31T12:21:02.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.578+0000 7f3f77fff640 1 -- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3f800c6f90 msgr2=0x7f3f800cb2e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3300/0 2024-08-31T12:21:02.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.578+0000 7f3f77fff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3f800c6f90 0x7f3f800cb2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:21:02.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.578+0000 7f3f84b5e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3f800c7dd0 0x7f3f801539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:02.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.578+0000 7f3f84b5e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3f800c7dd0 0x7f3f801539e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.19:53052/0 (socket says 172.21.15.19:53052) 2024-08-31T12:21:02.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.578+0000 7f3f84b5e640 1 -- 172.21.15.19:0/2222444813 learned_addr learned my addr 172.21.15.19:0/2222444813 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:21:02.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.579+0000 7f3f7ffff640 1 --2- 172.21.15.19:0/2222444813 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3f800719b0 0x7f3f80153070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:02.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.579+0000 7f3f84b5e640 1 -- 172.21.15.19:0/2222444813 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3f800c6f90 msgr2=0x7f3f800cb2e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:21:02.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.579+0000 7f3f84b5e640 1 --2- 172.21.15.19:0/2222444813 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3f800c6f90 0x7f3f800cb2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:02.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.579+0000 7f3f84b5e640 1 -- 172.21.15.19:0/2222444813 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3f800719b0 msgr2=0x7f3f80153070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:02.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.579+0000 7f3f84b5e640 1 --2- 172.21.15.19:0/2222444813 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3f800719b0 0x7f3f80153070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:02.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.579+0000 7f3f84b5e640 1 -- 172.21.15.19:0/2222444813 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f800cb820 con 0x7f3f800c7dd0 2024-08-31T12:21:02.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.579+0000 7f3f7ffff640 1 --2- 172.21.15.19:0/2222444813 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3f800719b0 0x7f3f80153070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-31T12:21:02.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.579+0000 7f3f84b5e640 1 --2- 172.21.15.19:0/2222444813 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3f800c7dd0 0x7f3f801539e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3f7800cb10 tx=0x7f3f7800a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:02.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.579+0000 7f3f7d7fa640 1 -- 172.21.15.19:0/2222444813 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f78016020 con 0x7f3f800c7dd0 2024-08-31T12:21:02.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.579+0000 7f3f7d7fa640 1 -- 172.21.15.19:0/2222444813 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3f7800ad60 con 0x7f3f800c7dd0 2024-08-31T12:21:02.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.579+0000 7f3f865c1640 1 -- 172.21.15.19:0/2222444813 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f80136ec0 con 0x7f3f800c7dd0 2024-08-31T12:21:02.582 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.579+0000 7f3f7d7fa640 1 -- 172.21.15.19:0/2222444813 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f78012680 con 0x7f3f800c7dd0 2024-08-31T12:21:02.582 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.579+0000 7f3f865c1640 1 -- 172.21.15.19:0/2222444813 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f800c3c80 con 0x7f3f800c7dd0 2024-08-31T12:21:02.582 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.580+0000 7f3f865c1640 1 -- 172.21.15.19:0/2222444813 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f80156250 con 0x7f3f800c7dd0 2024-08-31T12:21:02.583 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.581+0000 7f3f7d7fa640 1 -- 172.21.15.19:0/2222444813 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f3f78019020 con 0x7f3f800c7dd0 2024-08-31T12:21:02.583 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.581+0000 7f3f7d7fa640 1 --2- 172.21.15.19:0/2222444813 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3f60077a00 0x7f3f60079f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:02.583 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.581+0000 7f3f7d7fa640 1 -- 172.21.15.19:0/2222444813 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f3f78099ac0 con 0x7f3f800c7dd0 2024-08-31T12:21:02.583 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.582+0000 7f3f7ffff640 1 --2- 172.21.15.19:0/2222444813 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3f60077a00 0x7f3f60079f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:02.583 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.582+0000 7f3f7ffff640 1 --2- 172.21.15.19:0/2222444813 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3f60077a00 0x7f3f60079f10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3f70034000 tx=0x7f3f70004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:02.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.586+0000 7f3f7d7fa640 1 -- 172.21.15.19:0/2222444813 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f3f78020030 con 0x7f3f800c7dd0 2024-08-31T12:21:02.883 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.882+0000 7f3f865c1640 1 -- 172.21.15.19:0/2222444813 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3f801404e0 con 0x7f3f800c7dd0 2024-08-31T12:21:02.883 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.882+0000 7f3f7d7fa640 1 -- 172.21.15.19:0/2222444813 <== mon.2 v2:172.21.15.43:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f3f78062680 con 0x7f3f800c7dd0 2024-08-31T12:21:02.883 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:21:02.886 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.884+0000 7f3f865c1640 1 -- 172.21.15.19:0/2222444813 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3f60077a00 msgr2=0x7f3f60079f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:02.886 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.885+0000 7f3f865c1640 1 --2- 172.21.15.19:0/2222444813 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3f60077a00 0x7f3f60079f10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3f70034000 tx=0x7f3f70004730 comp rx=0 tx=0).stop 2024-08-31T12:21:02.886 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.885+0000 7f3f865c1640 1 -- 172.21.15.19:0/2222444813 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3f800c7dd0 msgr2=0x7f3f801539e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:02.886 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.885+0000 7f3f865c1640 1 --2- 172.21.15.19:0/2222444813 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3f800c7dd0 0x7f3f801539e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3f7800cb10 tx=0x7f3f7800a060 comp rx=0 tx=0).stop 2024-08-31T12:21:02.886 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.885+0000 7f3f865c1640 1 -- 172.21.15.19:0/2222444813 shutdown_connections 2024-08-31T12:21:02.886 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.885+0000 7f3f865c1640 1 --2- 172.21.15.19:0/2222444813 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3f60077a00 0x7f3f60079f10 secure :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3f70034000 tx=0x7f3f70004730 comp rx=0 tx=0).stop 2024-08-31T12:21:02.886 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.885+0000 7f3f865c1640 1 --2- 172.21.15.19:0/2222444813 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3f800c7dd0 0x7f3f801539e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:02.887 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.885+0000 7f3f865c1640 1 --2- 172.21.15.19:0/2222444813 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3f800c6f90 0x7f3f800cb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:02.887 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.885+0000 7f3f865c1640 1 --2- 172.21.15.19:0/2222444813 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3f800719b0 0x7f3f80153070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:02.887 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.885+0000 7f3f865c1640 1 -- 172.21.15.19:0/2222444813 >> 172.21.15.19:0/2222444813 conn(0x7f3f800cd7d0 msgr2=0x7f3f800c8a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:21:02.887 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.885+0000 7f3f865c1640 1 -- 172.21.15.19:0/2222444813 shutdown_connections 2024-08-31T12:21:02.887 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:02.885+0000 7f3f865c1640 1 -- 172.21.15.19:0/2222444813 wait complete. 2024-08-31T12:21:05.899 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:21:09.793 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:09.791+0000 7f95cb8fa640 1 -- 172.21.15.19:0/1799509313 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f95b0005350 msgr2=0x7f95b000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:09.793 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:09.791+0000 7f95cb8fa640 1 --2- 172.21.15.19:0/1799509313 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f95b0005350 0x7f95b000b9c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f95c406ed60 tx=0x7f95b0005830 comp rx=0 tx=0).stop 2024-08-31T12:21:09.793 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:09.791+0000 7f95cb8fa640 1 -- 172.21.15.19:0/1799509313 shutdown_connections 2024-08-31T12:21:09.793 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:09.791+0000 7f95cb8fa640 1 --2- 172.21.15.19:0/1799509313 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f95b0005350 0x7f95b000b9c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:09.793 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:09.791+0000 7f95cb8fa640 1 --2- 172.21.15.19:0/1799509313 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f95b0004800 0x7f95b0004c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:09.793 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:09.791+0000 7f95cb8fa640 1 --2- 172.21.15.19:0/1799509313 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f95b00039b0 0x7f95b0003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:09.793 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:09.791+0000 7f95cb8fa640 1 -- 172.21.15.19:0/1799509313 >> 172.21.15.19:0/1799509313 conn(0x7f95c406fe90 msgr2=0x7f95c40cd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:21:09.793 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:09.791+0000 7f95cb8fa640 1 -- 172.21.15.19:0/1799509313 shutdown_connections 2024-08-31T12:21:09.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:09.791+0000 7f95cb8fa640 1 -- 172.21.15.19:0/1799509313 wait complete. 2024-08-31T12:21:09.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:09.793+0000 7f95cb8fa640 1 Processor -- start 2024-08-31T12:21:09.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:09.793+0000 7f95cb8fa640 1 -- start start 2024-08-31T12:21:09.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:09.794+0000 7f95cb8fa640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f95b00039b0 0x7f95c4155980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:09.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:09.794+0000 7f95cb8fa640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f95b0004800 0x7f95c40ccc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:09.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:09.794+0000 7f95cb8fa640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f95b0005350 0x7f95c4156ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:09.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:09.794+0000 7f95cb8fa640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f95c4152290 con 0x7f95b00039b0 2024-08-31T12:21:09.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:09.794+0000 7f95cb8fa640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f95c40c7600 con 0x7f95b0004800 2024-08-31T12:21:09.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:09.794+0000 7f95cb8fa640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f95c4152410 con 0x7f95b0005350 2024-08-31T12:21:09.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:09.795+0000 7f95c9696640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f95b00039b0 0x7f95c4155980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:09.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:09.795+0000 7f95c9696640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f95b00039b0 0x7f95c4155980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:47664/0 (socket says 172.21.15.19:47664) 2024-08-31T12:21:09.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:09.795+0000 7f95c8e95640 1 -- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f95b0004800 msgr2=0x7f95c40ccc50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3300/0 2024-08-31T12:21:09.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:09.795+0000 7f95c9696640 1 -- 172.21.15.19:0/948930073 learned_addr learned my addr 172.21.15.19:0/948930073 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:21:09.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:09.795+0000 7f95c8e95640 1 --2- 172.21.15.19:0/948930073 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f95b0004800 0x7f95c40ccc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:21:09.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:09.795+0000 7f95c9e97640 1 --2- 172.21.15.19:0/948930073 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f95b0005350 0x7f95c4156ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:09.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:09.795+0000 7f95c9696640 1 -- 172.21.15.19:0/948930073 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f95b0005350 msgr2=0x7f95c4156ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:09.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:09.795+0000 7f95c9696640 1 --2- 172.21.15.19:0/948930073 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f95b0005350 0x7f95c4156ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:09.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:09.795+0000 7f95c9696640 1 -- 172.21.15.19:0/948930073 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f95b0004800 msgr2=0x7f95c40ccc50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:21:09.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:09.795+0000 7f95c9696640 1 --2- 172.21.15.19:0/948930073 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f95b0004800 0x7f95c40ccc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:09.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:09.795+0000 7f95c9696640 1 -- 172.21.15.19:0/948930073 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95c41f0c00 con 0x7f95b00039b0 2024-08-31T12:21:09.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:09.795+0000 7f95c9696640 1 --2- 172.21.15.19:0/948930073 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f95b00039b0 0x7f95c4155980 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f95c0012ac0 tx=0x7f95c0012f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:09.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:09.795+0000 7f95b9ffb640 1 -- 172.21.15.19:0/948930073 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c0011d50 con 0x7f95b00039b0 2024-08-31T12:21:09.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:09.795+0000 7f95cb8fa640 1 -- 172.21.15.19:0/948930073 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95c41f0920 con 0x7f95b00039b0 2024-08-31T12:21:09.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:09.795+0000 7f95b9ffb640 1 -- 172.21.15.19:0/948930073 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f95c0004020 con 0x7f95b00039b0 2024-08-31T12:21:09.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:09.795+0000 7f95b9ffb640 1 -- 172.21.15.19:0/948930073 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c0004470 con 0x7f95b00039b0 2024-08-31T12:21:09.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:09.796+0000 7f95cb8fa640 1 -- 172.21.15.19:0/948930073 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95c4146a90 con 0x7f95b00039b0 2024-08-31T12:21:09.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:09.797+0000 7f95cb8fa640 1 -- 172.21.15.19:0/948930073 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95c40c7c60 con 0x7f95b00039b0 2024-08-31T12:21:09.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:09.798+0000 7f95b9ffb640 1 -- 172.21.15.19:0/948930073 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f95c0024030 con 0x7f95b00039b0 2024-08-31T12:21:09.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:09.798+0000 7f95b9ffb640 1 --2- 172.21.15.19:0/948930073 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f95a8078230 0x7f95a807a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:09.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:09.798+0000 7f95b9ffb640 1 -- 172.21.15.19:0/948930073 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f95c009afe0 con 0x7f95b00039b0 2024-08-31T12:21:09.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:09.798+0000 7f95c8e95640 1 --2- 172.21.15.19:0/948930073 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f95a8078230 0x7f95a807a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:09.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:09.799+0000 7f95c8e95640 1 --2- 172.21.15.19:0/948930073 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f95a8078230 0x7f95a807a740 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f95c4144ff0 tx=0x7f95b4001180 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:09.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:09.803+0000 7f95b9ffb640 1 -- 172.21.15.19:0/948930073 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f95c002b020 con 0x7f95b00039b0 2024-08-31T12:21:10.101 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.099+0000 7f95cb8fa640 1 -- 172.21.15.19:0/948930073 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f95c40c82a0 con 0x7f95b00039b0 2024-08-31T12:21:10.101 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.100+0000 7f95b9ffb640 1 -- 172.21.15.19:0/948930073 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7f95c0063ce0 con 0x7f95b00039b0 2024-08-31T12:21:10.101 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":42,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:21:10.104 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.102+0000 7f95cb8fa640 1 -- 172.21.15.19:0/948930073 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f95a8078230 msgr2=0x7f95a807a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:10.104 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.102+0000 7f95cb8fa640 1 --2- 172.21.15.19:0/948930073 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f95a8078230 0x7f95a807a740 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f95c4144ff0 tx=0x7f95b4001180 comp rx=0 tx=0).stop 2024-08-31T12:21:10.104 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.102+0000 7f95cb8fa640 1 -- 172.21.15.19:0/948930073 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f95b00039b0 msgr2=0x7f95c4155980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:10.104 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.102+0000 7f95cb8fa640 1 --2- 172.21.15.19:0/948930073 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f95b00039b0 0x7f95c4155980 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f95c0012ac0 tx=0x7f95c0012f80 comp rx=0 tx=0).stop 2024-08-31T12:21:10.104 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.102+0000 7f95cb8fa640 1 -- 172.21.15.19:0/948930073 shutdown_connections 2024-08-31T12:21:10.104 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.102+0000 7f95cb8fa640 1 --2- 172.21.15.19:0/948930073 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f95a8078230 0x7f95a807a740 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:10.104 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.102+0000 7f95cb8fa640 1 --2- 172.21.15.19:0/948930073 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f95b0005350 0x7f95c4156ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:10.104 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.102+0000 7f95cb8fa640 1 --2- 172.21.15.19:0/948930073 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f95b0004800 0x7f95c40ccc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:10.104 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.102+0000 7f95cb8fa640 1 --2- 172.21.15.19:0/948930073 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f95b00039b0 0x7f95c4155980 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:10.104 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.102+0000 7f95cb8fa640 1 -- 172.21.15.19:0/948930073 >> 172.21.15.19:0/948930073 conn(0x7f95c406fe90 msgr2=0x7f95c40c8bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:21:10.104 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.102+0000 7f95cb8fa640 1 -- 172.21.15.19:0/948930073 shutdown_connections 2024-08-31T12:21:10.104 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.102+0000 7f95cb8fa640 1 -- 172.21.15.19:0/948930073 wait complete. 2024-08-31T12:21:10.115 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-08-31T12:21:10.115 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-08-31T12:21:10.115 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-31T12:21:10.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.239+0000 7fe2a360a640 1 -- 172.21.15.19:0/631828159 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe29c1532f0 msgr2=0x7fe29c15c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:10.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.239+0000 7fe2a360a640 1 --2- 172.21.15.19:0/631828159 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe29c1532f0 0x7fe29c15c4b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fe29c0cbdb0 tx=0x7fe28c02a560 comp rx=0 tx=0).stop 2024-08-31T12:21:10.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.239+0000 7fe2a360a640 1 -- 172.21.15.19:0/631828159 shutdown_connections 2024-08-31T12:21:10.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.239+0000 7fe2a360a640 1 --2- 172.21.15.19:0/631828159 >> v2:172.21.15.43:3300/0 conn(0x7fe29c0c9880 0x7fe29c15c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:10.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.239+0000 7fe2a360a640 1 --2- 172.21.15.19:0/631828159 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe29c1532f0 0x7fe29c15c4b0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:10.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.239+0000 7fe2a360a640 1 -- 172.21.15.19:0/631828159 >> 172.21.15.19:0/631828159 conn(0x7fe29c0cefb0 msgr2=0x7fe29c0cb340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:21:10.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.239+0000 7fe2a360a640 1 -- 172.21.15.19:0/631828159 shutdown_connections 2024-08-31T12:21:10.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.239+0000 7fe2a360a640 1 -- 172.21.15.19:0/631828159 wait complete. 2024-08-31T12:21:10.242 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.241+0000 7fe2a360a640 1 Processor -- start 2024-08-31T12:21:10.242 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.241+0000 7fe2a360a640 1 -- start start 2024-08-31T12:21:10.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.242+0000 7fe2a360a640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe29c0c9880 0x7fe29c154030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:10.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.242+0000 7fe2a360a640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe29c0ca7f0 0x7fe29c1f1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:10.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.242+0000 7fe2a360a640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe29c1532f0 0x7fe29c15bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:10.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.242+0000 7fe2a360a640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fe29c0c3860 con 0x7fe29c0ca7f0 2024-08-31T12:21:10.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.242+0000 7fe2a360a640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe29c15d330 con 0x7fe29c1532f0 2024-08-31T12:21:10.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.242+0000 7fe2a360a640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fe29c1f2630 con 0x7fe29c0c9880 2024-08-31T12:21:10.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.243+0000 7fe2a0ba5640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe29c0ca7f0 0x7fe29c1f1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:10.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.243+0000 7fe2a1ba7640 1 -- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe29c1532f0 msgr2=0x7fe29c15bd40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3300/0 2024-08-31T12:21:10.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.243+0000 7fe2a1ba7640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe29c1532f0 0x7fe29c15bd40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:21:10.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.243+0000 7fe2a0ba5640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe29c0ca7f0 0x7fe29c1f1c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:47672/0 (socket says 172.21.15.19:47672) 2024-08-31T12:21:10.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.243+0000 7fe2a0ba5640 1 -- 172.21.15.19:0/3526340621 learned_addr learned my addr 172.21.15.19:0/3526340621 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:21:10.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.243+0000 7fe2a13a6640 1 --2- 172.21.15.19:0/3526340621 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe29c0c9880 0x7fe29c154030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:10.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.243+0000 7fe2a0ba5640 1 -- 172.21.15.19:0/3526340621 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe29c0c9880 msgr2=0x7fe29c154030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:10.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.243+0000 7fe2a0ba5640 1 --2- 172.21.15.19:0/3526340621 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe29c0c9880 0x7fe29c154030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:10.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.243+0000 7fe2a0ba5640 1 -- 172.21.15.19:0/3526340621 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe29c1532f0 msgr2=0x7fe29c15bd40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:21:10.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.243+0000 7fe2a0ba5640 1 --2- 172.21.15.19:0/3526340621 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe29c1532f0 0x7fe29c15bd40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:10.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.243+0000 7fe2a0ba5640 1 -- 172.21.15.19:0/3526340621 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe29c1f2160 con 0x7fe29c0ca7f0 2024-08-31T12:21:10.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.243+0000 7fe2a0ba5640 1 --2- 172.21.15.19:0/3526340621 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe29c0ca7f0 0x7fe29c1f1c20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe28c009920 tx=0x7fe28c033d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:10.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.243+0000 7fe291ffb640 1 -- 172.21.15.19:0/3526340621 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe28c031d60 con 0x7fe29c0ca7f0 2024-08-31T12:21:10.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.243+0000 7fe291ffb640 1 -- 172.21.15.19:0/3526340621 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe28c0330d0 con 0x7fe29c0ca7f0 2024-08-31T12:21:10.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.243+0000 7fe291ffb640 1 -- 172.21.15.19:0/3526340621 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe28c033500 con 0x7fe29c0ca7f0 2024-08-31T12:21:10.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.243+0000 7fe2a360a640 1 -- 172.21.15.19:0/3526340621 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe29c0b0cb0 con 0x7fe29c0ca7f0 2024-08-31T12:21:10.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.244+0000 7fe2a360a640 1 -- 172.21.15.19:0/3526340621 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe29c06f920 con 0x7fe29c0ca7f0 2024-08-31T12:21:10.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.245+0000 7fe2a360a640 1 --2- 172.21.15.19:0/3526340621 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe29c06f4a0 0x7fe29c149150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:10.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.245+0000 7fe2a360a640 1 -- 172.21.15.19:0/3526340621 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fe29c0c39e0 con 0x7fe29c06f4a0 2024-08-31T12:21:10.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.245+0000 7fe2a13a6640 1 --2- 172.21.15.19:0/3526340621 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe29c06f4a0 0x7fe29c149150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:10.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.245+0000 7fe2a13a6640 1 -- 172.21.15.19:0/3526340621 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe29c15cac0 con 0x7fe29c06f4a0 2024-08-31T12:21:10.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.246+0000 7fe2a13a6640 1 --2- 172.21.15.19:0/3526340621 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe29c06f4a0 0x7fe29c149150 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe29800f540 tx=0x7fe29800fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:10.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.246+0000 7fe291ffb640 1 -- 172.21.15.19:0/3526340621 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe298018020 con 0x7fe29c06f4a0 2024-08-31T12:21:10.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.246+0000 7fe291ffb640 1 -- 172.21.15.19:0/3526340621 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe298014680 con 0x7fe29c06f4a0 2024-08-31T12:21:10.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.246+0000 7fe291ffb640 1 -- 172.21.15.19:0/3526340621 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe29c06f4a0 msgr2=0x7fe29c149150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:10.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.246+0000 7fe291ffb640 1 --2- 172.21.15.19:0/3526340621 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe29c06f4a0 0x7fe29c149150 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe29800f540 tx=0x7fe29800fa00 comp rx=0 tx=0).stop 2024-08-31T12:21:10.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.246+0000 7fe291ffb640 1 -- 172.21.15.19:0/3526340621 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe29c06f4a0 msgr2=0x7fe29c149150 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:21:10.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.246+0000 7fe291ffb640 1 --2- 172.21.15.19:0/3526340621 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe29c06f4a0 0x7fe29c149150 secure :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe29800f540 tx=0x7fe29800fa00 comp rx=0 tx=0).stop 2024-08-31T12:21:10.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.246+0000 7fe291ffb640 1 -- 172.21.15.19:0/3526340621 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fe28c0035f0 con 0x7fe29c0ca7f0 2024-08-31T12:21:10.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.247+0000 7fe291ffb640 1 --2- 172.21.15.19:0/3526340621 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe280077530 0x7fe280079a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:10.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.247+0000 7fe291ffb640 1 -- 172.21.15.19:0/3526340621 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fe28c0bac60 con 0x7fe29c0ca7f0 2024-08-31T12:21:10.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.247+0000 7fe2a13a6640 1 --2- 172.21.15.19:0/3526340621 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe280077530 0x7fe280079a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:10.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.248+0000 7fe2a13a6640 1 --2- 172.21.15.19:0/3526340621 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe280077530 0x7fe280079a40 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe29800fa40 tx=0x7fe298009030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:10.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.253+0000 7fe2a360a640 1 --2- 172.21.15.19:0/3526340621 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe29c0afac0 0x7fe29c149690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:10.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.253+0000 7fe2a360a640 1 -- 172.21.15.19:0/3526340621 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fe29c1568e0 con 0x7fe29c0afac0 2024-08-31T12:21:10.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.253+0000 7fe2a1ba7640 1 --2- 172.21.15.19:0/3526340621 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe29c0afac0 0x7fe29c149690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:10.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.253+0000 7fe2a1ba7640 1 -- 172.21.15.19:0/3526340621 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe29c0ca180 con 0x7fe29c0afac0 2024-08-31T12:21:10.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.253+0000 7fe2a1ba7640 1 --2- 172.21.15.19:0/3526340621 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe29c0afac0 0x7fe29c149690 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fe29c071300 tx=0x7fe294009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:10.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.254+0000 7fe291ffb640 1 -- 172.21.15.19:0/3526340621 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe294014420 con 0x7fe29c0afac0 2024-08-31T12:21:10.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.254+0000 7fe291ffb640 1 -- 172.21.15.19:0/3526340621 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5033 (secure 0 0 0) 0x7fe294010070 con 0x7fe29c0afac0 2024-08-31T12:21:10.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.254+0000 7fe291ffb640 1 -- 172.21.15.19:0/3526340621 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe29c0afac0 msgr2=0x7fe29c149690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:10.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.254+0000 7fe291ffb640 1 --2- 172.21.15.19:0/3526340621 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe29c0afac0 0x7fe29c149690 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fe29c071300 tx=0x7fe294009e90 comp rx=0 tx=0).stop 2024-08-31T12:21:10.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.254+0000 7fe291ffb640 1 -- 172.21.15.19:0/3526340621 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe29c0afac0 msgr2=0x7fe29c149690 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:21:10.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.254+0000 7fe291ffb640 1 --2- 172.21.15.19:0/3526340621 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe29c0afac0 0x7fe29c149690 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:10.256 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:21:10.256 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:21:10.256 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:21:10.256 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "leader", 2024-08-31T12:21:10.256 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 42, 2024-08-31T12:21:10.256 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 250245, 2024-08-31T12:21:10.256 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:21:10.256 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:21:10.256 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:21:10.256 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:21:10.256 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 0, 2024-08-31T12:21:10.256 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:21:10.257 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:21:10.257 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:21:10.257 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:21:10.257 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:21:10.257 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:21:10.257 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:21:10.257 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:21:10.257 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:21:10.257 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:21:10.257 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:21:10.257 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:21:10.257 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:21:10.257 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:21:10.258 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:21:10.258 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:21:10.258 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:21:10.258 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:21:10.258 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:21:10.258 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:21:10.258 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:21:10.258 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:21:10.258 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:21:10.258 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:21:10.258 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:21:10.258 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:21:10.258 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:21:10.259 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:21:10.259 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:10.259 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:10.259 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:21:10.259 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:21:10.259 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:21:10.259 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:21:10.259 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:21:10.259 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:21:10.259 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:21:10.259 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:21:10.259 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:21:10.259 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:21:10.259 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:21:10.260 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:21:10.260 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:21:10.260 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:21:10.260 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:21:10.260 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:21:10.260 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:21:10.260 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:21:10.260 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:21:10.260 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:21:10.260 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:21:10.260 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:21:10.260 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:21:10.260 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:21:10.260 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:21:10.261 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:21:10.261 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:21:10.261 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:21:10.261 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:21:10.261 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:21:10.261 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:10.261 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:21:10.261 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:10.261 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:21:10.261 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:21:10.261 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:21:10.261 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:21:10.262 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:10.262 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:21:10.262 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:21:10.262 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:10.262 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:10.262 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:10.262 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:21:10.262 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:21:10.262 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:10.262 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:10.262 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:10.263 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:10.263 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:21:10.263 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:21:10.263 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:21:10.263 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:21:10.263 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:21:10.263 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:10.263 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:10.263 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:21:10.263 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:21:10.263 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:21:10.263 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:21:10.264 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:10.264 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:21:10.264 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:21:10.264 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:10.264 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:10.264 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:10.264 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:21:10.264 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:21:10.264 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:10.264 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:10.264 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:10.264 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:10.264 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:21:10.264 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:21:10.265 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:21:10.265 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:21:10.265 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:21:10.265 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:10.265 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:10.265 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:21:10.265 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:21:10.265 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:21:10.265 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:21:10.265 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:10.265 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:21:10.265 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:21:10.265 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:10.265 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:10.266 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:10.266 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:21:10.266 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:21:10.266 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:10.266 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:10.266 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:10.266 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:10.266 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:21:10.266 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:21:10.266 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:21:10.266 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:21:10.266 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:21:10.266 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:10.266 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:10.267 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:10.267 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:21:10.267 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:21:10.267 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:10.267 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:21:10.267 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:21:10.267 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:21:10.267 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:10.267 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:21:10.267 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:21:10.267 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:10.267 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:21:10.267 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:21:10.267 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 6 2024-08-31T12:21:10.267 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:10.267 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:21:10.268 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:21:10.268 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:10.268 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:21:10.268 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:21:10.268 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 8 2024-08-31T12:21:10.268 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:10.268 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:21:10.268 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:21:10.268 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:10.268 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:21:10.268 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:21:10.268 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 6 2024-08-31T12:21:10.268 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:10.268 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:21:10.269 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:21:10.269 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:10.269 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:21:10.269 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:21:10.269 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:21:10.269 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:10.269 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:10.269 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:10.269 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:21:10.269 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:21:10.270 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.254+0000 7fe2a360a640 1 -- 172.21.15.19:0/3526340621 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe280077530 msgr2=0x7fe280079a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:10.270 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.254+0000 7fe2a360a640 1 --2- 172.21.15.19:0/3526340621 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe280077530 0x7fe280079a40 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe29800fa40 tx=0x7fe298009030 comp rx=0 tx=0).stop 2024-08-31T12:21:10.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.254+0000 7fe2a360a640 1 -- 172.21.15.19:0/3526340621 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe29c0ca7f0 msgr2=0x7fe29c1f1c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:10.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.254+0000 7fe2a360a640 1 --2- 172.21.15.19:0/3526340621 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe29c0ca7f0 0x7fe29c1f1c20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe28c009920 tx=0x7fe28c033d60 comp rx=0 tx=0).stop 2024-08-31T12:21:10.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.254+0000 7fe2a1ba7640 1 -- 172.21.15.19:0/3526340621 reap_dead start 2024-08-31T12:21:10.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.254+0000 7fe2a1ba7640 1 -- 172.21.15.19:0/3526340621 reap_dead start 2024-08-31T12:21:10.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.254+0000 7fe2a360a640 1 -- 172.21.15.19:0/3526340621 shutdown_connections 2024-08-31T12:21:10.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.254+0000 7fe2a360a640 1 -- 172.21.15.19:0/3526340621 >> 172.21.15.19:0/3526340621 conn(0x7fe29c0cefb0 msgr2=0x7fe29c0cb320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:21:10.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.254+0000 7fe2a360a640 1 -- 172.21.15.19:0/3526340621 shutdown_connections 2024-08-31T12:21:10.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.254+0000 7fe2a360a640 1 -- 172.21.15.19:0/3526340621 wait complete. 2024-08-31T12:21:10.272 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-08-31T12:21:10.272 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-08-31T12:21:10.272 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-31T12:21:10.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.390+0000 7fdeb4427640 1 -- 172.21.15.19:0/792793226 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fde98004800 msgr2=0x7fde98004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:10.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.390+0000 7fdeb4427640 1 --2- 172.21.15.19:0/792793226 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fde98004800 0x7fde98004c10 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fdeac08de60 tx=0x7fde98005830 comp rx=0 tx=0).stop 2024-08-31T12:21:10.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.390+0000 7fdeb29c4640 1 -- 172.21.15.19:0/792793226 reap_dead start 2024-08-31T12:21:10.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.391+0000 7fdeb4427640 1 -- 172.21.15.19:0/792793226 shutdown_connections 2024-08-31T12:21:10.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.391+0000 7fdeb4427640 1 -- 172.21.15.19:0/792793226 >> 172.21.15.19:0/792793226 conn(0x7fdeac0690d0 msgr2=0x7fdeac0d0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:21:10.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.391+0000 7fdeb4427640 1 -- 172.21.15.19:0/792793226 shutdown_connections 2024-08-31T12:21:10.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.391+0000 7fdeb4427640 1 -- 172.21.15.19:0/792793226 wait complete. 2024-08-31T12:21:10.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.392+0000 7fdeb4427640 1 Processor -- start 2024-08-31T12:21:10.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.392+0000 7fdeb4427640 1 -- start start 2024-08-31T12:21:10.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.394+0000 7fdeb4427640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fde98004800 0x7fdeac0c5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:10.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.394+0000 7fdeb4427640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdeac1f47f0 0x7fdeac0c2360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:10.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.394+0000 7fdeb4427640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdeac0c2b80 0x7fdeac1590a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:10.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.394+0000 7fdeb4427640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fdeac15b250 con 0x7fdeac0c2b80 2024-08-31T12:21:10.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.394+0000 7fdeb4427640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fdeac15af70 con 0x7fdeac1f47f0 2024-08-31T12:21:10.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.394+0000 7fdeb4427640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fdeac0c8610 con 0x7fde98004800 2024-08-31T12:21:10.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.394+0000 7fdeb19c2640 1 -- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdeac1f47f0 msgr2=0x7fdeac0c2360 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3300/0 2024-08-31T12:21:10.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.394+0000 7fdeb19c2640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdeac1f47f0 0x7fdeac0c2360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:21:10.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.394+0000 7fdeb29c4640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdeac0c2b80 0x7fdeac1590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:10.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.394+0000 7fdeb29c4640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdeac0c2b80 0x7fdeac1590a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:47716/0 (socket says 172.21.15.19:47716) 2024-08-31T12:21:10.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.394+0000 7fdeb29c4640 1 -- 172.21.15.19:0/355026070 learned_addr learned my addr 172.21.15.19:0/355026070 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:21:10.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.394+0000 7fdeb21c3640 1 --2- 172.21.15.19:0/355026070 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fde98004800 0x7fdeac0c5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:10.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.394+0000 7fdeb29c4640 1 -- 172.21.15.19:0/355026070 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fde98004800 msgr2=0x7fdeac0c5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:10.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.394+0000 7fdeb29c4640 1 --2- 172.21.15.19:0/355026070 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fde98004800 0x7fdeac0c5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:10.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.394+0000 7fdeb29c4640 1 -- 172.21.15.19:0/355026070 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdeac1f47f0 msgr2=0x7fdeac0c2360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:21:10.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.394+0000 7fdeb29c4640 1 --2- 172.21.15.19:0/355026070 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdeac1f47f0 0x7fdeac0c2360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:10.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.394+0000 7fdeb29c4640 1 -- 172.21.15.19:0/355026070 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeac14b080 con 0x7fdeac0c2b80 2024-08-31T12:21:10.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.395+0000 7fdeb29c4640 1 --2- 172.21.15.19:0/355026070 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdeac0c2b80 0x7fdeac1590a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fdeac0cb2b0 tx=0x7fdea4006720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:10.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.395+0000 7fdea2ffd640 1 -- 172.21.15.19:0/355026070 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea400fc90 con 0x7fdeac0c2b80 2024-08-31T12:21:10.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.395+0000 7fdea2ffd640 1 -- 172.21.15.19:0/355026070 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdea400fe30 con 0x7fdeac0c2b80 2024-08-31T12:21:10.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.395+0000 7fdeb4427640 1 -- 172.21.15.19:0/355026070 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdeac0c60b0 con 0x7fdeac0c2b80 2024-08-31T12:21:10.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.395+0000 7fdea2ffd640 1 -- 172.21.15.19:0/355026070 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea4011670 con 0x7fdeac0c2b80 2024-08-31T12:21:10.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.395+0000 7fdeb4427640 1 -- 172.21.15.19:0/355026070 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdeac14bb00 con 0x7fdeac0c2b80 2024-08-31T12:21:10.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.396+0000 7fdeb4427640 1 --2- 172.21.15.19:0/355026070 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdeac0c7030 0x7fdeac1595e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:10.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.396+0000 7fdeb4427640 1 -- 172.21.15.19:0/355026070 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fdeac1f4ef0 con 0x7fdeac0c7030 2024-08-31T12:21:10.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.396+0000 7fdeb21c3640 1 --2- 172.21.15.19:0/355026070 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdeac0c7030 0x7fdeac1595e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:10.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.396+0000 7fdeb21c3640 1 -- 172.21.15.19:0/355026070 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdeac0c9850 con 0x7fdeac0c7030 2024-08-31T12:21:10.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.397+0000 7fdeb21c3640 1 --2- 172.21.15.19:0/355026070 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdeac0c7030 0x7fdeac1595e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdea8001050 tx=0x7fdea800ec00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:10.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.397+0000 7fdea2ffd640 1 -- 172.21.15.19:0/355026070 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fdea4011810 con 0x7fdeac0c2b80 2024-08-31T12:21:10.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.397+0000 7fdea2ffd640 1 --2- 172.21.15.19:0/355026070 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fde90077a00 0x7fde90079f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:10.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.397+0000 7fdea2ffd640 1 -- 172.21.15.19:0/355026070 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fdea409a010 con 0x7fdeac0c2b80 2024-08-31T12:21:10.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.398+0000 7fdeb19c2640 1 --2- 172.21.15.19:0/355026070 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fde90077a00 0x7fde90079f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:10.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.398+0000 7fdea2ffd640 1 -- 172.21.15.19:0/355026070 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea8017070 con 0x7fdeac0c7030 2024-08-31T12:21:10.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.398+0000 7fdea2ffd640 1 -- 172.21.15.19:0/355026070 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fdea800ee50 con 0x7fdeac0c7030 2024-08-31T12:21:10.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.398+0000 7fdea2ffd640 1 -- 172.21.15.19:0/355026070 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdeac0c7030 msgr2=0x7fdeac1595e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:10.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.398+0000 7fdea2ffd640 1 --2- 172.21.15.19:0/355026070 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdeac0c7030 0x7fdeac1595e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdea8001050 tx=0x7fdea800ec00 comp rx=0 tx=0).stop 2024-08-31T12:21:10.401 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.398+0000 7fdea2ffd640 1 -- 172.21.15.19:0/355026070 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdeac0c7030 msgr2=0x7fdeac1595e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:21:10.401 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.398+0000 7fdea2ffd640 1 --2- 172.21.15.19:0/355026070 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdeac0c7030 0x7fdeac1595e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:10.401 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.398+0000 7fdeb19c2640 1 --2- 172.21.15.19:0/355026070 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fde90077a00 0x7fde90079f10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fde9c035000 tx=0x7fde9c0046c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:10.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.406+0000 7fdeb4427640 1 --2- 172.21.15.19:0/355026070 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdeac0c4630 0x7fdeac0cf670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:10.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.406+0000 7fdeb4427640 1 -- 172.21.15.19:0/355026070 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fdeac150ad0 con 0x7fdeac0c4630 2024-08-31T12:21:10.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.406+0000 7fdeb21c3640 1 --2- 172.21.15.19:0/355026070 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdeac0c4630 0x7fdeac0cf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:10.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.406+0000 7fdeb21c3640 1 -- 172.21.15.19:0/355026070 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdeac1507f0 con 0x7fdeac0c4630 2024-08-31T12:21:10.408 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.407+0000 7fdeb21c3640 1 --2- 172.21.15.19:0/355026070 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdeac0c4630 0x7fdeac0cf670 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fdea8019000 tx=0x7fdea8004800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:10.408 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.407+0000 7fdea2ffd640 1 -- 172.21.15.19:0/355026070 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea8017070 con 0x7fdeac0c4630 2024-08-31T12:21:10.408 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.407+0000 7fdea2ffd640 1 -- 172.21.15.19:0/355026070 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4529 (secure 0 0 0) 0x7fdea80064d0 con 0x7fdeac0c4630 2024-08-31T12:21:10.408 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.407+0000 7fdea2ffd640 1 -- 172.21.15.19:0/355026070 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdeac0c4630 msgr2=0x7fdeac0cf670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:10.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.407+0000 7fdea2ffd640 1 --2- 172.21.15.19:0/355026070 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdeac0c4630 0x7fdeac0cf670 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fdea8019000 tx=0x7fdea8004800 comp rx=0 tx=0).stop 2024-08-31T12:21:10.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.407+0000 7fdea2ffd640 1 -- 172.21.15.19:0/355026070 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdeac0c4630 msgr2=0x7fdeac0cf670 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:21:10.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.407+0000 7fdea2ffd640 1 --2- 172.21.15.19:0/355026070 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdeac0c4630 0x7fdeac0cf670 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fdea8019000 tx=0x7fdea8004800 comp rx=0 tx=0).stop 2024-08-31T12:21:10.409 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:21:10.409 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:21:10.409 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:21:10.409 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:21:10.409 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 42, 2024-08-31T12:21:10.409 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 31910, 2024-08-31T12:21:10.409 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:21:10.410 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:21:10.410 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:21:10.410 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:21:10.410 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 0, 2024-08-31T12:21:10.410 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:21:10.410 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:21:10.410 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:21:10.410 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:21:10.410 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:21:10.410 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:21:10.410 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:21:10.410 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:21:10.410 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:21:10.411 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:21:10.411 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:21:10.411 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:21:10.411 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:21:10.411 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:21:10.411 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:21:10.411 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:21:10.411 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:21:10.411 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:21:10.411 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:21:10.411 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:21:10.411 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:21:10.411 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:21:10.411 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:21:10.411 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:21:10.412 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:21:10.412 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:21:10.412 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:21:10.412 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:21:10.412 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:10.412 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:10.412 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:21:10.412 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:21:10.412 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:21:10.412 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:21:10.413 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:21:10.413 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:21:10.413 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:21:10.413 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:21:10.413 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:21:10.413 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:21:10.413 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:21:10.413 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:21:10.413 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:21:10.413 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:21:10.413 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:21:10.413 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:21:10.413 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:21:10.414 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:21:10.414 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:21:10.414 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:21:10.414 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:21:10.414 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:21:10.414 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:21:10.414 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:21:10.414 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:21:10.414 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:21:10.414 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:21:10.414 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:21:10.414 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:21:10.414 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:21:10.415 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:10.415 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:21:10.415 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:10.415 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:21:10.415 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:21:10.415 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:21:10.415 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:21:10.415 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:10.415 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:21:10.415 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:21:10.415 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:10.415 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:10.415 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:10.415 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:21:10.416 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:21:10.416 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:10.416 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:10.416 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:10.416 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:10.416 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:21:10.416 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:21:10.416 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:21:10.416 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:21:10.416 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:21:10.416 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:10.416 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:10.416 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:21:10.416 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:21:10.417 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:21:10.417 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:21:10.417 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:10.417 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:21:10.417 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:21:10.417 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:10.417 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:10.417 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:10.417 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:21:10.417 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:21:10.417 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:10.417 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:10.417 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:10.417 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:10.418 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:21:10.418 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:21:10.418 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:21:10.418 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:21:10.418 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:21:10.418 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:10.418 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:10.418 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:21:10.418 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:21:10.418 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:21:10.418 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:21:10.418 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:10.418 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:21:10.418 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:21:10.419 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:10.419 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:10.419 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:10.419 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:21:10.419 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:21:10.419 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:10.419 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:10.419 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:10.419 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:10.419 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:21:10.419 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:21:10.419 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:21:10.419 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:21:10.420 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:21:10.420 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:10.420 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:10.420 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:10.420 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:21:10.420 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:21:10.420 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:10.420 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:21:10.420 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:21:10.420 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:21:10.420 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:10.420 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:21:10.420 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:21:10.420 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:10.421 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:21:10.421 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:21:10.421 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:21:10.421 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:10.421 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:10.421 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:10.421 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:21:10.421 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:21:10.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.408+0000 7fdeb4427640 1 -- 172.21.15.19:0/355026070 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fde90077a00 msgr2=0x7fde90079f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:10.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.408+0000 7fdeb4427640 1 --2- 172.21.15.19:0/355026070 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fde90077a00 0x7fde90079f10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fde9c035000 tx=0x7fde9c0046c0 comp rx=0 tx=0).stop 2024-08-31T12:21:10.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.408+0000 7fdeb4427640 1 -- 172.21.15.19:0/355026070 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdeac0c2b80 msgr2=0x7fdeac1590a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:10.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.408+0000 7fdeb4427640 1 --2- 172.21.15.19:0/355026070 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdeac0c2b80 0x7fdeac1590a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fdeac0cb2b0 tx=0x7fdea4006720 comp rx=0 tx=0).stop 2024-08-31T12:21:10.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.408+0000 7fdeb29c4640 1 -- 172.21.15.19:0/355026070 reap_dead start 2024-08-31T12:21:10.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.408+0000 7fdeb29c4640 1 -- 172.21.15.19:0/355026070 reap_dead start 2024-08-31T12:21:10.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.408+0000 7fdeb4427640 1 -- 172.21.15.19:0/355026070 shutdown_connections 2024-08-31T12:21:10.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.408+0000 7fdeb4427640 1 -- 172.21.15.19:0/355026070 >> 172.21.15.19:0/355026070 conn(0x7fdeac0690d0 msgr2=0x7fdeac153f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:21:10.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.408+0000 7fdeb4427640 1 -- 172.21.15.19:0/355026070 shutdown_connections 2024-08-31T12:21:10.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:10.408+0000 7fdeb4427640 1 -- 172.21.15.19:0/355026070 wait complete. 2024-08-31T12:21:10.423 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-08-31T12:21:10.423 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-08-31T12:21:26.532 DEBUG:teuthology.orchestra.run.smithi019:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:21:26.535 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:21:30.424 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-08-31T12:21:30.425 INFO:tasks.ceph.mon.b:Restarting daemon 2024-08-31T12:21:30.425 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-08-31T12:21:30.428 INFO:tasks.ceph.mon.b:Started 2024-08-31T12:21:30.428 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-08-31T12:21:30.428 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:21:30.552 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.550+0000 7f6907d89640 1 -- 172.21.15.19:0/83577691 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f690015b830 msgr2=0x7f69000cc480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:30.552 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.550+0000 7f6907d89640 1 --2- 172.21.15.19:0/83577691 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f690015b830 0x7f69000cc480 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f68f8005fd0 tx=0x7f68f802a940 comp rx=0 tx=0).stop 2024-08-31T12:21:30.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.550+0000 7f6907d89640 1 -- 172.21.15.19:0/83577691 shutdown_connections 2024-08-31T12:21:30.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.550+0000 7f6907d89640 1 --2- 172.21.15.19:0/83577691 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f690015b830 0x7f69000cc480 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:30.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.550+0000 7f6907d89640 1 --2- 172.21.15.19:0/83577691 >> v2:172.21.15.43:3300/0 conn(0x7f69000ccbb0 0x7f69000cb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:30.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.550+0000 7f6907d89640 1 -- 172.21.15.19:0/83577691 >> 172.21.15.19:0/83577691 conn(0x7f6900074810 msgr2=0x7f6900068aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:21:30.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.550+0000 7f6907d89640 1 -- 172.21.15.19:0/83577691 shutdown_connections 2024-08-31T12:21:30.554 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.550+0000 7f6907d89640 1 -- 172.21.15.19:0/83577691 wait complete. 2024-08-31T12:21:30.559 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.552+0000 7f6907d89640 1 Processor -- start 2024-08-31T12:21:30.559 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.552+0000 7f6907d89640 1 -- start start 2024-08-31T12:21:30.559 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.553+0000 7f6907d89640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69000cc070 0x7f69000cf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:30.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.553+0000 7f6907d89640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f69000ccbb0 0x7f690014e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:30.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.553+0000 7f6907d89640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f690015b830 0x7f690014eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:30.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.553+0000 7f6907d89640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f69001557e0 con 0x7f69000ccbb0 2024-08-31T12:21:30.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.553+0000 7f6907d89640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6900155660 con 0x7f69000cc070 2024-08-31T12:21:30.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.553+0000 7f6907d89640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f690006e1a0 con 0x7f690015b830 2024-08-31T12:21:30.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.553+0000 7f6905324640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f69000ccbb0 0x7f690014e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:30.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.553+0000 7f6905b25640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69000cc070 0x7f69000cf2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:30.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.553+0000 7f6905324640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f69000ccbb0 0x7f690014e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:40094/0 (socket says 172.21.15.19:40094) 2024-08-31T12:21:30.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.553+0000 7f6905324640 1 -- 172.21.15.19:0/3490007298 learned_addr learned my addr 172.21.15.19:0/3490007298 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:21:30.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.554+0000 7f6906326640 1 --2- 172.21.15.19:0/3490007298 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f690015b830 0x7f690014eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:30.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.554+0000 7f6905b25640 1 -- 172.21.15.19:0/3490007298 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69000cc070 msgr2=0x7f69000cf2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-08-31T12:21:30.562 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.554+0000 7f6905b25640 1 -- 172.21.15.19:0/3490007298 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69000cc070 msgr2=0x7f69000cf2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-31T12:21:30.562 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.554+0000 7f6905b25640 1 --2- 172.21.15.19:0/3490007298 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69000cc070 0x7f69000cf2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-31T12:21:30.562 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.554+0000 7f6905b25640 1 --2- 172.21.15.19:0/3490007298 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69000cc070 0x7f69000cf2b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:21:30.562 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.554+0000 7f6906326640 1 -- 172.21.15.19:0/3490007298 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69000cc070 msgr2=0x7f69000cf2b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:21:30.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.554+0000 7f6906326640 1 --2- 172.21.15.19:0/3490007298 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69000cc070 0x7f69000cf2b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:30.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.554+0000 7f6906326640 1 -- 172.21.15.19:0/3490007298 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f69000ccbb0 msgr2=0x7f690014e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:30.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.554+0000 7f6906326640 1 --2- 172.21.15.19:0/3490007298 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f69000ccbb0 0x7f690014e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:30.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.554+0000 7f6906326640 1 -- 172.21.15.19:0/3490007298 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69000cfe50 con 0x7f690015b830 2024-08-31T12:21:30.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.554+0000 7f6905324640 1 --2- 172.21.15.19:0/3490007298 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f69000ccbb0 0x7f690014e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:21:30.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.554+0000 7f6906326640 1 --2- 172.21.15.19:0/3490007298 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f690015b830 0x7f690014eca0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f68f8009000 tx=0x7f68f8031ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:30.564 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.554+0000 7f68f67fc640 1 -- 172.21.15.19:0/3490007298 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f803e070 con 0x7f690015b830 2024-08-31T12:21:30.564 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.554+0000 7f68f67fc640 1 -- 172.21.15.19:0/3490007298 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f68f8004540 con 0x7f690015b830 2024-08-31T12:21:30.564 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.554+0000 7f6907d89640 1 -- 172.21.15.19:0/3490007298 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69000d0f80 con 0x7f690015b830 2024-08-31T12:21:30.564 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.554+0000 7f68f67fc640 1 -- 172.21.15.19:0/3490007298 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f8030430 con 0x7f690015b830 2024-08-31T12:21:30.564 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.554+0000 7f6907d89640 1 -- 172.21.15.19:0/3490007298 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f69000b4fd0 con 0x7f690015b830 2024-08-31T12:21:30.564 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.555+0000 7f6907d89640 1 -- 172.21.15.19:0/3490007298 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f690015d9e0 con 0x7f690015b830 2024-08-31T12:21:30.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.556+0000 7f68f67fc640 1 -- 172.21.15.19:0/3490007298 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f68f8043460 con 0x7f690015b830 2024-08-31T12:21:30.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.557+0000 7f68f67fc640 1 --2- 172.21.15.19:0/3490007298 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f68e4077a80 0x7f68e4079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:30.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.557+0000 7f68f67fc640 1 -- 172.21.15.19:0/3490007298 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f68f80bb0b0 con 0x7f690015b830 2024-08-31T12:21:30.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.557+0000 7f6905b25640 1 --2- 172.21.15.19:0/3490007298 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f68e4077a80 0x7f68e4079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:30.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.558+0000 7f6905b25640 1 --2- 172.21.15.19:0/3490007298 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f68e4077a80 0x7f68e4079f90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f68fc005fd0 tx=0x7f68fc005f60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:30.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.562+0000 7f68f67fc640 1 -- 172.21.15.19:0/3490007298 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f690015d9e0 con 0x7f690015b830 2024-08-31T12:21:30.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.861+0000 7f6907d89640 1 -- 172.21.15.19:0/3490007298 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f69000cb280 con 0x7f690015b830 2024-08-31T12:21:30.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.862+0000 7f68f67fc640 1 -- 172.21.15.19:0/3490007298 <== mon.2 v2:172.21.15.43:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f68f8083d80 con 0x7f690015b830 2024-08-31T12:21:30.863 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":42,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:21:30.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.865+0000 7f6907d89640 1 -- 172.21.15.19:0/3490007298 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f68e4077a80 msgr2=0x7f68e4079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:30.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.865+0000 7f6907d89640 1 --2- 172.21.15.19:0/3490007298 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f68e4077a80 0x7f68e4079f90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f68fc005fd0 tx=0x7f68fc005f60 comp rx=0 tx=0).stop 2024-08-31T12:21:30.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.865+0000 7f6907d89640 1 -- 172.21.15.19:0/3490007298 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f690015b830 msgr2=0x7f690014eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:30.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.865+0000 7f6907d89640 1 --2- 172.21.15.19:0/3490007298 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f690015b830 0x7f690014eca0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f68f8009000 tx=0x7f68f8031ea0 comp rx=0 tx=0).stop 2024-08-31T12:21:30.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.865+0000 7f6907d89640 1 -- 172.21.15.19:0/3490007298 shutdown_connections 2024-08-31T12:21:30.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.865+0000 7f6907d89640 1 --2- 172.21.15.19:0/3490007298 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f68e4077a80 0x7f68e4079f90 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:30.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.865+0000 7f6907d89640 1 --2- 172.21.15.19:0/3490007298 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f690015b830 0x7f690014eca0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:30.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.865+0000 7f6907d89640 1 --2- 172.21.15.19:0/3490007298 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f69000ccbb0 0x7f690014e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:30.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.865+0000 7f6907d89640 1 --2- 172.21.15.19:0/3490007298 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69000cc070 0x7f69000cf2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:30.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.865+0000 7f6907d89640 1 -- 172.21.15.19:0/3490007298 >> 172.21.15.19:0/3490007298 conn(0x7f6900074810 msgr2=0x7f690015ad70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:21:30.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.865+0000 7f6907d89640 1 -- 172.21.15.19:0/3490007298 shutdown_connections 2024-08-31T12:21:30.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:30.865+0000 7f6907d89640 1 -- 172.21.15.19:0/3490007298 wait complete. 2024-08-31T12:21:33.880 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:21:34.003 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.001+0000 7fe707f90640 1 -- 172.21.15.19:0/280377758 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe6ec005910 msgr2=0x7fe6ec005d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:34.004 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.001+0000 7fe707f90640 1 --2- 172.21.15.19:0/280377758 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe6ec005910 0x7fe6ec005d20 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe700073300 tx=0x7fe6ec006940 comp rx=0 tx=0).stop 2024-08-31T12:21:34.004 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.001+0000 7fe707f90640 1 -- 172.21.15.19:0/280377758 shutdown_connections 2024-08-31T12:21:34.004 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.001+0000 7fe707f90640 1 --2- 172.21.15.19:0/280377758 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe6ec005910 0x7fe6ec005d20 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:34.004 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.001+0000 7fe707f90640 1 -- 172.21.15.19:0/280377758 >> 172.21.15.19:0/280377758 conn(0x7fe70006cae0 msgr2=0x7fe70006cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:21:34.004 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.001+0000 7fe707f90640 1 -- 172.21.15.19:0/280377758 shutdown_connections 2024-08-31T12:21:34.004 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.001+0000 7fe707f90640 1 -- 172.21.15.19:0/280377758 wait complete. 2024-08-31T12:21:34.005 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.003+0000 7fe707f90640 1 Processor -- start 2024-08-31T12:21:34.005 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.003+0000 7fe707f90640 1 -- start start 2024-08-31T12:21:34.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.004+0000 7fe707f90640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe6ec005910 0x7fe7000cbe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:34.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.004+0000 7fe707f90640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe7001e4a70 0x7fe700145850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:34.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.004+0000 7fe707f90640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe700145d90 0x7fe70014c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:34.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.004+0000 7fe707f90640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fe7000c8c90 con 0x7fe7001e4a70 2024-08-31T12:21:34.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.004+0000 7fe707f90640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe700157ad0 con 0x7fe700145d90 2024-08-31T12:21:34.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.004+0000 7fe707f90640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fe700157410 con 0x7fe6ec005910 2024-08-31T12:21:34.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.004+0000 7fe70552b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe7001e4a70 0x7fe700145850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:34.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.005+0000 7fe70552b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe7001e4a70 0x7fe700145850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:40116/0 (socket says 172.21.15.19:40116) 2024-08-31T12:21:34.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.005+0000 7fe70552b640 1 -- 172.21.15.19:0/3465733312 learned_addr learned my addr 172.21.15.19:0/3465733312 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:21:34.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.005+0000 7fe705d2c640 1 --2- 172.21.15.19:0/3465733312 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe6ec005910 0x7fe7000cbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:34.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.005+0000 7fe70652d640 1 --2- 172.21.15.19:0/3465733312 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe700145d90 0x7fe70014c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:34.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.005+0000 7fe70552b640 1 -- 172.21.15.19:0/3465733312 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe6ec005910 msgr2=0x7fe7000cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:34.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.005+0000 7fe70552b640 1 --2- 172.21.15.19:0/3465733312 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe6ec005910 0x7fe7000cbe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:34.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.005+0000 7fe70552b640 1 -- 172.21.15.19:0/3465733312 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe700145d90 msgr2=0x7fe70014c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:34.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.005+0000 7fe70552b640 1 --2- 172.21.15.19:0/3465733312 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe700145d90 0x7fe70014c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:34.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.005+0000 7fe70552b640 1 -- 172.21.15.19:0/3465733312 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7000cc3c0 con 0x7fe7001e4a70 2024-08-31T12:21:34.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.005+0000 7fe70552b640 1 --2- 172.21.15.19:0/3465733312 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe7001e4a70 0x7fe700145850 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe6f002b040 tx=0x7fe6f00064a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:34.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.005+0000 7fe6f67fc640 1 -- 172.21.15.19:0/3465733312 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f0032030 con 0x7fe7001e4a70 2024-08-31T12:21:34.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.005+0000 7fe6f67fc640 1 -- 172.21.15.19:0/3465733312 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe6f00038e0 con 0x7fe7001e4a70 2024-08-31T12:21:34.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.005+0000 7fe6f67fc640 1 -- 172.21.15.19:0/3465733312 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f0003d30 con 0x7fe7001e4a70 2024-08-31T12:21:34.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.005+0000 7fe707f90640 1 -- 172.21.15.19:0/3465733312 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe70014b7d0 con 0x7fe7001e4a70 2024-08-31T12:21:34.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.005+0000 7fe707f90640 1 -- 172.21.15.19:0/3465733312 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7000cb770 con 0x7fe7001e4a70 2024-08-31T12:21:34.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.007+0000 7fe707f90640 1 -- 172.21.15.19:0/3465733312 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7000c1fa0 con 0x7fe7001e4a70 2024-08-31T12:21:34.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.007+0000 7fe6f67fc640 1 -- 172.21.15.19:0/3465733312 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fe6f0035020 con 0x7fe7001e4a70 2024-08-31T12:21:34.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.008+0000 7fe6f67fc640 1 --2- 172.21.15.19:0/3465733312 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe6e4077a50 0x7fe6e4079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:34.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.008+0000 7fe6f67fc640 1 -- 172.21.15.19:0/3465733312 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fe6f00b96a0 con 0x7fe7001e4a70 2024-08-31T12:21:34.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.008+0000 7fe705d2c640 1 --2- 172.21.15.19:0/3465733312 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe6e4077a50 0x7fe6e4079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:34.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.009+0000 7fe705d2c640 1 --2- 172.21.15.19:0/3465733312 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe6e4077a50 0x7fe6e4079f60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc006fd0 tx=0x7fe6fc006bc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:34.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.013+0000 7fe6f67fc640 1 -- 172.21.15.19:0/3465733312 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fe6f0047030 con 0x7fe7001e4a70 2024-08-31T12:21:34.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.316+0000 7fe707f90640 1 -- 172.21.15.19:0/3465733312 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe7000b29f0 con 0x7fe7001e4a70 2024-08-31T12:21:34.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.316+0000 7fe6f67fc640 1 -- 172.21.15.19:0/3465733312 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7fe6f0082470 con 0x7fe7001e4a70 2024-08-31T12:21:34.319 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":42,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:21:34.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.319+0000 7fe707f90640 1 -- 172.21.15.19:0/3465733312 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe6e4077a50 msgr2=0x7fe6e4079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:34.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.319+0000 7fe707f90640 1 --2- 172.21.15.19:0/3465733312 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe6e4077a50 0x7fe6e4079f60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc006fd0 tx=0x7fe6fc006bc0 comp rx=0 tx=0).stop 2024-08-31T12:21:34.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.319+0000 7fe707f90640 1 -- 172.21.15.19:0/3465733312 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe7001e4a70 msgr2=0x7fe700145850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:34.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.319+0000 7fe707f90640 1 --2- 172.21.15.19:0/3465733312 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe7001e4a70 0x7fe700145850 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe6f002b040 tx=0x7fe6f00064a0 comp rx=0 tx=0).stop 2024-08-31T12:21:34.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.319+0000 7fe707f90640 1 -- 172.21.15.19:0/3465733312 shutdown_connections 2024-08-31T12:21:34.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.319+0000 7fe707f90640 1 --2- 172.21.15.19:0/3465733312 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe6e4077a50 0x7fe6e4079f60 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:34.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.319+0000 7fe707f90640 1 --2- 172.21.15.19:0/3465733312 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe700145d90 0x7fe70014c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:34.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.319+0000 7fe707f90640 1 --2- 172.21.15.19:0/3465733312 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe7001e4a70 0x7fe700145850 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:34.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.319+0000 7fe707f90640 1 --2- 172.21.15.19:0/3465733312 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe6ec005910 0x7fe7000cbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:34.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.319+0000 7fe707f90640 1 -- 172.21.15.19:0/3465733312 >> 172.21.15.19:0/3465733312 conn(0x7fe70006cae0 msgr2=0x7fe70006cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:21:34.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.319+0000 7fe707f90640 1 -- 172.21.15.19:0/3465733312 shutdown_connections 2024-08-31T12:21:34.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:34.319+0000 7fe707f90640 1 -- 172.21.15.19:0/3465733312 wait complete. 2024-08-31T12:21:37.332 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:21:37.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.449+0000 7f58529ad640 1 -- 172.21.15.19:0/3464147181 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f584c0cb280 msgr2=0x7f584c15b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:37.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.449+0000 7f58529ad640 1 --2- 172.21.15.19:0/3464147181 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f584c0cb280 0x7f584c15b120 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f5834005fd0 tx=0x7f583402a670 comp rx=0 tx=0).stop 2024-08-31T12:21:37.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.449+0000 7f58529ad640 1 -- 172.21.15.19:0/3464147181 shutdown_connections 2024-08-31T12:21:37.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.449+0000 7f58529ad640 1 --2- 172.21.15.19:0/3464147181 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f584c15b660 0x7f584c165aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:37.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.449+0000 7f58529ad640 1 --2- 172.21.15.19:0/3464147181 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f584c0cb280 0x7f584c15b120 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:37.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.449+0000 7f58529ad640 1 -- 172.21.15.19:0/3464147181 >> 172.21.15.19:0/3464147181 conn(0x7f584c074810 msgr2=0x7f584c068aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:21:37.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.449+0000 7f58529ad640 1 -- 172.21.15.19:0/3464147181 shutdown_connections 2024-08-31T12:21:37.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.449+0000 7f58529ad640 1 -- 172.21.15.19:0/3464147181 wait complete. 2024-08-31T12:21:37.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.451+0000 7f58529ad640 1 Processor -- start 2024-08-31T12:21:37.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.451+0000 7f58529ad640 1 -- start start 2024-08-31T12:21:37.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.452+0000 7f58529ad640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f584c0cb280 0x7f584c1f1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:37.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.452+0000 7f58529ad640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f584c0ccbb0 0x7f584c1573a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:37.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.452+0000 7f58529ad640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f584c15b660 0x7f584c151490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:37.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.452+0000 7f58529ad640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f584c15ba70 con 0x7f584c15b660 2024-08-31T12:21:37.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.452+0000 7f58529ad640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f584c15f050 con 0x7f584c0cb280 2024-08-31T12:21:37.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.452+0000 7f58529ad640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f584c0cc780 con 0x7f584c0ccbb0 2024-08-31T12:21:37.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.453+0000 7f5850f4a640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f584c15b660 0x7f584c151490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:37.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.453+0000 7f584bfff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f584c0cb280 0x7f584c1f1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:37.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.453+0000 7f5850f4a640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f584c15b660 0x7f584c151490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:40124/0 (socket says 172.21.15.19:40124) 2024-08-31T12:21:37.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.453+0000 7f5850f4a640 1 -- 172.21.15.19:0/3882864758 learned_addr learned my addr 172.21.15.19:0/3882864758 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:21:37.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.453+0000 7f5843fff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f584c0ccbb0 0x7f584c1573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:37.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.453+0000 7f5850f4a640 1 -- 172.21.15.19:0/3882864758 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f584c0ccbb0 msgr2=0x7f584c1573a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:37.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.453+0000 7f5850f4a640 1 --2- 172.21.15.19:0/3882864758 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f584c0ccbb0 0x7f584c1573a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:37.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.453+0000 7f5850f4a640 1 -- 172.21.15.19:0/3882864758 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f584c0cb280 msgr2=0x7f584c1f1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:37.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.453+0000 7f5850f4a640 1 --2- 172.21.15.19:0/3882864758 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f584c0cb280 0x7f584c1f1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:37.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.453+0000 7f5850f4a640 1 -- 172.21.15.19:0/3882864758 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f584c1f2030 con 0x7f584c15b660 2024-08-31T12:21:37.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.453+0000 7f5850f4a640 1 --2- 172.21.15.19:0/3882864758 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f584c15b660 0x7f584c151490 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f584400ea40 tx=0x7f584400ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:37.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.453+0000 7f58497fa640 1 -- 172.21.15.19:0/3882864758 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f584400ce40 con 0x7f584c15b660 2024-08-31T12:21:37.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.453+0000 7f58497fa640 1 -- 172.21.15.19:0/3882864758 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5844004c10 con 0x7f584c15b660 2024-08-31T12:21:37.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.453+0000 7f58529ad640 1 -- 172.21.15.19:0/3882864758 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f584c0b4ea0 con 0x7f584c15b660 2024-08-31T12:21:37.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.453+0000 7f58497fa640 1 -- 172.21.15.19:0/3882864758 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5844005020 con 0x7f584c15b660 2024-08-31T12:21:37.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.453+0000 7f58529ad640 1 -- 172.21.15.19:0/3882864758 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f584c155a90 con 0x7f584c15b660 2024-08-31T12:21:37.459 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.455+0000 7f58529ad640 1 -- 172.21.15.19:0/3882864758 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f584c155660 con 0x7f584c15b660 2024-08-31T12:21:37.459 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.455+0000 7f58497fa640 1 -- 172.21.15.19:0/3882864758 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f58440051c0 con 0x7f584c15b660 2024-08-31T12:21:37.459 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.456+0000 7f58497fa640 1 --2- 172.21.15.19:0/3882864758 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f582c077a80 0x7f582c079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:37.459 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.456+0000 7f58497fa640 1 -- 172.21.15.19:0/3882864758 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f584409a460 con 0x7f584c15b660 2024-08-31T12:21:37.459 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.456+0000 7f584bfff640 1 --2- 172.21.15.19:0/3882864758 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f582c077a80 0x7f582c079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:37.459 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.457+0000 7f584bfff640 1 --2- 172.21.15.19:0/3882864758 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f582c077a80 0x7f582c079f90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f583c001720 tx=0x7f583c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:37.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.461+0000 7f58497fa640 1 -- 172.21.15.19:0/3882864758 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f5844062a70 con 0x7f584c15b660 2024-08-31T12:21:37.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.777+0000 7f58529ad640 1 -- 172.21.15.19:0/3882864758 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f584c0cac10 con 0x7f584c15b660 2024-08-31T12:21:37.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.777+0000 7f58497fa640 1 -- 172.21.15.19:0/3882864758 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f5844067230 con 0x7f584c15b660 2024-08-31T12:21:37.780 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":44,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:21:37.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.780+0000 7f58529ad640 1 -- 172.21.15.19:0/3882864758 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f582c077a80 msgr2=0x7f582c079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:37.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.780+0000 7f58529ad640 1 --2- 172.21.15.19:0/3882864758 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f582c077a80 0x7f582c079f90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f583c001720 tx=0x7f583c009040 comp rx=0 tx=0).stop 2024-08-31T12:21:37.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.780+0000 7f58529ad640 1 -- 172.21.15.19:0/3882864758 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f584c15b660 msgr2=0x7f584c151490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:37.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.780+0000 7f58529ad640 1 --2- 172.21.15.19:0/3882864758 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f584c15b660 0x7f584c151490 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f584400ea40 tx=0x7f584400ef00 comp rx=0 tx=0).stop 2024-08-31T12:21:37.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.780+0000 7f58529ad640 1 -- 172.21.15.19:0/3882864758 shutdown_connections 2024-08-31T12:21:37.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.780+0000 7f58529ad640 1 --2- 172.21.15.19:0/3882864758 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f582c077a80 0x7f582c079f90 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:37.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.780+0000 7f58529ad640 1 --2- 172.21.15.19:0/3882864758 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f584c15b660 0x7f584c151490 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:37.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.780+0000 7f58529ad640 1 --2- 172.21.15.19:0/3882864758 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f584c0ccbb0 0x7f584c1573a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:37.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.780+0000 7f58529ad640 1 --2- 172.21.15.19:0/3882864758 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f584c0cb280 0x7f584c1f1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:37.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.780+0000 7f58529ad640 1 -- 172.21.15.19:0/3882864758 >> 172.21.15.19:0/3882864758 conn(0x7f584c074810 msgr2=0x7f584c068aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:21:37.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.780+0000 7f58529ad640 1 -- 172.21.15.19:0/3882864758 shutdown_connections 2024-08-31T12:21:37.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.780+0000 7f58529ad640 1 -- 172.21.15.19:0/3882864758 wait complete. 2024-08-31T12:21:37.793 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-08-31T12:21:37.793 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-08-31T12:21:37.793 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-31T12:21:37.923 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.920+0000 7f910d407640 1 -- 172.21.15.19:0/1643425186 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9108155520 msgr2=0x7f910815b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:37.923 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.920+0000 7f910d407640 1 --2- 172.21.15.19:0/1643425186 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9108155520 0x7f910815b020 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f91080d1580 tx=0x7f90f802a5f0 comp rx=0 tx=0).stop 2024-08-31T12:21:37.923 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.920+0000 7f910d407640 1 -- 172.21.15.19:0/1643425186 shutdown_connections 2024-08-31T12:21:37.923 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.920+0000 7f910d407640 1 --2- 172.21.15.19:0/1643425186 >> v2:172.21.15.43:3300/0 conn(0x7f91080c5c20 0x7f91081659a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:37.923 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.920+0000 7f910d407640 1 --2- 172.21.15.19:0/1643425186 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9108155520 0x7f910815b020 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:37.923 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.920+0000 7f910d407640 1 -- 172.21.15.19:0/1643425186 >> 172.21.15.19:0/1643425186 conn(0x7f9108074a90 msgr2=0x7f9108068aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:21:37.923 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.920+0000 7f910d407640 1 -- 172.21.15.19:0/1643425186 shutdown_connections 2024-08-31T12:21:37.923 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.920+0000 7f910d407640 1 -- 172.21.15.19:0/1643425186 wait complete. 2024-08-31T12:21:37.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.923+0000 7f910d407640 1 Processor -- start 2024-08-31T12:21:37.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.923+0000 7f910d407640 1 -- start start 2024-08-31T12:21:37.928 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.924+0000 7f910d407640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f91080c5c20 0x7f910814e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:37.928 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.924+0000 7f910d407640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9108155520 0x7f9108159400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:37.928 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.924+0000 7f910d407640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f910815b560 0x7f910814cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:37.928 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.924+0000 7f910d407640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f91080d1140 con 0x7f91080c5c20 2024-08-31T12:21:37.928 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.924+0000 7f910d407640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f91080c6750 con 0x7f910815b560 2024-08-31T12:21:37.928 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.924+0000 7f910d407640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f91080cb800 con 0x7f9108155520 2024-08-31T12:21:37.928 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.925+0000 7f9106ffd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f91080c5c20 0x7f910814e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:37.928 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.925+0000 7f9106ffd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f91080c5c20 0x7f910814e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:40136/0 (socket says 172.21.15.19:40136) 2024-08-31T12:21:37.928 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.925+0000 7f9106ffd640 1 -- 172.21.15.19:0/795673864 learned_addr learned my addr 172.21.15.19:0/795673864 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:21:37.928 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.925+0000 7f91077fe640 1 --2- 172.21.15.19:0/795673864 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f910815b560 0x7f910814cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:37.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.925+0000 7f90fffff640 1 --2- 172.21.15.19:0/795673864 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9108155520 0x7f9108159400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:37.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.925+0000 7f9106ffd640 1 -- 172.21.15.19:0/795673864 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9108155520 msgr2=0x7f9108159400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:37.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.925+0000 7f9106ffd640 1 --2- 172.21.15.19:0/795673864 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9108155520 0x7f9108159400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:37.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.925+0000 7f9106ffd640 1 -- 172.21.15.19:0/795673864 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f910815b560 msgr2=0x7f910814cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:37.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.925+0000 7f9106ffd640 1 --2- 172.21.15.19:0/795673864 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f910815b560 0x7f910814cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:37.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.925+0000 7f9106ffd640 1 -- 172.21.15.19:0/795673864 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9108159940 con 0x7f91080c5c20 2024-08-31T12:21:37.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.925+0000 7f9106ffd640 1 --2- 172.21.15.19:0/795673864 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f91080c5c20 0x7f910814e670 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f90f8000c00 tx=0x7f90f8003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:37.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.926+0000 7f90ff7fe640 1 -- 172.21.15.19:0/795673864 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90f803e030 con 0x7f91080c5c20 2024-08-31T12:21:37.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.926+0000 7f90ff7fe640 1 -- 172.21.15.19:0/795673864 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f90f8003e00 con 0x7f91080c5c20 2024-08-31T12:21:37.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.926+0000 7f910d407640 1 -- 172.21.15.19:0/795673864 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91080d12c0 con 0x7f91080c5c20 2024-08-31T12:21:37.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.926+0000 7f90ff7fe640 1 -- 172.21.15.19:0/795673864 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90f8035040 con 0x7f91080c5c20 2024-08-31T12:21:37.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.926+0000 7f910d407640 1 -- 172.21.15.19:0/795673864 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f910814f050 con 0x7f91080c5c20 2024-08-31T12:21:37.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.927+0000 7f910d407640 1 --2- 172.21.15.19:0/795673864 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9108159b10 0x7f91080cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:37.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.927+0000 7f910d407640 1 -- 172.21.15.19:0/795673864 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f91080cb2f0 con 0x7f9108159b10 2024-08-31T12:21:37.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.927+0000 7f90fffff640 1 --2- 172.21.15.19:0/795673864 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9108159b10 0x7f91080cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:37.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.928+0000 7f90fffff640 1 -- 172.21.15.19:0/795673864 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f91080b3bb0 con 0x7f9108159b10 2024-08-31T12:21:37.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.928+0000 7f90fffff640 1 --2- 172.21.15.19:0/795673864 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9108159b10 0x7f91080cfc50 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f9108151c70 tx=0x7f90f000f9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:37.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.928+0000 7f90ff7fe640 1 -- 172.21.15.19:0/795673864 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90f00052e0 con 0x7f9108159b10 2024-08-31T12:21:37.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.928+0000 7f90ff7fe640 1 -- 172.21.15.19:0/795673864 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f90f8041030 con 0x7f91080c5c20 2024-08-31T12:21:37.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.929+0000 7f90ff7fe640 1 --2- 172.21.15.19:0/795673864 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f90e8077950 0x7f90e8079e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:37.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.929+0000 7f90ff7fe640 1 -- 172.21.15.19:0/795673864 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f90f80b9dd0 con 0x7f91080c5c20 2024-08-31T12:21:37.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.929+0000 7f90ff7fe640 1 -- 172.21.15.19:0/795673864 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f90f0005480 con 0x7f9108159b10 2024-08-31T12:21:37.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.929+0000 7f90ff7fe640 1 -- 172.21.15.19:0/795673864 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9108159b10 msgr2=0x7f91080cfc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:37.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.929+0000 7f90ff7fe640 1 --2- 172.21.15.19:0/795673864 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9108159b10 0x7f91080cfc50 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f9108151c70 tx=0x7f90f000f9d0 comp rx=0 tx=0).stop 2024-08-31T12:21:37.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.929+0000 7f90ff7fe640 1 -- 172.21.15.19:0/795673864 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9108159b10 msgr2=0x7f91080cfc50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:21:37.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.929+0000 7f90ff7fe640 1 --2- 172.21.15.19:0/795673864 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9108159b10 0x7f91080cfc50 secure :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f9108151c70 tx=0x7f90f000f9d0 comp rx=0 tx=0).stop 2024-08-31T12:21:37.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.929+0000 7f91077fe640 1 --2- 172.21.15.19:0/795673864 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f90e8077950 0x7f90e8079e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:37.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.930+0000 7f91077fe640 1 --2- 172.21.15.19:0/795673864 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f90e8077950 0x7f90e8079e60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f91080d14e0 tx=0x7f9100009560 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:37.939 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.937+0000 7f910d407640 1 --2- 172.21.15.19:0/795673864 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f91080ca780 0x7f91080d0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:37.939 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.937+0000 7f910d407640 1 -- 172.21.15.19:0/795673864 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f91080cbd70 con 0x7f91080ca780 2024-08-31T12:21:37.939 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.937+0000 7f90fffff640 1 --2- 172.21.15.19:0/795673864 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f91080ca780 0x7f91080d0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:37.940 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.938+0000 7f90fffff640 1 -- 172.21.15.19:0/795673864 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9108152540 con 0x7f91080ca780 2024-08-31T12:21:37.940 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.938+0000 7f90fffff640 1 --2- 172.21.15.19:0/795673864 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f91080ca780 0x7f91080d0190 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9108151c70 tx=0x7f90f0000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:37.940 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.938+0000 7f90ff7fe640 1 -- 172.21.15.19:0/795673864 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90f0006cb0 con 0x7f91080ca780 2024-08-31T12:21:37.941 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.939+0000 7f90ff7fe640 1 -- 172.21.15.19:0/795673864 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5044 (secure 0 0 0) 0x7f90f0006cb0 con 0x7f91080ca780 2024-08-31T12:21:37.941 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.939+0000 7f90ff7fe640 1 -- 172.21.15.19:0/795673864 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f91080ca780 msgr2=0x7f91080d0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:37.941 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.939+0000 7f90ff7fe640 1 --2- 172.21.15.19:0/795673864 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f91080ca780 0x7f91080d0190 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9108151c70 tx=0x7f90f0000ed0 comp rx=0 tx=0).stop 2024-08-31T12:21:37.941 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.939+0000 7f90ff7fe640 1 -- 172.21.15.19:0/795673864 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f91080ca780 msgr2=0x7f91080d0190 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:21:37.941 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.939+0000 7f90ff7fe640 1 --2- 172.21.15.19:0/795673864 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f91080ca780 0x7f91080d0190 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:37.942 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:21:37.942 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:21:37.942 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:21:37.942 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "leader", 2024-08-31T12:21:37.942 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 44, 2024-08-31T12:21:37.942 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 277930, 2024-08-31T12:21:37.942 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:21:37.942 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:21:37.942 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:21:37.942 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:21:37.942 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:21:37.943 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 1, 2024-08-31T12:21:37.943 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:21:37.943 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:21:37.943 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:21:37.943 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:21:37.943 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:21:37.943 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:21:37.943 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:21:37.943 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:21:37.943 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:21:37.943 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:21:37.943 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:21:37.943 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:21:37.944 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:21:37.944 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:21:37.944 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:21:37.944 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:21:37.944 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:21:37.944 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:21:37.944 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:21:37.944 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:21:37.944 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:21:37.944 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:21:37.944 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:21:37.944 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:21:37.945 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:21:37.945 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:21:37.945 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:21:37.945 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:21:37.945 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:37.945 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:37.945 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:21:37.945 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:21:37.945 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:21:37.945 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:21:37.945 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:21:37.945 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:21:37.945 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:21:37.946 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:21:37.946 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:21:37.946 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:21:37.946 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:21:37.946 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:21:37.946 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:21:37.946 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:21:37.946 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:21:37.946 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:21:37.946 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:21:37.946 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:21:37.946 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:21:37.946 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:21:37.946 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:21:37.947 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:21:37.947 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:21:37.947 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:21:37.947 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:21:37.947 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:21:37.947 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:21:37.947 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:21:37.947 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:21:37.947 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:21:37.947 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:37.947 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:21:37.947 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:37.947 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:21:37.947 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:21:37.947 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:21:37.948 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:21:37.948 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:37.948 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:21:37.948 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:21:37.948 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:37.948 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:37.948 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:37.948 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:21:37.948 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:21:37.948 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:37.948 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:37.948 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:37.948 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:37.948 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:21:37.949 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:21:37.949 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:21:37.949 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:21:37.949 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:21:37.949 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:37.949 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:37.949 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:21:37.949 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:21:37.949 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:21:37.949 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:21:37.949 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:37.949 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:21:37.950 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:21:37.950 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:37.950 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:37.950 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:37.950 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:21:37.950 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:21:37.950 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:37.950 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:37.950 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:37.950 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:37.950 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:21:37.950 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:21:37.951 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:21:37.951 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:21:37.951 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:21:37.951 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:37.951 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:37.951 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:21:37.951 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:21:37.951 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:21:37.951 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:21:37.951 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:37.951 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:21:37.951 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:21:37.951 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:37.952 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:37.952 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:37.952 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:21:37.952 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:21:37.952 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:37.952 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:37.952 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:37.952 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:37.952 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:21:37.952 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:21:37.952 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:21:37.952 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:21:37.953 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:21:37.953 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:37.953 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:37.953 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:37.953 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:21:37.953 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:21:37.953 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:37.953 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:21:37.953 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:21:37.953 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:21:37.953 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:37.953 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:21:37.954 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:21:37.954 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:37.954 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:21:37.954 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:21:37.954 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 6 2024-08-31T12:21:37.954 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:37.954 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:21:37.954 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:21:37.954 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:37.954 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:21:37.954 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:21:37.954 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 8 2024-08-31T12:21:37.954 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:37.955 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:21:37.955 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:21:37.955 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:37.955 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:21:37.955 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:21:37.955 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 6 2024-08-31T12:21:37.955 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:37.955 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:21:37.955 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:21:37.955 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:37.955 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:21:37.955 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:21:37.956 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:21:37.956 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:37.956 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:37.956 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:37.956 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:21:37.956 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:21:37.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.939+0000 7f910d407640 1 -- 172.21.15.19:0/795673864 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f90e8077950 msgr2=0x7f90e8079e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:37.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.939+0000 7f910d407640 1 --2- 172.21.15.19:0/795673864 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f90e8077950 0x7f90e8079e60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f91080d14e0 tx=0x7f9100009560 comp rx=0 tx=0).stop 2024-08-31T12:21:37.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.939+0000 7f910d407640 1 -- 172.21.15.19:0/795673864 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f91080c5c20 msgr2=0x7f910814e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:37.958 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.939+0000 7f910d407640 1 --2- 172.21.15.19:0/795673864 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f91080c5c20 0x7f910814e670 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f90f8000c00 tx=0x7f90f8003910 comp rx=0 tx=0).stop 2024-08-31T12:21:37.958 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.939+0000 7f91077fe640 1 -- 172.21.15.19:0/795673864 reap_dead start 2024-08-31T12:21:37.958 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.939+0000 7f91077fe640 1 -- 172.21.15.19:0/795673864 reap_dead start 2024-08-31T12:21:37.958 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.939+0000 7f910d407640 1 -- 172.21.15.19:0/795673864 shutdown_connections 2024-08-31T12:21:37.958 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.939+0000 7f910d407640 1 -- 172.21.15.19:0/795673864 >> 172.21.15.19:0/795673864 conn(0x7f9108074a90 msgr2=0x7f91080b5520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:21:37.958 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.939+0000 7f910d407640 1 -- 172.21.15.19:0/795673864 shutdown_connections 2024-08-31T12:21:37.958 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:37.939+0000 7f910d407640 1 -- 172.21.15.19:0/795673864 wait complete. 2024-08-31T12:21:37.958 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:21:37.958 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-08-31T12:21:37.959 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-31T12:21:38.078 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.076+0000 7f5c0b532640 1 -- 172.21.15.19:0/62533164 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5bf0004d90 msgr2=0x7f5bf0005220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:38.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.076+0000 7f5c0b532640 1 --2- 172.21.15.19:0/62533164 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5bf0004d90 0x7f5bf0005220 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc005fd0 tx=0x7f5bf0005830 comp rx=0 tx=0).stop 2024-08-31T12:21:38.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.076+0000 7f5c09acf640 1 -- 172.21.15.19:0/62533164 reap_dead start 2024-08-31T12:21:38.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.076+0000 7f5c0b532640 1 -- 172.21.15.19:0/62533164 shutdown_connections 2024-08-31T12:21:38.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.076+0000 7f5c0b532640 1 -- 172.21.15.19:0/62533164 >> 172.21.15.19:0/62533164 conn(0x7f5c040728b0 msgr2=0x7f5c04072cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:21:38.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.076+0000 7f5c0b532640 1 -- 172.21.15.19:0/62533164 shutdown_connections 2024-08-31T12:21:38.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.076+0000 7f5c0b532640 1 -- 172.21.15.19:0/62533164 wait complete. 2024-08-31T12:21:38.080 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.078+0000 7f5c0b532640 1 Processor -- start 2024-08-31T12:21:38.080 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.078+0000 7f5c0b532640 1 -- start start 2024-08-31T12:21:38.083 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.080+0000 7f5c0b532640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5bf0004d90 0x7f5c0414fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:38.083 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.080+0000 7f5c0b532640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5c041f2940 0x7f5c04148fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:38.083 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.080+0000 7f5c0b532640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c0415aac0 0x7f5c04149510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:38.083 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.080+0000 7f5c0b532640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f5c041535a0 con 0x7f5c041f2940 2024-08-31T12:21:38.083 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.080+0000 7f5c0b532640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5c040c4db0 con 0x7f5c0415aac0 2024-08-31T12:21:38.083 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.080+0000 7f5c0b532640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f5c04159c30 con 0x7f5bf0004d90 2024-08-31T12:21:38.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.080+0000 7f5c08acd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5c041f2940 0x7f5c04148fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:38.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.080+0000 7f5c08acd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5c041f2940 0x7f5c04148fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:40178/0 (socket says 172.21.15.19:40178) 2024-08-31T12:21:38.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.080+0000 7f5c08acd640 1 -- 172.21.15.19:0/3814982219 learned_addr learned my addr 172.21.15.19:0/3814982219 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:21:38.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.080+0000 7f5c092ce640 1 --2- 172.21.15.19:0/3814982219 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5bf0004d90 0x7f5c0414fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:38.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.080+0000 7f5c09acf640 1 --2- 172.21.15.19:0/3814982219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c0415aac0 0x7f5c04149510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:38.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.080+0000 7f5c08acd640 1 -- 172.21.15.19:0/3814982219 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5bf0004d90 msgr2=0x7f5c0414fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:38.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.080+0000 7f5c08acd640 1 --2- 172.21.15.19:0/3814982219 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5bf0004d90 0x7f5c0414fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:38.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.080+0000 7f5c08acd640 1 -- 172.21.15.19:0/3814982219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c0415aac0 msgr2=0x7f5c04149510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:38.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.080+0000 7f5c08acd640 1 --2- 172.21.15.19:0/3814982219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c0415aac0 0x7f5c04149510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:38.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.080+0000 7f5c08acd640 1 -- 172.21.15.19:0/3814982219 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c0415b030 con 0x7f5c041f2940 2024-08-31T12:21:38.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.081+0000 7f5c08acd640 1 --2- 172.21.15.19:0/3814982219 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5c041f2940 0x7f5c04148fd0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f5bf4006630 tx=0x7f5bf40039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:38.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.081+0000 7f5bf9ffb640 1 -- 172.21.15.19:0/3814982219 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf4015070 con 0x7f5c041f2940 2024-08-31T12:21:38.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.081+0000 7f5bf9ffb640 1 -- 172.21.15.19:0/3814982219 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5bf4011900 con 0x7f5c041f2940 2024-08-31T12:21:38.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.081+0000 7f5c0b532640 1 -- 172.21.15.19:0/3814982219 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c041e29c0 con 0x7f5c041f2940 2024-08-31T12:21:38.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.081+0000 7f5bf9ffb640 1 -- 172.21.15.19:0/3814982219 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf4011d50 con 0x7f5c041f2940 2024-08-31T12:21:38.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.081+0000 7f5c0b532640 1 -- 172.21.15.19:0/3814982219 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c041e2420 con 0x7f5c041f2940 2024-08-31T12:21:38.087 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.082+0000 7f5c0b532640 1 --2- 172.21.15.19:0/3814982219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c04158570 0x7f5c0414e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:38.087 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.082+0000 7f5c0b532640 1 -- 172.21.15.19:0/3814982219 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5c041f3070 con 0x7f5c04158570 2024-08-31T12:21:38.087 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.082+0000 7f5c092ce640 1 --2- 172.21.15.19:0/3814982219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c04158570 0x7f5c0414e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:38.087 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.083+0000 7f5c092ce640 1 -- 172.21.15.19:0/3814982219 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5c041e2660 con 0x7f5c04158570 2024-08-31T12:21:38.087 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.083+0000 7f5bf9ffb640 1 -- 172.21.15.19:0/3814982219 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f5bf4018070 con 0x7f5c041f2940 2024-08-31T12:21:38.087 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.083+0000 7f5c092ce640 1 --2- 172.21.15.19:0/3814982219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c04158570 0x7f5c0414e610 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5c00003e80 tx=0x7f5c00001800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:38.087 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.083+0000 7f5bf9ffb640 1 --2- 172.21.15.19:0/3814982219 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5be8077a50 0x7f5be8079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:38.087 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.083+0000 7f5bf9ffb640 1 -- 172.21.15.19:0/3814982219 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f5bf409a4e0 con 0x7f5c041f2940 2024-08-31T12:21:38.087 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.083+0000 7f5c09acf640 1 --2- 172.21.15.19:0/3814982219 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5be8077a50 0x7f5be8079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:38.087 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.084+0000 7f5bf9ffb640 1 -- 172.21.15.19:0/3814982219 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c0000f950 con 0x7f5c04158570 2024-08-31T12:21:38.087 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.084+0000 7f5bf9ffb640 1 -- 172.21.15.19:0/3814982219 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5c0000faf0 con 0x7f5c04158570 2024-08-31T12:21:38.087 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.084+0000 7f5bf9ffb640 1 -- 172.21.15.19:0/3814982219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c04158570 msgr2=0x7f5c0414e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:38.088 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.084+0000 7f5bf9ffb640 1 --2- 172.21.15.19:0/3814982219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c04158570 0x7f5c0414e610 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5c00003e80 tx=0x7f5c00001800 comp rx=0 tx=0).stop 2024-08-31T12:21:38.088 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.084+0000 7f5bf9ffb640 1 -- 172.21.15.19:0/3814982219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c04158570 msgr2=0x7f5c0414e610 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:21:38.088 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.084+0000 7f5bf9ffb640 1 --2- 172.21.15.19:0/3814982219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c04158570 0x7f5c0414e610 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:38.088 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.084+0000 7f5c09acf640 1 --2- 172.21.15.19:0/3814982219 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5be8077a50 0x7f5be8079f60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc035000 tx=0x7f5bfc02d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:38.093 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.091+0000 7f5c0b532640 1 --2- 172.21.15.19:0/3814982219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c0414eb50 0x7f5c0414ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:38.094 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.091+0000 7f5c0b532640 1 -- 172.21.15.19:0/3814982219 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5c0414f4a0 con 0x7f5c0414eb50 2024-08-31T12:21:38.094 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.092+0000 7f5c092ce640 1 --2- 172.21.15.19:0/3814982219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c0414eb50 0x7f5c0414ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:38.094 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.092+0000 7f5c092ce640 1 -- 172.21.15.19:0/3814982219 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5c040cb830 con 0x7f5c0414eb50 2024-08-31T12:21:38.095 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.092+0000 7f5c092ce640 1 --2- 172.21.15.19:0/3814982219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c0414eb50 0x7f5c0414ef60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5c00018000 tx=0x7f5c000012f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:38.095 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.092+0000 7f5bf9ffb640 1 -- 172.21.15.19:0/3814982219 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c0000f950 con 0x7f5c0414eb50 2024-08-31T12:21:38.095 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.093+0000 7f5bf9ffb640 1 -- 172.21.15.19:0/3814982219 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4539 (secure 0 0 0) 0x7f5c00011840 con 0x7f5c0414eb50 2024-08-31T12:21:38.095 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.093+0000 7f5bf9ffb640 1 -- 172.21.15.19:0/3814982219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c0414eb50 msgr2=0x7f5c0414ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:38.095 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.093+0000 7f5bf9ffb640 1 --2- 172.21.15.19:0/3814982219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c0414eb50 0x7f5c0414ef60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5c00018000 tx=0x7f5c000012f0 comp rx=0 tx=0).stop 2024-08-31T12:21:38.095 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.093+0000 7f5bf9ffb640 1 -- 172.21.15.19:0/3814982219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c0414eb50 msgr2=0x7f5c0414ef60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:21:38.095 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.093+0000 7f5bf9ffb640 1 --2- 172.21.15.19:0/3814982219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c0414eb50 0x7f5c0414ef60 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:38.095 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:21:38.096 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:21:38.096 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:21:38.096 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:21:38.096 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 44, 2024-08-31T12:21:38.096 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 7567, 2024-08-31T12:21:38.096 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:21:38.096 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:21:38.096 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:21:38.096 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:21:38.096 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:21:38.096 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 1, 2024-08-31T12:21:38.097 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:21:38.097 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:21:38.097 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:21:38.097 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:21:38.097 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:21:38.097 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:21:38.097 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:21:38.097 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:21:38.097 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:21:38.097 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:21:38.097 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:21:38.097 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:21:38.098 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:21:38.098 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:21:38.098 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:21:38.098 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:21:38.098 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:21:38.098 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:21:38.098 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:21:38.098 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:21:38.098 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:21:38.098 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:21:38.098 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:21:38.098 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:21:38.098 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:21:38.098 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:21:38.099 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:21:38.099 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:21:38.099 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:38.099 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:38.099 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:21:38.099 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:21:38.099 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:21:38.099 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:21:38.099 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:21:38.099 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:21:38.099 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:21:38.099 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:21:38.099 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:21:38.100 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:21:38.100 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:21:38.100 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:21:38.100 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:21:38.100 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:21:38.100 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:21:38.100 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:21:38.100 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:21:38.100 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:21:38.100 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:21:38.100 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:21:38.100 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:21:38.101 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:21:38.101 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:21:38.101 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:21:38.101 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:21:38.101 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:21:38.101 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:21:38.101 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:21:38.101 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:21:38.101 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:21:38.101 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:38.101 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:21:38.101 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:38.101 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:21:38.102 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:21:38.102 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:21:38.102 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:21:38.102 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:38.102 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:21:38.102 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:21:38.102 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:38.102 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:38.102 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:38.102 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:21:38.102 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:21:38.102 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:38.102 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:38.103 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:38.103 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:38.103 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:21:38.103 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:21:38.103 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:21:38.103 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:21:38.103 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:21:38.103 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:38.103 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:38.103 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:21:38.103 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:21:38.103 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:21:38.103 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:21:38.104 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:38.104 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:21:38.104 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:21:38.104 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:38.104 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:38.104 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:38.104 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:21:38.104 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:21:38.104 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:38.104 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:38.104 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:38.104 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:38.105 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:21:38.105 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:21:38.105 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:21:38.105 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:21:38.105 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:21:38.105 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:38.105 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:38.105 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:21:38.105 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:21:38.105 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:21:38.106 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:21:38.106 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:38.106 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:21:38.106 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:21:38.106 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:38.106 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:38.106 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:38.106 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:21:38.106 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:21:38.106 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:38.106 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:38.106 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:38.106 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:38.107 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:21:38.107 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:21:38.107 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:21:38.107 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:21:38.107 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:21:38.107 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:38.107 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:38.107 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:38.107 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:21:38.107 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:21:38.107 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:38.107 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:21:38.107 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:21:38.108 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:21:38.108 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:38.108 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:21:38.108 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:21:38.108 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:38.108 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:21:38.108 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:21:38.108 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:21:38.108 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:38.109 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:38.109 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:38.109 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:21:38.109 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:21:38.110 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.093+0000 7f5c0b532640 1 -- 172.21.15.19:0/3814982219 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5be8077a50 msgr2=0x7f5be8079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:38.110 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.093+0000 7f5c0b532640 1 --2- 172.21.15.19:0/3814982219 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5be8077a50 0x7f5be8079f60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc035000 tx=0x7f5bfc02d040 comp rx=0 tx=0).stop 2024-08-31T12:21:38.110 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.093+0000 7f5c0b532640 1 -- 172.21.15.19:0/3814982219 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5c041f2940 msgr2=0x7f5c04148fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:38.110 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.093+0000 7f5c0b532640 1 --2- 172.21.15.19:0/3814982219 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5c041f2940 0x7f5c04148fd0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f5bf4006630 tx=0x7f5bf40039c0 comp rx=0 tx=0).stop 2024-08-31T12:21:38.111 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.093+0000 7f5c09acf640 1 -- 172.21.15.19:0/3814982219 reap_dead start 2024-08-31T12:21:38.111 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.093+0000 7f5c0b532640 1 -- 172.21.15.19:0/3814982219 shutdown_connections 2024-08-31T12:21:38.111 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.093+0000 7f5c0b532640 1 --2- 172.21.15.19:0/3814982219 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5c041f2940 0x7f5c04148fd0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:38.111 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.093+0000 7f5c0b532640 1 -- 172.21.15.19:0/3814982219 >> 172.21.15.19:0/3814982219 conn(0x7f5c040728b0 msgr2=0x7f5c040635a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:21:38.111 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.093+0000 7f5c0b532640 1 -- 172.21.15.19:0/3814982219 shutdown_connections 2024-08-31T12:21:38.111 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.093+0000 7f5c0b532640 1 -- 172.21.15.19:0/3814982219 wait complete. 2024-08-31T12:21:38.111 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:21:38.111 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-08-31T12:21:38.111 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-31T12:21:38.234 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.231+0000 7f4440712640 1 -- 172.21.15.19:0/761115385 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4438070bf0 msgr2=0x7f4438071000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:38.234 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.232+0000 7f4440712640 1 --2- 172.21.15.19:0/761115385 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4438070bf0 0x7f4438071000 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f4430009be0 tx=0x7f443002a960 comp rx=0 tx=0).stop 2024-08-31T12:21:38.234 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.232+0000 7f4440712640 1 -- 172.21.15.19:0/761115385 shutdown_connections 2024-08-31T12:21:38.234 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.232+0000 7f4440712640 1 --2- 172.21.15.19:0/761115385 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4438070bf0 0x7f4438071000 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:38.234 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.232+0000 7f4440712640 1 -- 172.21.15.19:0/761115385 >> 172.21.15.19:0/761115385 conn(0x7f44380728b0 msgr2=0x7f4438072cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:21:38.235 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.232+0000 7f4440712640 1 -- 172.21.15.19:0/761115385 shutdown_connections 2024-08-31T12:21:38.235 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.232+0000 7f4440712640 1 -- 172.21.15.19:0/761115385 wait complete. 2024-08-31T12:21:38.235 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.234+0000 7f4440712640 1 Processor -- start 2024-08-31T12:21:38.235 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.234+0000 7f4440712640 1 -- start start 2024-08-31T12:21:38.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.235+0000 7f4440712640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4438070bf0 0x7f44381e3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:38.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.235+0000 7f4440712640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4438153190 0x7f44380cda60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:38.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.235+0000 7f4440712640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4438159820 0x7f443814afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:38.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.235+0000 7f4440712640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f44381535a0 con 0x7f4438153190 2024-08-31T12:21:38.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.235+0000 7f4440712640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f44380c4db0 con 0x7f4438159820 2024-08-31T12:21:38.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.235+0000 7f4440712640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f4438071540 con 0x7f4438070bf0 2024-08-31T12:21:38.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.235+0000 7f443dcad640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4438153190 0x7f44380cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:38.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.235+0000 7f443dcad640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4438153190 0x7f44380cda60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:45988/0 (socket says 172.21.15.19:45988) 2024-08-31T12:21:38.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.235+0000 7f443dcad640 1 -- 172.21.15.19:0/4181242515 learned_addr learned my addr 172.21.15.19:0/4181242515 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:21:38.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.235+0000 7f443ecaf640 1 --2- 172.21.15.19:0/4181242515 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4438159820 0x7f443814afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:38.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.235+0000 7f443e4ae640 1 --2- 172.21.15.19:0/4181242515 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4438070bf0 0x7f44381e3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:38.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.235+0000 7f443dcad640 1 -- 172.21.15.19:0/4181242515 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4438070bf0 msgr2=0x7f44381e3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:38.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.235+0000 7f443dcad640 1 --2- 172.21.15.19:0/4181242515 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4438070bf0 0x7f44381e3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:38.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.235+0000 7f443dcad640 1 -- 172.21.15.19:0/4181242515 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4438159820 msgr2=0x7f443814afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:38.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.235+0000 7f443dcad640 1 --2- 172.21.15.19:0/4181242515 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4438159820 0x7f443814afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:38.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.235+0000 7f443dcad640 1 -- 172.21.15.19:0/4181242515 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44381e4360 con 0x7f4438153190 2024-08-31T12:21:38.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.236+0000 7f443dcad640 1 --2- 172.21.15.19:0/4181242515 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4438153190 0x7f44380cda60 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f442800c3a0 tx=0x7f442800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:38.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.236+0000 7f442effd640 1 -- 172.21.15.19:0/4181242515 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4428017070 con 0x7f4438153190 2024-08-31T12:21:38.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.236+0000 7f442effd640 1 -- 172.21.15.19:0/4181242515 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f44280035f0 con 0x7f4438153190 2024-08-31T12:21:38.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.236+0000 7f442effd640 1 -- 172.21.15.19:0/4181242515 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4428003a40 con 0x7f4438153190 2024-08-31T12:21:38.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.236+0000 7f4440712640 1 -- 172.21.15.19:0/4181242515 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44380cebf0 con 0x7f4438153190 2024-08-31T12:21:38.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.236+0000 7f4440712640 1 -- 172.21.15.19:0/4181242515 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44380cf1b0 con 0x7f4438153190 2024-08-31T12:21:38.242 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.237+0000 7f4440712640 1 --2- 172.21.15.19:0/4181242515 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f443814b510 0x7f44381e2680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:38.242 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.237+0000 7f4440712640 1 -- 172.21.15.19:0/4181242515 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f44381e4830 con 0x7f443814b510 2024-08-31T12:21:38.242 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.237+0000 7f443e4ae640 1 --2- 172.21.15.19:0/4181242515 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f443814b510 0x7f44381e2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:38.242 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.237+0000 7f443e4ae640 1 -- 172.21.15.19:0/4181242515 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f44380ce9a0 con 0x7f443814b510 2024-08-31T12:21:38.242 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.238+0000 7f443e4ae640 1 --2- 172.21.15.19:0/4181242515 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f443814b510 0x7f44381e2680 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f443400f520 tx=0x7f443400f9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:38.242 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.238+0000 7f442effd640 1 -- 172.21.15.19:0/4181242515 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44340052e0 con 0x7f443814b510 2024-08-31T12:21:38.242 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.238+0000 7f442effd640 1 -- 172.21.15.19:0/4181242515 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f442801a070 con 0x7f4438153190 2024-08-31T12:21:38.242 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.238+0000 7f442effd640 1 --2- 172.21.15.19:0/4181242515 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f441c077950 0x7f441c079e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:38.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.239+0000 7f442effd640 1 -- 172.21.15.19:0/4181242515 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f4428098860 con 0x7f4438153190 2024-08-31T12:21:38.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.239+0000 7f443ecaf640 1 --2- 172.21.15.19:0/4181242515 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f441c077950 0x7f441c079e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:38.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.239+0000 7f442effd640 1 -- 172.21.15.19:0/4181242515 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4434018070 con 0x7f443814b510 2024-08-31T12:21:38.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.239+0000 7f442effd640 1 -- 172.21.15.19:0/4181242515 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f443814b510 msgr2=0x7f44381e2680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:38.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.239+0000 7f442effd640 1 --2- 172.21.15.19:0/4181242515 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f443814b510 0x7f44381e2680 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f443400f520 tx=0x7f443400f9e0 comp rx=0 tx=0).stop 2024-08-31T12:21:38.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.239+0000 7f442effd640 1 -- 172.21.15.19:0/4181242515 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f443814b510 msgr2=0x7f44381e2680 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:21:38.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.239+0000 7f442effd640 1 --2- 172.21.15.19:0/4181242515 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f443814b510 0x7f44381e2680 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f443400f520 tx=0x7f443400f9e0 comp rx=0 tx=0).stop 2024-08-31T12:21:38.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.239+0000 7f443ecaf640 1 --2- 172.21.15.19:0/4181242515 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f441c077950 0x7f441c079e60 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4430034000 tx=0x7f4430004ab0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:38.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.246+0000 7f4440712640 1 --2- 172.21.15.19:0/4181242515 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f44381e2bc0 0x7f44381f2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:38.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.246+0000 7f4440712640 1 -- 172.21.15.19:0/4181242515 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f4438071000 con 0x7f44381e2bc0 2024-08-31T12:21:38.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.246+0000 7f443e4ae640 1 --2- 172.21.15.19:0/4181242515 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f44381e2bc0 0x7f44381f2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:38.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.246+0000 7f443e4ae640 1 -- 172.21.15.19:0/4181242515 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f44381595c0 con 0x7f44381e2bc0 2024-08-31T12:21:38.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.246+0000 7f443e4ae640 1 --2- 172.21.15.19:0/4181242515 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f44381e2bc0 0x7f44381f2bb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f443400f520 tx=0x7f4434001080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:38.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.247+0000 7f442effd640 1 -- 172.21.15.19:0/4181242515 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4434003ad0 con 0x7f44381e2bc0 2024-08-31T12:21:38.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.247+0000 7f442effd640 1 -- 172.21.15.19:0/4181242515 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4540 (secure 0 0 0) 0x7f4434003ad0 con 0x7f44381e2bc0 2024-08-31T12:21:38.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.247+0000 7f442effd640 1 -- 172.21.15.19:0/4181242515 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f44381e2bc0 msgr2=0x7f44381f2bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:38.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.247+0000 7f442effd640 1 --2- 172.21.15.19:0/4181242515 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f44381e2bc0 0x7f44381f2bb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f443400f520 tx=0x7f4434001080 comp rx=0 tx=0).stop 2024-08-31T12:21:38.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.247+0000 7f442effd640 1 -- 172.21.15.19:0/4181242515 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f44381e2bc0 msgr2=0x7f44381f2bb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:21:38.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.247+0000 7f442effd640 1 --2- 172.21.15.19:0/4181242515 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f44381e2bc0 0x7f44381f2bb0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:38.250 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:21:38.250 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:21:38.250 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:21:38.250 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:21:38.250 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 44, 2024-08-31T12:21:38.250 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 59752, 2024-08-31T12:21:38.250 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:21:38.251 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:21:38.251 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:21:38.251 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:21:38.251 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:21:38.251 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 2, 2024-08-31T12:21:38.251 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:21:38.251 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:21:38.251 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:21:38.251 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:21:38.251 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:21:38.251 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:21:38.251 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:21:38.251 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:21:38.252 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:21:38.252 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:21:38.252 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:21:38.252 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:21:38.252 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:21:38.252 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:21:38.252 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:21:38.252 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:21:38.252 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:21:38.252 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:21:38.252 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:21:38.252 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:21:38.252 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:21:38.252 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:21:38.253 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:21:38.253 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:21:38.253 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:21:38.253 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:21:38.253 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:21:38.253 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:21:38.253 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:38.253 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:38.253 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:21:38.253 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:21:38.253 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:21:38.253 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:21:38.254 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:21:38.254 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:21:38.254 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:21:38.254 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:21:38.254 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:21:38.254 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:21:38.254 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:21:38.254 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:21:38.254 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:21:38.254 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:21:38.254 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:21:38.254 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:21:38.254 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:21:38.254 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:21:38.255 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:21:38.255 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:21:38.255 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:21:38.255 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:21:38.255 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:21:38.255 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:21:38.255 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:21:38.255 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:21:38.256 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:21:38.256 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:21:38.256 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:21:38.256 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:21:38.256 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:38.256 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:21:38.256 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:38.256 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:21:38.256 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:21:38.256 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:21:38.256 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:21:38.257 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:38.257 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:21:38.257 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:21:38.257 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:38.257 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:38.257 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:38.257 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:21:38.257 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:21:38.257 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:38.257 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:38.257 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:38.257 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:38.257 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:21:38.258 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:21:38.258 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:21:38.258 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:21:38.258 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:21:38.258 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:38.258 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:38.258 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:21:38.258 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:21:38.258 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:21:38.258 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:21:38.258 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:38.258 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:21:38.258 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:21:38.258 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:38.259 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:38.259 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:38.259 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:21:38.259 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:21:38.259 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:38.259 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:38.259 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:38.259 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:38.259 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:21:38.259 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:21:38.259 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:21:38.259 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:21:38.260 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:21:38.260 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:38.260 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:38.260 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:21:38.260 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:21:38.260 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:21:38.260 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:21:38.260 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:38.260 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:21:38.260 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:21:38.260 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:38.260 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:38.260 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:38.261 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:21:38.261 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:21:38.261 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:38.261 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:38.261 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:38.261 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:38.261 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:21:38.261 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:21:38.261 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:21:38.261 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:21:38.261 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:21:38.261 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:38.262 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:38.262 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:38.262 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:21:38.262 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:21:38.262 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:38.262 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:21:38.262 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:21:38.262 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:21:38.262 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:38.262 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:21:38.262 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:21:38.262 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:38.262 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:21:38.262 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:21:38.263 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:21:38.263 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:38.263 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:38.263 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:38.263 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:21:38.263 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:21:38.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.247+0000 7f4440712640 1 -- 172.21.15.19:0/4181242515 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f441c077950 msgr2=0x7f441c079e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:38.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.247+0000 7f4440712640 1 --2- 172.21.15.19:0/4181242515 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f441c077950 0x7f441c079e60 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4430034000 tx=0x7f4430004ab0 comp rx=0 tx=0).stop 2024-08-31T12:21:38.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.247+0000 7f4440712640 1 -- 172.21.15.19:0/4181242515 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4438153190 msgr2=0x7f44380cda60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:38.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.247+0000 7f4440712640 1 --2- 172.21.15.19:0/4181242515 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4438153190 0x7f44380cda60 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f442800c3a0 tx=0x7f442800c860 comp rx=0 tx=0).stop 2024-08-31T12:21:38.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.247+0000 7f443ecaf640 1 -- 172.21.15.19:0/4181242515 reap_dead start 2024-08-31T12:21:38.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.247+0000 7f443ecaf640 1 -- 172.21.15.19:0/4181242515 reap_dead start 2024-08-31T12:21:38.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.247+0000 7f4440712640 1 -- 172.21.15.19:0/4181242515 shutdown_connections 2024-08-31T12:21:38.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.247+0000 7f4440712640 1 -- 172.21.15.19:0/4181242515 >> 172.21.15.19:0/4181242515 conn(0x7f44380728b0 msgr2=0x7f44380cb3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:21:38.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.248+0000 7f4440712640 1 -- 172.21.15.19:0/4181242515 shutdown_connections 2024-08-31T12:21:38.266 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.248+0000 7f4440712640 1 -- 172.21.15.19:0/4181242515 wait complete. 2024-08-31T12:21:38.266 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:21:38.266 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-08-31T12:21:38.266 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-08-31T12:21:38.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.391+0000 7f514e98b640 1 -- 172.21.15.19:0/2606959182 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f51480719b0 msgr2=0x7f5148071dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:38.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.391+0000 7f514e98b640 1 --2- 172.21.15.19:0/2606959182 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f51480719b0 0x7f5148071dc0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f5140005fd0 tx=0x7f514002ab20 comp rx=0 tx=0).stop 2024-08-31T12:21:38.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.391+0000 7f514e98b640 1 -- 172.21.15.19:0/2606959182 shutdown_connections 2024-08-31T12:21:38.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.391+0000 7f514e98b640 1 --2- 172.21.15.19:0/2606959182 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f51480719b0 0x7f5148071dc0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:38.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.391+0000 7f514e98b640 1 --2- 172.21.15.19:0/2606959182 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f51480c7d60 0x7f51480c7140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:38.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.391+0000 7f514e98b640 1 -- 172.21.15.19:0/2606959182 >> 172.21.15.19:0/2606959182 conn(0x7f51480cd7d0 msgr2=0x7f51480cdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:21:38.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.391+0000 7f514e98b640 1 -- 172.21.15.19:0/2606959182 shutdown_connections 2024-08-31T12:21:38.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.391+0000 7f514e98b640 1 -- 172.21.15.19:0/2606959182 wait complete. 2024-08-31T12:21:38.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.393+0000 7f514e98b640 1 Processor -- start 2024-08-31T12:21:38.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.393+0000 7f514e98b640 1 -- start start 2024-08-31T12:21:38.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.394+0000 7f514e98b640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f51480719b0 0x7f5148159ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:38.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.394+0000 7f514e98b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f51480c7d60 0x7f5148138060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:38.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.394+0000 7f514e98b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5148140c10 0x7f51481e4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:38.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.394+0000 7f514e98b640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f51480c8170 con 0x7f51480c7d60 2024-08-31T12:21:38.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.394+0000 7f514e98b640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f51480c6920 con 0x7f51480719b0 2024-08-31T12:21:38.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.394+0000 7f514e98b640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f514815b2b0 con 0x7f5148140c10 2024-08-31T12:21:38.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.395+0000 7f51477fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f51480c7d60 0x7f5148138060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:38.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.395+0000 7f5147fff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f51480719b0 0x7f5148159ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:38.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.395+0000 7f51477fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f51480c7d60 0x7f5148138060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:46004/0 (socket says 172.21.15.19:46004) 2024-08-31T12:21:38.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.395+0000 7f51477fe640 1 -- 172.21.15.19:0/853827267 learned_addr learned my addr 172.21.15.19:0/853827267 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:21:38.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.395+0000 7f514cf28640 1 --2- 172.21.15.19:0/853827267 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5148140c10 0x7f51481e4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:38.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.395+0000 7f5147fff640 1 -- 172.21.15.19:0/853827267 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5148140c10 msgr2=0x7f51481e4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:38.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.395+0000 7f5147fff640 1 --2- 172.21.15.19:0/853827267 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5148140c10 0x7f51481e4e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:38.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.395+0000 7f5147fff640 1 -- 172.21.15.19:0/853827267 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f51480c7d60 msgr2=0x7f5148138060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:38.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.395+0000 7f5147fff640 1 --2- 172.21.15.19:0/853827267 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f51480c7d60 0x7f5148138060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:38.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.395+0000 7f5147fff640 1 -- 172.21.15.19:0/853827267 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5148152c40 con 0x7f51480719b0 2024-08-31T12:21:38.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.395+0000 7f51477fe640 1 --2- 172.21.15.19:0/853827267 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f51480c7d60 0x7f5148138060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:21:38.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.395+0000 7f5147fff640 1 --2- 172.21.15.19:0/853827267 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f51480719b0 0x7f5148159ba0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f513800c3c0 tx=0x7f513800c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:38.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.395+0000 7f5144ff9640 1 -- 172.21.15.19:0/853827267 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5138017070 con 0x7f51480719b0 2024-08-31T12:21:38.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.396+0000 7f514e98b640 1 -- 172.21.15.19:0/853827267 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5148152f20 con 0x7f51480719b0 2024-08-31T12:21:38.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.396+0000 7f5144ff9640 1 -- 172.21.15.19:0/853827267 <== mon.1 v2:172.21.15.43:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f51380053e0 con 0x7f51480719b0 2024-08-31T12:21:38.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.396+0000 7f5144ff9640 1 -- 172.21.15.19:0/853827267 <== mon.1 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5138003850 con 0x7f51480719b0 2024-08-31T12:21:38.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.396+0000 7f514e98b640 1 -- 172.21.15.19:0/853827267 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51481385a0 con 0x7f51480719b0 2024-08-31T12:21:38.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.397+0000 7f514e98b640 1 -- 172.21.15.19:0/853827267 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5148141240 con 0x7f51480719b0 2024-08-31T12:21:38.401 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.398+0000 7f5144ff9640 1 -- 172.21.15.19:0/853827267 <== mon.1 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f513801a070 con 0x7f51480719b0 2024-08-31T12:21:38.401 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.398+0000 7f5144ff9640 1 --2- 172.21.15.19:0/853827267 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5134077a50 0x7f5134079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:38.401 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.398+0000 7f5144ff9640 1 -- 172.21.15.19:0/853827267 <== mon.1 v2:172.21.15.43:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f5138098860 con 0x7f51480719b0 2024-08-31T12:21:38.401 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.398+0000 7f51477fe640 1 --2- 172.21.15.19:0/853827267 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5134077a50 0x7f5134079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:38.401 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.399+0000 7f51477fe640 1 --2- 172.21.15.19:0/853827267 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5134077a50 0x7f5134079f60 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f51481584b0 tx=0x7f513c00b390 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:38.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.403+0000 7f5144ff9640 1 -- 172.21.15.19:0/853827267 <== mon.1 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f5138060f10 con 0x7f51480719b0 2024-08-31T12:21:38.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.724+0000 7f514e98b640 1 -- 172.21.15.19:0/853827267 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f51480c9230 con 0x7f51480719b0 2024-08-31T12:21:38.727 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.726+0000 7f5144ff9640 1 -- 172.21.15.19:0/853827267 <== mon.1 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f51380656d0 con 0x7f51480719b0 2024-08-31T12:21:38.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.728+0000 7f514e98b640 1 -- 172.21.15.19:0/853827267 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5134077a50 msgr2=0x7f5134079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:38.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.728+0000 7f514e98b640 1 --2- 172.21.15.19:0/853827267 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5134077a50 0x7f5134079f60 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f51481584b0 tx=0x7f513c00b390 comp rx=0 tx=0).stop 2024-08-31T12:21:38.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.728+0000 7f514e98b640 1 -- 172.21.15.19:0/853827267 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f51480719b0 msgr2=0x7f5148159ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:38.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.728+0000 7f514e98b640 1 --2- 172.21.15.19:0/853827267 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f51480719b0 0x7f5148159ba0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f513800c3c0 tx=0x7f513800c880 comp rx=0 tx=0).stop 2024-08-31T12:21:38.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.728+0000 7f514e98b640 1 -- 172.21.15.19:0/853827267 shutdown_connections 2024-08-31T12:21:38.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.728+0000 7f514e98b640 1 --2- 172.21.15.19:0/853827267 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5134077a50 0x7f5134079f60 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:38.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.728+0000 7f514e98b640 1 --2- 172.21.15.19:0/853827267 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5148140c10 0x7f51481e4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:38.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.728+0000 7f514e98b640 1 --2- 172.21.15.19:0/853827267 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f51480c7d60 0x7f5148138060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:38.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.728+0000 7f514e98b640 1 --2- 172.21.15.19:0/853827267 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f51480719b0 0x7f5148159ba0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:38.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.728+0000 7f514e98b640 1 -- 172.21.15.19:0/853827267 >> 172.21.15.19:0/853827267 conn(0x7f51480cd7d0 msgr2=0x7f51480c9450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:21:38.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.728+0000 7f514e98b640 1 -- 172.21.15.19:0/853827267 shutdown_connections 2024-08-31T12:21:38.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:38.728+0000 7f514e98b640 1 -- 172.21.15.19:0/853827267 wait complete. 2024-08-31T12:21:38.745 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2024-08-31T12:21:48.746 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-08-31T12:21:48.747 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:21:48.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:48.869+0000 7fe491215640 1 -- 172.21.15.19:0/578620794 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe48c0ca7f0 msgr2=0x7fe48c1530f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:48.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:48.869+0000 7fe491215640 1 --2- 172.21.15.19:0/578620794 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe48c0ca7f0 0x7fe48c1530f0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fe47c005fd0 tx=0x7fe47c02a540 comp rx=0 tx=0).stop 2024-08-31T12:21:48.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:48.869+0000 7fe491215640 1 -- 172.21.15.19:0/578620794 shutdown_connections 2024-08-31T12:21:48.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:48.869+0000 7fe491215640 1 --2- 172.21.15.19:0/578620794 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe48c15c4b0 0x7fe48c15c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:48.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:48.869+0000 7fe491215640 1 --2- 172.21.15.19:0/578620794 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe48c0ca7f0 0x7fe48c1530f0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:48.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:48.869+0000 7fe491215640 1 -- 172.21.15.19:0/578620794 >> 172.21.15.19:0/578620794 conn(0x7fe48c0cef20 msgr2=0x7fe48c0cb340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:21:48.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:48.869+0000 7fe491215640 1 -- 172.21.15.19:0/578620794 shutdown_connections 2024-08-31T12:21:48.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:48.869+0000 7fe491215640 1 -- 172.21.15.19:0/578620794 wait complete. 2024-08-31T12:21:48.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:48.871+0000 7fe491215640 1 Processor -- start 2024-08-31T12:21:48.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:48.871+0000 7fe491215640 1 -- start start 2024-08-31T12:21:48.876 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:48.873+0000 7fe491215640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe48c0ca7f0 0x7fe48c06f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:48.876 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:48.873+0000 7fe491215640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe48c0cacb0 0x7fe48c0b3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:48.876 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:48.873+0000 7fe491215640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe48c15c4b0 0x7fe48c0b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:48.877 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:48.873+0000 7fe491215640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fe48c0c58d0 con 0x7fe48c0ca7f0 2024-08-31T12:21:48.877 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:48.873+0000 7fe491215640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe48c0c98b0 con 0x7fe48c0cacb0 2024-08-31T12:21:48.877 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:48.873+0000 7fe491215640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fe48c0c5a50 con 0x7fe48c15c4b0 2024-08-31T12:21:48.877 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:48.873+0000 7fe48ad9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe48c0ca7f0 0x7fe48c06f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:48.877 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:48.873+0000 7fe48ad9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe48c0ca7f0 0x7fe48c06f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:49082/0 (socket says 172.21.15.19:49082) 2024-08-31T12:21:48.877 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:48.873+0000 7fe48ad9d640 1 -- 172.21.15.19:0/1982262113 learned_addr learned my addr 172.21.15.19:0/1982262113 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:21:48.877 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:48.873+0000 7fe48b59e640 1 --2- 172.21.15.19:0/1982262113 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe48c15c4b0 0x7fe48c0b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:48.877 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:48.873+0000 7fe48a59c640 1 --2- 172.21.15.19:0/1982262113 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe48c0cacb0 0x7fe48c0b3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:48.877 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:48.873+0000 7fe48ad9d640 1 -- 172.21.15.19:0/1982262113 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe48c15c4b0 msgr2=0x7fe48c0b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:48.877 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:48.873+0000 7fe48ad9d640 1 --2- 172.21.15.19:0/1982262113 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe48c15c4b0 0x7fe48c0b41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:48.877 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:48.873+0000 7fe48ad9d640 1 -- 172.21.15.19:0/1982262113 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe48c0cacb0 msgr2=0x7fe48c0b3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:48.878 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:48.873+0000 7fe48ad9d640 1 --2- 172.21.15.19:0/1982262113 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe48c0cacb0 0x7fe48c0b3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:48.878 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:48.873+0000 7fe48ad9d640 1 -- 172.21.15.19:0/1982262113 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe48c06fbc0 con 0x7fe48c0ca7f0 2024-08-31T12:21:48.878 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:48.873+0000 7fe48ad9d640 1 --2- 172.21.15.19:0/1982262113 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe48c0ca7f0 0x7fe48c06f680 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe47c034040 tx=0x7fe47c0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:48.878 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:48.874+0000 7fe4737fe640 1 -- 172.21.15.19:0/1982262113 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe47c03e030 con 0x7fe48c0ca7f0 2024-08-31T12:21:48.878 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:48.874+0000 7fe4737fe640 1 -- 172.21.15.19:0/1982262113 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe47c003d90 con 0x7fe48c0ca7f0 2024-08-31T12:21:48.878 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:48.874+0000 7fe4737fe640 1 -- 172.21.15.19:0/1982262113 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe47c035040 con 0x7fe48c0ca7f0 2024-08-31T12:21:48.878 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:48.874+0000 7fe491215640 1 -- 172.21.15.19:0/1982262113 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe48c0b1200 con 0x7fe48c0ca7f0 2024-08-31T12:21:48.878 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:48.874+0000 7fe491215640 1 -- 172.21.15.19:0/1982262113 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe48c0b14a0 con 0x7fe48c0ca7f0 2024-08-31T12:21:48.879 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:48.875+0000 7fe491215640 1 -- 172.21.15.19:0/1982262113 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe48c0ca180 con 0x7fe48c0ca7f0 2024-08-31T12:21:48.879 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:48.876+0000 7fe4737fe640 1 -- 172.21.15.19:0/1982262113 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fe47c041030 con 0x7fe48c0ca7f0 2024-08-31T12:21:48.879 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:48.877+0000 7fe4737fe640 1 --2- 172.21.15.19:0/1982262113 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe478077a50 0x7fe478079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:48.879 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:48.877+0000 7fe4737fe640 1 -- 172.21.15.19:0/1982262113 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fe47c0b9e00 con 0x7fe48c0ca7f0 2024-08-31T12:21:48.879 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:48.877+0000 7fe48a59c640 1 --2- 172.21.15.19:0/1982262113 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe478077a50 0x7fe478079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:48.880 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:48.878+0000 7fe48a59c640 1 --2- 172.21.15.19:0/1982262113 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe478077a50 0x7fe478079f60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe48c06eb20 tx=0x7fe480009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:48.884 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:48.882+0000 7fe4737fe640 1 -- 172.21.15.19:0/1982262113 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fe47c082410 con 0x7fe48c0ca7f0 2024-08-31T12:21:49.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.151+0000 7fe491215640 1 -- 172.21.15.19:0/1982262113 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe48c0c3cc0 con 0x7fe48c0ca7f0 2024-08-31T12:21:49.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.151+0000 7fe4737fe640 1 -- 172.21.15.19:0/1982262113 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fe47c086bd0 con 0x7fe48c0ca7f0 2024-08-31T12:21:49.153 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":44,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:21:49.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.154+0000 7fe491215640 1 -- 172.21.15.19:0/1982262113 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe478077a50 msgr2=0x7fe478079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:49.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.154+0000 7fe491215640 1 --2- 172.21.15.19:0/1982262113 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe478077a50 0x7fe478079f60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe48c06eb20 tx=0x7fe480009040 comp rx=0 tx=0).stop 2024-08-31T12:21:49.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.154+0000 7fe491215640 1 -- 172.21.15.19:0/1982262113 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe48c0ca7f0 msgr2=0x7fe48c06f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:49.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.154+0000 7fe491215640 1 --2- 172.21.15.19:0/1982262113 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe48c0ca7f0 0x7fe48c06f680 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe47c034040 tx=0x7fe47c0038a0 comp rx=0 tx=0).stop 2024-08-31T12:21:49.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.154+0000 7fe491215640 1 -- 172.21.15.19:0/1982262113 shutdown_connections 2024-08-31T12:21:49.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.154+0000 7fe491215640 1 --2- 172.21.15.19:0/1982262113 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe478077a50 0x7fe478079f60 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:49.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.154+0000 7fe491215640 1 --2- 172.21.15.19:0/1982262113 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe48c15c4b0 0x7fe48c0b41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:49.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.154+0000 7fe491215640 1 --2- 172.21.15.19:0/1982262113 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe48c0cacb0 0x7fe48c0b3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:49.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.154+0000 7fe491215640 1 --2- 172.21.15.19:0/1982262113 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe48c0ca7f0 0x7fe48c06f680 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:49.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.154+0000 7fe491215640 1 -- 172.21.15.19:0/1982262113 >> 172.21.15.19:0/1982262113 conn(0x7fe48c0cef20 msgr2=0x7fe48c13f6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:21:49.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.154+0000 7fe491215640 1 -- 172.21.15.19:0/1982262113 shutdown_connections 2024-08-31T12:21:49.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.154+0000 7fe491215640 1 -- 172.21.15.19:0/1982262113 wait complete. 2024-08-31T12:21:49.169 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-08-31T12:21:49.169 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-08-31T12:21:49.169 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-08-31T12:21:49.169 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-31T12:21:49.295 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.293+0000 7ff600a9b640 1 -- 172.21.15.19:0/3971161254 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff5fc0cb2f0 msgr2=0x7ff5fc1669b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:49.295 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.293+0000 7ff600a9b640 1 --2- 172.21.15.19:0/3971161254 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff5fc0cb2f0 0x7ff5fc1669b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff5f4009be0 tx=0x7ff5f402a9b0 comp rx=0 tx=0).stop 2024-08-31T12:21:49.295 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.293+0000 7ff600a9b640 1 -- 172.21.15.19:0/3971161254 shutdown_connections 2024-08-31T12:21:49.295 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.293+0000 7ff600a9b640 1 --2- 172.21.15.19:0/3971161254 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff5fc0cb2f0 0x7ff5fc1669b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:49.295 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.293+0000 7ff600a9b640 1 -- 172.21.15.19:0/3971161254 >> 172.21.15.19:0/3971161254 conn(0x7ff5fc074a90 msgr2=0x7ff5fc068aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:21:49.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.293+0000 7ff600a9b640 1 -- 172.21.15.19:0/3971161254 shutdown_connections 2024-08-31T12:21:49.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.293+0000 7ff600a9b640 1 -- 172.21.15.19:0/3971161254 wait complete. 2024-08-31T12:21:49.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.295+0000 7ff600a9b640 1 Processor -- start 2024-08-31T12:21:49.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.295+0000 7ff600a9b640 1 -- start start 2024-08-31T12:21:49.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.297+0000 7ff600a9b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff5fc0c5c20 0x7ff5fc0cf890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:49.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.297+0000 7ff600a9b640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff5fc0cb2f0 0x7ff5fc157530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:49.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.297+0000 7ff600a9b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff5fc155520 0x7ff5fc1e5e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:49.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.297+0000 7ff600a9b640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7ff5fc0d1140 con 0x7ff5fc155520 2024-08-31T12:21:49.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.297+0000 7ff600a9b640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff5fc0c6750 con 0x7ff5fc0cb2f0 2024-08-31T12:21:49.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.297+0000 7ff600a9b640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7ff5fc15b730 con 0x7ff5fc0c5c20 2024-08-31T12:21:49.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.297+0000 7ff5fad9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff5fc155520 0x7ff5fc1e5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:49.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.297+0000 7ff5fad9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff5fc155520 0x7ff5fc1e5e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:49098/0 (socket says 172.21.15.19:49098) 2024-08-31T12:21:49.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.297+0000 7ff5fad9d640 1 -- 172.21.15.19:0/3329214309 learned_addr learned my addr 172.21.15.19:0/3329214309 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:21:49.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.297+0000 7ff5f9d9b640 1 --2- 172.21.15.19:0/3329214309 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff5fc0cb2f0 0x7ff5fc157530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:49.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.297+0000 7ff5fa59c640 1 --2- 172.21.15.19:0/3329214309 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff5fc0c5c20 0x7ff5fc0cf890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:49.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.297+0000 7ff5fad9d640 1 -- 172.21.15.19:0/3329214309 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff5fc0c5c20 msgr2=0x7ff5fc0cf890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:49.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.297+0000 7ff5fad9d640 1 --2- 172.21.15.19:0/3329214309 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff5fc0c5c20 0x7ff5fc0cf890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:49.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.297+0000 7ff5fad9d640 1 -- 172.21.15.19:0/3329214309 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff5fc0cb2f0 msgr2=0x7ff5fc157530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:49.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.297+0000 7ff5fad9d640 1 --2- 172.21.15.19:0/3329214309 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff5fc0cb2f0 0x7ff5fc157530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:49.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.297+0000 7ff5fad9d640 1 -- 172.21.15.19:0/3329214309 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5fc0cfdd0 con 0x7ff5fc155520 2024-08-31T12:21:49.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.297+0000 7ff5fad9d640 1 --2- 172.21.15.19:0/3329214309 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff5fc155520 0x7ff5fc1e5e00 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff5f4009d00 tx=0x7ff5f4030e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:49.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.298+0000 7ff5eaffd640 1 -- 172.21.15.19:0/3329214309 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f402f400 con 0x7ff5fc155520 2024-08-31T12:21:49.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.298+0000 7ff5eaffd640 1 -- 172.21.15.19:0/3329214309 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff5f4004540 con 0x7ff5fc155520 2024-08-31T12:21:49.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.298+0000 7ff600a9b640 1 -- 172.21.15.19:0/3329214309 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5fc0d0770 con 0x7ff5fc155520 2024-08-31T12:21:49.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.298+0000 7ff5eaffd640 1 -- 172.21.15.19:0/3329214309 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f4003e60 con 0x7ff5fc155520 2024-08-31T12:21:49.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.298+0000 7ff600a9b640 1 -- 172.21.15.19:0/3329214309 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5fc0d0cf0 con 0x7ff5fc155520 2024-08-31T12:21:49.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.299+0000 7ff600a9b640 1 --2- 172.21.15.19:0/3329214309 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff5fc14e7e0 0x7ff5fc0cecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:49.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.299+0000 7ff600a9b640 1 -- 172.21.15.19:0/3329214309 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7ff5fc158c60 con 0x7ff5fc14e7e0 2024-08-31T12:21:49.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.299+0000 7ff5fa59c640 1 --2- 172.21.15.19:0/3329214309 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff5fc14e7e0 0x7ff5fc0cecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:49.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.299+0000 7ff5fa59c640 1 -- 172.21.15.19:0/3329214309 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff5fc0cb800 con 0x7ff5fc14e7e0 2024-08-31T12:21:49.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.299+0000 7ff5fa59c640 1 --2- 172.21.15.19:0/3329214309 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff5fc14e7e0 0x7ff5fc0cecb0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec00f770 tx=0x7ff5ec00fc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:49.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.299+0000 7ff5eaffd640 1 -- 172.21.15.19:0/3329214309 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec0050d0 con 0x7ff5fc14e7e0 2024-08-31T12:21:49.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.300+0000 7ff5eaffd640 1 -- 172.21.15.19:0/3329214309 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7ff5f40413f0 con 0x7ff5fc155520 2024-08-31T12:21:49.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.300+0000 7ff5eaffd640 1 --2- 172.21.15.19:0/3329214309 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff5dc077980 0x7ff5dc079e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:49.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.300+0000 7ff5eaffd640 1 -- 172.21.15.19:0/3329214309 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7ff5f40ba890 con 0x7ff5fc155520 2024-08-31T12:21:49.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.300+0000 7ff5f9d9b640 1 --2- 172.21.15.19:0/3329214309 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff5dc077980 0x7ff5dc079e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:49.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.301+0000 7ff5eaffd640 1 -- 172.21.15.19:0/3329214309 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff5ec005270 con 0x7ff5fc14e7e0 2024-08-31T12:21:49.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.301+0000 7ff5eaffd640 1 -- 172.21.15.19:0/3329214309 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff5fc14e7e0 msgr2=0x7ff5fc0cecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:49.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.301+0000 7ff5eaffd640 1 --2- 172.21.15.19:0/3329214309 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff5fc14e7e0 0x7ff5fc0cecb0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec00f770 tx=0x7ff5ec00fc30 comp rx=0 tx=0).stop 2024-08-31T12:21:49.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.301+0000 7ff5eaffd640 1 -- 172.21.15.19:0/3329214309 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff5fc14e7e0 msgr2=0x7ff5fc0cecb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:21:49.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.301+0000 7ff5eaffd640 1 --2- 172.21.15.19:0/3329214309 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff5fc14e7e0 0x7ff5fc0cecb0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:49.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.301+0000 7ff5f9d9b640 1 --2- 172.21.15.19:0/3329214309 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff5dc077980 0x7ff5dc079e90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0000c00 tx=0x7ff5f0007eb0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:49.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.308+0000 7ff600a9b640 1 --2- 172.21.15.19:0/3329214309 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff5fc0cf1f0 0x7ff5fc14b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:49.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.308+0000 7ff600a9b640 1 -- 172.21.15.19:0/3329214309 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7ff5fc1529f0 con 0x7ff5fc0cf1f0 2024-08-31T12:21:49.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.308+0000 7ff5fa59c640 1 --2- 172.21.15.19:0/3329214309 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff5fc0cf1f0 0x7ff5fc14b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:49.311 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.309+0000 7ff5fa59c640 1 -- 172.21.15.19:0/3329214309 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff5fc0b3bb0 con 0x7ff5fc0cf1f0 2024-08-31T12:21:49.311 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.309+0000 7ff5fa59c640 1 --2- 172.21.15.19:0/3329214309 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff5fc0cf1f0 0x7ff5fc14b070 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec016000 tx=0x7ff5ec000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:49.311 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.309+0000 7ff5eaffd640 1 -- 172.21.15.19:0/3329214309 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec00de10 con 0x7ff5fc0cf1f0 2024-08-31T12:21:49.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.310+0000 7ff5eaffd640 1 -- 172.21.15.19:0/3329214309 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5045 (secure 0 0 0) 0x7ff5ec00de10 con 0x7ff5fc0cf1f0 2024-08-31T12:21:49.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.310+0000 7ff5eaffd640 1 -- 172.21.15.19:0/3329214309 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff5fc0cf1f0 msgr2=0x7ff5fc14b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:49.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.310+0000 7ff5eaffd640 1 --2- 172.21.15.19:0/3329214309 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff5fc0cf1f0 0x7ff5fc14b070 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec016000 tx=0x7ff5ec000fe0 comp rx=0 tx=0).stop 2024-08-31T12:21:49.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.310+0000 7ff5eaffd640 1 -- 172.21.15.19:0/3329214309 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff5fc0cf1f0 msgr2=0x7ff5fc14b070 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:21:49.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.310+0000 7ff5eaffd640 1 --2- 172.21.15.19:0/3329214309 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff5fc0cf1f0 0x7ff5fc14b070 secure :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec016000 tx=0x7ff5ec000fe0 comp rx=0 tx=0).stop 2024-08-31T12:21:49.313 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:21:49.313 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:21:49.313 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:21:49.313 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "leader", 2024-08-31T12:21:49.313 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 44, 2024-08-31T12:21:49.313 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 289301, 2024-08-31T12:21:49.313 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:21:49.313 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:21:49.313 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:21:49.313 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:21:49.313 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:21:49.314 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 13, 2024-08-31T12:21:49.314 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:21:49.314 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:21:49.314 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:21:49.314 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:21:49.314 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:21:49.314 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:21:49.314 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:21:49.314 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:21:49.314 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:21:49.314 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:21:49.314 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:21:49.314 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:21:49.315 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:21:49.315 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:21:49.315 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:21:49.315 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:21:49.315 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:21:49.315 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:21:49.315 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:21:49.315 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:21:49.315 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:21:49.315 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:21:49.315 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:21:49.315 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:21:49.315 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:21:49.316 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:21:49.316 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:21:49.316 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:21:49.316 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:49.316 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:49.316 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:21:49.316 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:21:49.316 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:21:49.316 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:21:49.316 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:21:49.316 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:21:49.316 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:21:49.316 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:21:49.317 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:21:49.317 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:21:49.317 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:21:49.317 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:21:49.317 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:21:49.317 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:21:49.317 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:21:49.317 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:21:49.317 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:21:49.317 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:21:49.317 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:21:49.317 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:21:49.317 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:21:49.318 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:21:49.318 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:21:49.318 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:21:49.318 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:21:49.318 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:21:49.318 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:21:49.318 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:21:49.318 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:21:49.318 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:21:49.318 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:49.318 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:21:49.318 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:49.319 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:21:49.319 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:21:49.319 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:21:49.319 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:21:49.319 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:49.319 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:21:49.319 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:21:49.319 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:49.319 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:49.319 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:49.319 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:21:49.319 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:21:49.319 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:49.320 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:49.320 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:49.320 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:49.320 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:21:49.320 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:21:49.320 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:21:49.320 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:21:49.320 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:21:49.320 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:49.320 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:49.320 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:21:49.320 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:21:49.320 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:21:49.321 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:21:49.321 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:49.321 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:21:49.321 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:21:49.321 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:49.321 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:49.321 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:49.321 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:21:49.321 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:21:49.321 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:49.321 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:49.321 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:49.321 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:49.322 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:21:49.322 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:21:49.322 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:21:49.322 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:21:49.322 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:21:49.322 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:49.322 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:49.322 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:21:49.322 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:21:49.322 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:21:49.322 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:21:49.322 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:49.322 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:21:49.323 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:21:49.323 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:49.323 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:49.323 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:49.323 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:21:49.323 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:21:49.323 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:49.323 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:49.323 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:49.323 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:49.323 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:21:49.323 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:21:49.323 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:21:49.324 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:21:49.324 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:21:49.324 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:49.324 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:49.324 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:49.324 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:21:49.324 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:21:49.324 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:49.324 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:21:49.324 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:21:49.324 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:21:49.324 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:49.324 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:21:49.325 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:21:49.325 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:49.325 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:21:49.325 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:21:49.325 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 6 2024-08-31T12:21:49.325 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:49.325 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:21:49.325 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:21:49.325 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:49.325 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:21:49.325 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:21:49.325 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 8 2024-08-31T12:21:49.325 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:49.326 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:21:49.326 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:21:49.326 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:49.326 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:21:49.326 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:21:49.326 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 6 2024-08-31T12:21:49.326 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:49.326 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:21:49.326 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:21:49.326 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:49.326 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:21:49.326 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:21:49.326 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:21:49.327 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:49.327 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:49.327 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:49.327 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:21:49.327 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:21:49.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.310+0000 7ff600a9b640 1 -- 172.21.15.19:0/3329214309 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff5dc077980 msgr2=0x7ff5dc079e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:49.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.310+0000 7ff600a9b640 1 --2- 172.21.15.19:0/3329214309 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff5dc077980 0x7ff5dc079e90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0000c00 tx=0x7ff5f0007eb0 comp rx=0 tx=0).stop 2024-08-31T12:21:49.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.310+0000 7ff600a9b640 1 -- 172.21.15.19:0/3329214309 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff5fc155520 msgr2=0x7ff5fc1e5e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:49.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.310+0000 7ff5fad9d640 1 -- 172.21.15.19:0/3329214309 reap_dead start 2024-08-31T12:21:49.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.310+0000 7ff600a9b640 1 --2- 172.21.15.19:0/3329214309 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff5fc155520 0x7ff5fc1e5e00 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff5f4009d00 tx=0x7ff5f4030e70 comp rx=0 tx=0).stop 2024-08-31T12:21:49.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.310+0000 7ff600a9b640 1 -- 172.21.15.19:0/3329214309 shutdown_connections 2024-08-31T12:21:49.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.310+0000 7ff600a9b640 1 --2- 172.21.15.19:0/3329214309 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff5fc155520 0x7ff5fc1e5e00 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:49.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.310+0000 7ff600a9b640 1 -- 172.21.15.19:0/3329214309 >> 172.21.15.19:0/3329214309 conn(0x7ff5fc074a90 msgr2=0x7ff5fc06dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:21:49.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.310+0000 7ff600a9b640 1 -- 172.21.15.19:0/3329214309 shutdown_connections 2024-08-31T12:21:49.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.310+0000 7ff600a9b640 1 -- 172.21.15.19:0/3329214309 wait complete. 2024-08-31T12:21:49.329 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:21:49.329 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-08-31T12:21:49.329 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-31T12:21:49.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.453+0000 7f0ae7b70640 1 -- 172.21.15.19:0/1532281689 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0ae00b2d80 msgr2=0x7f0ae00c8e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:49.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.453+0000 7f0ae7b70640 1 --2- 172.21.15.19:0/1532281689 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0ae00b2d80 0x7f0ae00c8e60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0adc009860 tx=0x7f0adc02a5c0 comp rx=0 tx=0).stop 2024-08-31T12:21:49.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.453+0000 7f0ae7b70640 1 -- 172.21.15.19:0/1532281689 shutdown_connections 2024-08-31T12:21:49.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.453+0000 7f0ae7b70640 1 --2- 172.21.15.19:0/1532281689 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0ae00b2d80 0x7f0ae00c8e60 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:49.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.453+0000 7f0ae7b70640 1 -- 172.21.15.19:0/1532281689 >> 172.21.15.19:0/1532281689 conn(0x7f0ae006d110 msgr2=0x7f0ae00cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:21:49.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.454+0000 7f0ae7b70640 1 -- 172.21.15.19:0/1532281689 shutdown_connections 2024-08-31T12:21:49.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.454+0000 7f0ae7b70640 1 -- 172.21.15.19:0/1532281689 wait complete. 2024-08-31T12:21:49.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.456+0000 7f0ae7b70640 1 Processor -- start 2024-08-31T12:21:49.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.456+0000 7f0ae7b70640 1 -- start start 2024-08-31T12:21:49.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.457+0000 7f0ae7b70640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0ae00b2d80 0x7f0ae0155e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:49.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.457+0000 7f0ae7b70640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0ae0153170 0x7f0ae01eee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:49.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.457+0000 7f0ae7b70640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ae0153580 0x7f0ae014b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:49.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.457+0000 7f0ae7b70640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae00690a0 con 0x7f0ae00b2d80 2024-08-31T12:21:49.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.457+0000 7f0ae7b70640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae00cd8a0 con 0x7f0ae0153580 2024-08-31T12:21:49.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.457+0000 7f0ae7b70640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f0ae00711e0 con 0x7f0ae0153170 2024-08-31T12:21:49.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.457+0000 7f0ae590c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0ae00b2d80 0x7f0ae0155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:49.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.457+0000 7f0ae590c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0ae00b2d80 0x7f0ae0155e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:49116/0 (socket says 172.21.15.19:49116) 2024-08-31T12:21:49.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.457+0000 7f0ae590c640 1 -- 172.21.15.19:0/2241068268 learned_addr learned my addr 172.21.15.19:0/2241068268 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:21:49.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.457+0000 7f0ae510b640 1 --2- 172.21.15.19:0/2241068268 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0ae0153170 0x7f0ae01eee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:49.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.457+0000 7f0ae610d640 1 --2- 172.21.15.19:0/2241068268 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ae0153580 0x7f0ae014b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:49.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.458+0000 7f0ae590c640 1 -- 172.21.15.19:0/2241068268 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0ae0153170 msgr2=0x7f0ae01eee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:49.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.458+0000 7f0ae590c640 1 --2- 172.21.15.19:0/2241068268 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0ae0153170 0x7f0ae01eee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:49.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.458+0000 7f0ae590c640 1 -- 172.21.15.19:0/2241068268 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ae0153580 msgr2=0x7f0ae014b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:49.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.458+0000 7f0ae590c640 1 --2- 172.21.15.19:0/2241068268 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ae0153580 0x7f0ae014b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:49.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.458+0000 7f0ae590c640 1 -- 172.21.15.19:0/2241068268 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ae0154c90 con 0x7f0ae00b2d80 2024-08-31T12:21:49.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.458+0000 7f0ae590c640 1 --2- 172.21.15.19:0/2241068268 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0ae00b2d80 0x7f0ae0155e50 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0adc02abb0 tx=0x7f0adc003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:49.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.458+0000 7f0ad67fc640 1 -- 172.21.15.19:0/2241068268 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0adc005c90 con 0x7f0ae00b2d80 2024-08-31T12:21:49.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.458+0000 7f0ad67fc640 1 -- 172.21.15.19:0/2241068268 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0adc043070 con 0x7f0ae00b2d80 2024-08-31T12:21:49.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.458+0000 7f0ad67fc640 1 -- 172.21.15.19:0/2241068268 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0adc031b20 con 0x7f0ae00b2d80 2024-08-31T12:21:49.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.458+0000 7f0ae7b70640 1 -- 172.21.15.19:0/2241068268 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ae00cd2c0 con 0x7f0ae00b2d80 2024-08-31T12:21:49.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.458+0000 7f0ae7b70640 1 -- 172.21.15.19:0/2241068268 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ae0154f50 con 0x7f0ae00b2d80 2024-08-31T12:21:49.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.459+0000 7f0ae7b70640 1 --2- 172.21.15.19:0/2241068268 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ae01ef3c0 0x7f0ae01f2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:49.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.459+0000 7f0ae7b70640 1 -- 172.21.15.19:0/2241068268 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae006a020 con 0x7f0ae01ef3c0 2024-08-31T12:21:49.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.459+0000 7f0ae510b640 1 --2- 172.21.15.19:0/2241068268 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ae01ef3c0 0x7f0ae01f2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:49.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.460+0000 7f0ae510b640 1 -- 172.21.15.19:0/2241068268 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0ae0159d80 con 0x7f0ae01ef3c0 2024-08-31T12:21:49.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.460+0000 7f0ae510b640 1 --2- 172.21.15.19:0/2241068268 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ae01ef3c0 0x7f0ae01f2470 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0ad00046b0 tx=0x7f0ad000ff70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:49.464 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.460+0000 7f0ad67fc640 1 -- 172.21.15.19:0/2241068268 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad0018020 con 0x7f0ae01ef3c0 2024-08-31T12:21:49.464 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.460+0000 7f0ad67fc640 1 -- 172.21.15.19:0/2241068268 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f0adc031cc0 con 0x7f0ae00b2d80 2024-08-31T12:21:49.464 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.461+0000 7f0ad67fc640 1 --2- 172.21.15.19:0/2241068268 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0ac4077950 0x7f0ac4079e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:49.464 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.461+0000 7f0ad67fc640 1 -- 172.21.15.19:0/2241068268 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f0adc0ba420 con 0x7f0ae00b2d80 2024-08-31T12:21:49.464 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.461+0000 7f0ad67fc640 1 -- 172.21.15.19:0/2241068268 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0ad0007e00 con 0x7f0ae01ef3c0 2024-08-31T12:21:49.464 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.461+0000 7f0ad67fc640 1 -- 172.21.15.19:0/2241068268 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ae01ef3c0 msgr2=0x7f0ae01f2470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:49.464 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.461+0000 7f0ad67fc640 1 --2- 172.21.15.19:0/2241068268 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ae01ef3c0 0x7f0ae01f2470 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0ad00046b0 tx=0x7f0ad000ff70 comp rx=0 tx=0).stop 2024-08-31T12:21:49.464 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.461+0000 7f0ad67fc640 1 -- 172.21.15.19:0/2241068268 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ae01ef3c0 msgr2=0x7f0ae01f2470 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:21:49.464 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.461+0000 7f0ad67fc640 1 --2- 172.21.15.19:0/2241068268 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ae01ef3c0 0x7f0ae01f2470 secure :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0ad00046b0 tx=0x7f0ad000ff70 comp rx=0 tx=0).stop 2024-08-31T12:21:49.464 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.461+0000 7f0ae610d640 1 --2- 172.21.15.19:0/2241068268 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0ac4077950 0x7f0ac4079e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:49.465 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.462+0000 7f0ae610d640 1 --2- 172.21.15.19:0/2241068268 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0ac4077950 0x7f0ac4079e60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0ae0159c00 tx=0x7f0ad8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:49.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.470+0000 7f0ae7b70640 1 --2- 172.21.15.19:0/2241068268 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ae01f29b0 0x7f0ae01f2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:49.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.470+0000 7f0ae7b70640 1 -- 172.21.15.19:0/2241068268 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae01f3300 con 0x7f0ae01f29b0 2024-08-31T12:21:49.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.470+0000 7f0ae510b640 1 --2- 172.21.15.19:0/2241068268 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ae01f29b0 0x7f0ae01f2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:49.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.471+0000 7f0ae510b640 1 -- 172.21.15.19:0/2241068268 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0ae00c8bf0 con 0x7f0ae01f29b0 2024-08-31T12:21:49.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.471+0000 7f0ae510b640 1 --2- 172.21.15.19:0/2241068268 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ae01f29b0 0x7f0ae01f2dc0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0ad0016000 tx=0x7f0ad0000ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:49.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.471+0000 7f0ad67fc640 1 -- 172.21.15.19:0/2241068268 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad0014670 con 0x7f0ae01f29b0 2024-08-31T12:21:49.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.471+0000 7f0ad67fc640 1 -- 172.21.15.19:0/2241068268 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4541 (secure 0 0 0) 0x7f0ad0014670 con 0x7f0ae01f29b0 2024-08-31T12:21:49.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.471+0000 7f0ad67fc640 1 -- 172.21.15.19:0/2241068268 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ae01f29b0 msgr2=0x7f0ae01f2dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:49.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.471+0000 7f0ad67fc640 1 --2- 172.21.15.19:0/2241068268 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ae01f29b0 0x7f0ae01f2dc0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0ad0016000 tx=0x7f0ad0000ed0 comp rx=0 tx=0).stop 2024-08-31T12:21:49.474 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.471+0000 7f0ad67fc640 1 -- 172.21.15.19:0/2241068268 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ae01f29b0 msgr2=0x7f0ae01f2dc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:21:49.474 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.471+0000 7f0ad67fc640 1 --2- 172.21.15.19:0/2241068268 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ae01f29b0 0x7f0ae01f2dc0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:49.474 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:21:49.474 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:21:49.474 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:21:49.474 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:21:49.474 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 44, 2024-08-31T12:21:49.474 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 18945, 2024-08-31T12:21:49.474 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:21:49.475 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:21:49.475 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:21:49.475 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:21:49.475 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:21:49.475 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 13, 2024-08-31T12:21:49.475 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:21:49.475 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:21:49.475 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:21:49.475 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:21:49.475 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:21:49.475 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:21:49.475 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:21:49.475 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:21:49.475 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:21:49.476 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:21:49.476 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:21:49.476 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:21:49.476 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:21:49.476 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:21:49.476 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:21:49.476 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:21:49.476 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:21:49.476 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:21:49.476 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:21:49.476 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:21:49.476 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:21:49.476 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:21:49.476 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:21:49.477 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:21:49.477 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:21:49.477 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:21:49.477 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:21:49.477 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:21:49.477 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:49.477 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:49.477 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:21:49.477 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:21:49.477 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:21:49.477 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:21:49.477 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:21:49.478 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:21:49.478 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:21:49.478 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:21:49.478 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:21:49.478 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:21:49.478 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:21:49.478 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:21:49.478 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:21:49.478 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:21:49.478 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:21:49.478 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:21:49.478 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:21:49.478 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:21:49.478 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:21:49.479 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:21:49.479 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:21:49.479 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:21:49.479 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:21:49.479 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:21:49.479 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:21:49.479 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:21:49.479 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:21:49.479 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:21:49.479 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:21:49.479 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:21:49.479 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:49.479 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:21:49.480 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:49.480 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:21:49.480 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:21:49.480 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:21:49.480 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:21:49.480 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:49.480 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:21:49.480 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:21:49.480 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:49.480 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:49.480 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:49.480 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:21:49.480 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:21:49.481 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:49.481 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:49.481 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:49.481 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:49.481 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:21:49.481 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:21:49.481 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:21:49.481 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:21:49.481 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:21:49.481 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:49.481 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:49.481 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:21:49.481 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:21:49.481 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:21:49.482 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:21:49.482 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:49.482 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:21:49.482 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:21:49.482 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:49.482 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:49.482 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:49.482 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:21:49.482 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:21:49.482 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:49.482 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:49.482 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:49.482 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:49.483 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:21:49.483 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:21:49.483 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:21:49.483 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:21:49.483 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:21:49.483 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:49.483 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:49.483 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:21:49.483 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:21:49.483 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:21:49.483 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:21:49.483 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:49.483 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:21:49.483 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:21:49.484 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:49.484 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:49.484 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:49.484 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:21:49.484 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:21:49.484 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:49.484 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:49.484 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:49.484 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:49.484 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:21:49.484 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:21:49.484 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:21:49.484 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:21:49.485 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:21:49.485 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:49.485 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:49.485 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:49.485 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:21:49.485 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:21:49.485 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:49.485 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:21:49.485 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:21:49.485 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:21:49.485 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:49.485 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:21:49.485 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:21:49.485 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:49.486 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:21:49.486 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:21:49.486 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:21:49.486 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:49.486 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:49.486 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:49.486 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:21:49.486 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:21:49.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.472+0000 7f0ae7b70640 1 -- 172.21.15.19:0/2241068268 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0ac4077950 msgr2=0x7f0ac4079e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:49.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.472+0000 7f0ae7b70640 1 --2- 172.21.15.19:0/2241068268 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0ac4077950 0x7f0ac4079e60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0ae0159c00 tx=0x7f0ad8009040 comp rx=0 tx=0).stop 2024-08-31T12:21:49.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.472+0000 7f0ae7b70640 1 -- 172.21.15.19:0/2241068268 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0ae00b2d80 msgr2=0x7f0ae0155e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:49.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.472+0000 7f0ae7b70640 1 --2- 172.21.15.19:0/2241068268 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0ae00b2d80 0x7f0ae0155e50 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0adc02abb0 tx=0x7f0adc003b90 comp rx=0 tx=0).stop 2024-08-31T12:21:49.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.472+0000 7f0ae610d640 1 -- 172.21.15.19:0/2241068268 reap_dead start 2024-08-31T12:21:49.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.472+0000 7f0ae7b70640 1 -- 172.21.15.19:0/2241068268 shutdown_connections 2024-08-31T12:21:49.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.472+0000 7f0ae7b70640 1 --2- 172.21.15.19:0/2241068268 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0ae00b2d80 0x7f0ae0155e50 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:49.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.472+0000 7f0ae7b70640 1 -- 172.21.15.19:0/2241068268 >> 172.21.15.19:0/2241068268 conn(0x7f0ae006d110 msgr2=0x7f0ae00635a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:21:49.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.472+0000 7f0ae7b70640 1 -- 172.21.15.19:0/2241068268 shutdown_connections 2024-08-31T12:21:49.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.472+0000 7f0ae7b70640 1 -- 172.21.15.19:0/2241068268 wait complete. 2024-08-31T12:21:49.488 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:21:49.488 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-08-31T12:21:49.489 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-31T12:21:49.606 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.604+0000 7f06ba03f640 1 -- 172.21.15.19:0/4245090278 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f06b406e0e0 msgr2=0x7f06b40c9850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:49.606 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.604+0000 7f06ba03f640 1 --2- 172.21.15.19:0/4245090278 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f06b406e0e0 0x7f06b40c9850 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f06b415b0f0 tx=0x7f06a802a5d0 comp rx=0 tx=0).stop 2024-08-31T12:21:49.606 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.604+0000 7f06ba03f640 1 -- 172.21.15.19:0/4245090278 shutdown_connections 2024-08-31T12:21:49.606 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.604+0000 7f06ba03f640 1 --2- 172.21.15.19:0/4245090278 >> v2:172.21.15.43:3300/0 conn(0x7f06b4155300 0x7f06b406d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:49.606 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.604+0000 7f06ba03f640 1 --2- 172.21.15.19:0/4245090278 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f06b406e0e0 0x7f06b40c9850 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:49.606 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.604+0000 7f06ba03f640 1 -- 172.21.15.19:0/4245090278 >> 172.21.15.19:0/4245090278 conn(0x7f06b40690d0 msgr2=0x7f06b40d0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:21:49.606 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.604+0000 7f06ba03f640 1 -- 172.21.15.19:0/4245090278 shutdown_connections 2024-08-31T12:21:49.606 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.604+0000 7f06ba03f640 1 -- 172.21.15.19:0/4245090278 wait complete. 2024-08-31T12:21:49.608 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.606+0000 7f06ba03f640 1 Processor -- start 2024-08-31T12:21:49.608 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.606+0000 7f06ba03f640 1 -- start start 2024-08-31T12:21:49.611 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.608+0000 7f06ba03f640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f06b406e0e0 0x7f06b40cf4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:49.611 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.608+0000 7f06ba03f640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06b40c8a40 0x7f06b40cd320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:49.611 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.608+0000 7f06ba03f640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f06b4155300 0x7f06b40c24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:49.611 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.608+0000 7f06ba03f640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f06b40c8330 con 0x7f06b406e0e0 2024-08-31T12:21:49.612 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.608+0000 7f06ba03f640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f06b415b250 con 0x7f06b40c8a40 2024-08-31T12:21:49.612 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.608+0000 7f06ba03f640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f06b40c8610 con 0x7f06b4155300 2024-08-31T12:21:49.612 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.608+0000 7f06b37fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f06b406e0e0 0x7f06b40cf4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:49.612 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.608+0000 7f06b37fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f06b406e0e0 0x7f06b40cf4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:49136/0 (socket says 172.21.15.19:49136) 2024-08-31T12:21:49.612 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.608+0000 7f06b37fe640 1 -- 172.21.15.19:0/2502909414 learned_addr learned my addr 172.21.15.19:0/2502909414 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:21:49.612 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.608+0000 7f06b3fff640 1 --2- 172.21.15.19:0/2502909414 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f06b4155300 0x7f06b40c24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:49.612 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.608+0000 7f06b2ffd640 1 --2- 172.21.15.19:0/2502909414 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06b40c8a40 0x7f06b40cd320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:49.612 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.608+0000 7f06b37fe640 1 -- 172.21.15.19:0/2502909414 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f06b4155300 msgr2=0x7f06b40c24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:49.612 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.608+0000 7f06b37fe640 1 --2- 172.21.15.19:0/2502909414 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f06b4155300 0x7f06b40c24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:49.612 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.608+0000 7f06b37fe640 1 -- 172.21.15.19:0/2502909414 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06b40c8a40 msgr2=0x7f06b40cd320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:49.612 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.609+0000 7f06b37fe640 1 --2- 172.21.15.19:0/2502909414 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06b40c8a40 0x7f06b40cd320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:49.612 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.609+0000 7f06b37fe640 1 -- 172.21.15.19:0/2502909414 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06b40c4670 con 0x7f06b406e0e0 2024-08-31T12:21:49.612 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.609+0000 7f06b37fe640 1 --2- 172.21.15.19:0/2502909414 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f06b406e0e0 0x7f06b40cf4c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f06a400b420 tx=0x7f06a400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:49.613 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.609+0000 7f069bfff640 1 -- 172.21.15.19:0/2502909414 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a4011830 con 0x7f06b406e0e0 2024-08-31T12:21:49.613 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.609+0000 7f069bfff640 1 -- 172.21.15.19:0/2502909414 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f06a4016070 con 0x7f06b406e0e0 2024-08-31T12:21:49.613 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.609+0000 7f06ba03f640 1 -- 172.21.15.19:0/2502909414 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06b40cfa00 con 0x7f06b406e0e0 2024-08-31T12:21:49.613 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.609+0000 7f069bfff640 1 -- 172.21.15.19:0/2502909414 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a4011d60 con 0x7f06b406e0e0 2024-08-31T12:21:49.613 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.609+0000 7f06ba03f640 1 -- 172.21.15.19:0/2502909414 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06b41f5880 con 0x7f06b406e0e0 2024-08-31T12:21:49.615 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.610+0000 7f06ba03f640 1 --2- 172.21.15.19:0/2502909414 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f06b40cec00 0x7f06b40c29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:49.615 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.610+0000 7f06ba03f640 1 -- 172.21.15.19:0/2502909414 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f06b41f5b60 con 0x7f06b40cec00 2024-08-31T12:21:49.615 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.610+0000 7f06b2ffd640 1 --2- 172.21.15.19:0/2502909414 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f06b40cec00 0x7f06b40c29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:49.615 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.611+0000 7f06b2ffd640 1 -- 172.21.15.19:0/2502909414 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f06b40c4840 con 0x7f06b40cec00 2024-08-31T12:21:49.615 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.611+0000 7f06b2ffd640 1 --2- 172.21.15.19:0/2502909414 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f06b40cec00 0x7f06b40c29e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f06b40d0ea0 tx=0x7f06a8004440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:49.615 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.611+0000 7f069bfff640 1 -- 172.21.15.19:0/2502909414 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f06a401b460 con 0x7f06b406e0e0 2024-08-31T12:21:49.615 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.611+0000 7f069bfff640 1 --2- 172.21.15.19:0/2502909414 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f06a0077a50 0x7f06a0079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:49.615 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.612+0000 7f069bfff640 1 -- 172.21.15.19:0/2502909414 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f06a4099140 con 0x7f06b406e0e0 2024-08-31T12:21:49.615 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.612+0000 7f06b3fff640 1 --2- 172.21.15.19:0/2502909414 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f06a0077a50 0x7f06a0079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:49.616 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.612+0000 7f069bfff640 1 -- 172.21.15.19:0/2502909414 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a8040070 con 0x7f06b40cec00 2024-08-31T12:21:49.616 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.612+0000 7f069bfff640 1 -- 172.21.15.19:0/2502909414 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f06a8003ac0 con 0x7f06b40cec00 2024-08-31T12:21:49.616 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.612+0000 7f069bfff640 1 -- 172.21.15.19:0/2502909414 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f06b40cec00 msgr2=0x7f06b40c29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:49.616 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.612+0000 7f069bfff640 1 --2- 172.21.15.19:0/2502909414 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f06b40cec00 0x7f06b40c29e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f06b40d0ea0 tx=0x7f06a8004440 comp rx=0 tx=0).stop 2024-08-31T12:21:49.616 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.612+0000 7f069bfff640 1 -- 172.21.15.19:0/2502909414 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f06b40cec00 msgr2=0x7f06b40c29e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:21:49.616 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.612+0000 7f069bfff640 1 --2- 172.21.15.19:0/2502909414 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f06b40cec00 0x7f06b40c29e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:49.616 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.612+0000 7f06b3fff640 1 --2- 172.21.15.19:0/2502909414 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f06a0077a50 0x7f06a0079f60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f06ac005fd0 tx=0x7f06ac009560 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:49.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.618+0000 7f06ba03f640 1 --2- 172.21.15.19:0/2502909414 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f06b40c6b50 0x7f06b41e53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:21:49.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.618+0000 7f06ba03f640 1 -- 172.21.15.19:0/2502909414 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f06b41f5630 con 0x7f06b40c6b50 2024-08-31T12:21:49.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.619+0000 7f06b2ffd640 1 --2- 172.21.15.19:0/2502909414 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f06b40c6b50 0x7f06b41e53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:21:49.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.619+0000 7f06b2ffd640 1 -- 172.21.15.19:0/2502909414 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f06b40c9850 con 0x7f06b40c6b50 2024-08-31T12:21:49.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.619+0000 7f06b2ffd640 1 --2- 172.21.15.19:0/2502909414 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f06b40c6b50 0x7f06b41e53f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f06a8000c00 tx=0x7f06a8009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:21:49.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.619+0000 7f069bfff640 1 -- 172.21.15.19:0/2502909414 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a8035040 con 0x7f06b40c6b50 2024-08-31T12:21:49.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.620+0000 7f069bfff640 1 -- 172.21.15.19:0/2502909414 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4541 (secure 0 0 0) 0x7f06a8044a80 con 0x7f06b40c6b50 2024-08-31T12:21:49.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.620+0000 7f069bfff640 1 -- 172.21.15.19:0/2502909414 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f06b40c6b50 msgr2=0x7f06b41e53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:49.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.620+0000 7f069bfff640 1 --2- 172.21.15.19:0/2502909414 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f06b40c6b50 0x7f06b41e53f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f06a8000c00 tx=0x7f06a8009260 comp rx=0 tx=0).stop 2024-08-31T12:21:49.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.620+0000 7f069bfff640 1 -- 172.21.15.19:0/2502909414 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f06b40c6b50 msgr2=0x7f06b41e53f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:21:49.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.620+0000 7f069bfff640 1 --2- 172.21.15.19:0/2502909414 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f06b40c6b50 0x7f06b41e53f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:21:49.622 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:21:49.623 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:21:49.623 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:21:49.623 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:21:49.623 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 44, 2024-08-31T12:21:49.623 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 71125, 2024-08-31T12:21:49.623 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:21:49.623 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:21:49.623 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:21:49.623 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:21:49.623 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:21:49.623 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 13, 2024-08-31T12:21:49.623 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:21:49.624 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:21:49.624 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:21:49.624 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:21:49.624 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:21:49.624 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:21:49.624 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:21:49.624 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:21:49.624 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:21:49.624 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:21:49.624 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:21:49.624 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:21:49.624 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:21:49.625 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:21:49.625 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:21:49.625 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:21:49.625 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:21:49.625 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:21:49.625 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:21:49.625 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:21:49.625 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:21:49.625 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:21:49.625 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:21:49.625 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:21:49.625 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:21:49.625 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:21:49.626 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:21:49.626 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:21:49.626 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:49.626 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:49.626 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:21:49.626 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:21:49.626 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:21:49.626 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:21:49.626 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:21:49.626 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:21:49.626 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:21:49.626 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:21:49.626 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:21:49.626 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:21:49.627 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:21:49.627 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:21:49.627 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:21:49.627 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:21:49.627 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:21:49.627 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:21:49.627 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:21:49.627 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:21:49.627 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:21:49.627 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:21:49.627 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:21:49.627 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:21:49.627 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:21:49.628 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:21:49.628 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:21:49.628 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:21:49.628 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:21:49.628 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:21:49.628 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:21:49.628 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:21:49.628 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:49.628 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:21:49.628 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:49.628 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:21:49.628 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:21:49.628 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:21:49.629 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:21:49.629 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:49.629 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:21:49.629 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:21:49.629 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:49.629 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:49.629 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:49.629 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:21:49.629 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:21:49.629 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:49.629 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:49.629 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:49.630 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:49.630 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:21:49.630 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:21:49.630 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:21:49.630 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:21:49.630 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:21:49.630 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:49.630 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:49.630 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:21:49.630 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:21:49.630 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:21:49.630 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:21:49.630 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:49.631 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:21:49.631 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:21:49.631 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:49.631 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:49.631 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:49.631 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:21:49.631 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:21:49.631 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:49.631 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:49.631 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:49.631 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:49.631 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:21:49.631 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:21:49.632 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:21:49.632 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:21:49.632 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:21:49.632 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:49.632 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:49.632 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:21:49.632 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:21:49.632 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:21:49.632 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:21:49.632 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:49.632 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:21:49.632 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:21:49.633 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:49.633 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:49.633 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:49.633 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:21:49.633 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:21:49.633 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:21:49.633 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:49.633 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:49.633 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:49.633 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:21:49.633 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:21:49.633 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:21:49.633 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:21:49.634 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:21:49.634 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:49.634 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:49.634 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:49.634 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:21:49.634 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:21:49.634 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:49.634 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:21:49.634 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:21:49.634 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:21:49.634 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:49.635 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:21:49.635 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:21:49.635 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:21:49.635 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:21:49.635 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:21:49.635 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:21:49.635 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:21:49.635 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:21:49.635 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:21:49.635 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:21:49.635 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:21:49.636 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.620+0000 7f06ba03f640 1 -- 172.21.15.19:0/2502909414 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f06a0077a50 msgr2=0x7f06a0079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:49.637 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.620+0000 7f06ba03f640 1 --2- 172.21.15.19:0/2502909414 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f06a0077a50 0x7f06a0079f60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f06ac005fd0 tx=0x7f06ac009560 comp rx=0 tx=0).stop 2024-08-31T12:21:49.637 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.620+0000 7f06ba03f640 1 -- 172.21.15.19:0/2502909414 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f06b406e0e0 msgr2=0x7f06b40cf4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:21:49.637 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.620+0000 7f06ba03f640 1 --2- 172.21.15.19:0/2502909414 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f06b406e0e0 0x7f06b40cf4c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f06a400b420 tx=0x7f06a400b8e0 comp rx=0 tx=0).stop 2024-08-31T12:21:49.637 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.620+0000 7f06b3fff640 1 -- 172.21.15.19:0/2502909414 reap_dead start 2024-08-31T12:21:49.637 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.620+0000 7f06b3fff640 1 -- 172.21.15.19:0/2502909414 reap_dead start 2024-08-31T12:21:49.637 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.620+0000 7f06ba03f640 1 -- 172.21.15.19:0/2502909414 shutdown_connections 2024-08-31T12:21:49.637 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.620+0000 7f06ba03f640 1 -- 172.21.15.19:0/2502909414 >> 172.21.15.19:0/2502909414 conn(0x7f06b40690d0 msgr2=0x7f06b4150d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:21:49.637 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.620+0000 7f06ba03f640 1 -- 172.21.15.19:0/2502909414 shutdown_connections 2024-08-31T12:21:49.637 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:21:49.620+0000 7f06ba03f640 1 -- 172.21.15.19:0/2502909414 wait complete. 2024-08-31T12:21:49.637 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:21:49.638 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-08-31T12:21:49.638 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2024-08-31T12:21:49.638 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-08-31T12:21:49.638 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-08-31T12:21:49.638 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-08-31T12:21:49.638 INFO:teuthology.orchestra.run:waiting for 300 2024-08-31T12:21:49.650 INFO:tasks.ceph.mon.c:Stopped 2024-08-31T12:21:49.651 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-08-31T12:21:49.651 INFO:tasks.ceph.mon.b:Sent signal 19 2024-08-31T12:21:49.651 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-08-31T12:21:56.589 DEBUG:teuthology.orchestra.run.smithi019:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:21:56.592 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:22:04.652 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-08-31T12:22:04.652 INFO:tasks.ceph.mon.b:Sent signal 18 2024-08-31T12:22:04.652 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-08-31T12:22:04.652 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:22:09.676 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:09.673+0000 7f641e719640 1 -- 172.21.15.19:0/3172839376 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f64000053a0 msgr2=0x7f640000ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:09.676 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:09.673+0000 7f641e719640 1 --2- 172.21.15.19:0/3172839376 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f64000053a0 0x7f640000ba20 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f64180746f0 tx=0x7f6400005830 comp rx=0 tx=0).stop 2024-08-31T12:22:09.677 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:09.674+0000 7f641e719640 1 -- 172.21.15.19:0/3172839376 shutdown_connections 2024-08-31T12:22:09.677 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:09.674+0000 7f641e719640 1 --2- 172.21.15.19:0/3172839376 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f64000053a0 0x7f640000ba20 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:09.677 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:09.674+0000 7f641e719640 1 --2- 172.21.15.19:0/3172839376 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6400004800 0x7f6400004c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:09.677 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:09.674+0000 7f641e719640 1 --2- 172.21.15.19:0/3172839376 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64000039b0 0x7f6400003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:09.677 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:09.674+0000 7f641e719640 1 -- 172.21.15.19:0/3172839376 >> 172.21.15.19:0/3172839376 conn(0x7f6418072c80 msgr2=0x7f6418073090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:22:09.677 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:09.674+0000 7f641e719640 1 -- 172.21.15.19:0/3172839376 shutdown_connections 2024-08-31T12:22:09.677 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:09.674+0000 7f641e719640 1 -- 172.21.15.19:0/3172839376 wait complete. 2024-08-31T12:22:09.678 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:09.676+0000 7f641e719640 1 Processor -- start 2024-08-31T12:22:09.678 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:09.676+0000 7f641e719640 1 -- start start 2024-08-31T12:22:09.681 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:09.677+0000 7f641e719640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64000039b0 0x7f6418149500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:09.681 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:09.677+0000 7f641e719640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6400004800 0x7f641815b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:09.681 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:09.677+0000 7f641e719640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64000053a0 0x7f641815baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:09.681 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:09.677+0000 7f641e719640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f64180721e0 con 0x7f6400004800 2024-08-31T12:22:09.681 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:09.677+0000 7f641e719640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f64180cae30 con 0x7f64000053a0 2024-08-31T12:22:09.681 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:09.677+0000 7f641e719640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f6418072360 con 0x7f64000039b0 2024-08-31T12:22:09.682 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:09.678+0000 7f6417fff640 1 -- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64000039b0 msgr2=0x7f6418149500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3301/0 2024-08-31T12:22:09.682 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:09.678+0000 7f6417fff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64000039b0 0x7f6418149500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:22:09.682 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:09.678+0000 7f64177fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6400004800 0x7f641815b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:09.682 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:09.678+0000 7f641ccb6640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64000053a0 0x7f641815baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:09.682 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:09.678+0000 7f64177fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6400004800 0x7f641815b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:60044/0 (socket says 172.21.15.19:60044) 2024-08-31T12:22:09.682 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:09.678+0000 7f64177fe640 1 -- 172.21.15.19:0/4049902961 learned_addr learned my addr 172.21.15.19:0/4049902961 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:22:09.682 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:09.678+0000 7f64177fe640 1 -- 172.21.15.19:0/4049902961 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64000039b0 msgr2=0x7f6418149500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:22:09.682 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:09.678+0000 7f64177fe640 1 --2- 172.21.15.19:0/4049902961 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64000039b0 0x7f6418149500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:09.682 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:09.678+0000 7f64177fe640 1 -- 172.21.15.19:0/4049902961 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64000053a0 msgr2=0x7f641815baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:09.682 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:09.678+0000 7f64177fe640 1 --2- 172.21.15.19:0/4049902961 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64000053a0 0x7f641815baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:09.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:09.678+0000 7f64177fe640 1 -- 172.21.15.19:0/4049902961 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64181540b0 con 0x7f6400004800 2024-08-31T12:22:09.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:09.678+0000 7f641ccb6640 1 --2- 172.21.15.19:0/4049902961 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64000053a0 0x7f641815baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:22:09.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:09.678+0000 7f64177fe640 1 --2- 172.21.15.19:0/4049902961 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6400004800 0x7f641815b560 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f640c00f630 tx=0x7f640c00faf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:22:09.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:09.679+0000 7f6414ff9640 1 -- 172.21.15.19:0/4049902961 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f640c00d7b0 con 0x7f6400004800 2024-08-31T12:22:09.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:09.679+0000 7f6414ff9640 1 -- 172.21.15.19:0/4049902961 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f640c016020 con 0x7f6400004800 2024-08-31T12:22:09.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:09.679+0000 7f641e719640 1 -- 172.21.15.19:0/4049902961 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64180cd720 con 0x7f6400004800 2024-08-31T12:22:09.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:09.679+0000 7f6414ff9640 1 -- 172.21.15.19:0/4049902961 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f640c00dbf0 con 0x7f6400004800 2024-08-31T12:22:09.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:09.679+0000 7f641e719640 1 -- 172.21.15.19:0/4049902961 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6418156550 con 0x7f6400004800 2024-08-31T12:22:09.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:09.680+0000 7f641e719640 1 -- 172.21.15.19:0/4049902961 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6418071b40 con 0x7f6400004800 2024-08-31T12:22:09.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:09.681+0000 7f6414ff9640 1 -- 172.21.15.19:0/4049902961 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f640c011430 con 0x7f6400004800 2024-08-31T12:22:09.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:09.681+0000 7f6414ff9640 1 --2- 172.21.15.19:0/4049902961 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f64040783e0 0x7f640407a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:09.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:09.682+0000 7f6414ff9640 1 -- 172.21.15.19:0/4049902961 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f640c099f00 con 0x7f6400004800 2024-08-31T12:22:09.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:09.682+0000 7f6417fff640 1 --2- 172.21.15.19:0/4049902961 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f64040783e0 0x7f640407a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:09.685 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:09.682+0000 7f6417fff640 1 --2- 172.21.15.19:0/4049902961 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f64040783e0 0x7f640407a8f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6408005cb0 tx=0x7f6408004600 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:22:09.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:09.687+0000 7f6414ff9640 1 -- 172.21.15.19:0/4049902961 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f640c020020 con 0x7f6400004800 2024-08-31T12:22:10.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.007+0000 7f641e719640 1 -- 172.21.15.19:0/4049902961 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f64180732e0 con 0x7f6400004800 2024-08-31T12:22:10.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.007+0000 7f6414ff9640 1 -- 172.21.15.19:0/4049902961 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7f640c062bf0 con 0x7f6400004800 2024-08-31T12:22:10.010 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":46,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:22:10.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.010+0000 7f641e719640 1 -- 172.21.15.19:0/4049902961 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f64040783e0 msgr2=0x7f640407a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:10.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.010+0000 7f641e719640 1 --2- 172.21.15.19:0/4049902961 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f64040783e0 0x7f640407a8f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6408005cb0 tx=0x7f6408004600 comp rx=0 tx=0).stop 2024-08-31T12:22:10.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.010+0000 7f641e719640 1 -- 172.21.15.19:0/4049902961 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6400004800 msgr2=0x7f641815b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:10.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.010+0000 7f641e719640 1 --2- 172.21.15.19:0/4049902961 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6400004800 0x7f641815b560 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f640c00f630 tx=0x7f640c00faf0 comp rx=0 tx=0).stop 2024-08-31T12:22:10.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.010+0000 7f641e719640 1 -- 172.21.15.19:0/4049902961 shutdown_connections 2024-08-31T12:22:10.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.010+0000 7f641e719640 1 --2- 172.21.15.19:0/4049902961 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f64040783e0 0x7f640407a8f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:10.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.010+0000 7f641e719640 1 --2- 172.21.15.19:0/4049902961 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64000053a0 0x7f641815baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:10.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.010+0000 7f641e719640 1 --2- 172.21.15.19:0/4049902961 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6400004800 0x7f641815b560 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:10.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.010+0000 7f641e719640 1 --2- 172.21.15.19:0/4049902961 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64000039b0 0x7f6418149500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:10.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.010+0000 7f641e719640 1 -- 172.21.15.19:0/4049902961 >> 172.21.15.19:0/4049902961 conn(0x7f6418072c80 msgr2=0x7f64180c48f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:22:10.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.011+0000 7f641e719640 1 -- 172.21.15.19:0/4049902961 shutdown_connections 2024-08-31T12:22:10.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.011+0000 7f641e719640 1 -- 172.21.15.19:0/4049902961 wait complete. 2024-08-31T12:22:10.025 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-08-31T12:22:10.025 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-08-31T12:22:10.025 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-31T12:22:10.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.152+0000 7f2d9f57b640 1 -- 172.21.15.19:0/2450121313 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2d84005350 msgr2=0x7f2d8400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:10.155 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.152+0000 7f2d9f57b640 1 --2- 172.21.15.19:0/2450121313 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2d84005350 0x7f2d8400b9c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2d90005fd0 tx=0x7f2d84005830 comp rx=0 tx=0).stop 2024-08-31T12:22:10.155 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.152+0000 7f2d9f57b640 1 -- 172.21.15.19:0/2450121313 shutdown_connections 2024-08-31T12:22:10.155 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.152+0000 7f2d9f57b640 1 --2- 172.21.15.19:0/2450121313 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2d84005350 0x7f2d8400b9c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:10.155 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.152+0000 7f2d9f57b640 1 -- 172.21.15.19:0/2450121313 >> 172.21.15.19:0/2450121313 conn(0x7f2d980690d0 msgr2=0x7f2d980d0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:22:10.155 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.152+0000 7f2d9f57b640 1 -- 172.21.15.19:0/2450121313 shutdown_connections 2024-08-31T12:22:10.155 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.152+0000 7f2d9f57b640 1 -- 172.21.15.19:0/2450121313 wait complete. 2024-08-31T12:22:10.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.154+0000 7f2d9f57b640 1 Processor -- start 2024-08-31T12:22:10.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.154+0000 7f2d9f57b640 1 -- start start 2024-08-31T12:22:10.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.155+0000 7f2d9f57b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2d84005350 0x7f2d9815df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:10.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.155+0000 7f2d9f57b640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d9815c530 0x7f2d981f45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:10.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.155+0000 7f2d9f57b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2d981f4b00 0x7f2d981508c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:10.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.155+0000 7f2d9f57b640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f2d9815b250 con 0x7f2d981f4b00 2024-08-31T12:22:10.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.155+0000 7f2d9f57b640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2d9815af70 con 0x7f2d9815c530 2024-08-31T12:22:10.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.155+0000 7f2d9f57b640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f2d980c8610 con 0x7f2d84005350 2024-08-31T12:22:10.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.156+0000 7f2d9d317640 1 -- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2d84005350 msgr2=0x7f2d9815df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3301/0 2024-08-31T12:22:10.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.156+0000 7f2d9d317640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2d84005350 0x7f2d9815df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:22:10.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.156+0000 7f2d9db18640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2d981f4b00 0x7f2d981508c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:10.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.156+0000 7f2d9db18640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2d981f4b00 0x7f2d981508c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:60072/0 (socket says 172.21.15.19:60072) 2024-08-31T12:22:10.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.156+0000 7f2d9db18640 1 -- 172.21.15.19:0/3468341166 learned_addr learned my addr 172.21.15.19:0/3468341166 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:22:10.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.156+0000 7f2d9cb16640 1 --2- 172.21.15.19:0/3468341166 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d9815c530 0x7f2d981f45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:10.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.156+0000 7f2d9db18640 1 -- 172.21.15.19:0/3468341166 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2d84005350 msgr2=0x7f2d9815df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:22:10.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.156+0000 7f2d9db18640 1 --2- 172.21.15.19:0/3468341166 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2d84005350 0x7f2d9815df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:10.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.156+0000 7f2d9db18640 1 -- 172.21.15.19:0/3468341166 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d9815c530 msgr2=0x7f2d981f45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:10.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.156+0000 7f2d9db18640 1 --2- 172.21.15.19:0/3468341166 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d9815c530 0x7f2d981f45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:10.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.156+0000 7f2d9db18640 1 -- 172.21.15.19:0/3468341166 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d980c5940 con 0x7f2d981f4b00 2024-08-31T12:22:10.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.156+0000 7f2d9db18640 1 --2- 172.21.15.19:0/3468341166 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2d981f4b00 0x7f2d981508c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2d90005fd0 tx=0x7f2d90033650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:22:10.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.156+0000 7f2d8dffb640 1 -- 172.21.15.19:0/3468341166 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d90038040 con 0x7f2d981f4b00 2024-08-31T12:22:10.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.156+0000 7f2d8dffb640 1 -- 172.21.15.19:0/3468341166 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2d9002b670 con 0x7f2d981f4b00 2024-08-31T12:22:10.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.156+0000 7f2d8dffb640 1 -- 172.21.15.19:0/3468341166 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d90004d40 con 0x7f2d981f4b00 2024-08-31T12:22:10.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.156+0000 7f2d9f57b640 1 -- 172.21.15.19:0/3468341166 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d9815e4b0 con 0x7f2d981f4b00 2024-08-31T12:22:10.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.157+0000 7f2d9f57b640 1 -- 172.21.15.19:0/3468341166 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d980c4c60 con 0x7f2d981f4b00 2024-08-31T12:22:10.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.158+0000 7f2d9f57b640 1 --2- 172.21.15.19:0/3468341166 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2d98150e00 0x7f2d981f0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:10.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.158+0000 7f2d9f57b640 1 -- 172.21.15.19:0/3468341166 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f2d980c5b10 con 0x7f2d98150e00 2024-08-31T12:22:10.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.158+0000 7f2d9d317640 1 --2- 172.21.15.19:0/3468341166 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2d98150e00 0x7f2d981f0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:10.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.158+0000 7f2d9d317640 1 -- 172.21.15.19:0/3468341166 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2d981f5080 con 0x7f2d98150e00 2024-08-31T12:22:10.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.158+0000 7f2d9d317640 1 --2- 172.21.15.19:0/3468341166 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2d98150e00 0x7f2d981f0d80 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f2d940036a0 tx=0x7f2d9400ce80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:22:10.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.158+0000 7f2d8dffb640 1 -- 172.21.15.19:0/3468341166 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d94017070 con 0x7f2d98150e00 2024-08-31T12:22:10.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.159+0000 7f2d8dffb640 1 -- 172.21.15.19:0/3468341166 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2d940074e0 con 0x7f2d98150e00 2024-08-31T12:22:10.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.159+0000 7f2d8dffb640 1 -- 172.21.15.19:0/3468341166 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2d98150e00 msgr2=0x7f2d981f0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:10.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.159+0000 7f2d8dffb640 1 --2- 172.21.15.19:0/3468341166 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2d98150e00 0x7f2d981f0d80 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f2d940036a0 tx=0x7f2d9400ce80 comp rx=0 tx=0).stop 2024-08-31T12:22:10.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.159+0000 7f2d8dffb640 1 -- 172.21.15.19:0/3468341166 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2d98150e00 msgr2=0x7f2d981f0d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:22:10.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.159+0000 7f2d8dffb640 1 --2- 172.21.15.19:0/3468341166 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2d98150e00 0x7f2d981f0d80 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:10.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.159+0000 7f2d8dffb640 1 -- 172.21.15.19:0/3468341166 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f2d90043030 con 0x7f2d981f4b00 2024-08-31T12:22:10.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.159+0000 7f2d8dffb640 1 --2- 172.21.15.19:0/3468341166 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2d7c0775e0 0x7f2d7c079af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:10.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.160+0000 7f2d8dffb640 1 -- 172.21.15.19:0/3468341166 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f2d900b9350 con 0x7f2d981f4b00 2024-08-31T12:22:10.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.160+0000 7f2d9d317640 1 --2- 172.21.15.19:0/3468341166 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2d7c0775e0 0x7f2d7c079af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:10.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.160+0000 7f2d9d317640 1 --2- 172.21.15.19:0/3468341166 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2d7c0775e0 0x7f2d7c079af0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2d940036a0 tx=0x7f2d94009000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:22:10.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.169+0000 7f2d9f57b640 1 --2- 172.21.15.19:0/3468341166 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2d981f12c0 0x7f2d981f16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:10.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.169+0000 7f2d9f57b640 1 -- 172.21.15.19:0/3468341166 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f2d9815d860 con 0x7f2d981f12c0 2024-08-31T12:22:10.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.169+0000 7f2d9cb16640 1 --2- 172.21.15.19:0/3468341166 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2d981f12c0 0x7f2d981f16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:10.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.169+0000 7f2d9cb16640 1 -- 172.21.15.19:0/3468341166 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2d981f1c10 con 0x7f2d981f12c0 2024-08-31T12:22:10.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.169+0000 7f2d9cb16640 1 --2- 172.21.15.19:0/3468341166 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2d981f12c0 0x7f2d981f16d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f2d880105f0 tx=0x7f2d88010ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:22:10.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.169+0000 7f2d8dffb640 1 -- 172.21.15.19:0/3468341166 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d88011960 con 0x7f2d981f12c0 2024-08-31T12:22:10.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.170+0000 7f2d8dffb640 1 -- 172.21.15.19:0/3468341166 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5033 (secure 0 0 0) 0x7f2d88015030 con 0x7f2d981f12c0 2024-08-31T12:22:10.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.170+0000 7f2d8dffb640 1 -- 172.21.15.19:0/3468341166 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2d981f12c0 msgr2=0x7f2d981f16d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:10.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.170+0000 7f2d8dffb640 1 --2- 172.21.15.19:0/3468341166 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2d981f12c0 0x7f2d981f16d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f2d880105f0 tx=0x7f2d88010ab0 comp rx=0 tx=0).stop 2024-08-31T12:22:10.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.170+0000 7f2d8dffb640 1 -- 172.21.15.19:0/3468341166 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2d981f12c0 msgr2=0x7f2d981f16d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:22:10.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.170+0000 7f2d8dffb640 1 --2- 172.21.15.19:0/3468341166 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2d981f12c0 0x7f2d981f16d0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:10.173 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:22:10.173 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:22:10.173 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:22:10.173 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "leader", 2024-08-31T12:22:10.173 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 46, 2024-08-31T12:22:10.173 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 310160, 2024-08-31T12:22:10.173 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:22:10.174 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:22:10.174 INFO:teuthology.orchestra.run.smithi019.stdout: 1 2024-08-31T12:22:10.174 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:22:10.174 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 0, 2024-08-31T12:22:10.174 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:22:10.174 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:22:10.174 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:22:10.174 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:22:10.174 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:22:10.174 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:22:10.174 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:22:10.174 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:22:10.175 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:22:10.175 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:22:10.175 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:22:10.175 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:22:10.175 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:22:10.175 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:22:10.175 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:22:10.175 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:22:10.175 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:22:10.175 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:22:10.175 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:22:10.175 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:22:10.175 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:22:10.176 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:22:10.176 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:22:10.176 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:22:10.176 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:22:10.176 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:22:10.176 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:22:10.176 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:22:10.176 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:10.176 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:10.176 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:22:10.176 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:22:10.176 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:22:10.176 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:22:10.177 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:22:10.177 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:22:10.177 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:22:10.177 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:22:10.177 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:22:10.177 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:22:10.177 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:22:10.177 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:22:10.177 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:22:10.177 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:22:10.177 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:22:10.177 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:22:10.177 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:22:10.178 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:22:10.178 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:22:10.178 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:22:10.178 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:22:10.178 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:22:10.178 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:22:10.178 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:22:10.178 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:22:10.178 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:22:10.178 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:22:10.178 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:22:10.178 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:22:10.178 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:22:10.179 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:10.179 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:22:10.179 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:10.179 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:22:10.179 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:22:10.179 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:22:10.179 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:22:10.179 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:10.179 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:22:10.179 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:22:10.179 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:10.179 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:10.179 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:10.180 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:22:10.180 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:22:10.180 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:10.180 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:10.180 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:10.180 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:10.180 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:22:10.180 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:22:10.180 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:22:10.180 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:22:10.180 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:22:10.180 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:10.180 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:10.180 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:22:10.181 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:22:10.181 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:22:10.181 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:22:10.181 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:10.181 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:22:10.181 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:22:10.181 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:10.181 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:10.181 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:10.181 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:22:10.181 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:22:10.181 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:10.181 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:10.182 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:10.182 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:10.182 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:22:10.182 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:22:10.182 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:22:10.182 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:22:10.182 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:22:10.182 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:10.182 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:10.182 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:22:10.182 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:22:10.182 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:22:10.182 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:22:10.183 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:10.183 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:22:10.183 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:22:10.183 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:10.183 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:10.183 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:10.183 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:22:10.183 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:22:10.183 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:10.183 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:10.183 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:10.183 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:10.183 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:22:10.184 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:22:10.184 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:22:10.184 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:22:10.184 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:22:10.184 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:10.184 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:10.184 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:10.184 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:22:10.184 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:22:10.184 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:10.184 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:22:10.184 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:22:10.185 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:22:10.185 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:10.185 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:22:10.185 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:22:10.185 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:10.185 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:22:10.185 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:22:10.185 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 6 2024-08-31T12:22:10.185 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:10.185 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:22:10.185 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:22:10.186 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:10.186 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:22:10.186 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:22:10.186 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 8 2024-08-31T12:22:10.186 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:10.186 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:22:10.186 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:22:10.186 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:10.186 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:22:10.186 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:22:10.186 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 6 2024-08-31T12:22:10.186 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:10.186 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:22:10.187 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:22:10.187 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:10.187 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:22:10.187 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:22:10.187 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:22:10.187 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:10.187 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:10.187 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:10.187 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:22:10.187 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:22:10.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.170+0000 7f2d9f57b640 1 -- 172.21.15.19:0/3468341166 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2d7c0775e0 msgr2=0x7f2d7c079af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:10.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.170+0000 7f2d9f57b640 1 --2- 172.21.15.19:0/3468341166 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2d7c0775e0 0x7f2d7c079af0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2d940036a0 tx=0x7f2d94009000 comp rx=0 tx=0).stop 2024-08-31T12:22:10.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.170+0000 7f2d9f57b640 1 -- 172.21.15.19:0/3468341166 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2d981f4b00 msgr2=0x7f2d981508c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:10.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.170+0000 7f2d9f57b640 1 --2- 172.21.15.19:0/3468341166 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2d981f4b00 0x7f2d981508c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2d90005fd0 tx=0x7f2d90033650 comp rx=0 tx=0).stop 2024-08-31T12:22:10.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.170+0000 7f2d9db18640 1 -- 172.21.15.19:0/3468341166 reap_dead start 2024-08-31T12:22:10.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.170+0000 7f2d9f57b640 1 -- 172.21.15.19:0/3468341166 shutdown_connections 2024-08-31T12:22:10.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.170+0000 7f2d9f57b640 1 --2- 172.21.15.19:0/3468341166 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2d981f4b00 0x7f2d981508c0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:10.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.170+0000 7f2d9f57b640 1 -- 172.21.15.19:0/3468341166 >> 172.21.15.19:0/3468341166 conn(0x7f2d980690d0 msgr2=0x7f2d98154360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:22:10.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.170+0000 7f2d9f57b640 1 -- 172.21.15.19:0/3468341166 shutdown_connections 2024-08-31T12:22:10.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.170+0000 7f2d9f57b640 1 -- 172.21.15.19:0/3468341166 wait complete. 2024-08-31T12:22:10.189 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-08-31T12:22:10.190 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-08-31T12:22:10.190 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-31T12:22:10.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.312+0000 7f190899e640 1 -- 172.21.15.19:0/1736751032 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1904159110 msgr2=0x7f19040ce840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:10.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.312+0000 7f190899e640 1 --2- 172.21.15.19:0/1736751032 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1904159110 0x7f19040ce840 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f18fc005fd0 tx=0x7f18fc02aac0 comp rx=0 tx=0).stop 2024-08-31T12:22:10.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.313+0000 7f190899e640 1 -- 172.21.15.19:0/1736751032 shutdown_connections 2024-08-31T12:22:10.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.313+0000 7f190899e640 1 --2- 172.21.15.19:0/1736751032 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1904159110 0x7f19040ce840 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:10.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.313+0000 7f190899e640 1 --2- 172.21.15.19:0/1736751032 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f19040c4c60 0x7f1904158bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:10.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.313+0000 7f190899e640 1 -- 172.21.15.19:0/1736751032 >> 172.21.15.19:0/1736751032 conn(0x7f1904068ec0 msgr2=0x7f19040692d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:22:10.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.313+0000 7f190899e640 1 -- 172.21.15.19:0/1736751032 shutdown_connections 2024-08-31T12:22:10.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.313+0000 7f190899e640 1 -- 172.21.15.19:0/1736751032 wait complete. 2024-08-31T12:22:10.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.315+0000 7f190899e640 1 Processor -- start 2024-08-31T12:22:10.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.315+0000 7f190899e640 1 -- start start 2024-08-31T12:22:10.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.317+0000 7f190899e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f19040c3c20 0x7f190414ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:10.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.317+0000 7f190899e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f19040c4c60 0x7f190414b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:10.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.317+0000 7f190899e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1904159110 0x7f1904157ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:10.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.317+0000 7f190899e640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f19040cb3d0 con 0x7f19040c3c20 2024-08-31T12:22:10.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.317+0000 7f190899e640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f19040cae70 con 0x7f1904159110 2024-08-31T12:22:10.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.317+0000 7f190899e640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f1904070d70 con 0x7f19040c4c60 2024-08-31T12:22:10.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.317+0000 7f19037fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f19040c3c20 0x7f190414ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:10.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.317+0000 7f1902ffd640 1 -- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f19040c4c60 msgr2=0x7f190414b990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3301/0 2024-08-31T12:22:10.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.317+0000 7f1902ffd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f19040c4c60 0x7f190414b990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:22:10.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.317+0000 7f19037fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f19040c3c20 0x7f190414ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:60120/0 (socket says 172.21.15.19:60120) 2024-08-31T12:22:10.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.317+0000 7f19037fe640 1 -- 172.21.15.19:0/3903436389 learned_addr learned my addr 172.21.15.19:0/3903436389 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:22:10.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.317+0000 7f1903fff640 1 --2- 172.21.15.19:0/3903436389 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1904159110 0x7f1904157ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:10.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.318+0000 7f19037fe640 1 -- 172.21.15.19:0/3903436389 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f19040c4c60 msgr2=0x7f190414b990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:22:10.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.318+0000 7f19037fe640 1 --2- 172.21.15.19:0/3903436389 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f19040c4c60 0x7f190414b990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:10.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.318+0000 7f19037fe640 1 -- 172.21.15.19:0/3903436389 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1904159110 msgr2=0x7f1904157ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:10.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.318+0000 7f19037fe640 1 --2- 172.21.15.19:0/3903436389 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1904159110 0x7f1904157ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:10.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.318+0000 7f19037fe640 1 -- 172.21.15.19:0/3903436389 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f190414bed0 con 0x7f19040c3c20 2024-08-31T12:22:10.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.318+0000 7f19037fe640 1 --2- 172.21.15.19:0/3903436389 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f19040c3c20 0x7f190414ab40 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f18f400c3f0 tx=0x7f18f400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:22:10.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.318+0000 7f18ebfff640 1 -- 172.21.15.19:0/3903436389 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f4017070 con 0x7f19040c3c20 2024-08-31T12:22:10.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.318+0000 7f18ebfff640 1 -- 172.21.15.19:0/3903436389 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f18f40035f0 con 0x7f19040c3c20 2024-08-31T12:22:10.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.318+0000 7f190899e640 1 -- 172.21.15.19:0/3903436389 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19041e4020 con 0x7f19040c3c20 2024-08-31T12:22:10.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.318+0000 7f18ebfff640 1 -- 172.21.15.19:0/3903436389 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f4003a40 con 0x7f19040c3c20 2024-08-31T12:22:10.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.318+0000 7f190899e640 1 -- 172.21.15.19:0/3903436389 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1904149d80 con 0x7f19040c3c20 2024-08-31T12:22:10.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.319+0000 7f190899e640 1 --2- 172.21.15.19:0/3903436389 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1904146bf0 0x7f1904147000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:10.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.319+0000 7f190899e640 1 -- 172.21.15.19:0/3903436389 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f19041f1220 con 0x7f1904146bf0 2024-08-31T12:22:10.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.320+0000 7f1902ffd640 1 --2- 172.21.15.19:0/3903436389 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1904146bf0 0x7f1904147000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:10.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.320+0000 7f1902ffd640 1 -- 172.21.15.19:0/3903436389 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1904147540 con 0x7f1904146bf0 2024-08-31T12:22:10.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.320+0000 7f1902ffd640 1 --2- 172.21.15.19:0/3903436389 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1904146bf0 0x7f1904147000 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1904158510 tx=0x7f18f800b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:22:10.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.320+0000 7f18ebfff640 1 -- 172.21.15.19:0/3903436389 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f18f401a070 con 0x7f19040c3c20 2024-08-31T12:22:10.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.321+0000 7f18ebfff640 1 --2- 172.21.15.19:0/3903436389 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f18f0077a30 0x7f18f0079f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:10.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.321+0000 7f18ebfff640 1 -- 172.21.15.19:0/3903436389 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f8016070 con 0x7f1904146bf0 2024-08-31T12:22:10.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.321+0000 7f18ebfff640 1 -- 172.21.15.19:0/3903436389 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f18f4098990 con 0x7f19040c3c20 2024-08-31T12:22:10.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.321+0000 7f18ebfff640 1 -- 172.21.15.19:0/3903436389 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f18f8003d70 con 0x7f1904146bf0 2024-08-31T12:22:10.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.321+0000 7f18ebfff640 1 -- 172.21.15.19:0/3903436389 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1904146bf0 msgr2=0x7f1904147000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:10.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.321+0000 7f1903fff640 1 --2- 172.21.15.19:0/3903436389 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f18f0077a30 0x7f18f0079f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:10.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.321+0000 7f18ebfff640 1 --2- 172.21.15.19:0/3903436389 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1904146bf0 0x7f1904147000 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1904158510 tx=0x7f18f800b870 comp rx=0 tx=0).stop 2024-08-31T12:22:10.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.321+0000 7f18ebfff640 1 -- 172.21.15.19:0/3903436389 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1904146bf0 msgr2=0x7f1904147000 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:22:10.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.321+0000 7f18ebfff640 1 --2- 172.21.15.19:0/3903436389 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1904146bf0 0x7f1904147000 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:10.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.322+0000 7f1903fff640 1 --2- 172.21.15.19:0/3903436389 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f18f0077a30 0x7f18f0079f40 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f18fc035000 tx=0x7f18fc038040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:22:10.331 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.329+0000 7f190899e640 1 --2- 172.21.15.19:0/3903436389 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19041f44e0 0x7f19041f48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:10.331 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.329+0000 7f190899e640 1 -- 172.21.15.19:0/3903436389 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f190406fba0 con 0x7f19041f44e0 2024-08-31T12:22:10.331 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.329+0000 7f1902ffd640 1 --2- 172.21.15.19:0/3903436389 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19041f44e0 0x7f19041f48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:10.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.329+0000 7f1902ffd640 1 -- 172.21.15.19:0/3903436389 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1904070fe0 con 0x7f19041f44e0 2024-08-31T12:22:10.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.330+0000 7f1902ffd640 1 --2- 172.21.15.19:0/3903436389 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19041f44e0 0x7f19041f48f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f18f8011000 tx=0x7f18f80059c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:22:10.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.330+0000 7f18ebfff640 1 -- 172.21.15.19:0/3903436389 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f8016040 con 0x7f19041f44e0 2024-08-31T12:22:10.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.330+0000 7f18ebfff640 1 -- 172.21.15.19:0/3903436389 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4529 (secure 0 0 0) 0x7f18f8005580 con 0x7f19041f44e0 2024-08-31T12:22:10.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.330+0000 7f18ebfff640 1 -- 172.21.15.19:0/3903436389 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19041f44e0 msgr2=0x7f19041f48f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:10.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.330+0000 7f18ebfff640 1 --2- 172.21.15.19:0/3903436389 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19041f44e0 0x7f19041f48f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f18f8011000 tx=0x7f18f80059c0 comp rx=0 tx=0).stop 2024-08-31T12:22:10.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.330+0000 7f18ebfff640 1 -- 172.21.15.19:0/3903436389 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19041f44e0 msgr2=0x7f19041f48f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:22:10.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.330+0000 7f18ebfff640 1 --2- 172.21.15.19:0/3903436389 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19041f44e0 0x7f19041f48f0 secure :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f18f8011000 tx=0x7f18f80059c0 comp rx=0 tx=0).stop 2024-08-31T12:22:10.333 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:22:10.333 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:22:10.333 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:22:10.333 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:22:10.334 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 46, 2024-08-31T12:22:10.334 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 39804, 2024-08-31T12:22:10.334 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:22:10.334 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:22:10.334 INFO:teuthology.orchestra.run.smithi019.stdout: 1 2024-08-31T12:22:10.334 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:22:10.334 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 0, 2024-08-31T12:22:10.335 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:22:10.335 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:22:10.335 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:22:10.335 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:22:10.335 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:22:10.335 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:22:10.335 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:22:10.335 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:22:10.336 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:22:10.336 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:22:10.336 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:22:10.336 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:22:10.336 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:22:10.336 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:22:10.336 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:22:10.336 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:22:10.336 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:22:10.336 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:22:10.336 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:22:10.336 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:22:10.336 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:22:10.337 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:22:10.337 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:22:10.337 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:22:10.337 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:22:10.337 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:22:10.337 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:22:10.337 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:22:10.337 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:10.337 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:10.337 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:22:10.338 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:22:10.338 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:22:10.338 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:22:10.338 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:22:10.338 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:22:10.338 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:22:10.338 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:22:10.338 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:22:10.338 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:22:10.338 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:22:10.338 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:22:10.338 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:22:10.339 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:22:10.339 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:22:10.339 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:22:10.339 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:22:10.339 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:22:10.339 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:22:10.339 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:22:10.339 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:22:10.339 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:22:10.339 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:22:10.339 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:22:10.339 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:22:10.339 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:22:10.340 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:22:10.340 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:22:10.340 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:22:10.340 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:22:10.340 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:10.340 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:22:10.340 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:10.340 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:22:10.340 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:22:10.340 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:22:10.341 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:22:10.341 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:10.341 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:22:10.341 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:22:10.341 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:10.341 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:10.341 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:10.341 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:22:10.341 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:22:10.341 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:10.341 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:10.341 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:10.341 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:10.342 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:22:10.342 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:22:10.342 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:22:10.342 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:22:10.342 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:22:10.342 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:10.342 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:10.342 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:22:10.342 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:22:10.342 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:22:10.342 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:22:10.342 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:10.343 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:22:10.343 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:22:10.343 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:10.343 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:10.343 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:10.343 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:22:10.343 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:22:10.343 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:10.343 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:10.343 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:10.344 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:10.344 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:22:10.344 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:22:10.344 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:22:10.344 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:22:10.344 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:22:10.344 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:10.344 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:10.344 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:22:10.344 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:22:10.345 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:22:10.345 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:22:10.345 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:10.345 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:22:10.345 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:22:10.345 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:10.345 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:10.345 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:10.345 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:22:10.345 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:22:10.345 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:10.346 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:10.346 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:10.346 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:10.346 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:22:10.346 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:22:10.346 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:22:10.346 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:22:10.346 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:22:10.346 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:10.346 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:10.346 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:10.346 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:22:10.346 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:22:10.347 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:10.347 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:22:10.347 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:22:10.347 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:22:10.347 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:10.347 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:22:10.347 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:22:10.347 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:10.347 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:22:10.347 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:22:10.347 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:22:10.347 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:10.347 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:10.347 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:10.348 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:22:10.348 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:22:10.349 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.331+0000 7f190899e640 1 -- 172.21.15.19:0/3903436389 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f18f0077a30 msgr2=0x7f18f0079f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:10.349 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.331+0000 7f190899e640 1 --2- 172.21.15.19:0/3903436389 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f18f0077a30 0x7f18f0079f40 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f18fc035000 tx=0x7f18fc038040 comp rx=0 tx=0).stop 2024-08-31T12:22:10.349 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.331+0000 7f190899e640 1 -- 172.21.15.19:0/3903436389 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f19040c3c20 msgr2=0x7f190414ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:10.349 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.331+0000 7f190899e640 1 --2- 172.21.15.19:0/3903436389 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f19040c3c20 0x7f190414ab40 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f18f400c3f0 tx=0x7f18f400c8b0 comp rx=0 tx=0).stop 2024-08-31T12:22:10.350 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.331+0000 7f1903fff640 1 -- 172.21.15.19:0/3903436389 reap_dead start 2024-08-31T12:22:10.350 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.331+0000 7f1903fff640 1 -- 172.21.15.19:0/3903436389 reap_dead start 2024-08-31T12:22:10.350 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.331+0000 7f190899e640 1 -- 172.21.15.19:0/3903436389 shutdown_connections 2024-08-31T12:22:10.350 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.331+0000 7f190899e640 1 -- 172.21.15.19:0/3903436389 >> 172.21.15.19:0/3903436389 conn(0x7f1904068ec0 msgr2=0x7f1904148eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:22:10.350 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.331+0000 7f190899e640 1 -- 172.21.15.19:0/3903436389 shutdown_connections 2024-08-31T12:22:10.350 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:10.331+0000 7f190899e640 1 -- 172.21.15.19:0/3903436389 wait complete. 2024-08-31T12:22:10.350 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-08-31T12:22:10.350 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-08-31T12:22:26.864 DEBUG:teuthology.orchestra.run.smithi019:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:22:26.867 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:22:30.351 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-08-31T12:22:30.351 INFO:tasks.ceph.mon.c:Restarting daemon 2024-08-31T12:22:30.351 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-08-31T12:22:30.354 INFO:tasks.ceph.mon.c:Started 2024-08-31T12:22:30.354 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-08-31T12:22:30.355 INFO:tasks.ceph.mon.b:Sent signal 19 2024-08-31T12:22:30.355 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-08-31T12:22:45.356 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-08-31T12:22:45.357 INFO:tasks.ceph.mon.b:Sent signal 18 2024-08-31T12:22:45.357 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-08-31T12:22:45.357 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:22:45.482 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.480+0000 7fc37e562640 1 -- 172.21.15.19:0/1447940759 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc378158170 msgr2=0x7fc378158580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:45.482 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.480+0000 7fc37e562640 1 --2- 172.21.15.19:0/1447940759 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc378158170 0x7fc378158580 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc370005fd0 tx=0x7fc37002a940 comp rx=0 tx=0).stop 2024-08-31T12:22:45.482 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.481+0000 7fc37e562640 1 -- 172.21.15.19:0/1447940759 shutdown_connections 2024-08-31T12:22:45.482 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.481+0000 7fc37e562640 1 --2- 172.21.15.19:0/1447940759 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc378158170 0x7fc378158580 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:45.482 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.481+0000 7fc37e562640 1 -- 172.21.15.19:0/1447940759 >> 172.21.15.19:0/1447940759 conn(0x7fc3780cdcb0 msgr2=0x7fc3780ca150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:22:45.482 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.481+0000 7fc37e562640 1 -- 172.21.15.19:0/1447940759 shutdown_connections 2024-08-31T12:22:45.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.481+0000 7fc37e562640 1 -- 172.21.15.19:0/1447940759 wait complete. 2024-08-31T12:22:45.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.482+0000 7fc37e562640 1 Processor -- start 2024-08-31T12:22:45.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.483+0000 7fc37e562640 1 -- start start 2024-08-31T12:22:45.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.484+0000 7fc37e562640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc3780c8050 0x7fc3781f4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:45.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.484+0000 7fc37e562640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc3780caad0 0x7fc378152e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:45.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.484+0000 7fc37e562640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc378158170 0x7fc378154c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:45.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.484+0000 7fc37e562640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fc378158ac0 con 0x7fc378158170 2024-08-31T12:22:45.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.484+0000 7fc37e562640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc3780c5870 con 0x7fc3780caad0 2024-08-31T12:22:45.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.484+0000 7fc37e562640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fc3780c7910 con 0x7fc3780c8050 2024-08-31T12:22:45.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.484+0000 7fc37caff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc378158170 0x7fc378154c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:45.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.484+0000 7fc3777fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc3780caad0 0x7fc378152e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:45.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.484+0000 7fc37caff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc378158170 0x7fc378154c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:59390/0 (socket says 172.21.15.19:59390) 2024-08-31T12:22:45.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.484+0000 7fc37caff640 1 -- 172.21.15.19:0/2983791369 learned_addr learned my addr 172.21.15.19:0/2983791369 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:22:45.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.484+0000 7fc377fff640 1 --2- 172.21.15.19:0/2983791369 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc3780c8050 0x7fc3781f4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:45.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.485+0000 7fc3777fe640 1 -- 172.21.15.19:0/2983791369 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc3780c8050 msgr2=0x7fc3781f4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:45.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.485+0000 7fc3777fe640 1 --2- 172.21.15.19:0/2983791369 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc3780c8050 0x7fc3781f4cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:45.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.485+0000 7fc3777fe640 1 -- 172.21.15.19:0/2983791369 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc378158170 msgr2=0x7fc378154c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:45.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.485+0000 7fc3777fe640 1 --2- 172.21.15.19:0/2983791369 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc378158170 0x7fc378154c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:45.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.485+0000 7fc3777fe640 1 -- 172.21.15.19:0/2983791369 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3781f5230 con 0x7fc3780caad0 2024-08-31T12:22:45.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.485+0000 7fc37caff640 1 --2- 172.21.15.19:0/2983791369 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc378158170 0x7fc378154c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:22:45.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.485+0000 7fc3777fe640 1 --2- 172.21.15.19:0/2983791369 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc3780caad0 0x7fc378152e40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc36c00c390 tx=0x7fc36c00c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:22:45.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.485+0000 7fc374ff9640 1 -- 172.21.15.19:0/2983791369 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc36c017070 con 0x7fc3780caad0 2024-08-31T12:22:45.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.485+0000 7fc374ff9640 1 -- 172.21.15.19:0/2983791369 <== mon.1 v2:172.21.15.43:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc36c005400 con 0x7fc3780caad0 2024-08-31T12:22:45.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.485+0000 7fc37e562640 1 -- 172.21.15.19:0/2983791369 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc37806cb80 con 0x7fc3780caad0 2024-08-31T12:22:45.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.485+0000 7fc374ff9640 1 -- 172.21.15.19:0/2983791369 <== mon.1 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc36c003850 con 0x7fc3780caad0 2024-08-31T12:22:45.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.485+0000 7fc37e562640 1 -- 172.21.15.19:0/2983791369 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc37806ce20 con 0x7fc3780caad0 2024-08-31T12:22:45.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.486+0000 7fc37e562640 1 -- 172.21.15.19:0/2983791369 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc378152150 con 0x7fc3780caad0 2024-08-31T12:22:45.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.487+0000 7fc374ff9640 1 -- 172.21.15.19:0/2983791369 <== mon.1 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fc36c01a070 con 0x7fc3780caad0 2024-08-31T12:22:45.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.488+0000 7fc374ff9640 1 --2- 172.21.15.19:0/2983791369 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc364077a80 0x7fc364079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:45.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.488+0000 7fc374ff9640 1 -- 172.21.15.19:0/2983791369 <== mon.1 v2:172.21.15.43:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fc36c098b60 con 0x7fc3780caad0 2024-08-31T12:22:45.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.488+0000 7fc377fff640 1 --2- 172.21.15.19:0/2983791369 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc364077a80 0x7fc364079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:45.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.488+0000 7fc377fff640 1 --2- 172.21.15.19:0/2983791369 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc364077a80 0x7fc364079f90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc368007fd0 tx=0x7fc368007eb0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:22:45.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.492+0000 7fc374ff9640 1 -- 172.21.15.19:0/2983791369 <== mon.1 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fc36c061160 con 0x7fc3780caad0 2024-08-31T12:22:45.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.837+0000 7fc37e562640 1 -- 172.21.15.19:0/2983791369 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc37806e800 con 0x7fc3780caad0 2024-08-31T12:22:45.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.837+0000 7fc374ff9640 1 -- 172.21.15.19:0/2983791369 <== mon.1 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fc36c065920 con 0x7fc3780caad0 2024-08-31T12:22:45.838 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":50,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:22:45.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.840+0000 7fc37e562640 1 -- 172.21.15.19:0/2983791369 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc364077a80 msgr2=0x7fc364079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:45.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.840+0000 7fc37e562640 1 --2- 172.21.15.19:0/2983791369 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc364077a80 0x7fc364079f90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc368007fd0 tx=0x7fc368007eb0 comp rx=0 tx=0).stop 2024-08-31T12:22:45.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.840+0000 7fc37e562640 1 -- 172.21.15.19:0/2983791369 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc3780caad0 msgr2=0x7fc378152e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:45.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.840+0000 7fc37e562640 1 --2- 172.21.15.19:0/2983791369 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc3780caad0 0x7fc378152e40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc36c00c390 tx=0x7fc36c00c850 comp rx=0 tx=0).stop 2024-08-31T12:22:45.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.840+0000 7fc37e562640 1 -- 172.21.15.19:0/2983791369 shutdown_connections 2024-08-31T12:22:45.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.840+0000 7fc37e562640 1 --2- 172.21.15.19:0/2983791369 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc364077a80 0x7fc364079f90 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:45.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.840+0000 7fc37e562640 1 --2- 172.21.15.19:0/2983791369 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc378158170 0x7fc378154c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:45.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.840+0000 7fc37e562640 1 --2- 172.21.15.19:0/2983791369 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc3780caad0 0x7fc378152e40 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:45.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.840+0000 7fc37e562640 1 --2- 172.21.15.19:0/2983791369 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc3780c8050 0x7fc3781f4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:45.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.840+0000 7fc37e562640 1 -- 172.21.15.19:0/2983791369 >> 172.21.15.19:0/2983791369 conn(0x7fc3780cdcb0 msgr2=0x7fc37814e120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:22:45.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.840+0000 7fc37e562640 1 -- 172.21.15.19:0/2983791369 shutdown_connections 2024-08-31T12:22:45.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.840+0000 7fc37e562640 1 -- 172.21.15.19:0/2983791369 wait complete. 2024-08-31T12:22:45.853 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-08-31T12:22:45.853 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-08-31T12:22:45.853 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-31T12:22:45.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.972+0000 7f97680cc640 1 -- 172.21.15.19:0/1345844646 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9760158330 msgr2=0x7f97600c3a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:45.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.972+0000 7f97680cc640 1 --2- 172.21.15.19:0/1345844646 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9760158330 0x7f97600c3a50 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f975800acf0 tx=0x7f975802bbd0 comp rx=0 tx=0).stop 2024-08-31T12:22:45.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.972+0000 7f97680cc640 1 -- 172.21.15.19:0/1345844646 shutdown_connections 2024-08-31T12:22:45.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.972+0000 7f97680cc640 1 --2- 172.21.15.19:0/1345844646 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9760158330 0x7f97600c3a50 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:45.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.972+0000 7f97680cc640 1 -- 172.21.15.19:0/1345844646 >> 172.21.15.19:0/1345844646 conn(0x7f9760070db0 msgr2=0x7f97600711c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:22:45.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.972+0000 7f97680cc640 1 -- 172.21.15.19:0/1345844646 shutdown_connections 2024-08-31T12:22:45.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.972+0000 7f97680cc640 1 -- 172.21.15.19:0/1345844646 wait complete. 2024-08-31T12:22:45.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.974+0000 7f97680cc640 1 Processor -- start 2024-08-31T12:22:45.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.974+0000 7f97680cc640 1 -- start start 2024-08-31T12:22:45.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.975+0000 7f97680cc640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f97600c3640 0x7f97600c6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:45.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.975+0000 7f97680cc640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9760158330 0x7f9760145820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:45.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.975+0000 7f97680cc640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f97601f1f60 0x7f9760145d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:45.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.976+0000 7f97680cc640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f97600747c0 con 0x7f97600c3640 2024-08-31T12:22:45.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.976+0000 7f97680cc640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9760157790 con 0x7f9760158330 2024-08-31T12:22:45.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.976+0000 7f97680cc640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f9760074940 con 0x7f97601f1f60 2024-08-31T12:22:45.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.976+0000 7f9765e68640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f97600c3640 0x7f97600c6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:45.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.976+0000 7f9765667640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9760158330 0x7f9760145820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:45.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.976+0000 7f9765e68640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f97600c3640 0x7f97600c6650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:59404/0 (socket says 172.21.15.19:59404) 2024-08-31T12:22:45.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.976+0000 7f9766669640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f97601f1f60 0x7f9760145d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:45.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.976+0000 7f9765e68640 1 -- 172.21.15.19:0/1689674343 learned_addr learned my addr 172.21.15.19:0/1689674343 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:22:45.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.976+0000 7f9765e68640 1 -- 172.21.15.19:0/1689674343 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f97601f1f60 msgr2=0x7f9760145d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:45.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.976+0000 7f9765e68640 1 --2- 172.21.15.19:0/1689674343 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f97601f1f60 0x7f9760145d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:45.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.976+0000 7f9765e68640 1 -- 172.21.15.19:0/1689674343 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9760158330 msgr2=0x7f9760145820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:45.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.976+0000 7f9765e68640 1 --2- 172.21.15.19:0/1689674343 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9760158330 0x7f9760145820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:45.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.976+0000 7f9765e68640 1 -- 172.21.15.19:0/1689674343 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9760157430 con 0x7f97600c3640 2024-08-31T12:22:45.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.976+0000 7f9765e68640 1 --2- 172.21.15.19:0/1689674343 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f97600c3640 0x7f97600c6650 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f975c00c460 tx=0x7f975c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:22:45.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.976+0000 7f97567fc640 1 -- 172.21.15.19:0/1689674343 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f975c0052d0 con 0x7f97600c3640 2024-08-31T12:22:45.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.976+0000 7f97567fc640 1 -- 172.21.15.19:0/1689674343 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f975c005490 con 0x7f97600c3640 2024-08-31T12:22:45.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.976+0000 7f97680cc640 1 -- 172.21.15.19:0/1689674343 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9760152060 con 0x7f97600c3640 2024-08-31T12:22:45.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.977+0000 7f97567fc640 1 -- 172.21.15.19:0/1689674343 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f975c012870 con 0x7f97600c3640 2024-08-31T12:22:45.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.977+0000 7f97680cc640 1 -- 172.21.15.19:0/1689674343 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97600c6f90 con 0x7f97600c3640 2024-08-31T12:22:45.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.978+0000 7f97680cc640 1 --2- 172.21.15.19:0/1689674343 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f97601f2380 0x7f976015a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:45.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.978+0000 7f97680cc640 1 -- 172.21.15.19:0/1689674343 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f97600c78a0 con 0x7f97601f2380 2024-08-31T12:22:45.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.978+0000 7f9765667640 1 --2- 172.21.15.19:0/1689674343 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f97601f2380 0x7f976015a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:45.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.978+0000 7f9765667640 1 -- 172.21.15.19:0/1689674343 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f97600c95a0 con 0x7f97601f2380 2024-08-31T12:22:45.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.978+0000 7f9765667640 1 --2- 172.21.15.19:0/1689674343 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f97601f2380 0x7f976015a9c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f97600c7410 tx=0x7f975000fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:22:45.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.978+0000 7f97567fc640 1 -- 172.21.15.19:0/1689674343 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f975c012a10 con 0x7f97600c3640 2024-08-31T12:22:45.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.979+0000 7f97567fc640 1 --2- 172.21.15.19:0/1689674343 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9744077a50 0x7f9744079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:45.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.979+0000 7f97567fc640 1 -- 172.21.15.19:0/1689674343 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f975c099140 con 0x7f97600c3640 2024-08-31T12:22:45.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.979+0000 7f97567fc640 1 -- 172.21.15.19:0/1689674343 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97500050d0 con 0x7f97601f2380 2024-08-31T12:22:45.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.979+0000 7f97567fc640 1 -- 172.21.15.19:0/1689674343 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9750018070 con 0x7f97601f2380 2024-08-31T12:22:45.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.979+0000 7f97567fc640 1 -- 172.21.15.19:0/1689674343 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f97601f2380 msgr2=0x7f976015a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:45.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.979+0000 7f97567fc640 1 --2- 172.21.15.19:0/1689674343 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f97601f2380 0x7f976015a9c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f97600c7410 tx=0x7f975000fc00 comp rx=0 tx=0).stop 2024-08-31T12:22:45.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.979+0000 7f97567fc640 1 -- 172.21.15.19:0/1689674343 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f97601f2380 msgr2=0x7f976015a9c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:22:45.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.979+0000 7f97567fc640 1 --2- 172.21.15.19:0/1689674343 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f97601f2380 0x7f976015a9c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:45.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.979+0000 7f9766669640 1 --2- 172.21.15.19:0/1689674343 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9744077a50 0x7f9744079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:45.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.980+0000 7f9766669640 1 --2- 172.21.15.19:0/1689674343 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9744077a50 0x7f9744079f60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9758035000 tx=0x7f975803a040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:22:45.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.985+0000 7f97680cc640 1 --2- 172.21.15.19:0/1689674343 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f976015af00 0x7f976015b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:45.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.985+0000 7f97680cc640 1 -- 172.21.15.19:0/1689674343 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f976015b850 con 0x7f976015af00 2024-08-31T12:22:45.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.986+0000 7f9765667640 1 --2- 172.21.15.19:0/1689674343 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f976015af00 0x7f976015b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:45.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.986+0000 7f9765667640 1 -- 172.21.15.19:0/1689674343 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f97600c9290 con 0x7f976015af00 2024-08-31T12:22:45.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.986+0000 7f9765667640 1 --2- 172.21.15.19:0/1689674343 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f976015af00 0x7f976015b310 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f9750014000 tx=0x7f9750000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:22:45.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.986+0000 7f97567fc640 1 -- 172.21.15.19:0/1689674343 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f975000d9f0 con 0x7f976015af00 2024-08-31T12:22:45.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.987+0000 7f97567fc640 1 -- 172.21.15.19:0/1689674343 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5044 (secure 0 0 0) 0x7f975000d9f0 con 0x7f976015af00 2024-08-31T12:22:45.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.987+0000 7f97567fc640 1 -- 172.21.15.19:0/1689674343 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f976015af00 msgr2=0x7f976015b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:45.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.987+0000 7f97567fc640 1 --2- 172.21.15.19:0/1689674343 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f976015af00 0x7f976015b310 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f9750014000 tx=0x7f9750000f40 comp rx=0 tx=0).stop 2024-08-31T12:22:45.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.987+0000 7f97567fc640 1 -- 172.21.15.19:0/1689674343 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f976015af00 msgr2=0x7f976015b310 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:22:45.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.987+0000 7f97567fc640 1 --2- 172.21.15.19:0/1689674343 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f976015af00 0x7f976015b310 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:45.988 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:22:45.989 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:22:45.989 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:22:45.989 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "leader", 2024-08-31T12:22:45.989 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 50, 2024-08-31T12:22:45.989 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 345977, 2024-08-31T12:22:45.989 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:22:45.989 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:22:45.989 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:22:45.989 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:22:45.989 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:22:45.989 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 0, 2024-08-31T12:22:45.989 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:22:45.989 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:22:45.989 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:22:45.990 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:22:45.990 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:22:45.990 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:22:45.990 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:22:45.990 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:22:45.990 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:22:45.990 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:22:45.990 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:22:45.990 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:22:45.990 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:22:45.990 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:22:45.990 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:22:45.990 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:22:45.990 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:22:45.990 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:22:45.991 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:22:45.991 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:22:45.991 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:22:45.991 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:22:45.991 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:22:45.991 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:22:45.991 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:22:45.991 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:22:45.991 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:22:45.991 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:22:45.991 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:45.991 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:45.991 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:22:45.991 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:22:45.991 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:22:45.992 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:22:45.992 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:22:45.992 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:22:45.992 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:22:45.992 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:22:45.992 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:22:45.992 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:22:45.992 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:22:45.992 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:22:45.992 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:22:45.992 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:22:45.992 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:22:45.992 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:22:45.993 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:22:45.993 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:22:45.993 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:22:45.993 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:22:45.993 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:22:45.993 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:22:45.993 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:22:45.993 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:22:45.993 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:22:45.993 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:22:45.993 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:22:45.993 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:22:45.993 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:22:45.993 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:22:45.994 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:45.994 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:22:45.994 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:45.994 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:22:45.994 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:22:45.994 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:22:45.994 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:22:45.994 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:45.994 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:22:45.994 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:22:45.994 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:45.994 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:45.995 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:45.995 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:22:45.995 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:22:45.995 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:45.995 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:45.995 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:45.995 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:45.995 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:22:45.995 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:22:45.995 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:22:45.995 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:22:45.995 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:22:45.995 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:45.995 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:45.996 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:22:45.996 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:22:45.996 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:22:45.996 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:22:45.996 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:45.996 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:22:45.996 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:22:45.996 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:45.996 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:45.996 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:45.996 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:22:45.996 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:22:45.996 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:45.997 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:45.997 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:45.997 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:45.997 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:22:45.997 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:22:45.997 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:22:45.997 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:22:45.997 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:22:45.997 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:45.997 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:45.997 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:22:45.997 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:22:45.998 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:22:45.998 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:22:45.998 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:45.998 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:22:45.998 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:22:45.998 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:45.998 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:45.998 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:45.998 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:22:45.998 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:22:45.998 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:45.998 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:45.998 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:45.998 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:45.998 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:22:45.999 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:22:45.999 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:22:45.999 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:22:45.999 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:22:45.999 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:45.999 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:45.999 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:45.999 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:22:45.999 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:22:45.999 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:45.999 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:22:45.999 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:22:45.999 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:22:45.999 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:46.000 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:22:46.000 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:22:46.000 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:46.000 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:22:46.000 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:22:46.000 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 6 2024-08-31T12:22:46.000 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:46.000 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:22:46.000 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:22:46.000 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:46.000 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:22:46.000 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:22:46.000 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 8 2024-08-31T12:22:46.000 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:46.001 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:22:46.001 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:22:46.001 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:46.001 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:22:46.001 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:22:46.001 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 6 2024-08-31T12:22:46.001 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:46.001 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:22:46.001 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:22:46.001 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:46.001 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:22:46.001 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:22:46.001 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:22:46.001 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:46.002 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:46.002 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:46.002 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:22:46.002 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:22:46.003 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.987+0000 7f97680cc640 1 -- 172.21.15.19:0/1689674343 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9744077a50 msgr2=0x7f9744079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:46.003 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.987+0000 7f97680cc640 1 --2- 172.21.15.19:0/1689674343 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9744077a50 0x7f9744079f60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9758035000 tx=0x7f975803a040 comp rx=0 tx=0).stop 2024-08-31T12:22:46.003 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.987+0000 7f97680cc640 1 -- 172.21.15.19:0/1689674343 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f97600c3640 msgr2=0x7f97600c6650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:46.003 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.987+0000 7f97680cc640 1 --2- 172.21.15.19:0/1689674343 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f97600c3640 0x7f97600c6650 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f975c00c460 tx=0x7f975c00c920 comp rx=0 tx=0).stop 2024-08-31T12:22:46.003 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.987+0000 7f9766669640 1 -- 172.21.15.19:0/1689674343 reap_dead start 2024-08-31T12:22:46.003 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.987+0000 7f97680cc640 1 -- 172.21.15.19:0/1689674343 shutdown_connections 2024-08-31T12:22:46.003 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.987+0000 7f97680cc640 1 --2- 172.21.15.19:0/1689674343 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f97600c3640 0x7f97600c6650 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:46.003 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.987+0000 7f97680cc640 1 -- 172.21.15.19:0/1689674343 >> 172.21.15.19:0/1689674343 conn(0x7f9760070db0 msgr2=0x7f9760146540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:22:46.004 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.987+0000 7f97680cc640 1 -- 172.21.15.19:0/1689674343 shutdown_connections 2024-08-31T12:22:46.004 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:45.987+0000 7f97680cc640 1 -- 172.21.15.19:0/1689674343 wait complete. 2024-08-31T12:22:46.004 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:22:46.004 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-08-31T12:22:46.004 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-31T12:22:46.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.146+0000 7fc9855e6640 1 -- 172.21.15.19:0/3212103161 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc968003fd0 msgr2=0x7fc968004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:46.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.146+0000 7fc9855e6640 1 --2- 172.21.15.19:0/3212103161 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc968003fd0 0x7fc968004800 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc974009b20 tx=0x7fc968005830 comp rx=0 tx=0).stop 2024-08-31T12:22:46.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.146+0000 7fc97f7fe640 1 -- 172.21.15.19:0/3212103161 reap_dead start 2024-08-31T12:22:46.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.146+0000 7fc9855e6640 1 -- 172.21.15.19:0/3212103161 shutdown_connections 2024-08-31T12:22:46.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.146+0000 7fc9855e6640 1 -- 172.21.15.19:0/3212103161 >> 172.21.15.19:0/3212103161 conn(0x7fc98006cae0 msgr2=0x7fc98006cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:22:46.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.146+0000 7fc9855e6640 1 -- 172.21.15.19:0/3212103161 shutdown_connections 2024-08-31T12:22:46.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.146+0000 7fc9855e6640 1 -- 172.21.15.19:0/3212103161 wait complete. 2024-08-31T12:22:46.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.148+0000 7fc9855e6640 1 Processor -- start 2024-08-31T12:22:46.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.148+0000 7fc9855e6640 1 -- start start 2024-08-31T12:22:46.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.150+0000 7fc9855e6640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc968003fd0 0x7fc9801efda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:46.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.150+0000 7fc9855e6640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc98015a320 0x7fc980142560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:46.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.150+0000 7fc9855e6640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc980143400 0x7fc980143810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:46.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.150+0000 7fc9855e6640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fc9800c3e90 con 0x7fc98015a320 2024-08-31T12:22:46.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.150+0000 7fc9855e6640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc980152380 con 0x7fc980143400 2024-08-31T12:22:46.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.150+0000 7fc9855e6640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fc9800c4010 con 0x7fc968003fd0 2024-08-31T12:22:46.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.150+0000 7fc97e7fc640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc98015a320 0x7fc980142560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:46.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.150+0000 7fc97e7fc640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc98015a320 0x7fc980142560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:59450/0 (socket says 172.21.15.19:59450) 2024-08-31T12:22:46.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.150+0000 7fc97e7fc640 1 -- 172.21.15.19:0/3391802564 learned_addr learned my addr 172.21.15.19:0/3391802564 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:22:46.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.150+0000 7fc97f7fe640 1 --2- 172.21.15.19:0/3391802564 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc980143400 0x7fc980143810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:46.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.150+0000 7fc97effd640 1 --2- 172.21.15.19:0/3391802564 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc968003fd0 0x7fc9801efda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:46.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.150+0000 7fc97e7fc640 1 -- 172.21.15.19:0/3391802564 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc968003fd0 msgr2=0x7fc9801efda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:46.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.150+0000 7fc97e7fc640 1 --2- 172.21.15.19:0/3391802564 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc968003fd0 0x7fc9801efda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:46.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.150+0000 7fc97e7fc640 1 -- 172.21.15.19:0/3391802564 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc980143400 msgr2=0x7fc980143810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:46.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.150+0000 7fc97e7fc640 1 --2- 172.21.15.19:0/3391802564 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc980143400 0x7fc980143810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:46.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.150+0000 7fc97e7fc640 1 -- 172.21.15.19:0/3391802564 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9801ef600 con 0x7fc98015a320 2024-08-31T12:22:46.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.151+0000 7fc97e7fc640 1 --2- 172.21.15.19:0/3391802564 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc98015a320 0x7fc980142560 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc97402b4d0 tx=0x7fc974033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:22:46.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.151+0000 7fc9677fe640 1 -- 172.21.15.19:0/3391802564 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc974038040 con 0x7fc98015a320 2024-08-31T12:22:46.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.151+0000 7fc9677fe640 1 -- 172.21.15.19:0/3391802564 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc974004020 con 0x7fc98015a320 2024-08-31T12:22:46.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.151+0000 7fc9855e6640 1 -- 172.21.15.19:0/3391802564 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9801f02e0 con 0x7fc98015a320 2024-08-31T12:22:46.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.151+0000 7fc9677fe640 1 -- 172.21.15.19:0/3391802564 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc974004450 con 0x7fc98015a320 2024-08-31T12:22:46.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.151+0000 7fc9855e6640 1 -- 172.21.15.19:0/3391802564 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc98015b510 con 0x7fc98015a320 2024-08-31T12:22:46.155 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.152+0000 7fc9855e6640 1 --2- 172.21.15.19:0/3391802564 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc98006f1a0 0x7fc98006f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:46.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.152+0000 7fc9855e6640 1 -- 172.21.15.19:0/3391802564 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc98015afe0 con 0x7fc98006f1a0 2024-08-31T12:22:46.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.152+0000 7fc97effd640 1 --2- 172.21.15.19:0/3391802564 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc98006f1a0 0x7fc98006f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:46.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.152+0000 7fc97effd640 1 -- 172.21.15.19:0/3391802564 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc9801f3af0 con 0x7fc98006f1a0 2024-08-31T12:22:46.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.153+0000 7fc97effd640 1 --2- 172.21.15.19:0/3391802564 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc98006f1a0 0x7fc98006f5b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc9700018d0 tx=0x7fc9700039c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:22:46.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.153+0000 7fc9677fe640 1 -- 172.21.15.19:0/3391802564 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fc974045020 con 0x7fc98015a320 2024-08-31T12:22:46.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.153+0000 7fc9677fe640 1 --2- 172.21.15.19:0/3391802564 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc96c077a80 0x7fc96c079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:46.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.153+0000 7fc9677fe640 1 -- 172.21.15.19:0/3391802564 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fc9740ba0a0 con 0x7fc98015a320 2024-08-31T12:22:46.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.153+0000 7fc97f7fe640 1 --2- 172.21.15.19:0/3391802564 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc96c077a80 0x7fc96c079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:46.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.154+0000 7fc9677fe640 1 -- 172.21.15.19:0/3391802564 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9700129d0 con 0x7fc98006f1a0 2024-08-31T12:22:46.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.154+0000 7fc9677fe640 1 -- 172.21.15.19:0/3391802564 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc970012b70 con 0x7fc98006f1a0 2024-08-31T12:22:46.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.154+0000 7fc9677fe640 1 -- 172.21.15.19:0/3391802564 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc98006f1a0 msgr2=0x7fc98006f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:46.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.154+0000 7fc9677fe640 1 --2- 172.21.15.19:0/3391802564 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc98006f1a0 0x7fc98006f5b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc9700018d0 tx=0x7fc9700039c0 comp rx=0 tx=0).stop 2024-08-31T12:22:46.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.154+0000 7fc9677fe640 1 -- 172.21.15.19:0/3391802564 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc98006f1a0 msgr2=0x7fc98006f5b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:22:46.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.154+0000 7fc9677fe640 1 --2- 172.21.15.19:0/3391802564 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc98006f1a0 0x7fc98006f5b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:46.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.154+0000 7fc97f7fe640 1 --2- 172.21.15.19:0/3391802564 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc96c077a80 0x7fc96c079f90 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc9800b2f40 tx=0x7fc978009500 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:22:46.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.161+0000 7fc9855e6640 1 --2- 172.21.15.19:0/3391802564 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc98006faf0 0x7fc9800cac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:46.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.161+0000 7fc9855e6640 1 -- 172.21.15.19:0/3391802564 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc980145030 con 0x7fc98006faf0 2024-08-31T12:22:46.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.161+0000 7fc97effd640 1 --2- 172.21.15.19:0/3391802564 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc98006faf0 0x7fc9800cac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:46.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.162+0000 7fc97effd640 1 -- 172.21.15.19:0/3391802564 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc980050d00 con 0x7fc98006faf0 2024-08-31T12:22:46.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.162+0000 7fc97effd640 1 --2- 172.21.15.19:0/3391802564 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc98006faf0 0x7fc9800cac10 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc970018000 tx=0x7fc9800690f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:22:46.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.162+0000 7fc9677fe640 1 -- 172.21.15.19:0/3391802564 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9700129d0 con 0x7fc98006faf0 2024-08-31T12:22:46.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.162+0000 7fc9677fe640 1 -- 172.21.15.19:0/3391802564 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4540 (secure 0 0 0) 0x7fc97000f7f0 con 0x7fc98006faf0 2024-08-31T12:22:46.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.162+0000 7fc9677fe640 1 -- 172.21.15.19:0/3391802564 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc98006faf0 msgr2=0x7fc9800cac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:46.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.162+0000 7fc9677fe640 1 --2- 172.21.15.19:0/3391802564 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc98006faf0 0x7fc9800cac10 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc970018000 tx=0x7fc9800690f0 comp rx=0 tx=0).stop 2024-08-31T12:22:46.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.162+0000 7fc9677fe640 1 -- 172.21.15.19:0/3391802564 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc98006faf0 msgr2=0x7fc9800cac10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:22:46.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.162+0000 7fc9677fe640 1 --2- 172.21.15.19:0/3391802564 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc98006faf0 0x7fc9800cac10 secure :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc970018000 tx=0x7fc9800690f0 comp rx=0 tx=0).stop 2024-08-31T12:22:46.164 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:22:46.164 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:22:46.164 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:22:46.165 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:22:46.165 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 50, 2024-08-31T12:22:46.165 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 75636, 2024-08-31T12:22:46.165 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:22:46.165 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:22:46.165 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:22:46.165 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:22:46.165 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:22:46.165 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 0, 2024-08-31T12:22:46.165 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:22:46.165 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:22:46.165 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:22:46.166 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:22:46.166 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:22:46.166 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:22:46.166 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:22:46.166 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:22:46.166 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:22:46.166 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:22:46.166 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:22:46.166 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:22:46.166 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:22:46.166 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:22:46.166 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:22:46.166 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:22:46.166 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:22:46.167 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:22:46.167 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:22:46.167 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:22:46.167 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:22:46.167 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:22:46.167 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:22:46.167 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:22:46.167 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:22:46.167 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:22:46.167 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:22:46.167 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:22:46.167 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:46.167 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:46.167 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:22:46.168 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:22:46.168 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:22:46.168 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:22:46.168 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:22:46.168 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:22:46.168 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:22:46.168 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:22:46.168 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:22:46.168 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:22:46.168 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:22:46.168 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:22:46.168 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:22:46.168 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:22:46.169 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:22:46.169 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:22:46.169 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:22:46.169 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:22:46.169 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:22:46.169 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:22:46.169 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:22:46.169 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:22:46.169 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:22:46.169 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:22:46.169 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:22:46.169 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:22:46.169 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:22:46.170 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:22:46.170 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:22:46.170 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:22:46.170 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:46.170 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:22:46.170 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:46.170 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:22:46.170 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:22:46.170 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:22:46.170 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:22:46.170 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:46.170 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:22:46.170 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:22:46.171 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:46.171 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:46.171 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:46.171 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:22:46.171 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:22:46.171 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:46.171 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:46.171 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:46.171 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:46.171 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:22:46.171 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:22:46.171 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:22:46.171 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:22:46.171 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:22:46.172 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:46.172 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:46.172 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:22:46.172 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:22:46.172 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:22:46.172 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:22:46.172 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:46.172 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:22:46.172 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:22:46.172 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:46.172 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:46.172 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:46.172 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:22:46.173 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:22:46.173 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:46.173 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:46.173 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:46.173 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:46.173 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:22:46.173 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:22:46.173 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:22:46.173 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:22:46.173 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:22:46.173 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:46.173 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:46.173 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:22:46.174 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:22:46.174 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:22:46.174 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:22:46.174 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:46.174 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:22:46.174 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:22:46.174 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:46.174 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:46.174 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:46.174 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:22:46.174 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:22:46.174 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:46.175 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:46.175 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:46.175 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:46.175 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:22:46.175 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:22:46.175 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:22:46.175 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:22:46.175 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:22:46.175 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:46.175 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:46.175 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:46.175 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:22:46.175 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:22:46.175 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:46.176 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:22:46.176 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:22:46.176 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:22:46.176 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:46.176 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:22:46.176 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:22:46.176 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:46.176 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:22:46.176 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:22:46.176 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:22:46.176 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:46.176 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:46.176 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:46.176 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:22:46.177 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:22:46.178 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.163+0000 7fc9855e6640 1 -- 172.21.15.19:0/3391802564 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc96c077a80 msgr2=0x7fc96c079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:46.178 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.163+0000 7fc9855e6640 1 --2- 172.21.15.19:0/3391802564 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc96c077a80 0x7fc96c079f90 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc9800b2f40 tx=0x7fc978009500 comp rx=0 tx=0).stop 2024-08-31T12:22:46.178 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.163+0000 7fc9855e6640 1 -- 172.21.15.19:0/3391802564 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc98015a320 msgr2=0x7fc980142560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:46.178 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.163+0000 7fc9855e6640 1 --2- 172.21.15.19:0/3391802564 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc98015a320 0x7fc980142560 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc97402b4d0 tx=0x7fc974033670 comp rx=0 tx=0).stop 2024-08-31T12:22:46.178 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.163+0000 7fc97f7fe640 1 -- 172.21.15.19:0/3391802564 reap_dead start 2024-08-31T12:22:46.178 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.163+0000 7fc9855e6640 1 -- 172.21.15.19:0/3391802564 shutdown_connections 2024-08-31T12:22:46.178 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.163+0000 7fc9855e6640 1 --2- 172.21.15.19:0/3391802564 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc98015a320 0x7fc980142560 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:46.178 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.163+0000 7fc9855e6640 1 -- 172.21.15.19:0/3391802564 >> 172.21.15.19:0/3391802564 conn(0x7fc98006cae0 msgr2=0x7fc980144920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:22:46.179 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.163+0000 7fc9855e6640 1 -- 172.21.15.19:0/3391802564 shutdown_connections 2024-08-31T12:22:46.179 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.163+0000 7fc9855e6640 1 -- 172.21.15.19:0/3391802564 wait complete. 2024-08-31T12:22:46.179 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:22:46.179 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-08-31T12:22:46.179 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-31T12:22:46.314 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.313+0000 7f1d434d5640 1 -- 172.21.15.19:0/362060314 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d28004800 msgr2=0x7f1d28004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:46.314 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.313+0000 7f1d434d5640 1 --2- 172.21.15.19:0/362060314 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d28004800 0x7f1d28004c10 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c073b10 tx=0x7f1d28005830 comp rx=0 tx=0).stop 2024-08-31T12:22:46.314 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.313+0000 7f1d434d5640 1 -- 172.21.15.19:0/362060314 shutdown_connections 2024-08-31T12:22:46.314 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.313+0000 7f1d434d5640 1 --2- 172.21.15.19:0/362060314 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d28004800 0x7f1d28004c10 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:46.314 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.313+0000 7f1d434d5640 1 -- 172.21.15.19:0/362060314 >> 172.21.15.19:0/362060314 conn(0x7f1d3c0ce0b0 msgr2=0x7f1d3c0c9f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:22:46.314 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.313+0000 7f1d434d5640 1 -- 172.21.15.19:0/362060314 shutdown_connections 2024-08-31T12:22:46.314 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.313+0000 7f1d434d5640 1 -- 172.21.15.19:0/362060314 wait complete. 2024-08-31T12:22:46.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.315+0000 7f1d434d5640 1 Processor -- start 2024-08-31T12:22:46.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.315+0000 7f1d434d5640 1 -- start start 2024-08-31T12:22:46.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.316+0000 7f1d434d5640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d28004800 0x7f1d3c1def50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:46.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.316+0000 7f1d434d5640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d3c06cd80 0x7f1d3c14adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:46.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.316+0000 7f1d434d5640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d3c14b300 0x7f1d3c1f25e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:46.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.316+0000 7f1d434d5640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f1d3c073d40 con 0x7f1d3c14b300 2024-08-31T12:22:46.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.316+0000 7f1d434d5640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1d3c0c76e0 con 0x7f1d3c06cd80 2024-08-31T12:22:46.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.316+0000 7f1d434d5640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f1d3c073ec0 con 0x7f1d28004800 2024-08-31T12:22:46.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.316+0000 7f1d41a72640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d3c14b300 0x7f1d3c1f25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:46.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.317+0000 7f1d41a72640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d3c14b300 0x7f1d3c1f25e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:59472/0 (socket says 172.21.15.19:59472) 2024-08-31T12:22:46.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.317+0000 7f1d41a72640 1 -- 172.21.15.19:0/1509239185 learned_addr learned my addr 172.21.15.19:0/1509239185 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:22:46.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.317+0000 7f1d41271640 1 --2- 172.21.15.19:0/1509239185 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d28004800 0x7f1d3c1def50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:46.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.317+0000 7f1d40a70640 1 --2- 172.21.15.19:0/1509239185 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d3c06cd80 0x7f1d3c14adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:46.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.317+0000 7f1d41a72640 1 -- 172.21.15.19:0/1509239185 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d28004800 msgr2=0x7f1d3c1def50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:46.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.317+0000 7f1d41a72640 1 --2- 172.21.15.19:0/1509239185 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d28004800 0x7f1d3c1def50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:46.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.317+0000 7f1d41a72640 1 -- 172.21.15.19:0/1509239185 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d3c06cd80 msgr2=0x7f1d3c14adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:46.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.317+0000 7f1d41a72640 1 --2- 172.21.15.19:0/1509239185 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d3c06cd80 0x7f1d3c14adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:46.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.317+0000 7f1d41a72640 1 -- 172.21.15.19:0/1509239185 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d3c1df490 con 0x7f1d3c14b300 2024-08-31T12:22:46.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.317+0000 7f1d41a72640 1 --2- 172.21.15.19:0/1509239185 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d3c14b300 0x7f1d3c1f25e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c0c6320 tx=0x7f1d3400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:22:46.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.317+0000 7f1d31ffb640 1 -- 172.21.15.19:0/1509239185 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d34004d20 con 0x7f1d3c14b300 2024-08-31T12:22:46.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.317+0000 7f1d31ffb640 1 -- 172.21.15.19:0/1509239185 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1d34003970 con 0x7f1d3c14b300 2024-08-31T12:22:46.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.317+0000 7f1d31ffb640 1 -- 172.21.15.19:0/1509239185 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d34003dc0 con 0x7f1d3c14b300 2024-08-31T12:22:46.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.317+0000 7f1d434d5640 1 -- 172.21.15.19:0/1509239185 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d3c154100 con 0x7f1d3c14b300 2024-08-31T12:22:46.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.318+0000 7f1d434d5640 1 -- 172.21.15.19:0/1509239185 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d3c1de470 con 0x7f1d3c14b300 2024-08-31T12:22:46.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.318+0000 7f1d434d5640 1 --2- 172.21.15.19:0/1509239185 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d3c1e3af0 0x7f1d3c1de6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:46.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.318+0000 7f1d434d5640 1 -- 172.21.15.19:0/1509239185 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f1d3c1df660 con 0x7f1d3c1e3af0 2024-08-31T12:22:46.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.319+0000 7f1d41271640 1 --2- 172.21.15.19:0/1509239185 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d3c1e3af0 0x7f1d3c1de6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:46.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.319+0000 7f1d41271640 1 -- 172.21.15.19:0/1509239185 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1d3c1526f0 con 0x7f1d3c1e3af0 2024-08-31T12:22:46.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.319+0000 7f1d41271640 1 --2- 172.21.15.19:0/1509239185 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d3c1e3af0 0x7f1d3c1de6b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c0526b0 tx=0x7f1d38003a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:22:46.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.319+0000 7f1d31ffb640 1 -- 172.21.15.19:0/1509239185 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d38012910 con 0x7f1d3c1e3af0 2024-08-31T12:22:46.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.320+0000 7f1d31ffb640 1 -- 172.21.15.19:0/1509239185 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f1d34004030 con 0x7f1d3c14b300 2024-08-31T12:22:46.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.320+0000 7f1d31ffb640 1 --2- 172.21.15.19:0/1509239185 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1d20077980 0x7f1d20079e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:46.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.320+0000 7f1d31ffb640 1 -- 172.21.15.19:0/1509239185 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f1d3409a480 con 0x7f1d3c14b300 2024-08-31T12:22:46.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.320+0000 7f1d40a70640 1 --2- 172.21.15.19:0/1509239185 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1d20077980 0x7f1d20079e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:46.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.320+0000 7f1d31ffb640 1 -- 172.21.15.19:0/1509239185 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1d38012ab0 con 0x7f1d3c1e3af0 2024-08-31T12:22:46.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.320+0000 7f1d31ffb640 1 -- 172.21.15.19:0/1509239185 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d3c1e3af0 msgr2=0x7f1d3c1de6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:46.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.320+0000 7f1d31ffb640 1 --2- 172.21.15.19:0/1509239185 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d3c1e3af0 0x7f1d3c1de6b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c0526b0 tx=0x7f1d38003a80 comp rx=0 tx=0).stop 2024-08-31T12:22:46.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.320+0000 7f1d31ffb640 1 -- 172.21.15.19:0/1509239185 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d3c1e3af0 msgr2=0x7f1d3c1de6b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:22:46.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.320+0000 7f1d31ffb640 1 --2- 172.21.15.19:0/1509239185 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d3c1e3af0 0x7f1d3c1de6b0 secure :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c0526b0 tx=0x7f1d38003a80 comp rx=0 tx=0).stop 2024-08-31T12:22:46.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.321+0000 7f1d40a70640 1 --2- 172.21.15.19:0/1509239185 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1d20077980 0x7f1d20079e90 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c032000 tx=0x7f1d2c001360 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:22:46.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.327+0000 7f1d434d5640 1 --2- 172.21.15.19:0/1509239185 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d3c1f2b20 0x7f1d3c1f2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:46.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.327+0000 7f1d434d5640 1 -- 172.21.15.19:0/1509239185 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f1d3c1f3470 con 0x7f1d3c1f2b20 2024-08-31T12:22:46.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.328+0000 7f1d41271640 1 --2- 172.21.15.19:0/1509239185 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d3c1f2b20 0x7f1d3c1f2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:46.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.328+0000 7f1d41271640 1 -- 172.21.15.19:0/1509239185 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1d3c1542d0 con 0x7f1d3c1f2b20 2024-08-31T12:22:46.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.328+0000 7f1d41271640 1 --2- 172.21.15.19:0/1509239185 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d3c1f2b20 0x7f1d3c1f2f30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c0526b0 tx=0x7f1d38006720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:22:46.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.328+0000 7f1d31ffb640 1 -- 172.21.15.19:0/1509239185 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d38029460 con 0x7f1d3c1f2b20 2024-08-31T12:22:46.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.329+0000 7f1d31ffb640 1 -- 172.21.15.19:0/1509239185 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4540 (secure 0 0 0) 0x7f1d38029460 con 0x7f1d3c1f2b20 2024-08-31T12:22:46.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.329+0000 7f1d31ffb640 1 -- 172.21.15.19:0/1509239185 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d3c1f2b20 msgr2=0x7f1d3c1f2f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:46.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.329+0000 7f1d31ffb640 1 --2- 172.21.15.19:0/1509239185 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d3c1f2b20 0x7f1d3c1f2f30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c0526b0 tx=0x7f1d38006720 comp rx=0 tx=0).stop 2024-08-31T12:22:46.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.329+0000 7f1d31ffb640 1 -- 172.21.15.19:0/1509239185 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d3c1f2b20 msgr2=0x7f1d3c1f2f30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:22:46.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.329+0000 7f1d31ffb640 1 --2- 172.21.15.19:0/1509239185 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d3c1f2b20 0x7f1d3c1f2f30 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:46.331 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:22:46.331 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:22:46.331 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:22:46.331 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:22:46.331 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 50, 2024-08-31T12:22:46.331 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 15874, 2024-08-31T12:22:46.331 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:22:46.331 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:22:46.331 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:22:46.331 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:22:46.331 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:22:46.331 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 0, 2024-08-31T12:22:46.331 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:22:46.332 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:22:46.332 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:22:46.332 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:22:46.332 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:22:46.332 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:22:46.332 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:22:46.332 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:22:46.332 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:22:46.332 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:22:46.332 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:22:46.332 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:22:46.332 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:22:46.332 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:22:46.333 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:22:46.333 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:22:46.333 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:22:46.333 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:22:46.333 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:22:46.333 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:22:46.333 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:22:46.333 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:22:46.333 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:22:46.333 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:22:46.333 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:22:46.333 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:22:46.333 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:22:46.333 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:22:46.334 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:46.334 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:46.334 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:22:46.334 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:22:46.334 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:22:46.334 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:22:46.334 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:22:46.334 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:22:46.334 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:22:46.334 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:22:46.334 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:22:46.334 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:22:46.334 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:22:46.334 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:22:46.335 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:22:46.335 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:22:46.335 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:22:46.335 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:22:46.335 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:22:46.335 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:22:46.335 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:22:46.335 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:22:46.335 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:22:46.335 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:22:46.335 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:22:46.335 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:22:46.335 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:22:46.335 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:22:46.336 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:22:46.336 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:22:46.336 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:22:46.336 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:22:46.336 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:46.336 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:22:46.336 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:46.336 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:22:46.336 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:22:46.336 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:22:46.336 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:22:46.336 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:46.336 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:22:46.336 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:22:46.337 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:46.337 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:46.337 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:46.337 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:22:46.337 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:22:46.337 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:46.337 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:46.337 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:46.337 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:46.337 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:22:46.337 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:22:46.337 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:22:46.337 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:22:46.337 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:22:46.338 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:46.338 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:46.338 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:22:46.338 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:22:46.338 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:22:46.338 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:22:46.338 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:46.338 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:22:46.338 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:22:46.338 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:46.338 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:46.338 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:46.338 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:22:46.338 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:22:46.339 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:46.339 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:46.339 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:46.339 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:46.339 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:22:46.339 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:22:46.339 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:22:46.339 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:22:46.339 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:22:46.339 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:46.339 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:46.339 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:22:46.339 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:22:46.339 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:22:46.339 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:22:46.340 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:46.340 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:22:46.340 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:22:46.340 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:46.340 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:46.340 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:46.340 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:22:46.340 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:22:46.340 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:46.340 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:46.340 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:46.340 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:46.340 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:22:46.341 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:22:46.341 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:22:46.341 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:22:46.341 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:22:46.341 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:46.341 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:46.341 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:46.341 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:22:46.341 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:22:46.341 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:46.341 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:22:46.341 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:22:46.341 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:22:46.341 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:46.342 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:22:46.342 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:22:46.342 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:46.342 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:22:46.342 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:22:46.342 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:22:46.342 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:46.342 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:46.342 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:46.342 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:22:46.342 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:22:46.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.329+0000 7f1d434d5640 1 -- 172.21.15.19:0/1509239185 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1d20077980 msgr2=0x7f1d20079e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:46.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.329+0000 7f1d434d5640 1 --2- 172.21.15.19:0/1509239185 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1d20077980 0x7f1d20079e90 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c032000 tx=0x7f1d2c001360 comp rx=0 tx=0).stop 2024-08-31T12:22:46.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.329+0000 7f1d434d5640 1 -- 172.21.15.19:0/1509239185 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d3c14b300 msgr2=0x7f1d3c1f25e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:46.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.329+0000 7f1d434d5640 1 --2- 172.21.15.19:0/1509239185 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d3c14b300 0x7f1d3c1f25e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c0c6320 tx=0x7f1d3400f790 comp rx=0 tx=0).stop 2024-08-31T12:22:46.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.329+0000 7f1d41a72640 1 -- 172.21.15.19:0/1509239185 reap_dead start 2024-08-31T12:22:46.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.329+0000 7f1d434d5640 1 -- 172.21.15.19:0/1509239185 shutdown_connections 2024-08-31T12:22:46.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.329+0000 7f1d434d5640 1 --2- 172.21.15.19:0/1509239185 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d3c14b300 0x7f1d3c1f25e0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:46.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.329+0000 7f1d434d5640 1 -- 172.21.15.19:0/1509239185 >> 172.21.15.19:0/1509239185 conn(0x7f1d3c0ce0b0 msgr2=0x7f1d3c070df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:22:46.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.329+0000 7f1d434d5640 1 -- 172.21.15.19:0/1509239185 shutdown_connections 2024-08-31T12:22:46.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.329+0000 7f1d434d5640 1 -- 172.21.15.19:0/1509239185 wait complete. 2024-08-31T12:22:46.344 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:22:46.345 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-08-31T12:22:46.345 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-08-31T12:22:46.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.467+0000 7f6a61d39640 1 -- 172.21.15.19:0/3241954089 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6a5c0c8a40 msgr2=0x7f6a5c0c9850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:46.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.467+0000 7f6a61d39640 1 --2- 172.21.15.19:0/3241954089 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6a5c0c8a40 0x7f6a5c0c9850 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c005fd0 tx=0x7f6a4c02a670 comp rx=0 tx=0).stop 2024-08-31T12:22:46.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.467+0000 7f6a61d39640 1 -- 172.21.15.19:0/3241954089 shutdown_connections 2024-08-31T12:22:46.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.467+0000 7f6a61d39640 1 --2- 172.21.15.19:0/3241954089 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6a5c06e0e0 0x7f6a5c155300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:46.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.467+0000 7f6a61d39640 1 --2- 172.21.15.19:0/3241954089 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6a5c0c8a40 0x7f6a5c0c9850 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:46.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.467+0000 7f6a61d39640 1 -- 172.21.15.19:0/3241954089 >> 172.21.15.19:0/3241954089 conn(0x7f6a5c0690d0 msgr2=0x7f6a5c0d0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:22:46.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.467+0000 7f6a61d39640 1 -- 172.21.15.19:0/3241954089 shutdown_connections 2024-08-31T12:22:46.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.467+0000 7f6a61d39640 1 -- 172.21.15.19:0/3241954089 wait complete. 2024-08-31T12:22:46.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.469+0000 7f6a61d39640 1 Processor -- start 2024-08-31T12:22:46.470 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.469+0000 7f6a61d39640 1 -- start start 2024-08-31T12:22:46.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.470+0000 7f6a61d39640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6a5c06e0e0 0x7f6a5c1e5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:46.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.470+0000 7f6a61d39640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6a5c0c8a40 0x7f6a5c0c5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:46.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.470+0000 7f6a61d39640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a5c15b6e0 0x7f6a5c0c26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:46.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.470+0000 7f6a61d39640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f6a5c0c8610 con 0x7f6a5c0c8a40 2024-08-31T12:22:46.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.470+0000 7f6a61d39640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6a5c0c8330 con 0x7f6a5c15b6e0 2024-08-31T12:22:46.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.470+0000 7f6a61d39640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f6a5c06d710 con 0x7f6a5c06e0e0 2024-08-31T12:22:46.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.470+0000 7f6a5affd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6a5c0c8a40 0x7f6a5c0c5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:46.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.470+0000 7f6a5affd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6a5c0c8a40 0x7f6a5c0c5ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:59494/0 (socket says 172.21.15.19:59494) 2024-08-31T12:22:46.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.470+0000 7f6a5affd640 1 -- 172.21.15.19:0/2569077123 learned_addr learned my addr 172.21.15.19:0/2569077123 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:22:46.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.470+0000 7f6a5b7fe640 1 --2- 172.21.15.19:0/2569077123 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6a5c06e0e0 0x7f6a5c1e5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:46.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.470+0000 7f6a5bfff640 1 --2- 172.21.15.19:0/2569077123 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a5c15b6e0 0x7f6a5c0c26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:46.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.471+0000 7f6a5b7fe640 1 -- 172.21.15.19:0/2569077123 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a5c15b6e0 msgr2=0x7f6a5c0c26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:46.474 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.471+0000 7f6a5b7fe640 1 --2- 172.21.15.19:0/2569077123 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a5c15b6e0 0x7f6a5c0c26d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:46.474 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.471+0000 7f6a5b7fe640 1 -- 172.21.15.19:0/2569077123 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6a5c0c8a40 msgr2=0x7f6a5c0c5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:46.474 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.471+0000 7f6a5b7fe640 1 --2- 172.21.15.19:0/2569077123 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6a5c0c8a40 0x7f6a5c0c5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:46.474 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.471+0000 7f6a5b7fe640 1 -- 172.21.15.19:0/2569077123 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a5c1e6320 con 0x7f6a5c06e0e0 2024-08-31T12:22:46.474 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.471+0000 7f6a5affd640 1 --2- 172.21.15.19:0/2569077123 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6a5c0c8a40 0x7f6a5c0c5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:22:46.474 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.471+0000 7f6a5b7fe640 1 --2- 172.21.15.19:0/2569077123 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6a5c06e0e0 0x7f6a5c1e5de0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c0d1820 tx=0x7f6a4c0038a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:22:46.474 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.471+0000 7f6a43fff640 1 -- 172.21.15.19:0/2569077123 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a4c03e030 con 0x7f6a5c06e0e0 2024-08-31T12:22:46.474 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.471+0000 7f6a43fff640 1 -- 172.21.15.19:0/2569077123 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6a4c003db0 con 0x7f6a5c06e0e0 2024-08-31T12:22:46.474 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.471+0000 7f6a61d39640 1 -- 172.21.15.19:0/2569077123 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a5c0c69d0 con 0x7f6a5c06e0e0 2024-08-31T12:22:46.474 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.471+0000 7f6a43fff640 1 -- 172.21.15.19:0/2569077123 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a4c035040 con 0x7f6a5c06e0e0 2024-08-31T12:22:46.474 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.471+0000 7f6a61d39640 1 -- 172.21.15.19:0/2569077123 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a5c0c6f40 con 0x7f6a5c06e0e0 2024-08-31T12:22:46.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.472+0000 7f6a61d39640 1 -- 172.21.15.19:0/2569077123 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a5c15abd0 con 0x7f6a5c06e0e0 2024-08-31T12:22:46.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.473+0000 7f6a43fff640 1 -- 172.21.15.19:0/2569077123 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f6a4c041030 con 0x7f6a5c06e0e0 2024-08-31T12:22:46.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.474+0000 7f6a43fff640 1 --2- 172.21.15.19:0/2569077123 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6a48077a80 0x7f6a48079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:46.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.474+0000 7f6a43fff640 1 -- 172.21.15.19:0/2569077123 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f6a4c0b9c70 con 0x7f6a5c06e0e0 2024-08-31T12:22:46.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.474+0000 7f6a5affd640 1 --2- 172.21.15.19:0/2569077123 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6a48077a80 0x7f6a48079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:46.476 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.475+0000 7f6a5affd640 1 --2- 172.21.15.19:0/2569077123 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6a48077a80 0x7f6a48079f90 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c0cb570 tx=0x7f6a50005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:22:46.480 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.479+0000 7f6a43fff640 1 -- 172.21.15.19:0/2569077123 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f6a5c15abd0 con 0x7f6a5c06e0e0 2024-08-31T12:22:46.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.768+0000 7f6a61d39640 1 -- 172.21.15.19:0/2569077123 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f6a5c0c9850 con 0x7f6a5c06e0e0 2024-08-31T12:22:46.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.770+0000 7f6a43fff640 1 -- 172.21.15.19:0/2569077123 <== mon.2 v2:172.21.15.43:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f6a4c082310 con 0x7f6a5c06e0e0 2024-08-31T12:22:46.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.773+0000 7f6a61d39640 1 -- 172.21.15.19:0/2569077123 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6a48077a80 msgr2=0x7f6a48079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:46.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.773+0000 7f6a61d39640 1 --2- 172.21.15.19:0/2569077123 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6a48077a80 0x7f6a48079f90 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c0cb570 tx=0x7f6a50005f90 comp rx=0 tx=0).stop 2024-08-31T12:22:46.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.773+0000 7f6a61d39640 1 -- 172.21.15.19:0/2569077123 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6a5c06e0e0 msgr2=0x7f6a5c1e5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:46.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.773+0000 7f6a61d39640 1 --2- 172.21.15.19:0/2569077123 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6a5c06e0e0 0x7f6a5c1e5de0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c0d1820 tx=0x7f6a4c0038a0 comp rx=0 tx=0).stop 2024-08-31T12:22:46.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.773+0000 7f6a61d39640 1 -- 172.21.15.19:0/2569077123 shutdown_connections 2024-08-31T12:22:46.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.773+0000 7f6a61d39640 1 --2- 172.21.15.19:0/2569077123 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6a48077a80 0x7f6a48079f90 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:46.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.773+0000 7f6a61d39640 1 --2- 172.21.15.19:0/2569077123 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a5c15b6e0 0x7f6a5c0c26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:46.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.773+0000 7f6a61d39640 1 --2- 172.21.15.19:0/2569077123 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6a5c0c8a40 0x7f6a5c0c5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:46.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.773+0000 7f6a61d39640 1 --2- 172.21.15.19:0/2569077123 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6a5c06e0e0 0x7f6a5c1e5de0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:46.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.773+0000 7f6a61d39640 1 -- 172.21.15.19:0/2569077123 >> 172.21.15.19:0/2569077123 conn(0x7f6a5c0690d0 msgr2=0x7f6a5c150da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:22:46.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.773+0000 7f6a61d39640 1 -- 172.21.15.19:0/2569077123 shutdown_connections 2024-08-31T12:22:46.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:46.773+0000 7f6a61d39640 1 -- 172.21.15.19:0/2569077123 wait complete. 2024-08-31T12:22:46.787 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2024-08-31T12:22:56.788 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-08-31T12:22:56.788 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:22:56.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:56.915+0000 7f745bfba640 1 -- 172.21.15.19:0/3587342677 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f745406fab0 msgr2=0x7f7454157f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:56.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:56.915+0000 7f745bfba640 1 --2- 172.21.15.19:0/3587342677 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f745406fab0 0x7f7454157f60 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f745000a9f0 tx=0x7f745002b850 comp rx=0 tx=0).stop 2024-08-31T12:22:56.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:56.915+0000 7f745bfba640 1 -- 172.21.15.19:0/3587342677 shutdown_connections 2024-08-31T12:22:56.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:56.915+0000 7f745bfba640 1 --2- 172.21.15.19:0/3587342677 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f74540c7810 0x7f74540c3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:56.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:56.915+0000 7f745bfba640 1 --2- 172.21.15.19:0/3587342677 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f745406fab0 0x7f7454157f60 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:56.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:56.915+0000 7f745bfba640 1 -- 172.21.15.19:0/3587342677 >> 172.21.15.19:0/3587342677 conn(0x7f745406cae0 msgr2=0x7f745406cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:22:56.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:56.915+0000 7f745bfba640 1 -- 172.21.15.19:0/3587342677 shutdown_connections 2024-08-31T12:22:56.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:56.915+0000 7f745bfba640 1 -- 172.21.15.19:0/3587342677 wait complete. 2024-08-31T12:22:56.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:56.917+0000 7f745bfba640 1 Processor -- start 2024-08-31T12:22:56.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:56.917+0000 7f745bfba640 1 -- start start 2024-08-31T12:22:56.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:56.918+0000 7f745bfba640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f745406fab0 0x7f74541f1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:56.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:56.918+0000 7f745bfba640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74540c7810 0x7f74541500f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:56.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:56.918+0000 7f745bfba640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f74541511b0 0x7f74540c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:56.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:56.918+0000 7f745bfba640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f7454157410 con 0x7f745406fab0 2024-08-31T12:22:56.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:56.918+0000 7f745bfba640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f74540c8c90 con 0x7f74540c7810 2024-08-31T12:22:56.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:56.918+0000 7f745bfba640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f74540c1ec0 con 0x7f74541511b0 2024-08-31T12:22:56.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:56.919+0000 7f7459d56640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f745406fab0 0x7f74541f1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:56.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:56.919+0000 7f7459d56640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f745406fab0 0x7f74541f1d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:47986/0 (socket says 172.21.15.19:47986) 2024-08-31T12:22:56.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:56.919+0000 7f7459d56640 1 -- 172.21.15.19:0/501943027 learned_addr learned my addr 172.21.15.19:0/501943027 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:22:56.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:56.919+0000 7f7459555640 1 --2- 172.21.15.19:0/501943027 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74540c7810 0x7f74541500f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:56.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:56.919+0000 7f745a557640 1 --2- 172.21.15.19:0/501943027 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f74541511b0 0x7f74540c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:56.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:56.919+0000 7f7459d56640 1 -- 172.21.15.19:0/501943027 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f74541511b0 msgr2=0x7f74540c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:56.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:56.919+0000 7f7459d56640 1 --2- 172.21.15.19:0/501943027 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f74541511b0 0x7f74540c4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:56.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:56.919+0000 7f7459d56640 1 -- 172.21.15.19:0/501943027 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74540c7810 msgr2=0x7f74541500f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:56.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:56.919+0000 7f7459d56640 1 --2- 172.21.15.19:0/501943027 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74540c7810 0x7f74541500f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:56.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:56.919+0000 7f7459d56640 1 -- 172.21.15.19:0/501943027 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74540cb030 con 0x7f745406fab0 2024-08-31T12:22:56.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:56.919+0000 7f7459d56640 1 --2- 172.21.15.19:0/501943027 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f745406fab0 0x7f74541f1d20 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f745000ab10 tx=0x7f7450004dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:22:56.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:56.919+0000 7f744a7fc640 1 -- 172.21.15.19:0/501943027 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7450041020 con 0x7f745406fab0 2024-08-31T12:22:56.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:56.919+0000 7f744a7fc640 1 -- 172.21.15.19:0/501943027 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7450031a50 con 0x7f745406fab0 2024-08-31T12:22:56.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:56.919+0000 7f745bfba640 1 -- 172.21.15.19:0/501943027 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74540748e0 con 0x7f745406fab0 2024-08-31T12:22:56.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:56.919+0000 7f744a7fc640 1 -- 172.21.15.19:0/501943027 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7450031ea0 con 0x7f745406fab0 2024-08-31T12:22:56.923 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:56.920+0000 7f745bfba640 1 -- 172.21.15.19:0/501943027 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74541f1520 con 0x7f745406fab0 2024-08-31T12:22:56.923 DEBUG:teuthology.orchestra.run.smithi019:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:22:56.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:56.921+0000 7f745bfba640 1 -- 172.21.15.19:0/501943027 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74540b2e20 con 0x7f745406fab0 2024-08-31T12:22:56.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:56.921+0000 7f744a7fc640 1 -- 172.21.15.19:0/501943027 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f7450044020 con 0x7f745406fab0 2024-08-31T12:22:56.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:56.922+0000 7f744a7fc640 1 --2- 172.21.15.19:0/501943027 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7438077a50 0x7f7438079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:56.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:56.922+0000 7f744a7fc640 1 -- 172.21.15.19:0/501943027 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f74500ba550 con 0x7f745406fab0 2024-08-31T12:22:56.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:56.922+0000 7f7459555640 1 --2- 172.21.15.19:0/501943027 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7438077a50 0x7f7438079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:56.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:56.923+0000 7f7459555640 1 --2- 172.21.15.19:0/501943027 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7438077a50 0x7f7438079f60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f74540ccc40 tx=0x7f744400a250 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:22:56.926 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:22:56.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:56.929+0000 7f744a7fc640 1 -- 172.21.15.19:0/501943027 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f745004b070 con 0x7f745406fab0 2024-08-31T12:22:57.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.216+0000 7f745bfba640 1 -- 172.21.15.19:0/501943027 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f745406f3d0 con 0x7f745406fab0 2024-08-31T12:22:57.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.217+0000 7f744a7fc640 1 -- 172.21.15.19:0/501943027 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f74500b8e20 con 0x7f745406fab0 2024-08-31T12:22:57.218 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":50,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:22:57.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.219+0000 7f745bfba640 1 -- 172.21.15.19:0/501943027 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7438077a50 msgr2=0x7f7438079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:57.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.219+0000 7f745bfba640 1 --2- 172.21.15.19:0/501943027 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7438077a50 0x7f7438079f60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f74540ccc40 tx=0x7f744400a250 comp rx=0 tx=0).stop 2024-08-31T12:22:57.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.219+0000 7f745bfba640 1 -- 172.21.15.19:0/501943027 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f745406fab0 msgr2=0x7f74541f1d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:57.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.219+0000 7f745bfba640 1 --2- 172.21.15.19:0/501943027 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f745406fab0 0x7f74541f1d20 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f745000ab10 tx=0x7f7450004dc0 comp rx=0 tx=0).stop 2024-08-31T12:22:57.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.219+0000 7f745bfba640 1 -- 172.21.15.19:0/501943027 shutdown_connections 2024-08-31T12:22:57.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.219+0000 7f745bfba640 1 --2- 172.21.15.19:0/501943027 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7438077a50 0x7f7438079f60 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:57.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.219+0000 7f745bfba640 1 --2- 172.21.15.19:0/501943027 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f74541511b0 0x7f74540c4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:57.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.219+0000 7f745bfba640 1 --2- 172.21.15.19:0/501943027 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74540c7810 0x7f74541500f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:57.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.219+0000 7f745bfba640 1 --2- 172.21.15.19:0/501943027 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f745406fab0 0x7f74541f1d20 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:57.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.219+0000 7f745bfba640 1 -- 172.21.15.19:0/501943027 >> 172.21.15.19:0/501943027 conn(0x7f745406cae0 msgr2=0x7f7454073300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:22:57.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.219+0000 7f745bfba640 1 -- 172.21.15.19:0/501943027 shutdown_connections 2024-08-31T12:22:57.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.220+0000 7f745bfba640 1 -- 172.21.15.19:0/501943027 wait complete. 2024-08-31T12:22:57.234 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-08-31T12:22:57.234 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-08-31T12:22:57.234 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-08-31T12:22:57.234 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-31T12:22:57.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.358+0000 7f67cac47640 1 -- 172.21.15.19:0/950461015 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67c40cb2b0 msgr2=0x7f67c414e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:57.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.358+0000 7f67cac47640 1 --2- 172.21.15.19:0/950461015 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67c40cb2b0 0x7f67c414e460 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f67c40710d0 tx=0x7f67b002a680 comp rx=0 tx=0).stop 2024-08-31T12:22:57.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.358+0000 7f67cac47640 1 -- 172.21.15.19:0/950461015 shutdown_connections 2024-08-31T12:22:57.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.358+0000 7f67cac47640 1 --2- 172.21.15.19:0/950461015 >> v2:172.21.15.43:3300/0 conn(0x7f67c414e9a0 0x7f67c40cc400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:57.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.358+0000 7f67cac47640 1 --2- 172.21.15.19:0/950461015 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67c40cb2b0 0x7f67c414e460 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:57.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.358+0000 7f67cac47640 1 -- 172.21.15.19:0/950461015 >> 172.21.15.19:0/950461015 conn(0x7f67c40ca670 msgr2=0x7f67c40cd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:22:57.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.358+0000 7f67cac47640 1 -- 172.21.15.19:0/950461015 shutdown_connections 2024-08-31T12:22:57.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.358+0000 7f67cac47640 1 -- 172.21.15.19:0/950461015 wait complete. 2024-08-31T12:22:57.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.360+0000 7f67cac47640 1 Processor -- start 2024-08-31T12:22:57.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.360+0000 7f67cac47640 1 -- start start 2024-08-31T12:22:57.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.362+0000 7f67cac47640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f67c40c5370 0x7f67c414ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:57.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.362+0000 7f67cac47640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67c40cb2b0 0x7f67c4074010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:57.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.362+0000 7f67cac47640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f67c414e9a0 0x7f67c4143db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:57.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.362+0000 7f67cac47640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f67c40c7650 con 0x7f67c40cb2b0 2024-08-31T12:22:57.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.362+0000 7f67cac47640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f67c40ce140 con 0x7f67c40c5370 2024-08-31T12:22:57.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.362+0000 7f67cac47640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f67c40c72e0 con 0x7f67c414e9a0 2024-08-31T12:22:57.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.362+0000 7f67c3fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67c40cb2b0 0x7f67c4074010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:57.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.362+0000 7f67c3fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67c40cb2b0 0x7f67c4074010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:48006/0 (socket says 172.21.15.19:48006) 2024-08-31T12:22:57.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.362+0000 7f67c3fff640 1 -- 172.21.15.19:0/1361192297 learned_addr learned my addr 172.21.15.19:0/1361192297 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:22:57.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.362+0000 7f67c89e3640 1 --2- 172.21.15.19:0/1361192297 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f67c40c5370 0x7f67c414ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:57.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.362+0000 7f67c91e4640 1 --2- 172.21.15.19:0/1361192297 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f67c414e9a0 0x7f67c4143db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:57.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.362+0000 7f67c3fff640 1 -- 172.21.15.19:0/1361192297 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f67c414e9a0 msgr2=0x7f67c4143db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:57.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.362+0000 7f67c3fff640 1 --2- 172.21.15.19:0/1361192297 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f67c414e9a0 0x7f67c4143db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:57.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.362+0000 7f67c3fff640 1 -- 172.21.15.19:0/1361192297 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f67c40c5370 msgr2=0x7f67c414ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:57.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.362+0000 7f67c3fff640 1 --2- 172.21.15.19:0/1361192297 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f67c40c5370 0x7f67c414ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:57.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.362+0000 7f67c3fff640 1 -- 172.21.15.19:0/1361192297 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67c41504f0 con 0x7f67c40cb2b0 2024-08-31T12:22:57.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.362+0000 7f67c3fff640 1 --2- 172.21.15.19:0/1361192297 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67c40cb2b0 0x7f67c4074010 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f67b0034040 tx=0x7f67b0003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:22:57.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.363+0000 7f67c17fa640 1 -- 172.21.15.19:0/1361192297 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67b003e030 con 0x7f67c40cb2b0 2024-08-31T12:22:57.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.363+0000 7f67c17fa640 1 -- 172.21.15.19:0/1361192297 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f67b0003d10 con 0x7f67c40cb2b0 2024-08-31T12:22:57.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.363+0000 7f67c17fa640 1 -- 172.21.15.19:0/1361192297 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67b0035040 con 0x7f67c40cb2b0 2024-08-31T12:22:57.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.363+0000 7f67cac47640 1 -- 172.21.15.19:0/1361192297 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67c414f9e0 con 0x7f67c40cb2b0 2024-08-31T12:22:57.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.363+0000 7f67cac47640 1 -- 172.21.15.19:0/1361192297 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67c415a430 con 0x7f67c40cb2b0 2024-08-31T12:22:57.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.364+0000 7f67cac47640 1 --2- 172.21.15.19:0/1361192297 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67c406d6d0 0x7f67c41442f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:57.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.364+0000 7f67cac47640 1 -- 172.21.15.19:0/1361192297 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f67c4157c90 con 0x7f67c406d6d0 2024-08-31T12:22:57.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.364+0000 7f67c89e3640 1 --2- 172.21.15.19:0/1361192297 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67c406d6d0 0x7f67c41442f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:57.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.364+0000 7f67c89e3640 1 -- 172.21.15.19:0/1361192297 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f67c406e760 con 0x7f67c406d6d0 2024-08-31T12:22:57.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.364+0000 7f67c89e3640 1 --2- 172.21.15.19:0/1361192297 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67c406d6d0 0x7f67c41442f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f67b800f570 tx=0x7f67b800fa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:22:57.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.365+0000 7f67c17fa640 1 -- 172.21.15.19:0/1361192297 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67b80052e0 con 0x7f67c406d6d0 2024-08-31T12:22:57.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.365+0000 7f67c17fa640 1 -- 172.21.15.19:0/1361192297 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f67b0041030 con 0x7f67c40cb2b0 2024-08-31T12:22:57.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.365+0000 7f67c17fa640 1 --2- 172.21.15.19:0/1361192297 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f67a8077980 0x7f67a8079e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:57.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.365+0000 7f67c17fa640 1 -- 172.21.15.19:0/1361192297 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f67b00b9db0 con 0x7f67c40cb2b0 2024-08-31T12:22:57.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.365+0000 7f67c17fa640 1 -- 172.21.15.19:0/1361192297 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f67b8018070 con 0x7f67c406d6d0 2024-08-31T12:22:57.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.365+0000 7f67c17fa640 1 -- 172.21.15.19:0/1361192297 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67c406d6d0 msgr2=0x7f67c41442f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:57.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.365+0000 7f67c17fa640 1 --2- 172.21.15.19:0/1361192297 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67c406d6d0 0x7f67c41442f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f67b800f570 tx=0x7f67b800fa30 comp rx=0 tx=0).stop 2024-08-31T12:22:57.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.365+0000 7f67c17fa640 1 -- 172.21.15.19:0/1361192297 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67c406d6d0 msgr2=0x7f67c41442f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:22:57.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.366+0000 7f67c17fa640 1 --2- 172.21.15.19:0/1361192297 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67c406d6d0 0x7f67c41442f0 secure :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f67b800f570 tx=0x7f67b800fa30 comp rx=0 tx=0).stop 2024-08-31T12:22:57.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.366+0000 7f67c91e4640 1 --2- 172.21.15.19:0/1361192297 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f67a8077980 0x7f67a8079e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:57.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.366+0000 7f67c91e4640 1 --2- 172.21.15.19:0/1361192297 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f67a8077980 0x7f67a8079e90 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f67c40c4710 tx=0x7f67bc009500 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:22:57.374 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.373+0000 7f67cac47640 1 --2- 172.21.15.19:0/1361192297 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67c402d2b0 0x7f67c402d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:57.374 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.373+0000 7f67cac47640 1 -- 172.21.15.19:0/1361192297 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f67c40cbfd0 con 0x7f67c402d2b0 2024-08-31T12:22:57.374 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.373+0000 7f67c89e3640 1 --2- 172.21.15.19:0/1361192297 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67c402d2b0 0x7f67c402d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:57.374 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.373+0000 7f67c89e3640 1 -- 172.21.15.19:0/1361192297 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f67c41e38a0 con 0x7f67c402d2b0 2024-08-31T12:22:57.374 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.373+0000 7f67c89e3640 1 --2- 172.21.15.19:0/1361192297 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67c402d2b0 0x7f67c402d6c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f67b800f570 tx=0x7f67b8000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:22:57.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.373+0000 7f67c17fa640 1 -- 172.21.15.19:0/1361192297 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67b8003ad0 con 0x7f67c402d2b0 2024-08-31T12:22:57.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.374+0000 7f67c17fa640 1 -- 172.21.15.19:0/1361192297 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5045 (secure 0 0 0) 0x7f67b8003ad0 con 0x7f67c402d2b0 2024-08-31T12:22:57.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.374+0000 7f67c17fa640 1 -- 172.21.15.19:0/1361192297 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67c402d2b0 msgr2=0x7f67c402d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:57.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.374+0000 7f67c17fa640 1 --2- 172.21.15.19:0/1361192297 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67c402d2b0 0x7f67c402d6c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f67b800f570 tx=0x7f67b8000ed0 comp rx=0 tx=0).stop 2024-08-31T12:22:57.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.374+0000 7f67c17fa640 1 -- 172.21.15.19:0/1361192297 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67c402d2b0 msgr2=0x7f67c402d6c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:22:57.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.374+0000 7f67c17fa640 1 --2- 172.21.15.19:0/1361192297 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67c402d2b0 0x7f67c402d6c0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:57.376 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:22:57.376 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:22:57.376 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:22:57.376 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "leader", 2024-08-31T12:22:57.376 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 50, 2024-08-31T12:22:57.376 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 357364, 2024-08-31T12:22:57.376 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:22:57.376 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:22:57.376 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:22:57.376 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:22:57.376 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:22:57.377 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 12, 2024-08-31T12:22:57.377 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:22:57.377 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:22:57.377 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:22:57.377 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:22:57.377 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:22:57.377 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:22:57.377 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:22:57.377 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:22:57.377 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:22:57.377 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:22:57.377 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:22:57.377 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:22:57.378 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:22:57.378 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:22:57.378 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:22:57.378 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:22:57.378 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:22:57.378 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:22:57.378 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:22:57.378 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:22:57.378 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:22:57.378 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:22:57.378 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:22:57.378 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:22:57.378 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:22:57.378 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:22:57.379 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:22:57.379 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:22:57.379 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:57.379 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:57.379 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:22:57.379 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:22:57.379 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:22:57.379 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:22:57.379 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:22:57.379 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:22:57.379 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:22:57.379 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:22:57.380 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:22:57.380 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:22:57.380 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:22:57.380 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:22:57.380 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:22:57.380 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:22:57.380 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:22:57.380 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:22:57.380 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:22:57.380 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:22:57.380 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:22:57.380 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:22:57.380 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:22:57.381 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:22:57.381 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:22:57.381 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:22:57.381 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:22:57.381 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:22:57.381 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:22:57.381 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:22:57.381 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:22:57.381 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:22:57.381 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:57.381 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:22:57.381 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:57.381 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:22:57.382 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:22:57.382 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:22:57.382 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:22:57.382 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:57.382 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:22:57.382 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:22:57.382 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:57.382 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:57.382 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:57.382 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:22:57.382 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:22:57.382 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:57.382 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:57.383 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:57.383 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:57.383 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:22:57.383 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:22:57.383 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:22:57.383 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:22:57.383 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:22:57.383 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:57.383 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:57.383 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:22:57.383 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:22:57.384 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:22:57.384 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:22:57.384 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:57.384 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:22:57.384 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:22:57.384 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:57.384 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:57.384 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:57.384 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:22:57.384 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:22:57.384 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:57.384 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:57.384 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:57.384 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:57.385 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:22:57.385 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:22:57.385 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:22:57.385 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:22:57.385 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:22:57.385 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:57.385 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:57.385 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:22:57.385 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:22:57.385 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:22:57.385 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:22:57.385 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:57.385 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:22:57.386 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:22:57.386 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:57.386 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:57.386 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:57.386 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:22:57.386 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:22:57.386 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:57.386 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:57.386 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:57.386 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:57.386 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:22:57.386 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:22:57.387 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:22:57.387 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:22:57.387 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:22:57.387 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:57.387 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:57.387 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:57.387 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:22:57.387 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:22:57.387 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:57.387 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:22:57.387 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:22:57.387 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:22:57.387 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:57.388 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:22:57.388 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:22:57.388 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:57.388 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:22:57.388 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:22:57.388 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 6 2024-08-31T12:22:57.388 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:57.388 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:22:57.388 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:22:57.388 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:57.388 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:22:57.388 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:22:57.388 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 8 2024-08-31T12:22:57.389 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:57.389 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:22:57.389 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:22:57.389 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:57.389 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:22:57.389 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:22:57.389 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 6 2024-08-31T12:22:57.389 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:57.389 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:22:57.389 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:22:57.389 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:57.389 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:22:57.389 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:22:57.390 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:22:57.390 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:57.390 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:57.390 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:57.390 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:22:57.390 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:22:57.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.374+0000 7f67cac47640 1 -- 172.21.15.19:0/1361192297 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f67a8077980 msgr2=0x7f67a8079e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:57.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.374+0000 7f67cac47640 1 --2- 172.21.15.19:0/1361192297 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f67a8077980 0x7f67a8079e90 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f67c40c4710 tx=0x7f67bc009500 comp rx=0 tx=0).stop 2024-08-31T12:22:57.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.374+0000 7f67cac47640 1 -- 172.21.15.19:0/1361192297 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67c40cb2b0 msgr2=0x7f67c4074010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:57.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.374+0000 7f67cac47640 1 --2- 172.21.15.19:0/1361192297 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67c40cb2b0 0x7f67c4074010 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f67b0034040 tx=0x7f67b0003820 comp rx=0 tx=0).stop 2024-08-31T12:22:57.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.374+0000 7f67c91e4640 1 -- 172.21.15.19:0/1361192297 reap_dead start 2024-08-31T12:22:57.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.374+0000 7f67cac47640 1 -- 172.21.15.19:0/1361192297 shutdown_connections 2024-08-31T12:22:57.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.374+0000 7f67cac47640 1 --2- 172.21.15.19:0/1361192297 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67c40cb2b0 0x7f67c4074010 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:57.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.374+0000 7f67cac47640 1 -- 172.21.15.19:0/1361192297 >> 172.21.15.19:0/1361192297 conn(0x7f67c40ca670 msgr2=0x7f67c40c4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:22:57.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.374+0000 7f67cac47640 1 -- 172.21.15.19:0/1361192297 shutdown_connections 2024-08-31T12:22:57.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.374+0000 7f67cac47640 1 -- 172.21.15.19:0/1361192297 wait complete. 2024-08-31T12:22:57.392 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:22:57.392 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-08-31T12:22:57.392 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-31T12:22:57.513 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.512+0000 7f503d4d5640 1 -- 172.21.15.19:0/1288065722 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f50380ca890 msgr2=0x7f503806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:57.513 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.512+0000 7f503d4d5640 1 --2- 172.21.15.19:0/1288065722 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f50380ca890 0x7f503806e7a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f5038073af0 tx=0x7f502802a680 comp rx=0 tx=0).stop 2024-08-31T12:22:57.514 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.512+0000 7f503d4d5640 1 -- 172.21.15.19:0/1288065722 shutdown_connections 2024-08-31T12:22:57.514 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.512+0000 7f503d4d5640 1 --2- 172.21.15.19:0/1288065722 >> v2:172.21.15.43:3300/0 conn(0x7f503815c4a0 0x7f503815c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:57.514 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.512+0000 7f503d4d5640 1 --2- 172.21.15.19:0/1288065722 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f50380ca890 0x7f503806e7a0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:57.514 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.512+0000 7f503d4d5640 1 -- 172.21.15.19:0/1288065722 >> 172.21.15.19:0/1288065722 conn(0x7f50380cda10 msgr2=0x7f50380c9e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:22:57.514 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.512+0000 7f503d4d5640 1 -- 172.21.15.19:0/1288065722 shutdown_connections 2024-08-31T12:22:57.514 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.512+0000 7f503d4d5640 1 -- 172.21.15.19:0/1288065722 wait complete. 2024-08-31T12:22:57.515 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.514+0000 7f503d4d5640 1 Processor -- start 2024-08-31T12:22:57.515 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.514+0000 7f503d4d5640 1 -- start start 2024-08-31T12:22:57.517 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.515+0000 7f503d4d5640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f50380ca890 0x7f5038155870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:57.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.515+0000 7f503d4d5640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5038152fb0 0x7f5038150860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:57.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.515+0000 7f503d4d5640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f503815c4a0 0x7f50380ba900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:57.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.515+0000 7f503d4d5640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f50381533c0 con 0x7f5038152fb0 2024-08-31T12:22:57.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.515+0000 7f503d4d5640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f503815cdf0 con 0x7f503815c4a0 2024-08-31T12:22:57.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.515+0000 7f503d4d5640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f5038153540 con 0x7f50380ca890 2024-08-31T12:22:57.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.515+0000 7f50367fc640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5038152fb0 0x7f5038150860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:57.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.516+0000 7f50367fc640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5038152fb0 0x7f5038150860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:48042/0 (socket says 172.21.15.19:48042) 2024-08-31T12:22:57.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.516+0000 7f50367fc640 1 -- 172.21.15.19:0/2842715596 learned_addr learned my addr 172.21.15.19:0/2842715596 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:22:57.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.516+0000 7f50377fe640 1 --2- 172.21.15.19:0/2842715596 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f503815c4a0 0x7f50380ba900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:57.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.516+0000 7f5036ffd640 1 --2- 172.21.15.19:0/2842715596 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f50380ca890 0x7f5038155870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:57.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.516+0000 7f50367fc640 1 -- 172.21.15.19:0/2842715596 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f50380ca890 msgr2=0x7f5038155870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:57.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.516+0000 7f50367fc640 1 --2- 172.21.15.19:0/2842715596 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f50380ca890 0x7f5038155870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:57.519 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.516+0000 7f50367fc640 1 -- 172.21.15.19:0/2842715596 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f503815c4a0 msgr2=0x7f50380ba900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:57.519 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.516+0000 7f50367fc640 1 --2- 172.21.15.19:0/2842715596 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f503815c4a0 0x7f50380ba900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:57.519 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.516+0000 7f50367fc640 1 -- 172.21.15.19:0/2842715596 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50380b9bd0 con 0x7f5038152fb0 2024-08-31T12:22:57.519 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.516+0000 7f50367fc640 1 --2- 172.21.15.19:0/2842715596 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5038152fb0 0x7f5038150860 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f502c00c3a0 tx=0x7f502c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:22:57.519 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.516+0000 7f501f7fe640 1 -- 172.21.15.19:0/2842715596 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f502c017070 con 0x7f5038152fb0 2024-08-31T12:22:57.519 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.516+0000 7f501f7fe640 1 -- 172.21.15.19:0/2842715596 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f502c0053c0 con 0x7f5038152fb0 2024-08-31T12:22:57.519 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.516+0000 7f503d4d5640 1 -- 172.21.15.19:0/2842715596 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5038155db0 con 0x7f5038152fb0 2024-08-31T12:22:57.519 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.516+0000 7f501f7fe640 1 -- 172.21.15.19:0/2842715596 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f502c003850 con 0x7f5038152fb0 2024-08-31T12:22:57.519 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.516+0000 7f503d4d5640 1 -- 172.21.15.19:0/2842715596 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5038155070 con 0x7f5038152fb0 2024-08-31T12:22:57.521 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.517+0000 7f503d4d5640 1 --2- 172.21.15.19:0/2842715596 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5038150da0 0x7f50380bae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:57.521 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.517+0000 7f503d4d5640 1 -- 172.21.15.19:0/2842715596 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f50380b9f20 con 0x7f5038150da0 2024-08-31T12:22:57.521 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.518+0000 7f5036ffd640 1 --2- 172.21.15.19:0/2842715596 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5038150da0 0x7f50380bae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:57.521 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.518+0000 7f5036ffd640 1 -- 172.21.15.19:0/2842715596 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f50381e4ee0 con 0x7f5038150da0 2024-08-31T12:22:57.521 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.518+0000 7f5036ffd640 1 --2- 172.21.15.19:0/2842715596 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5038150da0 0x7f50380bae40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f50380c9d80 tx=0x7f50280035f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:22:57.521 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.518+0000 7f501f7fe640 1 -- 172.21.15.19:0/2842715596 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5028040070 con 0x7f5038150da0 2024-08-31T12:22:57.521 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.519+0000 7f501f7fe640 1 -- 172.21.15.19:0/2842715596 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f502c01a070 con 0x7f5038152fb0 2024-08-31T12:22:57.521 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.519+0000 7f501f7fe640 1 --2- 172.21.15.19:0/2842715596 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5024077950 0x7f5024079e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:57.521 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.519+0000 7f501f7fe640 1 -- 172.21.15.19:0/2842715596 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f502c098a20 con 0x7f5038152fb0 2024-08-31T12:22:57.522 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.519+0000 7f50377fe640 1 --2- 172.21.15.19:0/2842715596 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5024077950 0x7f5024079e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:57.522 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.519+0000 7f501f7fe640 1 -- 172.21.15.19:0/2842715596 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5028003cd0 con 0x7f5038150da0 2024-08-31T12:22:57.522 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.519+0000 7f501f7fe640 1 -- 172.21.15.19:0/2842715596 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5038150da0 msgr2=0x7f50380bae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:57.522 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.519+0000 7f501f7fe640 1 --2- 172.21.15.19:0/2842715596 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5038150da0 0x7f50380bae40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f50380c9d80 tx=0x7f50280035f0 comp rx=0 tx=0).stop 2024-08-31T12:22:57.522 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.519+0000 7f501f7fe640 1 -- 172.21.15.19:0/2842715596 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5038150da0 msgr2=0x7f50380bae40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:22:57.522 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.519+0000 7f501f7fe640 1 --2- 172.21.15.19:0/2842715596 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5038150da0 0x7f50380bae40 secure :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f50380c9d80 tx=0x7f50280035f0 comp rx=0 tx=0).stop 2024-08-31T12:22:57.522 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.520+0000 7f50377fe640 1 --2- 172.21.15.19:0/2842715596 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5024077950 0x7f5024079e60 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5038070e50 tx=0x7f5030009560 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:22:57.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.526+0000 7f503d4d5640 1 --2- 172.21.15.19:0/2842715596 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5038155240 0x7f503815afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:57.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.526+0000 7f503d4d5640 1 -- 172.21.15.19:0/2842715596 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f50380c7190 con 0x7f5038155240 2024-08-31T12:22:57.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.526+0000 7f5036ffd640 1 --2- 172.21.15.19:0/2842715596 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5038155240 0x7f503815afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:57.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.526+0000 7f5036ffd640 1 -- 172.21.15.19:0/2842715596 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f50380c3f40 con 0x7f5038155240 2024-08-31T12:22:57.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.526+0000 7f5036ffd640 1 --2- 172.21.15.19:0/2842715596 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5038155240 0x7f503815afe0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5028000c00 tx=0x7f5028009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:22:57.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.527+0000 7f501f7fe640 1 -- 172.21.15.19:0/2842715596 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5028035040 con 0x7f5038155240 2024-08-31T12:22:57.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.527+0000 7f501f7fe640 1 -- 172.21.15.19:0/2842715596 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4541 (secure 0 0 0) 0x7f5028044ad0 con 0x7f5038155240 2024-08-31T12:22:57.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.527+0000 7f501f7fe640 1 -- 172.21.15.19:0/2842715596 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5038155240 msgr2=0x7f503815afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:57.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.527+0000 7f501f7fe640 1 --2- 172.21.15.19:0/2842715596 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5038155240 0x7f503815afe0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5028000c00 tx=0x7f5028009260 comp rx=0 tx=0).stop 2024-08-31T12:22:57.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.527+0000 7f501f7fe640 1 -- 172.21.15.19:0/2842715596 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5038155240 msgr2=0x7f503815afe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:22:57.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.527+0000 7f501f7fe640 1 --2- 172.21.15.19:0/2842715596 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5038155240 0x7f503815afe0 secure :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5028000c00 tx=0x7f5028009260 comp rx=0 tx=0).stop 2024-08-31T12:22:57.529 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:22:57.529 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:22:57.529 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:22:57.529 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:22:57.529 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 50, 2024-08-31T12:22:57.529 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 87000, 2024-08-31T12:22:57.529 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:22:57.529 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:22:57.529 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:22:57.529 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:22:57.529 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:22:57.529 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 12, 2024-08-31T12:22:57.530 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:22:57.530 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:22:57.530 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:22:57.530 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:22:57.530 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:22:57.530 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:22:57.530 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:22:57.530 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:22:57.530 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:22:57.530 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:22:57.530 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:22:57.530 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:22:57.530 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:22:57.531 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:22:57.531 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:22:57.531 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:22:57.531 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:22:57.531 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:22:57.531 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:22:57.531 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:22:57.531 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:22:57.531 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:22:57.531 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:22:57.531 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:22:57.531 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:22:57.531 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:22:57.532 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:22:57.532 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:22:57.532 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:57.532 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:57.532 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:22:57.532 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:22:57.532 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:22:57.532 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:22:57.532 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:22:57.532 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:22:57.532 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:22:57.532 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:22:57.532 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:22:57.532 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:22:57.533 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:22:57.533 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:22:57.533 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:22:57.533 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:22:57.533 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:22:57.533 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:22:57.533 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:22:57.533 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:22:57.533 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:22:57.533 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:22:57.533 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:22:57.533 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:22:57.533 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:22:57.534 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:22:57.534 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:22:57.534 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:22:57.534 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:22:57.534 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:22:57.534 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:22:57.534 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:22:57.534 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:57.534 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:22:57.534 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:57.534 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:22:57.534 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:22:57.534 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:22:57.534 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:22:57.535 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:57.535 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:22:57.535 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:22:57.535 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:57.535 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:57.535 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:57.535 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:22:57.535 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:22:57.535 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:57.535 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:57.535 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:57.535 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:57.535 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:22:57.535 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:22:57.536 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:22:57.536 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:22:57.536 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:22:57.536 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:57.536 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:57.536 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:22:57.536 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:22:57.536 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:22:57.536 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:22:57.536 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:57.536 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:22:57.536 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:22:57.536 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:57.537 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:57.537 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:57.537 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:22:57.537 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:22:57.537 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:57.537 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:57.537 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:57.537 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:57.537 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:22:57.537 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:22:57.537 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:22:57.537 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:22:57.537 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:22:57.538 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:57.538 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:57.538 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:22:57.538 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:22:57.538 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:22:57.538 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:22:57.538 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:57.538 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:22:57.538 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:22:57.538 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:57.538 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:57.538 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:57.538 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:22:57.538 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:22:57.538 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:57.539 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:57.539 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:57.539 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:57.539 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:22:57.539 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:22:57.539 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:22:57.539 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:22:57.539 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:22:57.539 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:57.539 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:57.539 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:57.539 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:22:57.539 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:22:57.539 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:57.539 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:22:57.540 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:22:57.540 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:22:57.540 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:57.540 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:22:57.540 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:22:57.540 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:57.540 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:22:57.540 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:22:57.540 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:22:57.540 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:57.540 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:57.540 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:57.540 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:22:57.541 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:22:57.542 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.527+0000 7f503d4d5640 1 -- 172.21.15.19:0/2842715596 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5024077950 msgr2=0x7f5024079e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:57.542 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.527+0000 7f503d4d5640 1 --2- 172.21.15.19:0/2842715596 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5024077950 0x7f5024079e60 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5038070e50 tx=0x7f5030009560 comp rx=0 tx=0).stop 2024-08-31T12:22:57.542 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.527+0000 7f503d4d5640 1 -- 172.21.15.19:0/2842715596 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5038152fb0 msgr2=0x7f5038150860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:57.542 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.527+0000 7f50377fe640 1 -- 172.21.15.19:0/2842715596 reap_dead start 2024-08-31T12:22:57.542 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.527+0000 7f503d4d5640 1 --2- 172.21.15.19:0/2842715596 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5038152fb0 0x7f5038150860 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f502c00c3a0 tx=0x7f502c00c860 comp rx=0 tx=0).stop 2024-08-31T12:22:57.542 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.527+0000 7f503d4d5640 1 -- 172.21.15.19:0/2842715596 shutdown_connections 2024-08-31T12:22:57.542 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.527+0000 7f503d4d5640 1 --2- 172.21.15.19:0/2842715596 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5038152fb0 0x7f5038150860 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:57.542 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.527+0000 7f503d4d5640 1 -- 172.21.15.19:0/2842715596 >> 172.21.15.19:0/2842715596 conn(0x7f50380cda10 msgr2=0x7f50381e47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:22:57.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.527+0000 7f503d4d5640 1 -- 172.21.15.19:0/2842715596 shutdown_connections 2024-08-31T12:22:57.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.527+0000 7f503d4d5640 1 -- 172.21.15.19:0/2842715596 wait complete. 2024-08-31T12:22:57.543 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:22:57.543 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-08-31T12:22:57.543 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-31T12:22:57.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.665+0000 7f11e9269640 1 -- 172.21.15.19:0/1983810765 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f11e40c3ff0 msgr2=0x7f11e40c8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:57.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.665+0000 7f11e9269640 1 --2- 172.21.15.19:0/1983810765 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f11e40c3ff0 0x7f11e40c8fc0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f11d4005fd0 tx=0x7f11d402a670 comp rx=0 tx=0).stop 2024-08-31T12:22:57.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.665+0000 7f11e9269640 1 -- 172.21.15.19:0/1983810765 shutdown_connections 2024-08-31T12:22:57.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.665+0000 7f11e9269640 1 --2- 172.21.15.19:0/1983810765 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f11e40ce550 0x7f11e40ce960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:57.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.665+0000 7f11e9269640 1 --2- 172.21.15.19:0/1983810765 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f11e40c3ff0 0x7f11e40c8fc0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:57.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.665+0000 7f11e9269640 1 -- 172.21.15.19:0/1983810765 >> 172.21.15.19:0/1983810765 conn(0x7f11e40694c0 msgr2=0x7f11e40728b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:22:57.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.665+0000 7f11e9269640 1 -- 172.21.15.19:0/1983810765 shutdown_connections 2024-08-31T12:22:57.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.665+0000 7f11e9269640 1 -- 172.21.15.19:0/1983810765 wait complete. 2024-08-31T12:22:57.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.667+0000 7f11e9269640 1 Processor -- start 2024-08-31T12:22:57.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.667+0000 7f11e9269640 1 -- start start 2024-08-31T12:22:57.671 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.669+0000 7f11e9269640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f11e4071dd0 0x7f11e4153a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:57.672 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.669+0000 7f11e9269640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f11e40c3ff0 0x7f11e41556d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:57.672 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.669+0000 7f11e9269640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f11e40ce550 0x7f11e4157af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:57.672 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.669+0000 7f11e9269640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f11e415c570 con 0x7f11e40c3ff0 2024-08-31T12:22:57.672 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.669+0000 7f11e9269640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f11e4159950 con 0x7f11e4071dd0 2024-08-31T12:22:57.672 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.669+0000 7f11e9269640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f11e415c6f0 con 0x7f11e40ce550 2024-08-31T12:22:57.672 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.669+0000 7f11e27fc640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f11e40c3ff0 0x7f11e41556d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:57.672 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.669+0000 7f11e27fc640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f11e40c3ff0 0x7f11e41556d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:48046/0 (socket says 172.21.15.19:48046) 2024-08-31T12:22:57.672 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.669+0000 7f11e27fc640 1 -- 172.21.15.19:0/4284196346 learned_addr learned my addr 172.21.15.19:0/4284196346 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:22:57.672 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.669+0000 7f11e37fe640 1 --2- 172.21.15.19:0/4284196346 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f11e40ce550 0x7f11e4157af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:57.672 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.669+0000 7f11e2ffd640 1 --2- 172.21.15.19:0/4284196346 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f11e4071dd0 0x7f11e4153a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:57.672 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.669+0000 7f11e27fc640 1 -- 172.21.15.19:0/4284196346 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f11e40ce550 msgr2=0x7f11e4157af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:57.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.669+0000 7f11e27fc640 1 --2- 172.21.15.19:0/4284196346 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f11e40ce550 0x7f11e4157af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:57.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.669+0000 7f11e27fc640 1 -- 172.21.15.19:0/4284196346 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f11e4071dd0 msgr2=0x7f11e4153a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:57.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.669+0000 7f11e27fc640 1 --2- 172.21.15.19:0/4284196346 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f11e4071dd0 0x7f11e4153a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:57.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.669+0000 7f11e27fc640 1 -- 172.21.15.19:0/4284196346 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11e4153f40 con 0x7f11e40c3ff0 2024-08-31T12:22:57.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.670+0000 7f11e27fc640 1 --2- 172.21.15.19:0/4284196346 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f11e40c3ff0 0x7f11e41556d0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f11d800c3a0 tx=0x7f11d800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:22:57.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.670+0000 7f11cb7fe640 1 -- 172.21.15.19:0/4284196346 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d8017070 con 0x7f11e40c3ff0 2024-08-31T12:22:57.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.670+0000 7f11cb7fe640 1 -- 172.21.15.19:0/4284196346 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f11d80053c0 con 0x7f11e40c3ff0 2024-08-31T12:22:57.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.670+0000 7f11e9269640 1 -- 172.21.15.19:0/4284196346 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11e415bc40 con 0x7f11e40c3ff0 2024-08-31T12:22:57.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.670+0000 7f11cb7fe640 1 -- 172.21.15.19:0/4284196346 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d8003850 con 0x7f11e40c3ff0 2024-08-31T12:22:57.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.670+0000 7f11e9269640 1 -- 172.21.15.19:0/4284196346 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11e4158ae0 con 0x7f11e40c3ff0 2024-08-31T12:22:57.675 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.671+0000 7f11e9269640 1 --2- 172.21.15.19:0/4284196346 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f11e415be60 0x7f11e4158030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:57.675 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.671+0000 7f11e9269640 1 -- 172.21.15.19:0/4284196346 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f11e4152020 con 0x7f11e415be60 2024-08-31T12:22:57.675 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.671+0000 7f11e2ffd640 1 --2- 172.21.15.19:0/4284196346 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f11e415be60 0x7f11e4158030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:57.675 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.671+0000 7f11e2ffd640 1 -- 172.21.15.19:0/4284196346 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f11e415cbe0 con 0x7f11e415be60 2024-08-31T12:22:57.676 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.672+0000 7f11e2ffd640 1 --2- 172.21.15.19:0/4284196346 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f11e415be60 0x7f11e4158030 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f11e40cc470 tx=0x7f11d40045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:22:57.676 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.672+0000 7f11cb7fe640 1 -- 172.21.15.19:0/4284196346 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d4040070 con 0x7f11e415be60 2024-08-31T12:22:57.676 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.672+0000 7f11cb7fe640 1 -- 172.21.15.19:0/4284196346 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f11d801a070 con 0x7f11e40c3ff0 2024-08-31T12:22:57.676 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.673+0000 7f11cb7fe640 1 --2- 172.21.15.19:0/4284196346 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f11d0077980 0x7f11d0079e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:57.676 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.673+0000 7f11cb7fe640 1 -- 172.21.15.19:0/4284196346 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f11d8098a10 con 0x7f11e40c3ff0 2024-08-31T12:22:57.676 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.673+0000 7f11e37fe640 1 --2- 172.21.15.19:0/4284196346 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f11d0077980 0x7f11d0079e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:57.676 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.673+0000 7f11cb7fe640 1 -- 172.21.15.19:0/4284196346 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f11d4003bc0 con 0x7f11e415be60 2024-08-31T12:22:57.676 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.673+0000 7f11cb7fe640 1 -- 172.21.15.19:0/4284196346 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f11e415be60 msgr2=0x7f11e4158030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:57.676 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.673+0000 7f11cb7fe640 1 --2- 172.21.15.19:0/4284196346 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f11e415be60 0x7f11e4158030 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f11e40cc470 tx=0x7f11d40045e0 comp rx=0 tx=0).stop 2024-08-31T12:22:57.676 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.673+0000 7f11cb7fe640 1 -- 172.21.15.19:0/4284196346 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f11e415be60 msgr2=0x7f11e4158030 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:22:57.676 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.673+0000 7f11cb7fe640 1 --2- 172.21.15.19:0/4284196346 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f11e415be60 0x7f11e4158030 secure :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f11e40cc470 tx=0x7f11d40045e0 comp rx=0 tx=0).stop 2024-08-31T12:22:57.676 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.674+0000 7f11e37fe640 1 --2- 172.21.15.19:0/4284196346 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f11d0077980 0x7f11d0079e90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f11e4159c70 tx=0x7f11dc009500 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:22:57.680 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.680+0000 7f11e9269640 1 --2- 172.21.15.19:0/4284196346 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f11e41f27f0 0x7f11e41f2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:57.680 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.680+0000 7f11e9269640 1 -- 172.21.15.19:0/4284196346 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f11e40c9530 con 0x7f11e41f27f0 2024-08-31T12:22:57.681 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.680+0000 7f11e2ffd640 1 --2- 172.21.15.19:0/4284196346 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f11e41f27f0 0x7f11e41f2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:57.682 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.680+0000 7f11e2ffd640 1 -- 172.21.15.19:0/4284196346 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f11e415c270 con 0x7f11e41f27f0 2024-08-31T12:22:57.682 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.680+0000 7f11e2ffd640 1 --2- 172.21.15.19:0/4284196346 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f11e41f27f0 0x7f11e41f2c00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f11d4009850 tx=0x7f11d4009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:22:57.682 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.681+0000 7f11cb7fe640 1 -- 172.21.15.19:0/4284196346 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d4035040 con 0x7f11e41f27f0 2024-08-31T12:22:57.682 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.681+0000 7f11cb7fe640 1 -- 172.21.15.19:0/4284196346 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4541 (secure 0 0 0) 0x7f11d40466e0 con 0x7f11e41f27f0 2024-08-31T12:22:57.682 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.681+0000 7f11cb7fe640 1 -- 172.21.15.19:0/4284196346 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f11e41f27f0 msgr2=0x7f11e41f2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:57.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.681+0000 7f11cb7fe640 1 --2- 172.21.15.19:0/4284196346 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f11e41f27f0 0x7f11e41f2c00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f11d4009850 tx=0x7f11d4009260 comp rx=0 tx=0).stop 2024-08-31T12:22:57.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.681+0000 7f11cb7fe640 1 -- 172.21.15.19:0/4284196346 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f11e41f27f0 msgr2=0x7f11e41f2c00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:22:57.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.681+0000 7f11cb7fe640 1 --2- 172.21.15.19:0/4284196346 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f11e41f27f0 0x7f11e41f2c00 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:57.683 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:22:57.683 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:22:57.683 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:22:57.683 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:22:57.683 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 50, 2024-08-31T12:22:57.683 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 27227, 2024-08-31T12:22:57.683 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:22:57.683 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:22:57.684 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:22:57.684 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:22:57.684 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:22:57.684 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 12, 2024-08-31T12:22:57.684 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:22:57.684 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:22:57.684 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:22:57.684 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:22:57.684 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:22:57.684 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:22:57.684 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:22:57.684 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:22:57.685 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:22:57.685 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:22:57.685 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:22:57.685 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:22:57.685 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:22:57.685 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:22:57.685 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:22:57.685 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:22:57.685 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:22:57.685 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:22:57.685 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:22:57.685 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:22:57.685 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:22:57.685 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:22:57.686 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:22:57.686 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:22:57.686 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:22:57.686 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:22:57.686 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:22:57.686 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:22:57.686 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:57.686 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:57.686 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:22:57.686 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:22:57.686 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:22:57.686 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:22:57.686 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:22:57.687 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:22:57.687 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:22:57.687 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:22:57.687 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:22:57.687 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:22:57.687 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:22:57.687 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:22:57.687 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:22:57.687 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:22:57.687 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:22:57.687 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:22:57.687 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:22:57.687 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:22:57.688 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:22:57.688 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:22:57.688 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:22:57.688 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:22:57.688 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:22:57.688 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:22:57.688 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:22:57.688 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:22:57.688 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:22:57.688 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:22:57.688 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:22:57.688 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:22:57.689 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:57.689 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:22:57.689 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:57.689 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:22:57.689 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:22:57.689 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:22:57.689 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:22:57.689 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:57.689 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:22:57.689 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:22:57.689 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:57.689 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:57.690 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:57.690 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:22:57.690 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:22:57.690 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:57.690 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:57.690 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:57.690 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:57.690 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:22:57.690 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:22:57.690 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:22:57.690 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:22:57.690 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:22:57.690 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:57.691 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:57.691 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:22:57.691 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:22:57.691 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:22:57.691 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:22:57.691 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:57.691 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:22:57.691 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:22:57.691 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:57.691 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:57.691 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:57.691 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:22:57.691 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:22:57.692 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:57.692 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:57.692 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:57.692 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:57.692 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:22:57.692 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:22:57.692 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:22:57.692 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:22:57.692 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:22:57.692 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:57.692 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:57.692 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:22:57.692 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:22:57.693 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:22:57.693 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:22:57.693 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:57.693 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:22:57.693 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:22:57.693 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:57.693 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:57.693 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:57.693 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:22:57.693 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:22:57.693 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:22:57.693 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:57.693 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:57.694 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:57.694 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:22:57.694 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:22:57.694 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:22:57.694 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:22:57.694 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:22:57.694 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:57.694 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:57.694 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:57.694 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:22:57.694 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:22:57.694 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:57.694 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:22:57.695 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:22:57.695 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:22:57.695 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:57.695 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:22:57.695 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:22:57.695 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:22:57.695 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:22:57.695 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:22:57.695 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:22:57.695 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:22:57.695 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:22:57.695 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:22:57.695 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:22:57.695 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:22:57.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.681+0000 7f11e9269640 1 -- 172.21.15.19:0/4284196346 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f11d0077980 msgr2=0x7f11d0079e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:57.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.681+0000 7f11e9269640 1 --2- 172.21.15.19:0/4284196346 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f11d0077980 0x7f11d0079e90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f11e4159c70 tx=0x7f11dc009500 comp rx=0 tx=0).stop 2024-08-31T12:22:57.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.681+0000 7f11e9269640 1 -- 172.21.15.19:0/4284196346 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f11e40c3ff0 msgr2=0x7f11e41556d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:57.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.681+0000 7f11e9269640 1 --2- 172.21.15.19:0/4284196346 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f11e40c3ff0 0x7f11e41556d0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f11d800c3a0 tx=0x7f11d800c860 comp rx=0 tx=0).stop 2024-08-31T12:22:57.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.681+0000 7f11e37fe640 1 -- 172.21.15.19:0/4284196346 reap_dead start 2024-08-31T12:22:57.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.681+0000 7f11e9269640 1 -- 172.21.15.19:0/4284196346 shutdown_connections 2024-08-31T12:22:57.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.681+0000 7f11e9269640 1 --2- 172.21.15.19:0/4284196346 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f11e40c3ff0 0x7f11e41556d0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:57.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.681+0000 7f11e9269640 1 -- 172.21.15.19:0/4284196346 >> 172.21.15.19:0/4284196346 conn(0x7f11e40694c0 msgr2=0x7f11e414e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:22:57.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.681+0000 7f11e9269640 1 -- 172.21.15.19:0/4284196346 shutdown_connections 2024-08-31T12:22:57.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.682+0000 7f11e9269640 1 -- 172.21.15.19:0/4284196346 wait complete. 2024-08-31T12:22:57.698 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:22:57.698 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-08-31T12:22:57.698 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-08-31T12:22:57.698 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-08-31T12:22:57.698 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-08-31T12:22:57.698 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-08-31T12:22:57.699 INFO:teuthology.orchestra.run:waiting for 300 2024-08-31T12:22:57.710 INFO:tasks.ceph.mon.c:Stopped 2024-08-31T12:22:57.710 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-08-31T12:22:57.711 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:22:57.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.836+0000 7faeef19e640 1 -- 172.21.15.19:0/877049788 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faee80c7ac0 msgr2=0x7faee8152040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:57.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.836+0000 7faeef19e640 1 --2- 172.21.15.19:0/877049788 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faee80c7ac0 0x7faee8152040 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7faedc009820 tx=0x7faedc02a530 comp rx=0 tx=0).stop 2024-08-31T12:22:57.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.836+0000 7faeef19e640 1 -- 172.21.15.19:0/877049788 shutdown_connections 2024-08-31T12:22:57.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.836+0000 7faeef19e640 1 --2- 172.21.15.19:0/877049788 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faee80c89a0 0x7faee80c8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:57.839 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.836+0000 7faeef19e640 1 --2- 172.21.15.19:0/877049788 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faee80c7ac0 0x7faee8152040 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:57.839 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.836+0000 7faeef19e640 1 -- 172.21.15.19:0/877049788 >> 172.21.15.19:0/877049788 conn(0x7faee806cae0 msgr2=0x7faee806cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:22:57.839 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.837+0000 7faeef19e640 1 -- 172.21.15.19:0/877049788 shutdown_connections 2024-08-31T12:22:57.839 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.837+0000 7faeef19e640 1 -- 172.21.15.19:0/877049788 wait complete. 2024-08-31T12:22:57.839 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.838+0000 7faeef19e640 1 Processor -- start 2024-08-31T12:22:57.839 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.838+0000 7faeef19e640 1 -- start start 2024-08-31T12:22:57.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.840+0000 7faeef19e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faee80b28d0 0x7faee8144570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:57.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.840+0000 7faeef19e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faee80c7ac0 0x7faee8142f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:57.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.840+0000 7faeef19e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faee80c89a0 0x7faee81e3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:57.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.840+0000 7faeef19e640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7faee80c92f0 con 0x7faee80c7ac0 2024-08-31T12:22:57.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.840+0000 7faeef19e640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7faee8068fd0 con 0x7faee80b28d0 2024-08-31T12:22:57.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.840+0000 7faeef19e640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7faee806da00 con 0x7faee80c89a0 2024-08-31T12:22:57.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.840+0000 7faee7fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faee80c7ac0 0x7faee8142f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:57.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.840+0000 7faeed73b640 1 -- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faee80c89a0 msgr2=0x7faee81e3780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3301/0 2024-08-31T12:22:57.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.840+0000 7faeed73b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faee80c89a0 0x7faee81e3780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:22:57.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.840+0000 7faee7fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faee80c7ac0 0x7faee8142f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:48072/0 (socket says 172.21.15.19:48072) 2024-08-31T12:22:57.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.840+0000 7faee7fff640 1 -- 172.21.15.19:0/2285676540 learned_addr learned my addr 172.21.15.19:0/2285676540 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:22:57.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.841+0000 7faeecf3a640 1 --2- 172.21.15.19:0/2285676540 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faee80b28d0 0x7faee8144570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:57.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.841+0000 7faee7fff640 1 -- 172.21.15.19:0/2285676540 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faee80c89a0 msgr2=0x7faee81e3780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:22:57.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.841+0000 7faee7fff640 1 --2- 172.21.15.19:0/2285676540 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faee80c89a0 0x7faee81e3780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:57.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.841+0000 7faee7fff640 1 -- 172.21.15.19:0/2285676540 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faee80b28d0 msgr2=0x7faee8144570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:57.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.841+0000 7faee7fff640 1 --2- 172.21.15.19:0/2285676540 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faee80b28d0 0x7faee8144570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:57.844 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.841+0000 7faee7fff640 1 -- 172.21.15.19:0/2285676540 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faee81421f0 con 0x7faee80c7ac0 2024-08-31T12:22:57.844 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.841+0000 7faee7fff640 1 --2- 172.21.15.19:0/2285676540 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faee80c7ac0 0x7faee8142f60 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7faed400c3f0 tx=0x7faed400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:22:57.844 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.841+0000 7faee57fa640 1 -- 172.21.15.19:0/2285676540 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faed4017070 con 0x7faee80c7ac0 2024-08-31T12:22:57.844 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.841+0000 7faeef19e640 1 -- 172.21.15.19:0/2285676540 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faee8144ab0 con 0x7faee80c7ac0 2024-08-31T12:22:57.844 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.841+0000 7faee57fa640 1 -- 172.21.15.19:0/2285676540 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faed40035f0 con 0x7faee80c7ac0 2024-08-31T12:22:57.844 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.841+0000 7faee57fa640 1 -- 172.21.15.19:0/2285676540 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faed4003a40 con 0x7faee80c7ac0 2024-08-31T12:22:57.844 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.841+0000 7faeef19e640 1 -- 172.21.15.19:0/2285676540 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faee8143970 con 0x7faee80c7ac0 2024-08-31T12:22:57.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.842+0000 7faeef19e640 1 -- 172.21.15.19:0/2285676540 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faee8155e20 con 0x7faee80c7ac0 2024-08-31T12:22:57.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.843+0000 7faee57fa640 1 -- 172.21.15.19:0/2285676540 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7faed401a070 con 0x7faee80c7ac0 2024-08-31T12:22:57.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.844+0000 7faee57fa640 1 --2- 172.21.15.19:0/2285676540 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7faecc077a30 0x7faecc079f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:22:57.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.844+0000 7faee57fa640 1 -- 172.21.15.19:0/2285676540 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7faed4098a70 con 0x7faee80c7ac0 2024-08-31T12:22:57.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.844+0000 7faeecf3a640 1 --2- 172.21.15.19:0/2285676540 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7faecc077a30 0x7faecc079f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:22:57.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.845+0000 7faeecf3a640 1 --2- 172.21.15.19:0/2285676540 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7faecc077a30 0x7faecc079f40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7faedc034000 tx=0x7faedc004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:22:57.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:57.849+0000 7faee57fa640 1 -- 172.21.15.19:0/2285676540 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7faed40616e0 con 0x7faee80c7ac0 2024-08-31T12:22:58.169 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:58.168+0000 7faeef19e640 1 -- 172.21.15.19:0/2285676540 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7faee80c3e90 con 0x7faee80c7ac0 2024-08-31T12:22:58.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:58.169+0000 7faee57fa640 1 -- 172.21.15.19:0/2285676540 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7faed4060fe0 con 0x7faee80c7ac0 2024-08-31T12:22:58.174 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":50,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:22:58.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:58.171+0000 7faeef19e640 1 -- 172.21.15.19:0/2285676540 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7faecc077a30 msgr2=0x7faecc079f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:58.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:58.171+0000 7faeef19e640 1 --2- 172.21.15.19:0/2285676540 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7faecc077a30 0x7faecc079f40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7faedc034000 tx=0x7faedc004730 comp rx=0 tx=0).stop 2024-08-31T12:22:58.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:58.171+0000 7faeef19e640 1 -- 172.21.15.19:0/2285676540 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faee80c7ac0 msgr2=0x7faee8142f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:22:58.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:58.171+0000 7faeef19e640 1 --2- 172.21.15.19:0/2285676540 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faee80c7ac0 0x7faee8142f60 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7faed400c3f0 tx=0x7faed400c8b0 comp rx=0 tx=0).stop 2024-08-31T12:22:58.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:58.171+0000 7faeef19e640 1 -- 172.21.15.19:0/2285676540 shutdown_connections 2024-08-31T12:22:58.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:58.171+0000 7faeef19e640 1 --2- 172.21.15.19:0/2285676540 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7faecc077a30 0x7faecc079f40 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:58.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:58.171+0000 7faeef19e640 1 --2- 172.21.15.19:0/2285676540 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faee80c89a0 0x7faee81e3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:58.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:58.171+0000 7faeef19e640 1 --2- 172.21.15.19:0/2285676540 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faee80c7ac0 0x7faee8142f60 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:58.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:58.171+0000 7faeef19e640 1 --2- 172.21.15.19:0/2285676540 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faee80b28d0 0x7faee8144570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:22:58.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:58.171+0000 7faeef19e640 1 -- 172.21.15.19:0/2285676540 >> 172.21.15.19:0/2285676540 conn(0x7faee806cae0 msgr2=0x7faee81419c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:22:58.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:58.171+0000 7faeef19e640 1 -- 172.21.15.19:0/2285676540 shutdown_connections 2024-08-31T12:22:58.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:22:58.171+0000 7faeef19e640 1 -- 172.21.15.19:0/2285676540 wait complete. 2024-08-31T12:23:01.186 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:23:01.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.306+0000 7f2ae79a8640 1 -- 172.21.15.19:0/3518982084 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2ae00ccbb0 msgr2=0x7f2ae00cb280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:01.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.306+0000 7f2ae79a8640 1 --2- 172.21.15.19:0/3518982084 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2ae00ccbb0 0x7f2ae00cb280 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f2adc009830 tx=0x7f2adc02a640 comp rx=0 tx=0).stop 2024-08-31T12:23:01.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.307+0000 7f2ae79a8640 1 -- 172.21.15.19:0/3518982084 shutdown_connections 2024-08-31T12:23:01.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.307+0000 7f2ae79a8640 1 --2- 172.21.15.19:0/3518982084 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2ae00ccbb0 0x7f2ae00cb280 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:01.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.307+0000 7f2ae79a8640 1 -- 172.21.15.19:0/3518982084 >> 172.21.15.19:0/3518982084 conn(0x7f2ae0074810 msgr2=0x7f2ae0068aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:23:01.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.307+0000 7f2ae79a8640 1 -- 172.21.15.19:0/3518982084 shutdown_connections 2024-08-31T12:23:01.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.307+0000 7f2ae79a8640 1 -- 172.21.15.19:0/3518982084 wait complete. 2024-08-31T12:23:01.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.309+0000 7f2ae79a8640 1 Processor -- start 2024-08-31T12:23:01.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.309+0000 7f2ae79a8640 1 -- start start 2024-08-31T12:23:01.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.310+0000 7f2ae79a8640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2ae00cc070 0x7f2ae015c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:01.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.311+0000 7f2ae79a8640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ae00ccbb0 0x7f2ae01e7d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:01.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.311+0000 7f2ae79a8640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2ae015b830 0x7f2ae014bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:01.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.311+0000 7f2ae79a8640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae01557e0 con 0x7f2ae015b830 2024-08-31T12:23:01.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.311+0000 7f2ae79a8640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae0155660 con 0x7f2ae00ccbb0 2024-08-31T12:23:01.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.311+0000 7f2ae79a8640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f2ae006e1a0 con 0x7f2ae00cc070 2024-08-31T12:23:01.314 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.311+0000 7f2ae5744640 1 -- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2ae00cc070 msgr2=0x7f2ae015c3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3301/0 2024-08-31T12:23:01.314 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.311+0000 7f2ae5744640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2ae00cc070 0x7f2ae015c3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:23:01.314 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.311+0000 7f2ae5f45640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2ae015b830 0x7f2ae014bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:01.314 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.311+0000 7f2ae4f43640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ae00ccbb0 0x7f2ae01e7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:01.314 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.311+0000 7f2ae5f45640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2ae015b830 0x7f2ae014bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:44150/0 (socket says 172.21.15.19:44150) 2024-08-31T12:23:01.314 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.311+0000 7f2ae4f43640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ae00ccbb0 0x7f2ae01e7d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:42720/0 (socket says 172.21.15.19:42720) 2024-08-31T12:23:01.314 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.311+0000 7f2ae5f45640 1 -- 172.21.15.19:0/670241711 learned_addr learned my addr 172.21.15.19:0/670241711 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:23:01.314 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.311+0000 7f2ae5f45640 1 -- 172.21.15.19:0/670241711 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2ae00cc070 msgr2=0x7f2ae015c3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:23:01.314 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.311+0000 7f2ae5f45640 1 --2- 172.21.15.19:0/670241711 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2ae00cc070 0x7f2ae015c3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:01.314 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.311+0000 7f2ae5f45640 1 -- 172.21.15.19:0/670241711 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ae00ccbb0 msgr2=0x7f2ae01e7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:01.314 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.311+0000 7f2ae5f45640 1 --2- 172.21.15.19:0/670241711 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ae00ccbb0 0x7f2ae01e7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:01.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.311+0000 7f2ae5f45640 1 -- 172.21.15.19:0/670241711 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ae015c920 con 0x7f2ae015b830 2024-08-31T12:23:01.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.311+0000 7f2ae4f43640 1 --2- 172.21.15.19:0/670241711 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ae00ccbb0 0x7f2ae01e7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:23:01.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.312+0000 7f2ae5f45640 1 --2- 172.21.15.19:0/670241711 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2ae015b830 0x7f2ae014bfa0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f2ad800cae0 tx=0x7f2ad800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:01.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.312+0000 7f2ad5ffb640 1 -- 172.21.15.19:0/670241711 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad8016020 con 0x7f2ae015b830 2024-08-31T12:23:01.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.312+0000 7f2ad5ffb640 1 -- 172.21.15.19:0/670241711 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2ad800ad30 con 0x7f2ae015b830 2024-08-31T12:23:01.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.312+0000 7f2ad5ffb640 1 -- 172.21.15.19:0/670241711 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad8012680 con 0x7f2ae015b830 2024-08-31T12:23:01.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.312+0000 7f2ae79a8640 1 -- 172.21.15.19:0/670241711 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ae00ca930 con 0x7f2ae015b830 2024-08-31T12:23:01.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.312+0000 7f2ae79a8640 1 -- 172.21.15.19:0/670241711 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ae015dca0 con 0x7f2ae015b830 2024-08-31T12:23:01.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.313+0000 7f2ae79a8640 1 -- 172.21.15.19:0/670241711 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ae0069cc0 con 0x7f2ae015b830 2024-08-31T12:23:01.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.314+0000 7f2ad5ffb640 1 -- 172.21.15.19:0/670241711 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f2ad8019020 con 0x7f2ae015b830 2024-08-31T12:23:01.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.315+0000 7f2ad5ffb640 1 --2- 172.21.15.19:0/670241711 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2ac4077a00 0x7f2ac4079f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:01.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.315+0000 7f2ad5ffb640 1 -- 172.21.15.19:0/670241711 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f2ad8099c20 con 0x7f2ae015b830 2024-08-31T12:23:01.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.315+0000 7f2ae5744640 1 --2- 172.21.15.19:0/670241711 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2ac4077a00 0x7f2ac4079f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:01.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.316+0000 7f2ae5744640 1 --2- 172.21.15.19:0/670241711 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2ac4077a00 0x7f2ac4079f10 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2adc034000 tx=0x7f2adc004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:01.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.320+0000 7f2ad5ffb640 1 -- 172.21.15.19:0/670241711 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f2ae0069cc0 con 0x7f2ae015b830 2024-08-31T12:23:01.614 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.613+0000 7f2ae79a8640 1 -- 172.21.15.19:0/670241711 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2ae0155250 con 0x7f2ae015b830 2024-08-31T12:23:01.614 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.613+0000 7f2ad5ffb640 1 -- 172.21.15.19:0/670241711 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f2ae0155250 con 0x7f2ae015b830 2024-08-31T12:23:01.615 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":50,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:23:01.617 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.616+0000 7f2ae79a8640 1 -- 172.21.15.19:0/670241711 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2ac4077a00 msgr2=0x7f2ac4079f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:01.617 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.616+0000 7f2ae79a8640 1 --2- 172.21.15.19:0/670241711 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2ac4077a00 0x7f2ac4079f10 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2adc034000 tx=0x7f2adc004730 comp rx=0 tx=0).stop 2024-08-31T12:23:01.617 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.616+0000 7f2ae79a8640 1 -- 172.21.15.19:0/670241711 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2ae015b830 msgr2=0x7f2ae014bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:01.617 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.616+0000 7f2ae79a8640 1 --2- 172.21.15.19:0/670241711 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2ae015b830 0x7f2ae014bfa0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f2ad800cae0 tx=0x7f2ad800a060 comp rx=0 tx=0).stop 2024-08-31T12:23:01.617 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.616+0000 7f2ae79a8640 1 -- 172.21.15.19:0/670241711 shutdown_connections 2024-08-31T12:23:01.617 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.616+0000 7f2ae79a8640 1 --2- 172.21.15.19:0/670241711 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2ac4077a00 0x7f2ac4079f10 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:01.617 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.616+0000 7f2ae79a8640 1 --2- 172.21.15.19:0/670241711 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2ae015b830 0x7f2ae014bfa0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:01.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.616+0000 7f2ae79a8640 1 --2- 172.21.15.19:0/670241711 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ae00ccbb0 0x7f2ae01e7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:01.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.616+0000 7f2ae79a8640 1 --2- 172.21.15.19:0/670241711 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2ae00cc070 0x7f2ae015c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:01.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.616+0000 7f2ae79a8640 1 -- 172.21.15.19:0/670241711 >> 172.21.15.19:0/670241711 conn(0x7f2ae0074810 msgr2=0x7f2ae0150880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:23:01.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.616+0000 7f2ae79a8640 1 -- 172.21.15.19:0/670241711 shutdown_connections 2024-08-31T12:23:01.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:01.616+0000 7f2ae79a8640 1 -- 172.21.15.19:0/670241711 wait complete. 2024-08-31T12:23:04.629 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:23:04.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:04.755+0000 7ffbb0152640 1 -- 172.21.15.19:0/2046984099 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffba815cbc0 msgr2=0x7ffba81531f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:04.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:04.755+0000 7ffbb0152640 1 --2- 172.21.15.19:0/2046984099 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffba815cbc0 0x7ffba81531f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ffba0005fd0 tx=0x7ffba002a8b0 comp rx=0 tx=0).stop 2024-08-31T12:23:04.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:04.755+0000 7ffbb0152640 1 -- 172.21.15.19:0/2046984099 shutdown_connections 2024-08-31T12:23:04.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:04.755+0000 7ffbb0152640 1 --2- 172.21.15.19:0/2046984099 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffba815cbc0 0x7ffba81531f0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:04.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:04.755+0000 7ffbb0152640 1 -- 172.21.15.19:0/2046984099 >> 172.21.15.19:0/2046984099 conn(0x7ffba80cefb0 msgr2=0x7ffba80cb340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:23:04.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:04.755+0000 7ffbb0152640 1 -- 172.21.15.19:0/2046984099 shutdown_connections 2024-08-31T12:23:04.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:04.755+0000 7ffbb0152640 1 -- 172.21.15.19:0/2046984099 wait complete. 2024-08-31T12:23:04.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:04.757+0000 7ffbb0152640 1 Processor -- start 2024-08-31T12:23:04.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:04.757+0000 7ffbb0152640 1 -- start start 2024-08-31T12:23:04.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:04.759+0000 7ffbb0152640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffba80ca7d0 0x7ffba80b2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:04.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:04.759+0000 7ffbb0152640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffba80cac90 0x7ffba80b3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:04.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:04.759+0000 7ffbb0152640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffba815cbc0 0x7ffba81e36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:04.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:04.759+0000 7ffbb0152640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7ffba80714f0 con 0x7ffba80ca7d0 2024-08-31T12:23:04.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:04.759+0000 7ffbb0152640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ffba80c9bb0 con 0x7ffba80cac90 2024-08-31T12:23:04.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:04.759+0000 7ffbb0152640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7ffba815d3a0 con 0x7ffba815cbc0 2024-08-31T12:23:04.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:04.759+0000 7ffbae6ef640 1 -- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffba815cbc0 msgr2=0x7ffba81e36d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3301/0 2024-08-31T12:23:04.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:04.759+0000 7ffbae6ef640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffba815cbc0 0x7ffba81e36d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:23:04.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:04.759+0000 7ffbadeee640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffba80ca7d0 0x7ffba80b2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:04.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:04.759+0000 7ffbadeee640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffba80ca7d0 0x7ffba80b2470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:44176/0 (socket says 172.21.15.19:44176) 2024-08-31T12:23:04.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:04.759+0000 7ffbadeee640 1 -- 172.21.15.19:0/2403834921 learned_addr learned my addr 172.21.15.19:0/2403834921 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:23:04.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:04.759+0000 7ffbad6ed640 1 --2- 172.21.15.19:0/2403834921 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffba80cac90 0x7ffba80b3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:04.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:04.760+0000 7ffbadeee640 1 -- 172.21.15.19:0/2403834921 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffba815cbc0 msgr2=0x7ffba81e36d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:23:04.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:04.760+0000 7ffbadeee640 1 --2- 172.21.15.19:0/2403834921 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffba815cbc0 0x7ffba81e36d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:04.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:04.760+0000 7ffbadeee640 1 -- 172.21.15.19:0/2403834921 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffba80cac90 msgr2=0x7ffba80b3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:04.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:04.760+0000 7ffbadeee640 1 --2- 172.21.15.19:0/2403834921 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffba80cac90 0x7ffba80b3030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:04.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:04.760+0000 7ffbadeee640 1 -- 172.21.15.19:0/2403834921 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffba80ce010 con 0x7ffba80ca7d0 2024-08-31T12:23:04.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:04.760+0000 7ffbadeee640 1 --2- 172.21.15.19:0/2403834921 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffba80ca7d0 0x7ffba80b2470 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7ffba400c400 tx=0x7ffba400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:04.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:04.760+0000 7ffb9e7fc640 1 -- 172.21.15.19:0/2403834921 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba4011790 con 0x7ffba80ca7d0 2024-08-31T12:23:04.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:04.760+0000 7ffb9e7fc640 1 -- 172.21.15.19:0/2403834921 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffba4017070 con 0x7ffba80ca7d0 2024-08-31T12:23:04.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:04.760+0000 7ffb9e7fc640 1 -- 172.21.15.19:0/2403834921 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba4011ca0 con 0x7ffba80ca7d0 2024-08-31T12:23:04.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:04.760+0000 7ffbb0152640 1 -- 172.21.15.19:0/2403834921 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffba80ce2f0 con 0x7ffba80ca7d0 2024-08-31T12:23:04.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:04.760+0000 7ffbb0152640 1 -- 172.21.15.19:0/2403834921 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffba80b3e20 con 0x7ffba80ca7d0 2024-08-31T12:23:04.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:04.761+0000 7ffbb0152640 1 -- 172.21.15.19:0/2403834921 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffba80ca160 con 0x7ffba80ca7d0 2024-08-31T12:23:04.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:04.762+0000 7ffb9e7fc640 1 -- 172.21.15.19:0/2403834921 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7ffba401c460 con 0x7ffba80ca7d0 2024-08-31T12:23:04.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:04.762+0000 7ffb9e7fc640 1 --2- 172.21.15.19:0/2403834921 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ffb8c077a00 0x7ffb8c079f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:04.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:04.762+0000 7ffb9e7fc640 1 -- 172.21.15.19:0/2403834921 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7ffba4099390 con 0x7ffba80ca7d0 2024-08-31T12:23:04.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:04.763+0000 7ffbad6ed640 1 --2- 172.21.15.19:0/2403834921 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ffb8c077a00 0x7ffb8c079f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:04.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:04.763+0000 7ffbad6ed640 1 --2- 172.21.15.19:0/2403834921 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ffb8c077a00 0x7ffb8c079f10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ffba80c5120 tx=0x7ffb98009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:04.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:04.767+0000 7ffb9e7fc640 1 -- 172.21.15.19:0/2403834921 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7ffba4062110 con 0x7ffba80ca7d0 2024-08-31T12:23:05.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:05.054+0000 7ffbb0152640 1 -- 172.21.15.19:0/2403834921 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ffba81e4f50 con 0x7ffba80ca7d0 2024-08-31T12:23:05.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:05.054+0000 7ffb9e7fc640 1 -- 172.21.15.19:0/2403834921 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7ffba4061a10 con 0x7ffba80ca7d0 2024-08-31T12:23:05.056 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":50,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":19,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:23:05.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:05.056+0000 7ffbb0152640 1 -- 172.21.15.19:0/2403834921 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ffb8c077a00 msgr2=0x7ffb8c079f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:05.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:05.057+0000 7ffbb0152640 1 --2- 172.21.15.19:0/2403834921 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ffb8c077a00 0x7ffb8c079f10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ffba80c5120 tx=0x7ffb98009040 comp rx=0 tx=0).stop 2024-08-31T12:23:05.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:05.057+0000 7ffbb0152640 1 -- 172.21.15.19:0/2403834921 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffba80ca7d0 msgr2=0x7ffba80b2470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:05.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:05.057+0000 7ffbb0152640 1 --2- 172.21.15.19:0/2403834921 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffba80ca7d0 0x7ffba80b2470 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7ffba400c400 tx=0x7ffba400c8c0 comp rx=0 tx=0).stop 2024-08-31T12:23:05.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:05.057+0000 7ffbb0152640 1 -- 172.21.15.19:0/2403834921 shutdown_connections 2024-08-31T12:23:05.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:05.057+0000 7ffbb0152640 1 --2- 172.21.15.19:0/2403834921 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ffb8c077a00 0x7ffb8c079f10 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:05.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:05.057+0000 7ffbb0152640 1 --2- 172.21.15.19:0/2403834921 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffba815cbc0 0x7ffba81e36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:05.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:05.057+0000 7ffbb0152640 1 --2- 172.21.15.19:0/2403834921 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffba80cac90 0x7ffba80b3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:05.059 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:05.057+0000 7ffbb0152640 1 --2- 172.21.15.19:0/2403834921 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffba80ca7d0 0x7ffba80b2470 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:05.059 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:05.057+0000 7ffbb0152640 1 -- 172.21.15.19:0/2403834921 >> 172.21.15.19:0/2403834921 conn(0x7ffba80cefb0 msgr2=0x7ffba814ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:23:05.059 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:05.057+0000 7ffbb0152640 1 -- 172.21.15.19:0/2403834921 shutdown_connections 2024-08-31T12:23:05.059 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:05.057+0000 7ffbb0152640 1 -- 172.21.15.19:0/2403834921 wait complete. 2024-08-31T12:23:08.070 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:23:11.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:11.904+0000 7fb6f2579640 1 -- 172.21.15.19:0/3143211892 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb6d40039b0 msgr2=0x7fb6d4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:11.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:11.904+0000 7fb6f2579640 1 --2- 172.21.15.19:0/3143211892 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb6d40039b0 0x7fb6d4003e00 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc001950 tx=0x7fb6d4005830 comp rx=0 tx=0).stop 2024-08-31T12:23:11.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:11.904+0000 7fb6f2579640 1 -- 172.21.15.19:0/3143211892 shutdown_connections 2024-08-31T12:23:11.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:11.904+0000 7fb6f2579640 1 --2- 172.21.15.19:0/3143211892 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb6d40053f0 0x7fb6d400ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:11.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:11.904+0000 7fb6f2579640 1 --2- 172.21.15.19:0/3143211892 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb6d4004800 0x7fb6d4004c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:11.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:11.904+0000 7fb6f2579640 1 --2- 172.21.15.19:0/3143211892 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb6d40039b0 0x7fb6d4003e00 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:11.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:11.904+0000 7fb6f2579640 1 -- 172.21.15.19:0/3143211892 >> 172.21.15.19:0/3143211892 conn(0x7fb6ec06cae0 msgr2=0x7fb6ec06cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:23:11.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:11.904+0000 7fb6f2579640 1 -- 172.21.15.19:0/3143211892 shutdown_connections 2024-08-31T12:23:11.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:11.904+0000 7fb6f2579640 1 -- 172.21.15.19:0/3143211892 wait complete. 2024-08-31T12:23:11.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:11.907+0000 7fb6f2579640 1 Processor -- start 2024-08-31T12:23:11.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:11.907+0000 7fb6f2579640 1 -- start start 2024-08-31T12:23:11.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:11.909+0000 7fb6f2579640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb6d40039b0 0x7fb6ec154090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:11.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:11.909+0000 7fb6f2579640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb6d4004800 0x7fb6ec0cbfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:11.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:11.909+0000 7fb6f2579640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb6ec0b1bb0 0x7fb6ec0b1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:11.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:11.909+0000 7fb6f2579640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fb6ec06da00 con 0x7fb6d4004800 2024-08-31T12:23:11.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:11.909+0000 7fb6f2579640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb6ec068fd0 con 0x7fb6d40039b0 2024-08-31T12:23:11.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:11.909+0000 7fb6f2579640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fb6ec0c3e90 con 0x7fb6ec0b1bb0 2024-08-31T12:23:11.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:11.910+0000 7fb6eb7fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb6d4004800 0x7fb6ec0cbfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:11.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:11.910+0000 7fb6f0b16640 1 -- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb6ec0b1bb0 msgr2=0x7fb6ec0b1fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3301/0 2024-08-31T12:23:11.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:11.910+0000 7fb6f0b16640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb6ec0b1bb0 0x7fb6ec0b1fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:23:11.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:11.910+0000 7fb6eb7fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb6d4004800 0x7fb6ec0cbfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:48728/0 (socket says 172.21.15.19:48728) 2024-08-31T12:23:11.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:11.910+0000 7fb6eb7fe640 1 -- 172.21.15.19:0/2250219594 learned_addr learned my addr 172.21.15.19:0/2250219594 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:23:11.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:11.910+0000 7fb6ebfff640 1 --2- 172.21.15.19:0/2250219594 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb6d40039b0 0x7fb6ec154090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:11.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:11.910+0000 7fb6eb7fe640 1 -- 172.21.15.19:0/2250219594 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb6ec0b1bb0 msgr2=0x7fb6ec0b1fc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:23:11.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:11.910+0000 7fb6eb7fe640 1 --2- 172.21.15.19:0/2250219594 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb6ec0b1bb0 0x7fb6ec0b1fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:11.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:11.910+0000 7fb6eb7fe640 1 -- 172.21.15.19:0/2250219594 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb6d40039b0 msgr2=0x7fb6ec154090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:11.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:11.910+0000 7fb6eb7fe640 1 --2- 172.21.15.19:0/2250219594 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb6d40039b0 0x7fb6ec154090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:11.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:11.910+0000 7fb6eb7fe640 1 -- 172.21.15.19:0/2250219594 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6ec1f1750 con 0x7fb6d4004800 2024-08-31T12:23:11.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:11.910+0000 7fb6eb7fe640 1 --2- 172.21.15.19:0/2250219594 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb6d4004800 0x7fb6ec0cbfa0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fb6e0012a20 tx=0x7fb6e0012ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:11.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:11.910+0000 7fb6e8ff9640 1 -- 172.21.15.19:0/2250219594 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6e0021020 con 0x7fb6d4004800 2024-08-31T12:23:11.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:11.910+0000 7fb6e8ff9640 1 -- 172.21.15.19:0/2250219594 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb6e001d440 con 0x7fb6d4004800 2024-08-31T12:23:11.914 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:11.910+0000 7fb6f2579640 1 -- 172.21.15.19:0/2250219594 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6ec06fdd0 con 0x7fb6d4004800 2024-08-31T12:23:11.914 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:11.910+0000 7fb6e8ff9640 1 -- 172.21.15.19:0/2250219594 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6e001d890 con 0x7fb6d4004800 2024-08-31T12:23:11.914 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:11.911+0000 7fb6f2579640 1 -- 172.21.15.19:0/2250219594 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6ec157140 con 0x7fb6d4004800 2024-08-31T12:23:11.914 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:11.912+0000 7fb6f2579640 1 -- 172.21.15.19:0/2250219594 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6ec158b20 con 0x7fb6d4004800 2024-08-31T12:23:11.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:11.913+0000 7fb6e8ff9640 1 -- 172.21.15.19:0/2250219594 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fb6e0024070 con 0x7fb6d4004800 2024-08-31T12:23:11.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:11.913+0000 7fb6e8ff9640 1 --2- 172.21.15.19:0/2250219594 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb6d8078220 0x7fb6d807a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:11.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:11.913+0000 7fb6e8ff9640 1 -- 172.21.15.19:0/2250219594 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fb6e009ac50 con 0x7fb6d4004800 2024-08-31T12:23:11.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:11.914+0000 7fb6ebfff640 1 --2- 172.21.15.19:0/2250219594 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb6d8078220 0x7fb6d807a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:11.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:11.914+0000 7fb6ebfff640 1 --2- 172.21.15.19:0/2250219594 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb6d8078220 0x7fb6d807a730 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc0011b0 tx=0x7fb6dc001420 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:11.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:11.918+0000 7fb6e8ff9640 1 -- 172.21.15.19:0/2250219594 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fb6e00a0020 con 0x7fb6d4004800 2024-08-31T12:23:12.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.223+0000 7fb6f2579640 1 -- 172.21.15.19:0/2250219594 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb6ec14bf20 con 0x7fb6d4004800 2024-08-31T12:23:12.225 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.223+0000 7fb6e8ff9640 1 -- 172.21.15.19:0/2250219594 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7fb6e00639d0 con 0x7fb6d4004800 2024-08-31T12:23:12.225 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":54,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:23:12.227 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.226+0000 7fb6f2579640 1 -- 172.21.15.19:0/2250219594 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb6d8078220 msgr2=0x7fb6d807a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:12.227 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.226+0000 7fb6f2579640 1 --2- 172.21.15.19:0/2250219594 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb6d8078220 0x7fb6d807a730 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc0011b0 tx=0x7fb6dc001420 comp rx=0 tx=0).stop 2024-08-31T12:23:12.228 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.226+0000 7fb6f2579640 1 -- 172.21.15.19:0/2250219594 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb6d4004800 msgr2=0x7fb6ec0cbfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:12.228 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.226+0000 7fb6f2579640 1 --2- 172.21.15.19:0/2250219594 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb6d4004800 0x7fb6ec0cbfa0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fb6e0012a20 tx=0x7fb6e0012ee0 comp rx=0 tx=0).stop 2024-08-31T12:23:12.228 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.226+0000 7fb6f2579640 1 -- 172.21.15.19:0/2250219594 shutdown_connections 2024-08-31T12:23:12.228 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.226+0000 7fb6f2579640 1 --2- 172.21.15.19:0/2250219594 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb6d8078220 0x7fb6d807a730 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:12.228 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.226+0000 7fb6f2579640 1 --2- 172.21.15.19:0/2250219594 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb6ec0b1bb0 0x7fb6ec0b1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:12.228 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.226+0000 7fb6f2579640 1 --2- 172.21.15.19:0/2250219594 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb6d4004800 0x7fb6ec0cbfa0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:12.228 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.226+0000 7fb6f2579640 1 --2- 172.21.15.19:0/2250219594 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb6d40039b0 0x7fb6ec154090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:12.228 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.226+0000 7fb6f2579640 1 -- 172.21.15.19:0/2250219594 >> 172.21.15.19:0/2250219594 conn(0x7fb6ec06cae0 msgr2=0x7fb6ec14faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:23:12.228 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.226+0000 7fb6f2579640 1 -- 172.21.15.19:0/2250219594 shutdown_connections 2024-08-31T12:23:12.228 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.226+0000 7fb6f2579640 1 -- 172.21.15.19:0/2250219594 wait complete. 2024-08-31T12:23:12.239 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-08-31T12:23:12.240 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-08-31T12:23:12.240 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-31T12:23:12.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.361+0000 7f835755f640 1 -- 172.21.15.19:0/3154936019 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83500c9bb0 msgr2=0x7f83500c7000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:12.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.361+0000 7f835755f640 1 --2- 172.21.15.19:0/3154936019 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83500c9bb0 0x7f83500c7000 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8340009830 tx=0x7f834002a540 comp rx=0 tx=0).stop 2024-08-31T12:23:12.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.361+0000 7f835755f640 1 -- 172.21.15.19:0/3154936019 shutdown_connections 2024-08-31T12:23:12.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.361+0000 7f835755f640 1 --2- 172.21.15.19:0/3154936019 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83500c9bb0 0x7f83500c7000 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:12.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.361+0000 7f835755f640 1 -- 172.21.15.19:0/3154936019 >> 172.21.15.19:0/3154936019 conn(0x7f83500700a0 msgr2=0x7f83500cd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:23:12.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.361+0000 7f835755f640 1 -- 172.21.15.19:0/3154936019 shutdown_connections 2024-08-31T12:23:12.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.361+0000 7f835755f640 1 -- 172.21.15.19:0/3154936019 wait complete. 2024-08-31T12:23:12.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.363+0000 7f835755f640 1 Processor -- start 2024-08-31T12:23:12.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.363+0000 7f835755f640 1 -- start start 2024-08-31T12:23:12.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.364+0000 7f835755f640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83500c7f70 0x7f83501e1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:12.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.364+0000 7f835755f640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f83500c9660 0x7f835006ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:12.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.364+0000 7f835755f640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83500c9bb0 0x7f8350142e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:12.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.364+0000 7f835755f640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f83500c5640 con 0x7f83500c7f70 2024-08-31T12:23:12.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.364+0000 7f835755f640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f83501521f0 con 0x7f83500c9bb0 2024-08-31T12:23:12.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.364+0000 7f835755f640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f83501e3fd0 con 0x7f83500c9660 2024-08-31T12:23:12.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.364+0000 7f83552fb640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83500c7f70 0x7f83501e1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:12.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.364+0000 7f8354afa640 1 -- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f83500c9660 msgr2=0x7f835006ccc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3301/0 2024-08-31T12:23:12.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.364+0000 7f83552fb640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83500c7f70 0x7f83501e1fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:48730/0 (socket says 172.21.15.19:48730) 2024-08-31T12:23:12.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.364+0000 7f83552fb640 1 -- 172.21.15.19:0/4271865849 learned_addr learned my addr 172.21.15.19:0/4271865849 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:23:12.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.364+0000 7f8354afa640 1 --2- 172.21.15.19:0/4271865849 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f83500c9660 0x7f835006ccc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:23:12.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.364+0000 7f8355afc640 1 --2- 172.21.15.19:0/4271865849 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83500c9bb0 0x7f8350142e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:12.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.364+0000 7f83552fb640 1 -- 172.21.15.19:0/4271865849 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f83500c9660 msgr2=0x7f835006ccc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:23:12.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.364+0000 7f83552fb640 1 --2- 172.21.15.19:0/4271865849 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f83500c9660 0x7f835006ccc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:12.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.364+0000 7f83552fb640 1 -- 172.21.15.19:0/4271865849 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83500c9bb0 msgr2=0x7f8350142e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:12.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.364+0000 7f83552fb640 1 --2- 172.21.15.19:0/4271865849 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83500c9bb0 0x7f8350142e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:12.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.364+0000 7f83552fb640 1 -- 172.21.15.19:0/4271865849 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83501576c0 con 0x7f83500c7f70 2024-08-31T12:23:12.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.365+0000 7f83552fb640 1 --2- 172.21.15.19:0/4271865849 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83500c7f70 0x7f83501e1fa0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f834c00c430 tx=0x7f834c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:12.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.365+0000 7f8345ffb640 1 -- 172.21.15.19:0/4271865849 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f834c017070 con 0x7f83500c7f70 2024-08-31T12:23:12.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.365+0000 7f835755f640 1 -- 172.21.15.19:0/4271865849 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83500cb4f0 con 0x7f83500c7f70 2024-08-31T12:23:12.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.365+0000 7f8345ffb640 1 -- 172.21.15.19:0/4271865849 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f834c0053c0 con 0x7f83500c7f70 2024-08-31T12:23:12.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.365+0000 7f8345ffb640 1 -- 172.21.15.19:0/4271865849 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f834c003850 con 0x7f83500c7f70 2024-08-31T12:23:12.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.365+0000 7f835755f640 1 -- 172.21.15.19:0/4271865849 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83501f4b90 con 0x7f83500c7f70 2024-08-31T12:23:12.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.366+0000 7f835755f640 1 --2- 172.21.15.19:0/4271865849 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83501f5670 0x7f8350143360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:12.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.366+0000 7f835755f640 1 -- 172.21.15.19:0/4271865849 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f83500c57c0 con 0x7f83501f5670 2024-08-31T12:23:12.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.366+0000 7f8354afa640 1 --2- 172.21.15.19:0/4271865849 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83501f5670 0x7f8350143360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:12.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.367+0000 7f8354afa640 1 -- 172.21.15.19:0/4271865849 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f835006de90 con 0x7f83501f5670 2024-08-31T12:23:12.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.367+0000 7f8345ffb640 1 -- 172.21.15.19:0/4271865849 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f834c01a070 con 0x7f83500c7f70 2024-08-31T12:23:12.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.367+0000 7f8354afa640 1 --2- 172.21.15.19:0/4271865849 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83501f5670 0x7f8350143360 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f83500c9ff0 tx=0x7f8340003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:12.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.367+0000 7f8345ffb640 1 --2- 172.21.15.19:0/4271865849 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8334077a00 0x7f8334079f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:12.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.368+0000 7f8345ffb640 1 -- 172.21.15.19:0/4271865849 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f834c098ab0 con 0x7f83500c7f70 2024-08-31T12:23:12.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.368+0000 7f8355afc640 1 --2- 172.21.15.19:0/4271865849 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8334077a00 0x7f8334079f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:12.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.368+0000 7f8345ffb640 1 -- 172.21.15.19:0/4271865849 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8340003df0 con 0x7f83501f5670 2024-08-31T12:23:12.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.368+0000 7f8345ffb640 1 -- 172.21.15.19:0/4271865849 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8340005020 con 0x7f83501f5670 2024-08-31T12:23:12.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.368+0000 7f8345ffb640 1 -- 172.21.15.19:0/4271865849 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83501f5670 msgr2=0x7f8350143360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:12.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.368+0000 7f8345ffb640 1 --2- 172.21.15.19:0/4271865849 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83501f5670 0x7f8350143360 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f83500c9ff0 tx=0x7f8340003820 comp rx=0 tx=0).stop 2024-08-31T12:23:12.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.368+0000 7f8345ffb640 1 -- 172.21.15.19:0/4271865849 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83501f5670 msgr2=0x7f8350143360 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:23:12.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.368+0000 7f8345ffb640 1 --2- 172.21.15.19:0/4271865849 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83501f5670 0x7f8350143360 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:12.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.369+0000 7f8355afc640 1 --2- 172.21.15.19:0/4271865849 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8334077a00 0x7f8334079f10 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f83480015a0 tx=0x7f8348009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:12.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.375+0000 7f835755f640 1 --2- 172.21.15.19:0/4271865849 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83501f4dd0 0x7f83500cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:12.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.375+0000 7f835755f640 1 -- 172.21.15.19:0/4271865849 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f8350064210 con 0x7f83501f4dd0 2024-08-31T12:23:12.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.375+0000 7f8354afa640 1 --2- 172.21.15.19:0/4271865849 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83501f4dd0 0x7f83500cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:12.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.375+0000 7f8354afa640 1 -- 172.21.15.19:0/4271865849 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f83501e24e0 con 0x7f83501f4dd0 2024-08-31T12:23:12.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.375+0000 7f8354afa640 1 --2- 172.21.15.19:0/4271865849 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83501f4dd0 0x7f83500cc8a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f8340009830 tx=0x7f834002a540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:12.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.376+0000 7f8345ffb640 1 -- 172.21.15.19:0/4271865849 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f834003b820 con 0x7f83501f4dd0 2024-08-31T12:23:12.378 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.376+0000 7f8345ffb640 1 -- 172.21.15.19:0/4271865849 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5033 (secure 0 0 0) 0x7f834003b820 con 0x7f83501f4dd0 2024-08-31T12:23:12.378 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.376+0000 7f8345ffb640 1 -- 172.21.15.19:0/4271865849 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83501f4dd0 msgr2=0x7f83500cc8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:12.378 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.376+0000 7f8345ffb640 1 --2- 172.21.15.19:0/4271865849 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83501f4dd0 0x7f83500cc8a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f8340009830 tx=0x7f834002a540 comp rx=0 tx=0).stop 2024-08-31T12:23:12.378 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.376+0000 7f8345ffb640 1 -- 172.21.15.19:0/4271865849 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83501f4dd0 msgr2=0x7f83500cc8a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:23:12.378 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.376+0000 7f8345ffb640 1 --2- 172.21.15.19:0/4271865849 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83501f4dd0 0x7f83500cc8a0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:12.378 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:23:12.378 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:23:12.378 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:23:12.378 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "leader", 2024-08-31T12:23:12.378 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 54, 2024-08-31T12:23:12.379 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 372367, 2024-08-31T12:23:12.379 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:23:12.379 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:23:12.379 INFO:teuthology.orchestra.run.smithi019.stdout: 1 2024-08-31T12:23:12.379 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:23:12.379 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 0, 2024-08-31T12:23:12.379 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:23:12.379 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:23:12.379 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:23:12.379 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:23:12.379 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:23:12.379 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:23:12.379 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:23:12.379 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:23:12.380 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:23:12.380 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:23:12.380 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:23:12.380 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:23:12.380 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:23:12.380 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:23:12.380 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:23:12.380 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:23:12.380 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:23:12.381 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:23:12.381 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:23:12.381 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:23:12.381 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:23:12.381 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:23:12.381 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:23:12.381 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:23:12.381 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:23:12.381 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:23:12.381 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:23:12.381 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:23:12.381 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:12.382 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:12.382 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:23:12.382 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:23:12.382 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:23:12.382 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:23:12.382 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:23:12.382 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:23:12.382 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:23:12.382 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:23:12.382 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:23:12.382 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:23:12.382 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:23:12.382 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:23:12.382 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:23:12.383 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:23:12.383 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:23:12.383 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:23:12.383 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:23:12.383 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:23:12.383 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:23:12.383 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:23:12.383 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:23:12.383 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:23:12.383 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:23:12.383 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:23:12.383 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:23:12.383 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:23:12.383 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:23:12.383 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:23:12.384 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:23:12.384 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:23:12.384 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:12.384 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:23:12.384 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:12.384 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:23:12.384 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:23:12.384 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:23:12.384 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:23:12.384 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:12.384 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:23:12.384 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:23:12.385 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:12.385 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:12.385 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:12.385 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:23:12.385 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:23:12.385 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:12.386 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:12.386 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:12.386 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:12.386 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:23:12.386 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:23:12.386 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:23:12.386 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:23:12.386 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:23:12.386 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:12.386 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:12.386 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:23:12.386 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:23:12.386 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:23:12.387 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:23:12.387 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:12.387 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:23:12.387 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:23:12.387 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:12.387 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:12.387 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:12.387 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:23:12.387 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:23:12.387 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:12.387 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:12.387 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:12.388 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:12.388 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:23:12.388 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:23:12.388 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:23:12.388 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:23:12.388 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:23:12.388 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:12.388 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:12.388 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:23:12.388 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:23:12.388 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:23:12.388 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:23:12.388 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:12.388 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:23:12.389 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:23:12.389 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:12.389 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:12.389 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:12.389 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:23:12.389 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:23:12.389 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:12.389 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:12.389 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:12.389 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:12.389 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:23:12.389 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:23:12.389 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:23:12.390 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:23:12.390 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:23:12.390 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:12.390 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:12.390 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:12.390 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:23:12.390 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:23:12.390 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:12.390 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:23:12.390 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:23:12.390 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:23:12.390 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:12.390 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:23:12.391 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:23:12.391 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:12.391 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:23:12.391 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:23:12.391 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 6 2024-08-31T12:23:12.391 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:12.391 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:23:12.391 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:23:12.391 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:12.391 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:23:12.391 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:23:12.391 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 8 2024-08-31T12:23:12.391 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:12.392 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:23:12.392 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:23:12.392 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:12.392 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:23:12.392 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:23:12.392 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 6 2024-08-31T12:23:12.392 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:12.392 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:23:12.392 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:23:12.392 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:12.392 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:23:12.392 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:23:12.393 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:23:12.393 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:12.393 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:12.393 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:12.393 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:23:12.393 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:23:12.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.376+0000 7f835755f640 1 -- 172.21.15.19:0/4271865849 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8334077a00 msgr2=0x7f8334079f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:12.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.376+0000 7f835755f640 1 --2- 172.21.15.19:0/4271865849 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8334077a00 0x7f8334079f10 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f83480015a0 tx=0x7f8348009040 comp rx=0 tx=0).stop 2024-08-31T12:23:12.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.376+0000 7f835755f640 1 -- 172.21.15.19:0/4271865849 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83500c7f70 msgr2=0x7f83501e1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:12.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.376+0000 7f835755f640 1 --2- 172.21.15.19:0/4271865849 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83500c7f70 0x7f83501e1fa0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f834c00c430 tx=0x7f834c00c8f0 comp rx=0 tx=0).stop 2024-08-31T12:23:12.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.376+0000 7f8355afc640 1 -- 172.21.15.19:0/4271865849 reap_dead start 2024-08-31T12:23:12.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.376+0000 7f835755f640 1 -- 172.21.15.19:0/4271865849 shutdown_connections 2024-08-31T12:23:12.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.376+0000 7f835755f640 1 --2- 172.21.15.19:0/4271865849 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83500c7f70 0x7f83501e1fa0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:12.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.376+0000 7f835755f640 1 -- 172.21.15.19:0/4271865849 >> 172.21.15.19:0/4271865849 conn(0x7f83500700a0 msgr2=0x7f8350148330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:23:12.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.376+0000 7f835755f640 1 -- 172.21.15.19:0/4271865849 shutdown_connections 2024-08-31T12:23:12.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.377+0000 7f835755f640 1 -- 172.21.15.19:0/4271865849 wait complete. 2024-08-31T12:23:12.395 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-08-31T12:23:12.395 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-08-31T12:23:12.396 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-31T12:23:12.517 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.516+0000 7f6a65e0e640 1 -- 172.21.15.19:0/154709294 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6a601587f0 msgr2=0x7f6a6014e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:12.517 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.516+0000 7f6a65e0e640 1 --2- 172.21.15.19:0/154709294 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6a601587f0 0x7f6a6014e720 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f6a600c7ed0 tx=0x7f6a5402a530 comp rx=0 tx=0).stop 2024-08-31T12:23:12.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.516+0000 7f6a65e0e640 1 -- 172.21.15.19:0/154709294 shutdown_connections 2024-08-31T12:23:12.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.516+0000 7f6a65e0e640 1 --2- 172.21.15.19:0/154709294 >> v2:172.21.15.43:3300/0 conn(0x7f6a600c85a0 0x7f6a600c89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:12.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.516+0000 7f6a65e0e640 1 --2- 172.21.15.19:0/154709294 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6a601587f0 0x7f6a6014e720 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:12.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.516+0000 7f6a65e0e640 1 -- 172.21.15.19:0/154709294 >> 172.21.15.19:0/154709294 conn(0x7f6a6006cae0 msgr2=0x7f6a6006cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:23:12.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.516+0000 7f6a65e0e640 1 -- 172.21.15.19:0/154709294 shutdown_connections 2024-08-31T12:23:12.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.516+0000 7f6a65e0e640 1 -- 172.21.15.19:0/154709294 wait complete. 2024-08-31T12:23:12.519 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.518+0000 7f6a65e0e640 1 Processor -- start 2024-08-31T12:23:12.519 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.518+0000 7f6a65e0e640 1 -- start start 2024-08-31T12:23:12.521 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.519+0000 7f6a65e0e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6a600c7ac0 0x7f6a601de1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:12.521 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.519+0000 7f6a65e0e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6a600c85a0 0x7f6a6006f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:12.521 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.519+0000 7f6a65e0e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a601587f0 0x7f6a601df870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:12.522 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.519+0000 7f6a65e0e640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f6a60152380 con 0x7f6a600c85a0 2024-08-31T12:23:12.522 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.519+0000 7f6a65e0e640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6a600c8ef0 con 0x7f6a601587f0 2024-08-31T12:23:12.522 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.519+0000 7f6a65e0e640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f6a600c3e90 con 0x7f6a600c7ac0 2024-08-31T12:23:12.522 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.519+0000 7f6a5f7fe640 1 -- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6a600c7ac0 msgr2=0x7f6a601de1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3301/0 2024-08-31T12:23:12.522 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.519+0000 7f6a5f7fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6a600c7ac0 0x7f6a601de1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:23:12.522 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.519+0000 7f6a5effd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6a600c85a0 0x7f6a6006f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:12.522 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.519+0000 7f6a5effd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6a600c85a0 0x7f6a6006f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:48750/0 (socket says 172.21.15.19:48750) 2024-08-31T12:23:12.522 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.519+0000 7f6a5effd640 1 -- 172.21.15.19:0/298474945 learned_addr learned my addr 172.21.15.19:0/298474945 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:23:12.522 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.519+0000 7f6a5ffff640 1 --2- 172.21.15.19:0/298474945 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a601587f0 0x7f6a601df870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:12.522 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.520+0000 7f6a5effd640 1 -- 172.21.15.19:0/298474945 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6a600c7ac0 msgr2=0x7f6a601de1a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:23:12.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.520+0000 7f6a5effd640 1 --2- 172.21.15.19:0/298474945 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6a600c7ac0 0x7f6a601de1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:12.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.520+0000 7f6a5effd640 1 -- 172.21.15.19:0/298474945 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a601587f0 msgr2=0x7f6a601df870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:12.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.520+0000 7f6a5effd640 1 --2- 172.21.15.19:0/298474945 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a601587f0 0x7f6a601df870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:12.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.520+0000 7f6a5effd640 1 -- 172.21.15.19:0/298474945 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a601de6e0 con 0x7f6a600c85a0 2024-08-31T12:23:12.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.520+0000 7f6a5effd640 1 --2- 172.21.15.19:0/298474945 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6a600c85a0 0x7f6a6006f380 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f6a540098f0 tx=0x7f6a54033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:12.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.520+0000 7f6a47fff640 1 -- 172.21.15.19:0/298474945 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a54031d60 con 0x7f6a600c85a0 2024-08-31T12:23:12.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.520+0000 7f6a47fff640 1 -- 172.21.15.19:0/298474945 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6a54033e10 con 0x7f6a600c85a0 2024-08-31T12:23:12.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.520+0000 7f6a65e0e640 1 -- 172.21.15.19:0/298474945 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a600cc120 con 0x7f6a600c85a0 2024-08-31T12:23:12.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.520+0000 7f6a47fff640 1 -- 172.21.15.19:0/298474945 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a54005260 con 0x7f6a600c85a0 2024-08-31T12:23:12.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.520+0000 7f6a65e0e640 1 -- 172.21.15.19:0/298474945 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a600b24c0 con 0x7f6a600c85a0 2024-08-31T12:23:12.525 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.521+0000 7f6a65e0e640 1 --2- 172.21.15.19:0/298474945 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a600cae90 0x7f6a601dfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:12.525 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.521+0000 7f6a65e0e640 1 -- 172.21.15.19:0/298474945 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6a600ccaf0 con 0x7f6a600cae90 2024-08-31T12:23:12.525 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.521+0000 7f6a5f7fe640 1 --2- 172.21.15.19:0/298474945 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a600cae90 0x7f6a601dfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:12.525 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.522+0000 7f6a5f7fe640 1 -- 172.21.15.19:0/298474945 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6a600c89b0 con 0x7f6a600cae90 2024-08-31T12:23:12.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.522+0000 7f6a47fff640 1 -- 172.21.15.19:0/298474945 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f6a54005400 con 0x7f6a600c85a0 2024-08-31T12:23:12.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.522+0000 7f6a5f7fe640 1 --2- 172.21.15.19:0/298474945 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a600cae90 0x7f6a601dfdb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6a50000f20 tx=0x7f6a5000c7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:12.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.522+0000 7f6a47fff640 1 --2- 172.21.15.19:0/298474945 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6a4c0779b0 0x7f6a4c079ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:12.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.522+0000 7f6a47fff640 1 -- 172.21.15.19:0/298474945 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f6a540ba840 con 0x7f6a600c85a0 2024-08-31T12:23:12.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.523+0000 7f6a5ffff640 1 --2- 172.21.15.19:0/298474945 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6a4c0779b0 0x7f6a4c079ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:12.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.523+0000 7f6a47fff640 1 -- 172.21.15.19:0/298474945 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a500074e0 con 0x7f6a600cae90 2024-08-31T12:23:12.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.523+0000 7f6a47fff640 1 -- 172.21.15.19:0/298474945 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6a50004790 con 0x7f6a600cae90 2024-08-31T12:23:12.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.523+0000 7f6a47fff640 1 -- 172.21.15.19:0/298474945 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a600cae90 msgr2=0x7f6a601dfdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:12.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.523+0000 7f6a47fff640 1 --2- 172.21.15.19:0/298474945 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a600cae90 0x7f6a601dfdb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6a50000f20 tx=0x7f6a5000c7a0 comp rx=0 tx=0).stop 2024-08-31T12:23:12.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.523+0000 7f6a47fff640 1 -- 172.21.15.19:0/298474945 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a600cae90 msgr2=0x7f6a601dfdb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:23:12.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.523+0000 7f6a47fff640 1 --2- 172.21.15.19:0/298474945 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a600cae90 0x7f6a601dfdb0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:12.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.523+0000 7f6a5ffff640 1 --2- 172.21.15.19:0/298474945 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6a4c0779b0 0x7f6a4c079ec0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6a58005fd0 tx=0x7f6a58009560 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:12.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.529+0000 7f6a65e0e640 1 --2- 172.21.15.19:0/298474945 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a601f4f70 0x7f6a601f5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:12.531 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.530+0000 7f6a65e0e640 1 -- 172.21.15.19:0/298474945 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6a600b2270 con 0x7f6a601f4f70 2024-08-31T12:23:12.531 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.530+0000 7f6a5f7fe640 1 --2- 172.21.15.19:0/298474945 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a601f4f70 0x7f6a601f5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:12.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.530+0000 7f6a5f7fe640 1 -- 172.21.15.19:0/298474945 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6a600c78a0 con 0x7f6a601f4f70 2024-08-31T12:23:12.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.530+0000 7f6a5f7fe640 1 --2- 172.21.15.19:0/298474945 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a601f4f70 0x7f6a601f5380 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6a50000f20 tx=0x7f6a50000fe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:12.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.531+0000 7f6a47fff640 1 -- 172.21.15.19:0/298474945 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a50018040 con 0x7f6a601f4f70 2024-08-31T12:23:12.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.531+0000 7f6a47fff640 1 -- 172.21.15.19:0/298474945 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4530 (secure 0 0 0) 0x7f6a50013410 con 0x7f6a601f4f70 2024-08-31T12:23:12.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.531+0000 7f6a47fff640 1 -- 172.21.15.19:0/298474945 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a601f4f70 msgr2=0x7f6a601f5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:12.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.531+0000 7f6a47fff640 1 --2- 172.21.15.19:0/298474945 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a601f4f70 0x7f6a601f5380 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6a50000f20 tx=0x7f6a50000fe0 comp rx=0 tx=0).stop 2024-08-31T12:23:12.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.531+0000 7f6a47fff640 1 -- 172.21.15.19:0/298474945 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a601f4f70 msgr2=0x7f6a601f5380 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:23:12.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.531+0000 7f6a47fff640 1 --2- 172.21.15.19:0/298474945 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a601f4f70 0x7f6a601f5380 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:12.533 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:23:12.533 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:23:12.533 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:23:12.533 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:23:12.533 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 54, 2024-08-31T12:23:12.533 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 102004, 2024-08-31T12:23:12.533 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:23:12.533 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:23:12.533 INFO:teuthology.orchestra.run.smithi019.stdout: 1 2024-08-31T12:23:12.534 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:23:12.534 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 0, 2024-08-31T12:23:12.534 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:23:12.534 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:23:12.534 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:23:12.534 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:23:12.534 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:23:12.534 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:23:12.534 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:23:12.534 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:23:12.534 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:23:12.534 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:23:12.534 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:23:12.534 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:23:12.534 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:23:12.535 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:23:12.535 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:23:12.535 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:23:12.535 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:23:12.535 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:23:12.535 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:23:12.535 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:23:12.535 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:23:12.535 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:23:12.535 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:23:12.535 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:23:12.535 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:23:12.536 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:23:12.536 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:23:12.536 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:23:12.536 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:12.536 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:12.536 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:23:12.536 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:23:12.536 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:23:12.536 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:23:12.536 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:23:12.536 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:23:12.536 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:23:12.537 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:23:12.537 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:23:12.537 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:23:12.537 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:23:12.537 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:23:12.537 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:23:12.537 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:23:12.537 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:23:12.537 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:23:12.537 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:23:12.537 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:23:12.537 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:23:12.538 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:23:12.538 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:23:12.538 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:23:12.538 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:23:12.538 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:23:12.538 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:23:12.538 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:23:12.538 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:23:12.538 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:23:12.538 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:23:12.538 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:23:12.538 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:12.538 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:23:12.538 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:12.539 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:23:12.539 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:23:12.539 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:23:12.539 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:23:12.539 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:12.539 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:23:12.539 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:23:12.539 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:12.539 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:12.539 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:12.539 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:23:12.539 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:23:12.539 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:12.539 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:12.540 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:12.540 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:12.540 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:23:12.540 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:23:12.540 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:23:12.540 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:23:12.540 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:23:12.540 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:12.540 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:12.540 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:23:12.540 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:23:12.541 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:23:12.541 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:23:12.541 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:12.541 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:23:12.541 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:23:12.541 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:12.541 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:12.541 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:12.541 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:23:12.541 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:23:12.541 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:12.541 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:12.542 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:12.542 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:12.542 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:23:12.542 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:23:12.542 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:23:12.542 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:23:12.542 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:23:12.542 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:12.542 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:12.542 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:23:12.542 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:23:12.543 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:23:12.543 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:23:12.543 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:12.543 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:23:12.543 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:23:12.543 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:12.543 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:12.543 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:12.543 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:23:12.543 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:23:12.543 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:12.543 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:12.543 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:12.544 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:12.544 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:23:12.544 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:23:12.544 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:23:12.544 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:23:12.544 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:23:12.544 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:12.544 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:12.544 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:12.544 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:23:12.544 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:23:12.544 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:12.544 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:23:12.544 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:23:12.545 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:23:12.545 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:12.545 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:23:12.545 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:23:12.545 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:12.545 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:23:12.545 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:23:12.545 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:23:12.545 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:12.545 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:12.545 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:12.545 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:23:12.545 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:23:12.547 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.531+0000 7f6a65e0e640 1 -- 172.21.15.19:0/298474945 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6a4c0779b0 msgr2=0x7f6a4c079ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:12.547 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.531+0000 7f6a65e0e640 1 --2- 172.21.15.19:0/298474945 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6a4c0779b0 0x7f6a4c079ec0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6a58005fd0 tx=0x7f6a58009560 comp rx=0 tx=0).stop 2024-08-31T12:23:12.547 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.531+0000 7f6a65e0e640 1 -- 172.21.15.19:0/298474945 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6a600c85a0 msgr2=0x7f6a6006f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:12.547 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.531+0000 7f6a65e0e640 1 --2- 172.21.15.19:0/298474945 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6a600c85a0 0x7f6a6006f380 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f6a540098f0 tx=0x7f6a54033c20 comp rx=0 tx=0).stop 2024-08-31T12:23:12.547 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.531+0000 7f6a5ffff640 1 -- 172.21.15.19:0/298474945 reap_dead start 2024-08-31T12:23:12.547 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.531+0000 7f6a65e0e640 1 -- 172.21.15.19:0/298474945 shutdown_connections 2024-08-31T12:23:12.547 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.531+0000 7f6a65e0e640 1 --2- 172.21.15.19:0/298474945 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6a600c85a0 0x7f6a6006f380 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:12.547 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.531+0000 7f6a65e0e640 1 -- 172.21.15.19:0/298474945 >> 172.21.15.19:0/298474945 conn(0x7f6a6006cae0 msgr2=0x7f6a60157ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:23:12.547 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.531+0000 7f6a65e0e640 1 -- 172.21.15.19:0/298474945 shutdown_connections 2024-08-31T12:23:12.547 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:12.531+0000 7f6a65e0e640 1 -- 172.21.15.19:0/298474945 wait complete. 2024-08-31T12:23:12.548 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-08-31T12:23:12.548 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-08-31T12:23:26.983 DEBUG:teuthology.orchestra.run.smithi019:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:23:26.985 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:23:32.548 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-08-31T12:23:32.549 INFO:tasks.ceph.mon.c:Restarting daemon 2024-08-31T12:23:32.549 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-08-31T12:23:32.551 INFO:tasks.ceph.mon.c:Started 2024-08-31T12:23:32.551 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-08-31T12:23:32.552 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:23:32.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:32.671+0000 7fa6d5313640 1 -- 172.21.15.19:0/2828061115 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6d00cacd0 msgr2=0x7fa6d00c4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:32.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:32.671+0000 7fa6d5313640 1 --2- 172.21.15.19:0/2828061115 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6d00cacd0 0x7fa6d00c4060 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0009830 tx=0x7fa6c002a530 comp rx=0 tx=0).stop 2024-08-31T12:23:32.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:32.671+0000 7fa6d5313640 1 -- 172.21.15.19:0/2828061115 shutdown_connections 2024-08-31T12:23:32.674 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:32.671+0000 7fa6d5313640 1 --2- 172.21.15.19:0/2828061115 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa6d0153070 0x7fa6d0072710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:32.674 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:32.671+0000 7fa6d5313640 1 --2- 172.21.15.19:0/2828061115 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6d00cacd0 0x7fa6d00c4060 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:32.674 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:32.671+0000 7fa6d5313640 1 -- 172.21.15.19:0/2828061115 >> 172.21.15.19:0/2828061115 conn(0x7fa6d0071f60 msgr2=0x7fa6d006cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:23:32.674 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:32.671+0000 7fa6d5313640 1 -- 172.21.15.19:0/2828061115 shutdown_connections 2024-08-31T12:23:32.674 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:32.671+0000 7fa6d5313640 1 -- 172.21.15.19:0/2828061115 wait complete. 2024-08-31T12:23:32.675 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:32.673+0000 7fa6d5313640 1 Processor -- start 2024-08-31T12:23:32.675 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:32.673+0000 7fa6d5313640 1 -- start start 2024-08-31T12:23:32.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:32.675+0000 7fa6d5313640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa6d0072c50 0x7fa6d00ae930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:32.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:32.675+0000 7fa6d5313640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6d00cacd0 0x7fa6d01f47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:32.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:32.675+0000 7fa6d5313640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa6d0153070 0x7fa6d00693d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:32.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:32.675+0000 7fa6d5313640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d00cb0e0 con 0x7fa6d0153070 2024-08-31T12:23:32.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:32.675+0000 7fa6d5313640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d00caa40 con 0x7fa6d00cacd0 2024-08-31T12:23:32.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:32.675+0000 7fa6d5313640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fa6d0153480 con 0x7fa6d0072c50 2024-08-31T12:23:32.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:32.675+0000 7fa6cf7fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa6d0153070 0x7fa6d00693d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:32.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:32.676+0000 7fa6cf7fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa6d0153070 0x7fa6d00693d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:35736/0 (socket says 172.21.15.19:35736) 2024-08-31T12:23:32.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:32.676+0000 7fa6cf7fe640 1 -- 172.21.15.19:0/1401748660 learned_addr learned my addr 172.21.15.19:0/1401748660 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:23:32.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:32.676+0000 7fa6ce7fc640 1 --2- 172.21.15.19:0/1401748660 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6d00cacd0 0x7fa6d01f47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:32.680 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:32.676+0000 7fa6cf7fe640 1 -- 172.21.15.19:0/1401748660 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa6d0072c50 msgr2=0x7fa6d00ae930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:32.680 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:32.676+0000 7fa6cf7fe640 1 --2- 172.21.15.19:0/1401748660 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa6d0072c50 0x7fa6d00ae930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:32.680 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:32.676+0000 7fa6cf7fe640 1 -- 172.21.15.19:0/1401748660 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6d00cacd0 msgr2=0x7fa6d01f47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:32.680 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:32.676+0000 7fa6cf7fe640 1 --2- 172.21.15.19:0/1401748660 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6d00cacd0 0x7fa6d01f47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:32.680 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:32.676+0000 7fa6cf7fe640 1 -- 172.21.15.19:0/1401748660 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6d006e600 con 0x7fa6d0153070 2024-08-31T12:23:32.680 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:32.676+0000 7fa6cf7fe640 1 --2- 172.21.15.19:0/1401748660 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa6d0153070 0x7fa6d00693d0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa6c800cae0 tx=0x7fa6c800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:32.680 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:32.676+0000 7fa6b77fe640 1 -- 172.21.15.19:0/1401748660 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c8016020 con 0x7fa6d0153070 2024-08-31T12:23:32.680 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:32.676+0000 7fa6b77fe640 1 -- 172.21.15.19:0/1401748660 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa6c800ad30 con 0x7fa6d0153070 2024-08-31T12:23:32.680 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:32.676+0000 7fa6d5313640 1 -- 172.21.15.19:0/1401748660 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6d01f1c80 con 0x7fa6d0153070 2024-08-31T12:23:32.680 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:32.677+0000 7fa6b77fe640 1 -- 172.21.15.19:0/1401748660 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c8012680 con 0x7fa6d0153070 2024-08-31T12:23:32.680 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:32.677+0000 7fa6d5313640 1 -- 172.21.15.19:0/1401748660 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6d006eb40 con 0x7fa6d0153070 2024-08-31T12:23:32.681 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:32.678+0000 7fa6d5313640 1 -- 172.21.15.19:0/1401748660 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6d01f21a0 con 0x7fa6d0153070 2024-08-31T12:23:32.681 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:32.679+0000 7fa6b77fe640 1 -- 172.21.15.19:0/1401748660 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fa6c8019020 con 0x7fa6d0153070 2024-08-31T12:23:32.682 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:32.679+0000 7fa6b77fe640 1 --2- 172.21.15.19:0/1401748660 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa6bc077a50 0x7fa6bc079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:32.682 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:32.679+0000 7fa6b77fe640 1 -- 172.21.15.19:0/1401748660 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fa6c8099ee0 con 0x7fa6d0153070 2024-08-31T12:23:32.682 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:32.680+0000 7fa6ceffd640 1 --2- 172.21.15.19:0/1401748660 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa6bc077a50 0x7fa6bc079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:32.682 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:32.680+0000 7fa6ceffd640 1 --2- 172.21.15.19:0/1401748660 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa6bc077a50 0x7fa6bc079f60 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0034000 tx=0x7fa6c00046c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:32.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:32.685+0000 7fa6b77fe640 1 -- 172.21.15.19:0/1401748660 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fa6d01f21a0 con 0x7fa6d0153070 2024-08-31T12:23:33.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:33.056+0000 7fa6d5313640 1 -- 172.21.15.19:0/1401748660 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa6d0072710 con 0x7fa6d0153070 2024-08-31T12:23:33.059 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:33.056+0000 7fa6b77fe640 1 -- 172.21.15.19:0/1401748660 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7fa6c8020020 con 0x7fa6d0153070 2024-08-31T12:23:33.059 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":54,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:23:33.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:33.059+0000 7fa6d5313640 1 -- 172.21.15.19:0/1401748660 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa6bc077a50 msgr2=0x7fa6bc079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:33.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:33.059+0000 7fa6d5313640 1 --2- 172.21.15.19:0/1401748660 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa6bc077a50 0x7fa6bc079f60 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0034000 tx=0x7fa6c00046c0 comp rx=0 tx=0).stop 2024-08-31T12:23:33.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:33.059+0000 7fa6d5313640 1 -- 172.21.15.19:0/1401748660 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa6d0153070 msgr2=0x7fa6d00693d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:33.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:33.059+0000 7fa6d5313640 1 --2- 172.21.15.19:0/1401748660 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa6d0153070 0x7fa6d00693d0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa6c800cae0 tx=0x7fa6c800a060 comp rx=0 tx=0).stop 2024-08-31T12:23:33.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:33.059+0000 7fa6d5313640 1 -- 172.21.15.19:0/1401748660 shutdown_connections 2024-08-31T12:23:33.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:33.059+0000 7fa6d5313640 1 --2- 172.21.15.19:0/1401748660 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa6bc077a50 0x7fa6bc079f60 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:33.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:33.059+0000 7fa6d5313640 1 --2- 172.21.15.19:0/1401748660 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa6d0153070 0x7fa6d00693d0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:33.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:33.059+0000 7fa6d5313640 1 --2- 172.21.15.19:0/1401748660 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6d00cacd0 0x7fa6d01f47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:33.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:33.059+0000 7fa6d5313640 1 --2- 172.21.15.19:0/1401748660 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa6d0072c50 0x7fa6d00ae930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:33.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:33.059+0000 7fa6d5313640 1 -- 172.21.15.19:0/1401748660 >> 172.21.15.19:0/1401748660 conn(0x7fa6d0071f60 msgr2=0x7fa6d01491b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:23:33.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:33.059+0000 7fa6d5313640 1 -- 172.21.15.19:0/1401748660 shutdown_connections 2024-08-31T12:23:33.064 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:33.059+0000 7fa6d5313640 1 -- 172.21.15.19:0/1401748660 wait complete. 2024-08-31T12:23:36.073 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:23:36.191 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.188+0000 7f186cfa8640 1 -- 172.21.15.19:0/1312982864 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f18680c8000 msgr2=0x7f18680c5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:36.191 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.188+0000 7f186cfa8640 1 --2- 172.21.15.19:0/1312982864 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f18680c8000 0x7f18680c5430 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f18680c77b0 tx=0x7f186002ba10 comp rx=0 tx=0).stop 2024-08-31T12:23:36.191 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.188+0000 7f186cfa8640 1 -- 172.21.15.19:0/1312982864 shutdown_connections 2024-08-31T12:23:36.191 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.188+0000 7f186cfa8640 1 --2- 172.21.15.19:0/1312982864 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f18680c8000 0x7f18680c5430 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:36.191 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.188+0000 7f186cfa8640 1 --2- 172.21.15.19:0/1312982864 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f18680c6390 0x7f18680c7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:36.191 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.188+0000 7f186cfa8640 1 -- 172.21.15.19:0/1312982864 >> 172.21.15.19:0/1312982864 conn(0x7f18680cc770 msgr2=0x7f18680ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:23:36.191 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.188+0000 7f186cfa8640 1 -- 172.21.15.19:0/1312982864 shutdown_connections 2024-08-31T12:23:36.191 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.188+0000 7f186cfa8640 1 -- 172.21.15.19:0/1312982864 wait complete. 2024-08-31T12:23:36.192 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.191+0000 7f186cfa8640 1 Processor -- start 2024-08-31T12:23:36.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.191+0000 7f186cfa8640 1 -- start start 2024-08-31T12:23:36.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.192+0000 7f186cfa8640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f18680c3a60 0x7f18681f1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:36.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.192+0000 7f186cfa8640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f18680c6390 0x7f186814a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:36.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.192+0000 7f186cfa8640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f18680c8000 0x7f1868150140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:36.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.192+0000 7f186cfa8640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f1868070b20 con 0x7f18680c8000 2024-08-31T12:23:36.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.192+0000 7f186cfa8640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1868070840 con 0x7f18680c3a60 2024-08-31T12:23:36.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.192+0000 7f186cfa8640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f1868071400 con 0x7f18680c6390 2024-08-31T12:23:36.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.192+0000 7f1866d9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f18680c8000 0x7f1868150140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:36.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.192+0000 7f1866d9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f18680c8000 0x7f1868150140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:35754/0 (socket says 172.21.15.19:35754) 2024-08-31T12:23:36.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.192+0000 7f1866d9d640 1 -- 172.21.15.19:0/3650808622 learned_addr learned my addr 172.21.15.19:0/3650808622 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:23:36.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.192+0000 7f186659c640 1 --2- 172.21.15.19:0/3650808622 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f18680c3a60 0x7f18681f1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:36.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.192+0000 7f1865d9b640 1 --2- 172.21.15.19:0/3650808622 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f18680c6390 0x7f186814a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:36.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.192+0000 7f1865d9b640 1 -- 172.21.15.19:0/3650808622 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f18680c6390 msgr2=0x7f186814a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-08-31T12:23:36.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.192+0000 7f1865d9b640 1 -- 172.21.15.19:0/3650808622 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f18680c6390 msgr2=0x7f186814a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-31T12:23:36.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.192+0000 7f1865d9b640 1 --2- 172.21.15.19:0/3650808622 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f18680c6390 0x7f186814a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-31T12:23:36.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.192+0000 7f1865d9b640 1 --2- 172.21.15.19:0/3650808622 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f18680c6390 0x7f186814a110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:23:36.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.193+0000 7f1866d9d640 1 -- 172.21.15.19:0/3650808622 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f18680c6390 msgr2=0x7f186814a110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:23:36.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.193+0000 7f1866d9d640 1 --2- 172.21.15.19:0/3650808622 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f18680c6390 0x7f186814a110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:36.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.193+0000 7f1866d9d640 1 -- 172.21.15.19:0/3650808622 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f18680c3a60 msgr2=0x7f18681f1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:36.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.193+0000 7f1866d9d640 1 --2- 172.21.15.19:0/3650808622 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f18680c3a60 0x7f18681f1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:36.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.193+0000 7f1866d9d640 1 -- 172.21.15.19:0/3650808622 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1868153ae0 con 0x7f18680c8000 2024-08-31T12:23:36.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.193+0000 7f1866d9d640 1 --2- 172.21.15.19:0/3650808622 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f18680c8000 0x7f1868150140 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f186000ae70 tx=0x7f1860003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:36.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.193+0000 7f1856ffd640 1 -- 172.21.15.19:0/3650808622 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1860034030 con 0x7f18680c8000 2024-08-31T12:23:36.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.193+0000 7f1856ffd640 1 -- 172.21.15.19:0/3650808622 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1860004540 con 0x7f18680c8000 2024-08-31T12:23:36.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.193+0000 7f186cfa8640 1 -- 172.21.15.19:0/3650808622 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18681f22a0 con 0x7f18680c8000 2024-08-31T12:23:36.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.193+0000 7f1856ffd640 1 -- 172.21.15.19:0/3650808622 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1860039450 con 0x7f18680c8000 2024-08-31T12:23:36.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.193+0000 7f186cfa8640 1 -- 172.21.15.19:0/3650808622 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1868158380 con 0x7f18680c8000 2024-08-31T12:23:36.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.194+0000 7f186cfa8640 1 -- 172.21.15.19:0/3650808622 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18680c7ac0 con 0x7f18680c8000 2024-08-31T12:23:36.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.195+0000 7f1856ffd640 1 -- 172.21.15.19:0/3650808622 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f1860037070 con 0x7f18680c8000 2024-08-31T12:23:36.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.196+0000 7f1856ffd640 1 --2- 172.21.15.19:0/3650808622 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1848077a80 0x7f1848079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:36.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.196+0000 7f1856ffd640 1 -- 172.21.15.19:0/3650808622 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f18600babe0 con 0x7f18680c8000 2024-08-31T12:23:36.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.196+0000 7f186659c640 1 --2- 172.21.15.19:0/3650808622 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1848077a80 0x7f1848079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:36.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.197+0000 7f186659c640 1 --2- 172.21.15.19:0/3650808622 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1848077a80 0x7f1848079f90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1858001720 tx=0x7f1858009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:36.203 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.201+0000 7f1856ffd640 1 -- 172.21.15.19:0/3650808622 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f1860083840 con 0x7f18680c8000 2024-08-31T12:23:36.493 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.491+0000 7f186cfa8640 1 -- 172.21.15.19:0/3650808622 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f186806fa50 con 0x7f18680c8000 2024-08-31T12:23:36.493 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.491+0000 7f1856ffd640 1 -- 172.21.15.19:0/3650808622 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f1860083140 con 0x7f18680c8000 2024-08-31T12:23:36.493 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":56,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:23:36.496 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.494+0000 7f186cfa8640 1 -- 172.21.15.19:0/3650808622 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1848077a80 msgr2=0x7f1848079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:36.496 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.494+0000 7f186cfa8640 1 --2- 172.21.15.19:0/3650808622 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1848077a80 0x7f1848079f90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1858001720 tx=0x7f1858009040 comp rx=0 tx=0).stop 2024-08-31T12:23:36.496 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.494+0000 7f186cfa8640 1 -- 172.21.15.19:0/3650808622 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f18680c8000 msgr2=0x7f1868150140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:36.496 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.494+0000 7f186cfa8640 1 --2- 172.21.15.19:0/3650808622 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f18680c8000 0x7f1868150140 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f186000ae70 tx=0x7f1860003f10 comp rx=0 tx=0).stop 2024-08-31T12:23:36.496 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.494+0000 7f186cfa8640 1 -- 172.21.15.19:0/3650808622 shutdown_connections 2024-08-31T12:23:36.496 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.494+0000 7f186cfa8640 1 --2- 172.21.15.19:0/3650808622 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1848077a80 0x7f1848079f90 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:36.497 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.494+0000 7f186cfa8640 1 --2- 172.21.15.19:0/3650808622 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f18680c8000 0x7f1868150140 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:36.497 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.494+0000 7f186cfa8640 1 --2- 172.21.15.19:0/3650808622 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f18680c6390 0x7f186814a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:36.497 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.494+0000 7f186cfa8640 1 --2- 172.21.15.19:0/3650808622 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f18680c3a60 0x7f18681f1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:36.497 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.494+0000 7f186cfa8640 1 -- 172.21.15.19:0/3650808622 >> 172.21.15.19:0/3650808622 conn(0x7f18680cc770 msgr2=0x7f1868154390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:23:36.497 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.494+0000 7f186cfa8640 1 -- 172.21.15.19:0/3650808622 shutdown_connections 2024-08-31T12:23:36.497 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.494+0000 7f186cfa8640 1 -- 172.21.15.19:0/3650808622 wait complete. 2024-08-31T12:23:36.508 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-08-31T12:23:36.509 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-08-31T12:23:36.509 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-31T12:23:36.638 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.635+0000 7fb598938640 1 -- 172.21.15.19:0/204653440 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb58c036400 con 0x7fb594158010 2024-08-31T12:23:36.638 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.635+0000 7fb59bb9e640 1 -- 172.21.15.19:0/204653440 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb594158010 msgr2=0x7fb594158420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:36.638 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.635+0000 7fb59bb9e640 1 --2- 172.21.15.19:0/204653440 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb594158010 0x7fb594158420 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fb58c005fd0 tx=0x7fb58c02aa90 comp rx=0 tx=0).stop 2024-08-31T12:23:36.638 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.636+0000 7fb59bb9e640 1 -- 172.21.15.19:0/204653440 shutdown_connections 2024-08-31T12:23:36.638 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.636+0000 7fb59bb9e640 1 --2- 172.21.15.19:0/204653440 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb594158010 0x7fb594158420 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:36.638 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.636+0000 7fb59bb9e640 1 --2- 172.21.15.19:0/204653440 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb5940c5440 0x7fb594074860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:36.638 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.636+0000 7fb59bb9e640 1 -- 172.21.15.19:0/204653440 >> 172.21.15.19:0/204653440 conn(0x7fb5940ce0b0 msgr2=0x7fb5940c9f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:23:36.638 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.636+0000 7fb59bb9e640 1 -- 172.21.15.19:0/204653440 shutdown_connections 2024-08-31T12:23:36.638 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.636+0000 7fb59bb9e640 1 -- 172.21.15.19:0/204653440 wait complete. 2024-08-31T12:23:36.639 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.637+0000 7fb59bb9e640 1 Processor -- start 2024-08-31T12:23:36.639 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.637+0000 7fb59bb9e640 1 -- start start 2024-08-31T12:23:36.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.639+0000 7fb59bb9e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb5940c5440 0x7fb5941f5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:36.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.639+0000 7fb59bb9e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb5940c7e60 0x7fb594146140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:36.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.639+0000 7fb59bb9e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb594158010 0x7fb59414cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:36.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.639+0000 7fb59bb9e640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fb594158960 con 0x7fb5940c5440 2024-08-31T12:23:36.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.639+0000 7fb59bb9e640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb5940c76e0 con 0x7fb594158010 2024-08-31T12:23:36.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.639+0000 7fb59bb9e640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fb594074da0 con 0x7fb5940c7e60 2024-08-31T12:23:36.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.639+0000 7fb59993a640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb5940c5440 0x7fb5941f5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:36.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.639+0000 7fb59993a640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb5940c5440 0x7fb5941f5130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:35772/0 (socket says 172.21.15.19:35772) 2024-08-31T12:23:36.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.639+0000 7fb59993a640 1 -- 172.21.15.19:0/4109142713 learned_addr learned my addr 172.21.15.19:0/4109142713 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:23:36.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.639+0000 7fb59a13b640 1 --2- 172.21.15.19:0/4109142713 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb594158010 0x7fb59414cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:36.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.639+0000 7fb599139640 1 --2- 172.21.15.19:0/4109142713 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb5940c7e60 0x7fb594146140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:36.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.639+0000 7fb59993a640 1 -- 172.21.15.19:0/4109142713 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb5940c7e60 msgr2=0x7fb594146140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:36.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.639+0000 7fb59993a640 1 --2- 172.21.15.19:0/4109142713 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb5940c7e60 0x7fb594146140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:36.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.639+0000 7fb59993a640 1 -- 172.21.15.19:0/4109142713 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb594158010 msgr2=0x7fb59414cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:36.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.639+0000 7fb59993a640 1 --2- 172.21.15.19:0/4109142713 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb594158010 0x7fb59414cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:36.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.639+0000 7fb59993a640 1 -- 172.21.15.19:0/4109142713 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5941f5670 con 0x7fb5940c5440 2024-08-31T12:23:36.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.639+0000 7fb599139640 1 --2- 172.21.15.19:0/4109142713 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb5940c7e60 0x7fb594146140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-31T12:23:36.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.639+0000 7fb59a13b640 1 --2- 172.21.15.19:0/4109142713 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb594158010 0x7fb59414cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-31T12:23:36.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.640+0000 7fb59993a640 1 --2- 172.21.15.19:0/4109142713 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb5940c5440 0x7fb5941f5130 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fb59000d7a0 tx=0x7fb59000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:36.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.640+0000 7fb58a7fc640 1 -- 172.21.15.19:0/4109142713 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb590017070 con 0x7fb5940c5440 2024-08-31T12:23:36.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.640+0000 7fb58a7fc640 1 -- 172.21.15.19:0/4109142713 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb590009cc0 con 0x7fb5940c5440 2024-08-31T12:23:36.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.640+0000 7fb59bb9e640 1 -- 172.21.15.19:0/4109142713 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5941f20c0 con 0x7fb5940c5440 2024-08-31T12:23:36.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.640+0000 7fb58a7fc640 1 -- 172.21.15.19:0/4109142713 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5900106c0 con 0x7fb5940c5440 2024-08-31T12:23:36.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.640+0000 7fb59bb9e640 1 -- 172.21.15.19:0/4109142713 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb59406d0a0 con 0x7fb5940c5440 2024-08-31T12:23:36.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.641+0000 7fb59bb9e640 1 --2- 172.21.15.19:0/4109142713 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb59414d520 0x7fb5940b9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:36.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.641+0000 7fb59bb9e640 1 -- 172.21.15.19:0/4109142713 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fb5940cadc0 con 0x7fb59414d520 2024-08-31T12:23:36.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.641+0000 7fb599139640 1 --2- 172.21.15.19:0/4109142713 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb59414d520 0x7fb5940b9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:36.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.641+0000 7fb599139640 1 -- 172.21.15.19:0/4109142713 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb5940c5850 con 0x7fb59414d520 2024-08-31T12:23:36.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.642+0000 7fb599139640 1 --2- 172.21.15.19:0/4109142713 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb59414d520 0x7fb5940b9320 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fb58400d3d0 tx=0x7fb58400d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:36.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.642+0000 7fb58a7fc640 1 -- 172.21.15.19:0/4109142713 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb584016020 con 0x7fb59414d520 2024-08-31T12:23:36.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.642+0000 7fb58a7fc640 1 -- 172.21.15.19:0/4109142713 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fb59001a070 con 0x7fb5940c5440 2024-08-31T12:23:36.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.642+0000 7fb58a7fc640 1 --2- 172.21.15.19:0/4109142713 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb578077950 0x7fb578079e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:36.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.642+0000 7fb58a7fc640 1 -- 172.21.15.19:0/4109142713 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fb5900997b0 con 0x7fb5940c5440 2024-08-31T12:23:36.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.643+0000 7fb59a13b640 1 --2- 172.21.15.19:0/4109142713 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb578077950 0x7fb578079e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:36.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.643+0000 7fb58a7fc640 1 -- 172.21.15.19:0/4109142713 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb584012800 con 0x7fb59414d520 2024-08-31T12:23:36.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.643+0000 7fb58a7fc640 1 -- 172.21.15.19:0/4109142713 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb59414d520 msgr2=0x7fb5940b9320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:36.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.643+0000 7fb58a7fc640 1 --2- 172.21.15.19:0/4109142713 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb59414d520 0x7fb5940b9320 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fb58400d3d0 tx=0x7fb58400d890 comp rx=0 tx=0).stop 2024-08-31T12:23:36.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.643+0000 7fb58a7fc640 1 -- 172.21.15.19:0/4109142713 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb59414d520 msgr2=0x7fb5940b9320 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:23:36.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.643+0000 7fb58a7fc640 1 --2- 172.21.15.19:0/4109142713 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb59414d520 0x7fb5940b9320 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:36.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.643+0000 7fb59a13b640 1 --2- 172.21.15.19:0/4109142713 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb578077950 0x7fb578079e60 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb58c035000 tx=0x7fb58c004ab0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:36.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.649+0000 7fb59bb9e640 1 --2- 172.21.15.19:0/4109142713 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb5940b9860 0x7fb5940b9c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:36.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.649+0000 7fb59bb9e640 1 -- 172.21.15.19:0/4109142713 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fb594074860 con 0x7fb5940b9860 2024-08-31T12:23:36.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.649+0000 7fb599139640 1 --2- 172.21.15.19:0/4109142713 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb5940b9860 0x7fb5940b9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:36.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.650+0000 7fb599139640 1 -- 172.21.15.19:0/4109142713 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb594146680 con 0x7fb5940b9860 2024-08-31T12:23:36.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.650+0000 7fb599139640 1 --2- 172.21.15.19:0/4109142713 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb5940b9860 0x7fb5940b9c70 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fb58400d3d0 tx=0x7fb584000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:36.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.650+0000 7fb58a7fc640 1 -- 172.21.15.19:0/4109142713 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb58401de60 con 0x7fb5940b9860 2024-08-31T12:23:36.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.650+0000 7fb58a7fc640 1 -- 172.21.15.19:0/4109142713 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5044 (secure 0 0 0) 0x7fb58401d020 con 0x7fb5940b9860 2024-08-31T12:23:36.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.650+0000 7fb58a7fc640 1 -- 172.21.15.19:0/4109142713 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb5940b9860 msgr2=0x7fb5940b9c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:36.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.650+0000 7fb58a7fc640 1 --2- 172.21.15.19:0/4109142713 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb5940b9860 0x7fb5940b9c70 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fb58400d3d0 tx=0x7fb584000ed0 comp rx=0 tx=0).stop 2024-08-31T12:23:36.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.650+0000 7fb58a7fc640 1 -- 172.21.15.19:0/4109142713 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb5940b9860 msgr2=0x7fb5940b9c70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:23:36.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.650+0000 7fb58a7fc640 1 --2- 172.21.15.19:0/4109142713 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb5940b9860 0x7fb5940b9c70 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:36.653 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:23:36.653 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:23:36.654 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:23:36.654 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "leader", 2024-08-31T12:23:36.654 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 56, 2024-08-31T12:23:36.654 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 396642, 2024-08-31T12:23:36.654 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:23:36.654 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:23:36.654 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:23:36.654 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:23:36.654 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:23:36.654 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 2, 2024-08-31T12:23:36.654 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:23:36.654 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:23:36.654 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:23:36.654 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:23:36.655 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:23:36.655 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:23:36.655 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:23:36.655 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:23:36.655 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:23:36.655 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:23:36.655 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:23:36.655 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:23:36.655 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:23:36.655 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:23:36.655 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:23:36.655 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:23:36.655 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:23:36.656 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:23:36.656 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:23:36.656 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:23:36.656 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:23:36.656 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:23:36.656 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:23:36.656 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:23:36.656 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:23:36.656 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:23:36.656 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:23:36.656 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:23:36.656 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:36.657 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:36.657 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:23:36.657 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:23:36.657 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:23:36.657 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:23:36.657 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:23:36.657 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:23:36.657 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:23:36.658 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:23:36.658 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:23:36.658 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:23:36.658 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:23:36.658 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:23:36.658 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:23:36.658 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:23:36.658 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:23:36.658 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:23:36.658 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:23:36.658 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:23:36.658 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:23:36.658 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:23:36.659 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:23:36.659 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:23:36.659 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:23:36.659 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:23:36.659 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:23:36.659 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:23:36.659 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:23:36.659 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:23:36.659 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:23:36.659 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:23:36.659 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:36.659 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:23:36.660 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:36.660 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:23:36.660 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:23:36.660 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:23:36.660 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:23:36.660 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:36.660 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:23:36.661 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:23:36.661 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:36.661 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:36.661 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:36.661 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:23:36.661 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:23:36.661 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:36.661 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:36.661 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:36.661 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:36.661 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:23:36.661 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:23:36.661 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:23:36.662 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:23:36.662 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:23:36.662 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:36.662 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:36.662 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:23:36.662 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:23:36.662 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:23:36.662 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:23:36.662 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:36.662 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:23:36.662 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:23:36.662 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:36.662 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:36.663 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:36.663 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:23:36.663 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:23:36.663 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:36.663 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:36.663 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:36.663 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:36.663 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:23:36.663 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:23:36.663 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:23:36.663 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:23:36.663 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:23:36.663 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:36.664 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:36.664 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:23:36.664 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:23:36.664 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:23:36.664 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:23:36.664 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:36.664 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:23:36.664 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:23:36.664 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:36.664 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:36.664 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:36.664 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:23:36.664 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:23:36.665 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:36.665 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:36.665 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:36.665 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:36.665 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:23:36.665 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:23:36.665 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:23:36.665 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:23:36.665 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:23:36.665 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:36.665 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:36.665 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:36.665 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:23:36.666 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:23:36.666 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:36.666 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:23:36.666 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:23:36.666 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:23:36.666 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:36.666 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:23:36.666 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:23:36.666 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:36.666 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:23:36.667 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:23:36.667 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 6 2024-08-31T12:23:36.667 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:36.667 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:23:36.667 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:23:36.667 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:36.667 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:23:36.667 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:23:36.667 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 8 2024-08-31T12:23:36.667 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:36.667 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:23:36.667 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:23:36.667 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:36.668 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:23:36.668 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:23:36.668 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 6 2024-08-31T12:23:36.668 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:36.668 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:23:36.668 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:23:36.668 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:36.668 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:23:36.668 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:23:36.668 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:23:36.668 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:36.668 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:36.668 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:36.669 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:23:36.669 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:23:36.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.651+0000 7fb59bb9e640 1 -- 172.21.15.19:0/4109142713 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb578077950 msgr2=0x7fb578079e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:36.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.651+0000 7fb59bb9e640 1 --2- 172.21.15.19:0/4109142713 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb578077950 0x7fb578079e60 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb58c035000 tx=0x7fb58c004ab0 comp rx=0 tx=0).stop 2024-08-31T12:23:36.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.651+0000 7fb59bb9e640 1 -- 172.21.15.19:0/4109142713 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb5940c5440 msgr2=0x7fb5941f5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:36.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.651+0000 7fb59bb9e640 1 --2- 172.21.15.19:0/4109142713 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb5940c5440 0x7fb5941f5130 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fb59000d7a0 tx=0x7fb59000dc60 comp rx=0 tx=0).stop 2024-08-31T12:23:36.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.651+0000 7fb59a13b640 1 -- 172.21.15.19:0/4109142713 reap_dead start 2024-08-31T12:23:36.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.651+0000 7fb59a13b640 1 -- 172.21.15.19:0/4109142713 reap_dead start 2024-08-31T12:23:36.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.651+0000 7fb59bb9e640 1 -- 172.21.15.19:0/4109142713 shutdown_connections 2024-08-31T12:23:36.671 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.651+0000 7fb59bb9e640 1 -- 172.21.15.19:0/4109142713 >> 172.21.15.19:0/4109142713 conn(0x7fb5940ce0b0 msgr2=0x7fb59414faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:23:36.671 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.651+0000 7fb59bb9e640 1 -- 172.21.15.19:0/4109142713 shutdown_connections 2024-08-31T12:23:36.671 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.651+0000 7fb59bb9e640 1 -- 172.21.15.19:0/4109142713 wait complete. 2024-08-31T12:23:36.671 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:23:36.671 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-08-31T12:23:36.671 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-31T12:23:36.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.793+0000 7fb32538b640 1 -- 172.21.15.19:0/4163869108 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb3200c5c20 msgr2=0x7fb3200cbf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:36.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.793+0000 7fb32538b640 1 --2- 172.21.15.19:0/4163869108 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb3200c5c20 0x7fb3200cbf90 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb3200d1580 tx=0x7fb31402a5d0 comp rx=0 tx=0).stop 2024-08-31T12:23:36.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.793+0000 7fb32538b640 1 -- 172.21.15.19:0/4163869108 shutdown_connections 2024-08-31T12:23:36.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.793+0000 7fb32538b640 1 --2- 172.21.15.19:0/4163869108 >> v2:172.21.15.43:3300/0 conn(0x7fb3200cb2f0 0x7fb3200cc4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:36.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.793+0000 7fb32538b640 1 --2- 172.21.15.19:0/4163869108 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb3200c5c20 0x7fb3200cbf90 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:36.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.793+0000 7fb32538b640 1 -- 172.21.15.19:0/4163869108 >> 172.21.15.19:0/4163869108 conn(0x7fb320074a90 msgr2=0x7fb320068aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:23:36.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.793+0000 7fb32538b640 1 -- 172.21.15.19:0/4163869108 shutdown_connections 2024-08-31T12:23:36.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.793+0000 7fb32538b640 1 -- 172.21.15.19:0/4163869108 wait complete. 2024-08-31T12:23:36.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.796+0000 7fb32538b640 1 Processor -- start 2024-08-31T12:23:36.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.796+0000 7fb32538b640 1 -- start start 2024-08-31T12:23:36.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.797+0000 7fb32538b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb3200c5c20 0x7fb320145f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:36.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.797+0000 7fb32538b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb3200cb2f0 0x7fb3201464d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:36.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.797+0000 7fb32538b640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb320155520 0x7fb3200ce760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:36.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.798+0000 7fb32538b640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fb3200d1140 con 0x7fb3200c5c20 2024-08-31T12:23:36.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.798+0000 7fb32538b640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb3200c6750 con 0x7fb320155520 2024-08-31T12:23:36.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.798+0000 7fb32538b640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fb32015b730 con 0x7fb3200cb2f0 2024-08-31T12:23:36.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.798+0000 7fb31effd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb3200c5c20 0x7fb320145f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:36.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.798+0000 7fb31effd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb3200c5c20 0x7fb320145f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:35816/0 (socket says 172.21.15.19:35816) 2024-08-31T12:23:36.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.798+0000 7fb31effd640 1 -- 172.21.15.19:0/1849030112 learned_addr learned my addr 172.21.15.19:0/1849030112 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:23:36.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.798+0000 7fb31f7fe640 1 --2- 172.21.15.19:0/1849030112 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb320155520 0x7fb3200ce760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:36.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.798+0000 7fb31e7fc640 1 --2- 172.21.15.19:0/1849030112 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb3200cb2f0 0x7fb3201464d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:36.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.798+0000 7fb31e7fc640 1 -- 172.21.15.19:0/1849030112 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb3200cb2f0 msgr2=0x7fb3201464d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-08-31T12:23:36.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.798+0000 7fb31e7fc640 1 -- 172.21.15.19:0/1849030112 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb3200cb2f0 msgr2=0x7fb3201464d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-31T12:23:36.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.798+0000 7fb31e7fc640 1 --2- 172.21.15.19:0/1849030112 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb3200cb2f0 0x7fb3201464d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-31T12:23:36.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.798+0000 7fb31e7fc640 1 --2- 172.21.15.19:0/1849030112 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb3200cb2f0 0x7fb3201464d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:23:36.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.798+0000 7fb31effd640 1 -- 172.21.15.19:0/1849030112 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb3200cb2f0 msgr2=0x7fb3201464d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:23:36.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.798+0000 7fb31effd640 1 --2- 172.21.15.19:0/1849030112 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb3200cb2f0 0x7fb3201464d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:36.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.798+0000 7fb31effd640 1 -- 172.21.15.19:0/1849030112 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb320155520 msgr2=0x7fb3200ce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:36.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.798+0000 7fb31effd640 1 --2- 172.21.15.19:0/1849030112 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb320155520 0x7fb3200ce760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:36.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.798+0000 7fb31effd640 1 -- 172.21.15.19:0/1849030112 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb32015b4d0 con 0x7fb3200c5c20 2024-08-31T12:23:36.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.798+0000 7fb31effd640 1 --2- 172.21.15.19:0/1849030112 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb3200c5c20 0x7fb320145f90 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb31000c460 tx=0x7fb31000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:36.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.799+0000 7fb3077fe640 1 -- 172.21.15.19:0/1849030112 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb310016020 con 0x7fb3200c5c20 2024-08-31T12:23:36.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.799+0000 7fb3077fe640 1 -- 172.21.15.19:0/1849030112 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb310012940 con 0x7fb3200c5c20 2024-08-31T12:23:36.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.799+0000 7fb32538b640 1 -- 172.21.15.19:0/1849030112 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb32015a430 con 0x7fb3200c5c20 2024-08-31T12:23:36.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.799+0000 7fb3077fe640 1 -- 172.21.15.19:0/1849030112 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb310012d90 con 0x7fb3200c5c20 2024-08-31T12:23:36.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.799+0000 7fb32538b640 1 -- 172.21.15.19:0/1849030112 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb32015a9f0 con 0x7fb3200c5c20 2024-08-31T12:23:36.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.800+0000 7fb32538b640 1 --2- 172.21.15.19:0/1849030112 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb3200d00f0 0x7fb3200ceca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:36.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.800+0000 7fb32538b640 1 -- 172.21.15.19:0/1849030112 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb32015a1e0 con 0x7fb3200d00f0 2024-08-31T12:23:36.805 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.800+0000 7fb31e7fc640 1 --2- 172.21.15.19:0/1849030112 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb3200d00f0 0x7fb3200ceca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:36.805 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.801+0000 7fb31e7fc640 1 -- 172.21.15.19:0/1849030112 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb3200cf1e0 con 0x7fb3200d00f0 2024-08-31T12:23:36.805 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.801+0000 7fb31e7fc640 1 --2- 172.21.15.19:0/1849030112 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb3200d00f0 0x7fb3200ceca0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb32008e3d0 tx=0x7fb314009540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:36.805 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.801+0000 7fb3077fe640 1 -- 172.21.15.19:0/1849030112 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fb310019020 con 0x7fb3200c5c20 2024-08-31T12:23:36.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.801+0000 7fb3077fe640 1 --2- 172.21.15.19:0/1849030112 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb30c077a80 0x7fb30c079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:36.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.802+0000 7fb3077fe640 1 -- 172.21.15.19:0/1849030112 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fb3100994e0 con 0x7fb3200c5c20 2024-08-31T12:23:36.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.802+0000 7fb31f7fe640 1 --2- 172.21.15.19:0/1849030112 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb30c077a80 0x7fb30c079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:36.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.802+0000 7fb3077fe640 1 -- 172.21.15.19:0/1849030112 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb314003bc0 con 0x7fb3200d00f0 2024-08-31T12:23:36.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.802+0000 7fb3077fe640 1 -- 172.21.15.19:0/1849030112 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb314040070 con 0x7fb3200d00f0 2024-08-31T12:23:36.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.802+0000 7fb3077fe640 1 -- 172.21.15.19:0/1849030112 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb3200d00f0 msgr2=0x7fb3200ceca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:36.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.802+0000 7fb3077fe640 1 --2- 172.21.15.19:0/1849030112 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb3200d00f0 0x7fb3200ceca0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb32008e3d0 tx=0x7fb314009540 comp rx=0 tx=0).stop 2024-08-31T12:23:36.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.802+0000 7fb3077fe640 1 -- 172.21.15.19:0/1849030112 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb3200d00f0 msgr2=0x7fb3200ceca0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:23:36.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.802+0000 7fb3077fe640 1 --2- 172.21.15.19:0/1849030112 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb3200d00f0 0x7fb3200ceca0 secure :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb32008e3d0 tx=0x7fb314009540 comp rx=0 tx=0).stop 2024-08-31T12:23:36.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.802+0000 7fb31f7fe640 1 --2- 172.21.15.19:0/1849030112 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb30c077a80 0x7fb30c079f90 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb3200b4420 tx=0x7fb318009500 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:36.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.811+0000 7fb32538b640 1 --2- 172.21.15.19:0/1849030112 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb3200d0a20 0x7fb3201f27b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:36.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.811+0000 7fb32538b640 1 -- 172.21.15.19:0/1849030112 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb3200cb800 con 0x7fb3200d0a20 2024-08-31T12:23:36.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.812+0000 7fb31e7fc640 1 --2- 172.21.15.19:0/1849030112 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb3200d0a20 0x7fb3201f27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:36.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.812+0000 7fb31e7fc640 1 -- 172.21.15.19:0/1849030112 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb3200b3bb0 con 0x7fb3200d0a20 2024-08-31T12:23:36.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.812+0000 7fb31e7fc640 1 --2- 172.21.15.19:0/1849030112 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb3200d0a20 0x7fb3201f27b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb320051420 tx=0x7fb314005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:36.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.812+0000 7fb3077fe640 1 -- 172.21.15.19:0/1849030112 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb314003e70 con 0x7fb3200d0a20 2024-08-31T12:23:36.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.813+0000 7fb3077fe640 1 -- 172.21.15.19:0/1849030112 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4541 (secure 0 0 0) 0x7fb31404e070 con 0x7fb3200d0a20 2024-08-31T12:23:36.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.813+0000 7fb3077fe640 1 -- 172.21.15.19:0/1849030112 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb3200d0a20 msgr2=0x7fb3201f27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:36.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.813+0000 7fb3077fe640 1 --2- 172.21.15.19:0/1849030112 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb3200d0a20 0x7fb3201f27b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb320051420 tx=0x7fb314005f90 comp rx=0 tx=0).stop 2024-08-31T12:23:36.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.813+0000 7fb3077fe640 1 -- 172.21.15.19:0/1849030112 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb3200d0a20 msgr2=0x7fb3201f27b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:23:36.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.813+0000 7fb3077fe640 1 --2- 172.21.15.19:0/1849030112 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb3200d0a20 0x7fb3201f27b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:36.816 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:23:36.816 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:23:36.816 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:23:36.816 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:23:36.816 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 56, 2024-08-31T12:23:36.816 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 126287, 2024-08-31T12:23:36.816 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:23:36.816 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:23:36.816 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:23:36.816 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:23:36.816 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:23:36.816 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 2, 2024-08-31T12:23:36.817 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:23:36.817 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:23:36.817 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:23:36.817 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:23:36.817 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:23:36.817 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:23:36.817 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:23:36.817 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:23:36.817 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:23:36.817 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:23:36.817 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:23:36.817 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:23:36.817 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:23:36.818 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:23:36.818 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:23:36.818 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:23:36.818 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:23:36.818 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:23:36.818 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:23:36.818 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:23:36.818 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:23:36.818 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:23:36.818 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:23:36.818 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:23:36.818 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:23:36.818 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:23:36.819 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:23:36.819 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:23:36.819 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:36.819 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:36.819 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:23:36.819 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:23:36.819 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:23:36.819 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:23:36.819 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:23:36.819 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:23:36.819 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:23:36.820 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:23:36.820 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:23:36.820 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:23:36.820 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:23:36.820 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:23:36.820 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:23:36.820 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:23:36.820 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:23:36.820 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:23:36.820 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:23:36.820 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:23:36.820 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:23:36.820 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:23:36.821 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:23:36.821 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:23:36.821 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:23:36.821 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:23:36.821 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:23:36.821 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:23:36.821 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:23:36.821 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:23:36.821 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:23:36.821 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:23:36.821 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:36.821 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:23:36.821 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:36.822 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:23:36.822 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:23:36.822 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:23:36.822 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:23:36.822 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:36.822 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:23:36.822 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:23:36.822 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:36.822 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:36.822 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:36.822 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:23:36.823 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:23:36.823 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:36.823 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:36.823 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:36.823 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:36.823 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:23:36.823 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:23:36.823 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:23:36.823 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:23:36.823 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:23:36.823 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:36.823 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:36.823 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:23:36.824 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:23:36.824 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:23:36.824 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:23:36.824 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:36.824 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:23:36.824 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:23:36.824 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:36.824 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:36.824 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:36.824 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:23:36.824 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:23:36.824 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:36.824 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:36.825 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:36.825 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:36.825 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:23:36.825 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:23:36.825 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:23:36.825 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:23:36.825 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:23:36.825 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:36.825 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:36.825 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:23:36.826 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:23:36.826 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:23:36.826 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:23:36.826 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:36.826 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:23:36.826 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:23:36.826 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:36.826 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:36.826 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:36.826 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:23:36.826 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:23:36.826 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:36.826 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:36.826 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:36.827 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:36.827 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:23:36.827 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:23:36.827 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:23:36.827 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:23:36.827 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:23:36.827 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:36.827 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:36.827 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:36.827 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:23:36.827 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:23:36.827 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:36.827 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:23:36.827 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:23:36.828 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:23:36.828 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:36.828 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:23:36.828 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:23:36.828 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:36.828 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:23:36.828 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:23:36.828 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:23:36.828 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:36.828 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:36.828 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:36.828 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:23:36.828 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:23:36.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.813+0000 7fb32538b640 1 -- 172.21.15.19:0/1849030112 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb30c077a80 msgr2=0x7fb30c079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:36.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.813+0000 7fb32538b640 1 --2- 172.21.15.19:0/1849030112 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb30c077a80 0x7fb30c079f90 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb3200b4420 tx=0x7fb318009500 comp rx=0 tx=0).stop 2024-08-31T12:23:36.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.813+0000 7fb32538b640 1 -- 172.21.15.19:0/1849030112 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb3200c5c20 msgr2=0x7fb320145f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:36.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.813+0000 7fb32538b640 1 --2- 172.21.15.19:0/1849030112 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb3200c5c20 0x7fb320145f90 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb31000c460 tx=0x7fb31000c920 comp rx=0 tx=0).stop 2024-08-31T12:23:36.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.813+0000 7fb31f7fe640 1 -- 172.21.15.19:0/1849030112 reap_dead start 2024-08-31T12:23:36.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.813+0000 7fb31f7fe640 1 -- 172.21.15.19:0/1849030112 reap_dead start 2024-08-31T12:23:36.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.813+0000 7fb32538b640 1 -- 172.21.15.19:0/1849030112 shutdown_connections 2024-08-31T12:23:36.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.813+0000 7fb32538b640 1 -- 172.21.15.19:0/1849030112 >> 172.21.15.19:0/1849030112 conn(0x7fb320074a90 msgr2=0x7fb3200b4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:23:36.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.813+0000 7fb32538b640 1 -- 172.21.15.19:0/1849030112 shutdown_connections 2024-08-31T12:23:36.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.813+0000 7fb32538b640 1 -- 172.21.15.19:0/1849030112 wait complete. 2024-08-31T12:23:36.831 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:23:36.831 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-08-31T12:23:36.831 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-31T12:23:36.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.954+0000 7f57e26bd640 1 -- 172.21.15.19:0/1357730973 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f57dc1587f0 msgr2=0x7f57dc0c1ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:36.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.954+0000 7f57e26bd640 1 --2- 172.21.15.19:0/1357730973 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f57dc1587f0 0x7f57dc0c1ec0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f57d0009830 tx=0x7f57d002a510 comp rx=0 tx=0).stop 2024-08-31T12:23:36.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.954+0000 7f57e26bd640 1 -- 172.21.15.19:0/1357730973 shutdown_connections 2024-08-31T12:23:36.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.954+0000 7f57e26bd640 1 --2- 172.21.15.19:0/1357730973 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f57dc1587f0 0x7f57dc0c1ec0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:36.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.954+0000 7f57e26bd640 1 --2- 172.21.15.19:0/1357730973 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f57dc0c7ac0 0x7f57dc14e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:36.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.954+0000 7f57e26bd640 1 -- 172.21.15.19:0/1357730973 >> 172.21.15.19:0/1357730973 conn(0x7f57dc06cae0 msgr2=0x7f57dc06cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:23:36.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.954+0000 7f57e26bd640 1 -- 172.21.15.19:0/1357730973 shutdown_connections 2024-08-31T12:23:36.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.954+0000 7f57e26bd640 1 -- 172.21.15.19:0/1357730973 wait complete. 2024-08-31T12:23:36.959 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.957+0000 7f57e26bd640 1 Processor -- start 2024-08-31T12:23:36.959 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.957+0000 7f57e26bd640 1 -- start start 2024-08-31T12:23:36.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.959+0000 7f57e26bd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f57dc0c2400 0x7f57dc1f2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:36.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.959+0000 7f57e26bd640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f57dc0c7ac0 0x7f57dc154830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:36.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.959+0000 7f57e26bd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f57dc1587f0 0x7f57dc154d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:36.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.959+0000 7f57e26bd640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f57dc0c3e90 con 0x7f57dc1587f0 2024-08-31T12:23:36.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.959+0000 7f57e26bd640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f57dc152380 con 0x7f57dc0c7ac0 2024-08-31T12:23:36.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.959+0000 7f57e26bd640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f57dc0c4010 con 0x7f57dc0c2400 2024-08-31T12:23:36.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.959+0000 7f57e0c5a640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f57dc1587f0 0x7f57dc154d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:36.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.959+0000 7f57e0c5a640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f57dc1587f0 0x7f57dc154d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:35840/0 (socket says 172.21.15.19:35840) 2024-08-31T12:23:36.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.959+0000 7f57e0c5a640 1 -- 172.21.15.19:0/1662601233 learned_addr learned my addr 172.21.15.19:0/1662601233 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:23:36.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.959+0000 7f57dbfff640 1 --2- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f57dc0c2400 0x7f57dc1f2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:36.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.959+0000 7f57db7fe640 1 --2- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f57dc0c7ac0 0x7f57dc154830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:36.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.959+0000 7f57dbfff640 1 -- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f57dc0c2400 msgr2=0x7f57dc1f2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-08-31T12:23:36.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.959+0000 7f57dbfff640 1 -- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f57dc0c2400 msgr2=0x7f57dc1f2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-31T12:23:36.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.959+0000 7f57dbfff640 1 --2- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f57dc0c2400 0x7f57dc1f2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-31T12:23:36.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.959+0000 7f57dbfff640 1 --2- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f57dc0c2400 0x7f57dc1f2120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:23:36.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.959+0000 7f57e0c5a640 1 -- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f57dc0c2400 msgr2=0x7f57dc1f2120 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:23:36.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.959+0000 7f57e0c5a640 1 --2- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f57dc0c2400 0x7f57dc1f2120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:36.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.959+0000 7f57e0c5a640 1 -- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f57dc0c7ac0 msgr2=0x7f57dc154830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:36.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.959+0000 7f57e0c5a640 1 --2- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f57dc0c7ac0 0x7f57dc154830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:36.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.959+0000 7f57e0c5a640 1 -- 172.21.15.19:0/1662601233 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57dc0c70f0 con 0x7f57dc1587f0 2024-08-31T12:23:36.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.960+0000 7f57e0c5a640 1 --2- 172.21.15.19:0/1662601233 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f57dc1587f0 0x7f57dc154d70 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f57d400cb10 tx=0x7f57d400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:36.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.960+0000 7f57d8ff9640 1 -- 172.21.15.19:0/1662601233 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57d4016020 con 0x7f57dc1587f0 2024-08-31T12:23:36.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.960+0000 7f57d8ff9640 1 -- 172.21.15.19:0/1662601233 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f57d400ad60 con 0x7f57dc1587f0 2024-08-31T12:23:36.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.960+0000 7f57d8ff9640 1 -- 172.21.15.19:0/1662601233 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57d4012680 con 0x7f57dc1587f0 2024-08-31T12:23:36.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.960+0000 7f57e26bd640 1 -- 172.21.15.19:0/1662601233 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57dc1f4cd0 con 0x7f57dc1587f0 2024-08-31T12:23:36.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.960+0000 7f57e26bd640 1 -- 172.21.15.19:0/1662601233 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57dc1f5290 con 0x7f57dc1587f0 2024-08-31T12:23:36.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.961+0000 7f57e26bd640 1 --2- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f57dc0c41c0 0x7f57dc0c4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:36.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.961+0000 7f57e26bd640 1 -- 172.21.15.19:0/1662601233 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f57dc0c6ba0 con 0x7f57dc0c41c0 2024-08-31T12:23:36.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.962+0000 7f57dbfff640 1 --2- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f57dc0c41c0 0x7f57dc0c4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:36.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.962+0000 7f57dbfff640 1 -- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f57dc0c41c0 msgr2=0x7f57dc0c4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 20 2024-08-31T12:23:36.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.962+0000 7f57dbfff640 1 -- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f57dc0c41c0 msgr2=0x7f57dc0c4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-31T12:23:36.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.962+0000 7f57dbfff640 1 --2- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f57dc0c41c0 0x7f57dc0c4cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-31T12:23:36.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.962+0000 7f57dbfff640 1 --2- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f57dc0c41c0 0x7f57dc0c4cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:23:36.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.962+0000 7f57d8ff9640 1 -- 172.21.15.19:0/1662601233 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f57d4019020 con 0x7f57dc1587f0 2024-08-31T12:23:36.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.963+0000 7f57d8ff9640 1 --2- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f57c8077a50 0x7f57c8079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:36.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.963+0000 7f57d8ff9640 1 -- 172.21.15.19:0/1662601233 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f57d40999b0 con 0x7f57dc1587f0 2024-08-31T12:23:36.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.963+0000 7f57db7fe640 1 --2- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f57c8077a50 0x7f57c8079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:36.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:36.964+0000 7f57db7fe640 1 --2- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f57c8077a50 0x7f57c8079f60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f57d0034000 tx=0x7f57d0004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:37.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:37.164+0000 7f57dbfff640 1 --2- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f57dc0c41c0 0x7f57dc0c4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:37.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:37.164+0000 7f57dbfff640 1 -- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f57dc0c41c0 msgr2=0x7f57dc0c4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 21 2024-08-31T12:23:37.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:37.164+0000 7f57dbfff640 1 -- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f57dc0c41c0 msgr2=0x7f57dc0c4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-31T12:23:37.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:37.164+0000 7f57dbfff640 1 --2- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f57dc0c41c0 0x7f57dc0c4cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-31T12:23:37.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:37.164+0000 7f57dbfff640 1 --2- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f57dc0c41c0 0x7f57dc0c4cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-31T12:23:37.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:37.565+0000 7f57dbfff640 1 --2- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f57dc0c41c0 0x7f57dc0c4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:37.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:37.566+0000 7f57dbfff640 1 -- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f57dc0c41c0 msgr2=0x7f57dc0c4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 21 2024-08-31T12:23:37.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:37.566+0000 7f57dbfff640 1 -- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f57dc0c41c0 msgr2=0x7f57dc0c4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-31T12:23:37.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:37.566+0000 7f57dbfff640 1 --2- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f57dc0c41c0 0x7f57dc0c4cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-31T12:23:37.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:37.566+0000 7f57dbfff640 1 --2- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f57dc0c41c0 0x7f57dc0c4cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-31T12:23:38.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.367+0000 7f57dbfff640 1 --2- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f57dc0c41c0 0x7f57dc0c4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:38.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.367+0000 7f57dbfff640 1 -- 172.21.15.19:0/1662601233 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f57dc0c6e60 con 0x7f57dc0c41c0 2024-08-31T12:23:38.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.368+0000 7f57dbfff640 1 --2- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f57dc0c41c0 0x7f57dc0c4cf0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f57cc005df0 tx=0x7f57cc011f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:38.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.368+0000 7f57d8ff9640 1 -- 172.21.15.19:0/1662601233 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57cc013260 con 0x7f57dc0c41c0 2024-08-31T12:23:38.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.368+0000 7f57d8ff9640 1 -- 172.21.15.19:0/1662601233 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f57cc013260 con 0x7f57dc0c41c0 2024-08-31T12:23:38.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.368+0000 7f57d8ff9640 1 -- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f57dc0c41c0 msgr2=0x7f57dc0c4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:38.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.368+0000 7f57d8ff9640 1 --2- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f57dc0c41c0 0x7f57dc0c4cf0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f57cc005df0 tx=0x7f57cc011f40 comp rx=0 tx=0).stop 2024-08-31T12:23:38.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.368+0000 7f57d8ff9640 1 -- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f57dc0c41c0 msgr2=0x7f57dc0c4cf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:23:38.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.368+0000 7f57d8ff9640 1 --2- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f57dc0c41c0 0x7f57dc0c4cf0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:38.378 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.376+0000 7f57e26bd640 1 --2- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f57dc0c45d0 0x7f57dc0c5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:38.378 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.376+0000 7f57e26bd640 1 -- 172.21.15.19:0/1662601233 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f57dc155790 con 0x7f57dc0c45d0 2024-08-31T12:23:38.378 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.376+0000 7f57dbfff640 1 --2- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f57dc0c45d0 0x7f57dc0c5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:38.379 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.377+0000 7f57dbfff640 1 -- 172.21.15.19:0/1662601233 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f57dc0c78a0 con 0x7f57dc0c45d0 2024-08-31T12:23:38.379 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.377+0000 7f57dbfff640 1 --2- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f57dc0c45d0 0x7f57dc0c5230 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f57cc005df0 tx=0x7f57cc004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:38.379 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.377+0000 7f57d8ff9640 1 -- 172.21.15.19:0/1662601233 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57cc012660 con 0x7f57dc0c45d0 2024-08-31T12:23:38.380 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.377+0000 7f57d8ff9640 1 -- 172.21.15.19:0/1662601233 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4539 (secure 0 0 0) 0x7f57cc012660 con 0x7f57dc0c45d0 2024-08-31T12:23:38.380 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.377+0000 7f57d8ff9640 1 -- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f57dc0c45d0 msgr2=0x7f57dc0c5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:38.380 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.377+0000 7f57d8ff9640 1 --2- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f57dc0c45d0 0x7f57dc0c5230 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f57cc005df0 tx=0x7f57cc004730 comp rx=0 tx=0).stop 2024-08-31T12:23:38.380 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.377+0000 7f57d8ff9640 1 -- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f57dc0c45d0 msgr2=0x7f57dc0c5230 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:23:38.380 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.377+0000 7f57d8ff9640 1 --2- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f57dc0c45d0 0x7f57dc0c5230 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:38.381 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:23:38.381 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:23:38.381 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:23:38.381 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:23:38.381 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 56, 2024-08-31T12:23:38.381 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 5735, 2024-08-31T12:23:38.381 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:23:38.381 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:23:38.381 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:23:38.381 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:23:38.381 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:23:38.382 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 4, 2024-08-31T12:23:38.382 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:23:38.382 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:23:38.382 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:23:38.382 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:23:38.382 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:23:38.382 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:23:38.382 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:23:38.382 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:23:38.382 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:23:38.382 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:23:38.382 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:23:38.382 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:23:38.383 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:23:38.383 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:23:38.383 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:23:38.383 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:23:38.383 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:23:38.383 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:23:38.383 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:23:38.383 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:23:38.383 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:23:38.383 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:23:38.383 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:23:38.383 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:23:38.383 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:23:38.384 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:23:38.384 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:23:38.384 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:23:38.384 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:38.384 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:38.384 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:23:38.384 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:23:38.384 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:23:38.384 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:23:38.384 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:23:38.384 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:23:38.384 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:23:38.384 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:23:38.385 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:23:38.385 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:23:38.385 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:23:38.385 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:23:38.385 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:23:38.385 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:23:38.385 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:23:38.385 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:23:38.385 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:23:38.385 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:23:38.385 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:23:38.385 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:23:38.385 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:23:38.386 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:23:38.386 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:23:38.386 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:23:38.386 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:23:38.386 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:23:38.386 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:23:38.386 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:23:38.386 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:23:38.386 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:23:38.386 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:38.386 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:23:38.386 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:38.386 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:23:38.387 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:23:38.387 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:23:38.387 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:23:38.387 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:38.387 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:23:38.387 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:23:38.387 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:38.387 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:38.387 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:38.387 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:23:38.387 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:23:38.387 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:38.387 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:38.388 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:38.388 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:38.388 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:23:38.388 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:23:38.388 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:23:38.388 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:23:38.388 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:23:38.388 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:38.388 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:38.388 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:23:38.388 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:23:38.388 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:23:38.388 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:23:38.389 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:38.389 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:23:38.389 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:23:38.389 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:38.389 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:38.389 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:38.389 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:23:38.389 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:23:38.389 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:38.389 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:38.389 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:38.389 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:38.389 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:23:38.390 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:23:38.390 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:23:38.390 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:23:38.390 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:23:38.390 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:38.390 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:38.390 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:23:38.390 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:23:38.390 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:23:38.390 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:23:38.390 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:38.390 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:23:38.390 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:23:38.391 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:38.391 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:38.391 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:38.391 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:23:38.391 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:23:38.391 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:38.391 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:38.391 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:38.391 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:38.391 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:23:38.391 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:23:38.391 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:23:38.391 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:23:38.391 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:23:38.392 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:38.392 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:38.392 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:38.392 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:23:38.392 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:23:38.392 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:38.392 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:23:38.392 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:23:38.392 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:23:38.392 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:38.392 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:23:38.392 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:23:38.393 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:38.393 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:23:38.393 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:23:38.393 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:23:38.393 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:38.393 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:38.393 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:38.393 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:23:38.393 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:23:38.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.378+0000 7f57e26bd640 1 -- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f57c8077a50 msgr2=0x7f57c8079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:38.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.378+0000 7f57e26bd640 1 --2- 172.21.15.19:0/1662601233 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f57c8077a50 0x7f57c8079f60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f57d0034000 tx=0x7f57d0004730 comp rx=0 tx=0).stop 2024-08-31T12:23:38.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.378+0000 7f57e26bd640 1 -- 172.21.15.19:0/1662601233 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f57dc1587f0 msgr2=0x7f57dc154d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:38.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.378+0000 7f57e26bd640 1 --2- 172.21.15.19:0/1662601233 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f57dc1587f0 0x7f57dc154d70 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f57d400cb10 tx=0x7f57d400a060 comp rx=0 tx=0).stop 2024-08-31T12:23:38.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.378+0000 7f57e0c5a640 1 -- 172.21.15.19:0/1662601233 reap_dead start 2024-08-31T12:23:38.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.378+0000 7f57e0c5a640 1 -- 172.21.15.19:0/1662601233 reap_dead start 2024-08-31T12:23:38.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.378+0000 7f57e26bd640 1 -- 172.21.15.19:0/1662601233 shutdown_connections 2024-08-31T12:23:38.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.378+0000 7f57e26bd640 1 -- 172.21.15.19:0/1662601233 >> 172.21.15.19:0/1662601233 conn(0x7f57dc06cae0 msgr2=0x7f57dc141830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:23:38.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.378+0000 7f57e26bd640 1 -- 172.21.15.19:0/1662601233 shutdown_connections 2024-08-31T12:23:38.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.378+0000 7f57e26bd640 1 -- 172.21.15.19:0/1662601233 wait complete. 2024-08-31T12:23:38.395 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:23:38.396 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-08-31T12:23:38.396 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-08-31T12:23:38.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.518+0000 7efcffa94640 1 -- 172.21.15.19:0/974071935 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efce4004800 msgr2=0x7efce4004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:38.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.518+0000 7efcffa94640 1 --2- 172.21.15.19:0/974071935 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efce4004800 0x7efce4004c10 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7efce8006fc0 tx=0x7efce4005830 comp rx=0 tx=0).stop 2024-08-31T12:23:38.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.518+0000 7efcffa94640 1 -- 172.21.15.19:0/974071935 shutdown_connections 2024-08-31T12:23:38.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.518+0000 7efcffa94640 1 --2- 172.21.15.19:0/974071935 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efce4004800 0x7efce4004c10 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:38.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.518+0000 7efcffa94640 1 -- 172.21.15.19:0/974071935 >> 172.21.15.19:0/974071935 conn(0x7efcf80690d0 msgr2=0x7efcf80d0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:23:38.521 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.518+0000 7efcffa94640 1 -- 172.21.15.19:0/974071935 shutdown_connections 2024-08-31T12:23:38.521 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.518+0000 7efcffa94640 1 -- 172.21.15.19:0/974071935 wait complete. 2024-08-31T12:23:38.522 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.520+0000 7efcffa94640 1 Processor -- start 2024-08-31T12:23:38.522 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.520+0000 7efcffa94640 1 -- start start 2024-08-31T12:23:38.525 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.522+0000 7efcffa94640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efce4004800 0x7efcf80c2b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:38.525 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.522+0000 7efcffa94640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efcf815c970 0x7efcf815b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:38.525 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.522+0000 7efcffa94640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efcf8150a80 0x7efcf8150e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:38.525 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.522+0000 7efcffa94640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7efcf815b250 con 0x7efce4004800 2024-08-31T12:23:38.525 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.522+0000 7efcffa94640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7efcf815af70 con 0x7efcf8150a80 2024-08-31T12:23:38.525 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.522+0000 7efcffa94640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7efcf80c8610 con 0x7efcf815c970 2024-08-31T12:23:38.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.522+0000 7efcfd830640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efce4004800 0x7efcf80c2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:38.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.522+0000 7efcfd830640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efce4004800 0x7efcf80c2b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:40714/0 (socket says 172.21.15.19:40714) 2024-08-31T12:23:38.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.522+0000 7efcfd830640 1 -- 172.21.15.19:0/2142280754 learned_addr learned my addr 172.21.15.19:0/2142280754 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:23:38.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.522+0000 7efcfe031640 1 --2- 172.21.15.19:0/2142280754 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efcf8150a80 0x7efcf8150e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:38.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.522+0000 7efcfd02f640 1 --2- 172.21.15.19:0/2142280754 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efcf815c970 0x7efcf815b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:38.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.522+0000 7efcfd830640 1 -- 172.21.15.19:0/2142280754 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efcf815c970 msgr2=0x7efcf815b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:38.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.522+0000 7efcfd830640 1 --2- 172.21.15.19:0/2142280754 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efcf815c970 0x7efcf815b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:38.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.522+0000 7efcfd830640 1 -- 172.21.15.19:0/2142280754 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efcf8150a80 msgr2=0x7efcf8150e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:38.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.522+0000 7efcfd830640 1 --2- 172.21.15.19:0/2142280754 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efcf8150a80 0x7efcf8150e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:38.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.522+0000 7efcfd830640 1 -- 172.21.15.19:0/2142280754 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcf8157360 con 0x7efce4004800 2024-08-31T12:23:38.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.522+0000 7efcfd830640 1 --2- 172.21.15.19:0/2142280754 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efce4004800 0x7efcf80c2b20 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7efcf4011b10 tx=0x7efcf4009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:38.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.523+0000 7efcee7fc640 1 -- 172.21.15.19:0/2142280754 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf400fce0 con 0x7efce4004800 2024-08-31T12:23:38.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.523+0000 7efcee7fc640 1 -- 172.21.15.19:0/2142280754 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efcf400fe80 con 0x7efce4004800 2024-08-31T12:23:38.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.523+0000 7efcee7fc640 1 -- 172.21.15.19:0/2142280754 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf4004280 con 0x7efce4004800 2024-08-31T12:23:38.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.523+0000 7efcffa94640 1 -- 172.21.15.19:0/2142280754 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcf8157640 con 0x7efce4004800 2024-08-31T12:23:38.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.523+0000 7efcffa94640 1 -- 172.21.15.19:0/2142280754 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcf81eb680 con 0x7efce4004800 2024-08-31T12:23:38.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.524+0000 7efcffa94640 1 -- 172.21.15.19:0/2142280754 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcf80c9a10 con 0x7efce4004800 2024-08-31T12:23:38.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.524+0000 7efcee7fc640 1 -- 172.21.15.19:0/2142280754 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7efcf40035f0 con 0x7efce4004800 2024-08-31T12:23:38.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.525+0000 7efcee7fc640 1 --2- 172.21.15.19:0/2142280754 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7efcdc077a50 0x7efcdc079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:38.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.525+0000 7efcee7fc640 1 -- 172.21.15.19:0/2142280754 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7efcf409a140 con 0x7efce4004800 2024-08-31T12:23:38.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.525+0000 7efcfd02f640 1 --2- 172.21.15.19:0/2142280754 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7efcdc077a50 0x7efcdc079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:38.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.526+0000 7efcfd02f640 1 --2- 172.21.15.19:0/2142280754 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7efcdc077a50 0x7efcdc079f60 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7efce8033000 tx=0x7efce8038040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:38.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.532+0000 7efcee7fc640 1 -- 172.21.15.19:0/2142280754 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7efcf4062870 con 0x7efce4004800 2024-08-31T12:23:38.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.812+0000 7efcffa94640 1 -- 172.21.15.19:0/2142280754 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7efcf815c590 con 0x7efce4004800 2024-08-31T12:23:38.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.813+0000 7efcee7fc640 1 -- 172.21.15.19:0/2142280754 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7efcf4067030 con 0x7efce4004800 2024-08-31T12:23:38.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.816+0000 7efcffa94640 1 -- 172.21.15.19:0/2142280754 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7efcdc077a50 msgr2=0x7efcdc079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:38.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.816+0000 7efcffa94640 1 --2- 172.21.15.19:0/2142280754 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7efcdc077a50 0x7efcdc079f60 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7efce8033000 tx=0x7efce8038040 comp rx=0 tx=0).stop 2024-08-31T12:23:38.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.816+0000 7efcffa94640 1 -- 172.21.15.19:0/2142280754 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efce4004800 msgr2=0x7efcf80c2b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:38.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.816+0000 7efcffa94640 1 --2- 172.21.15.19:0/2142280754 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efce4004800 0x7efcf80c2b20 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7efcf4011b10 tx=0x7efcf4009f10 comp rx=0 tx=0).stop 2024-08-31T12:23:38.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.816+0000 7efcffa94640 1 -- 172.21.15.19:0/2142280754 shutdown_connections 2024-08-31T12:23:38.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.816+0000 7efcffa94640 1 --2- 172.21.15.19:0/2142280754 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7efcdc077a50 0x7efcdc079f60 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:38.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.816+0000 7efcffa94640 1 --2- 172.21.15.19:0/2142280754 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efcf8150a80 0x7efcf8150e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:38.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.816+0000 7efcffa94640 1 --2- 172.21.15.19:0/2142280754 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efcf815c970 0x7efcf815b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:38.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.816+0000 7efcffa94640 1 --2- 172.21.15.19:0/2142280754 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efce4004800 0x7efcf80c2b20 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:38.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.816+0000 7efcffa94640 1 -- 172.21.15.19:0/2142280754 >> 172.21.15.19:0/2142280754 conn(0x7efcf80690d0 msgr2=0x7efcf80d0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:23:38.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.816+0000 7efcffa94640 1 -- 172.21.15.19:0/2142280754 shutdown_connections 2024-08-31T12:23:38.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:38.816+0000 7efcffa94640 1 -- 172.21.15.19:0/2142280754 wait complete. 2024-08-31T12:23:38.830 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2024-08-31T12:23:48.831 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-08-31T12:23:48.831 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:23:48.956 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:48.954+0000 7f8fa8339640 1 -- 172.21.15.19:0/1631631713 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8fa01511b0 msgr2=0x7f8fa01515c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:48.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:48.954+0000 7f8fa8339640 1 --2- 172.21.15.19:0/1631631713 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8fa01511b0 0x7f8fa01515c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8f98005fd0 tx=0x7f8f9802b980 comp rx=0 tx=0).stop 2024-08-31T12:23:48.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:48.954+0000 7f8fa8339640 1 -- 172.21.15.19:0/1631631713 shutdown_connections 2024-08-31T12:23:48.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:48.954+0000 7f8fa8339640 1 --2- 172.21.15.19:0/1631631713 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8fa01511b0 0x7f8fa01515c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:48.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:48.954+0000 7f8fa8339640 1 --2- 172.21.15.19:0/1631631713 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8fa0157f60 0x7f8fa00c7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:48.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:48.954+0000 7f8fa8339640 1 -- 172.21.15.19:0/1631631713 >> 172.21.15.19:0/1631631713 conn(0x7f8fa006cae0 msgr2=0x7f8fa006cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:23:48.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:48.954+0000 7f8fa8339640 1 -- 172.21.15.19:0/1631631713 shutdown_connections 2024-08-31T12:23:48.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:48.954+0000 7f8fa8339640 1 -- 172.21.15.19:0/1631631713 wait complete. 2024-08-31T12:23:48.958 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:48.956+0000 7f8fa8339640 1 Processor -- start 2024-08-31T12:23:48.958 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:48.956+0000 7f8fa8339640 1 -- start start 2024-08-31T12:23:48.961 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:48.957+0000 7f8fa8339640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8fa006fab0 0x7f8fa01ecf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:48.961 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:48.958+0000 7f8fa8339640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8fa01511b0 0x7f8fa01e3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:48.961 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:48.958+0000 7f8fa8339640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8fa0157f60 0x7f8fa0146070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:48.961 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:48.958+0000 7f8fa8339640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f8fa0151b00 con 0x7f8fa0157f60 2024-08-31T12:23:48.961 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:48.958+0000 7f8fa8339640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8fa0157760 con 0x7f8fa01511b0 2024-08-31T12:23:48.961 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:48.958+0000 7f8fa8339640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f8fa00c8c90 con 0x7f8fa006fab0 2024-08-31T12:23:48.961 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:48.958+0000 7f8fa68d6640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8fa0157f60 0x7f8fa0146070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:48.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:48.958+0000 7f8fa68d6640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8fa0157f60 0x7f8fa0146070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:43196/0 (socket says 172.21.15.19:43196) 2024-08-31T12:23:48.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:48.958+0000 7f8fa68d6640 1 -- 172.21.15.19:0/2968101142 learned_addr learned my addr 172.21.15.19:0/2968101142 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:23:48.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:48.958+0000 7f8fa58d4640 1 --2- 172.21.15.19:0/2968101142 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8fa01511b0 0x7f8fa01e3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:48.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:48.958+0000 7f8fa60d5640 1 --2- 172.21.15.19:0/2968101142 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8fa006fab0 0x7f8fa01ecf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:48.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:48.958+0000 7f8fa68d6640 1 -- 172.21.15.19:0/2968101142 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8fa006fab0 msgr2=0x7f8fa01ecf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:48.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:48.958+0000 7f8fa68d6640 1 --2- 172.21.15.19:0/2968101142 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8fa006fab0 0x7f8fa01ecf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:48.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:48.958+0000 7f8fa68d6640 1 -- 172.21.15.19:0/2968101142 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8fa01511b0 msgr2=0x7f8fa01e3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:48.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:48.958+0000 7f8fa68d6640 1 --2- 172.21.15.19:0/2968101142 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8fa01511b0 0x7f8fa01e3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:48.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:48.958+0000 7f8fa68d6640 1 -- 172.21.15.19:0/2968101142 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fa01ed480 con 0x7f8fa0157f60 2024-08-31T12:23:48.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:48.958+0000 7f8fa58d4640 1 --2- 172.21.15.19:0/2968101142 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8fa01511b0 0x7f8fa01e3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:23:48.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:48.958+0000 7f8fa68d6640 1 --2- 172.21.15.19:0/2968101142 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8fa0157f60 0x7f8fa0146070 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f8f98006010 tx=0x7f8f98003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:48.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:48.959+0000 7f8f96ffd640 1 -- 172.21.15.19:0/2968101142 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f98035040 con 0x7f8fa0157f60 2024-08-31T12:23:48.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:48.959+0000 7f8f96ffd640 1 -- 172.21.15.19:0/2968101142 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8f9803db40 con 0x7f8fa0157f60 2024-08-31T12:23:48.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:48.959+0000 7f8fa8339640 1 -- 172.21.15.19:0/2968101142 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fa00cc810 con 0x7f8fa0157f60 2024-08-31T12:23:48.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:48.959+0000 7f8f96ffd640 1 -- 172.21.15.19:0/2968101142 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f98004040 con 0x7f8fa0157f60 2024-08-31T12:23:48.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:48.959+0000 7f8fa8339640 1 -- 172.21.15.19:0/2968101142 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fa00cc420 con 0x7f8fa0157f60 2024-08-31T12:23:48.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:48.960+0000 7f8fa8339640 1 -- 172.21.15.19:0/2968101142 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fa01ed650 con 0x7f8fa0157f60 2024-08-31T12:23:48.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:48.961+0000 7f8f96ffd640 1 -- 172.21.15.19:0/2968101142 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f8f98045020 con 0x7f8fa0157f60 2024-08-31T12:23:48.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:48.961+0000 7f8f96ffd640 1 --2- 172.21.15.19:0/2968101142 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8f84077a80 0x7f8f84079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:48.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:48.961+0000 7f8f96ffd640 1 -- 172.21.15.19:0/2968101142 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f8f980bcff0 con 0x7f8fa0157f60 2024-08-31T12:23:48.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:48.961+0000 7f8fa60d5640 1 --2- 172.21.15.19:0/2968101142 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8f84077a80 0x7f8f84079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:48.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:48.962+0000 7f8fa60d5640 1 --2- 172.21.15.19:0/2968101142 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8f84077a80 0x7f8f84079f90 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c007fd0 tx=0x7f8f9c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:48.969 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:48.967+0000 7f8f96ffd640 1 -- 172.21.15.19:0/2968101142 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f8f98085690 con 0x7f8fa0157f60 2024-08-31T12:23:49.264 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.262+0000 7f8fa8339640 1 -- 172.21.15.19:0/2968101142 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8fa00b29f0 con 0x7f8fa0157f60 2024-08-31T12:23:49.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.263+0000 7f8f96ffd640 1 -- 172.21.15.19:0/2968101142 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f8f98089e50 con 0x7f8fa0157f60 2024-08-31T12:23:49.265 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":56,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:23:49.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.265+0000 7f8fa8339640 1 -- 172.21.15.19:0/2968101142 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8f84077a80 msgr2=0x7f8f84079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:49.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.265+0000 7f8fa8339640 1 --2- 172.21.15.19:0/2968101142 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8f84077a80 0x7f8f84079f90 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c007fd0 tx=0x7f8f9c009040 comp rx=0 tx=0).stop 2024-08-31T12:23:49.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.265+0000 7f8fa8339640 1 -- 172.21.15.19:0/2968101142 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8fa0157f60 msgr2=0x7f8fa0146070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:49.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.265+0000 7f8fa8339640 1 --2- 172.21.15.19:0/2968101142 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8fa0157f60 0x7f8fa0146070 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f8f98006010 tx=0x7f8f98003f10 comp rx=0 tx=0).stop 2024-08-31T12:23:49.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.265+0000 7f8fa8339640 1 -- 172.21.15.19:0/2968101142 shutdown_connections 2024-08-31T12:23:49.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.265+0000 7f8fa8339640 1 --2- 172.21.15.19:0/2968101142 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8f84077a80 0x7f8f84079f90 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:49.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.265+0000 7f8fa8339640 1 --2- 172.21.15.19:0/2968101142 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8fa0157f60 0x7f8fa0146070 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:49.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.265+0000 7f8fa8339640 1 --2- 172.21.15.19:0/2968101142 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8fa01511b0 0x7f8fa01e3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:49.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.265+0000 7f8fa8339640 1 --2- 172.21.15.19:0/2968101142 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8fa006fab0 0x7f8fa01ecf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:49.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.265+0000 7f8fa8339640 1 -- 172.21.15.19:0/2968101142 >> 172.21.15.19:0/2968101142 conn(0x7f8fa006cae0 msgr2=0x7f8fa01556d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:23:49.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.265+0000 7f8fa8339640 1 -- 172.21.15.19:0/2968101142 shutdown_connections 2024-08-31T12:23:49.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.265+0000 7f8fa8339640 1 -- 172.21.15.19:0/2968101142 wait complete. 2024-08-31T12:23:49.280 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-08-31T12:23:49.280 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-08-31T12:23:49.280 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-08-31T12:23:49.280 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-31T12:23:49.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.419+0000 7f8e8a370640 1 -- 172.21.15.19:0/994030831 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e840719b0 msgr2=0x7f8e84071dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:49.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.419+0000 7f8e8a370640 1 --2- 172.21.15.19:0/994030831 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e840719b0 0x7f8e84071dc0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f8e840750c0 tx=0x7f8e7c02aa30 comp rx=0 tx=0).stop 2024-08-31T12:23:49.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.419+0000 7f8e8a370640 1 -- 172.21.15.19:0/994030831 shutdown_connections 2024-08-31T12:23:49.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.419+0000 7f8e8a370640 1 --2- 172.21.15.19:0/994030831 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e840719b0 0x7f8e84071dc0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:49.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.419+0000 7f8e8a370640 1 --2- 172.21.15.19:0/994030831 >> v2:172.21.15.43:3300/0 conn(0x7f8e840c7d60 0x7f8e840c7140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:49.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.419+0000 7f8e8a370640 1 -- 172.21.15.19:0/994030831 >> 172.21.15.19:0/994030831 conn(0x7f8e840cd7d0 msgr2=0x7f8e840cdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:23:49.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.419+0000 7f8e8a370640 1 -- 172.21.15.19:0/994030831 shutdown_connections 2024-08-31T12:23:49.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.419+0000 7f8e8a370640 1 -- 172.21.15.19:0/994030831 wait complete. 2024-08-31T12:23:49.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.435+0000 7f8e8a370640 1 Processor -- start 2024-08-31T12:23:49.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.436+0000 7f8e8a370640 1 -- start start 2024-08-31T12:23:49.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.437+0000 7f8e8a370640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e840719b0 0x7f8e841f1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:49.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.437+0000 7f8e8a370640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8e840c7d60 0x7f8e84154f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:49.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.437+0000 7f8e8a370640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8e84140c10 0x7f8e84152d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:49.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.437+0000 7f8e8a370640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f8e840c8170 con 0x7f8e840719b0 2024-08-31T12:23:49.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.437+0000 7f8e8a370640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8e840c6920 con 0x7f8e84140c10 2024-08-31T12:23:49.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.437+0000 7f8e8a370640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f8e8415b2b0 con 0x7f8e840c7d60 2024-08-31T12:23:49.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.438+0000 7f8e83fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e840719b0 0x7f8e841f1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:49.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.438+0000 7f8e83fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e840719b0 0x7f8e841f1a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:43224/0 (socket says 172.21.15.19:43224) 2024-08-31T12:23:49.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.438+0000 7f8e83fff640 1 -- 172.21.15.19:0/3808558828 learned_addr learned my addr 172.21.15.19:0/3808558828 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:23:49.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.438+0000 7f8e8890d640 1 --2- 172.21.15.19:0/3808558828 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8e84140c10 0x7f8e84152d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:49.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.438+0000 7f8e837fe640 1 --2- 172.21.15.19:0/3808558828 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8e840c7d60 0x7f8e84154f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:49.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.438+0000 7f8e83fff640 1 -- 172.21.15.19:0/3808558828 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8e840c7d60 msgr2=0x7f8e84154f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:49.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.438+0000 7f8e83fff640 1 --2- 172.21.15.19:0/3808558828 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8e840c7d60 0x7f8e84154f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:49.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.438+0000 7f8e83fff640 1 -- 172.21.15.19:0/3808558828 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8e84140c10 msgr2=0x7f8e84152d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:49.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.438+0000 7f8e83fff640 1 --2- 172.21.15.19:0/3808558828 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8e84140c10 0x7f8e84152d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:49.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.438+0000 7f8e83fff640 1 -- 172.21.15.19:0/3808558828 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e841f1f40 con 0x7f8e840719b0 2024-08-31T12:23:49.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.438+0000 7f8e83fff640 1 --2- 172.21.15.19:0/3808558828 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e840719b0 0x7f8e841f1a00 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f8e7400b440 tx=0x7f8e7400b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:49.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.438+0000 7f8e80ff9640 1 -- 172.21.15.19:0/3808558828 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e74015020 con 0x7f8e840719b0 2024-08-31T12:23:49.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.438+0000 7f8e80ff9640 1 -- 172.21.15.19:0/3808558828 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8e74011940 con 0x7f8e840719b0 2024-08-31T12:23:49.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.438+0000 7f8e80ff9640 1 -- 172.21.15.19:0/3808558828 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e74011d90 con 0x7f8e840719b0 2024-08-31T12:23:49.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.438+0000 7f8e8a370640 1 -- 172.21.15.19:0/3808558828 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e8413bb20 con 0x7f8e840719b0 2024-08-31T12:23:49.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.439+0000 7f8e8a370640 1 -- 172.21.15.19:0/3808558828 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e841f5530 con 0x7f8e840719b0 2024-08-31T12:23:49.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.440+0000 7f8e8a370640 1 --2- 172.21.15.19:0/3808558828 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e8413a1c0 0x7f8e8406cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:49.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.440+0000 7f8e8a370640 1 -- 172.21.15.19:0/3808558828 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f8e8413b8d0 con 0x7f8e8413a1c0 2024-08-31T12:23:49.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.440+0000 7f8e837fe640 1 --2- 172.21.15.19:0/3808558828 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e8413a1c0 0x7f8e8406cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:49.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.440+0000 7f8e837fe640 1 -- 172.21.15.19:0/3808558828 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8e841f2110 con 0x7f8e8413a1c0 2024-08-31T12:23:49.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.440+0000 7f8e837fe640 1 --2- 172.21.15.19:0/3808558828 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e8413a1c0 0x7f8e8406cae0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f8e78006fd0 tx=0x7f8e7800ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:49.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.440+0000 7f8e80ff9640 1 -- 172.21.15.19:0/3808558828 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e78017070 con 0x7f8e8413a1c0 2024-08-31T12:23:49.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.441+0000 7f8e80ff9640 1 -- 172.21.15.19:0/3808558828 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f8e74018020 con 0x7f8e840719b0 2024-08-31T12:23:49.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.441+0000 7f8e80ff9640 1 --2- 172.21.15.19:0/3808558828 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8e70077980 0x7f8e70079e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:49.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.441+0000 7f8e80ff9640 1 -- 172.21.15.19:0/3808558828 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f8e740994f0 con 0x7f8e840719b0 2024-08-31T12:23:49.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.441+0000 7f8e8890d640 1 --2- 172.21.15.19:0/3808558828 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8e70077980 0x7f8e70079e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:49.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.441+0000 7f8e80ff9640 1 -- 172.21.15.19:0/3808558828 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8e7800c770 con 0x7f8e8413a1c0 2024-08-31T12:23:49.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.441+0000 7f8e80ff9640 1 -- 172.21.15.19:0/3808558828 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e8413a1c0 msgr2=0x7f8e8406cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:49.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.441+0000 7f8e80ff9640 1 --2- 172.21.15.19:0/3808558828 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e8413a1c0 0x7f8e8406cae0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f8e78006fd0 tx=0x7f8e7800ec00 comp rx=0 tx=0).stop 2024-08-31T12:23:49.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.442+0000 7f8e80ff9640 1 -- 172.21.15.19:0/3808558828 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e8413a1c0 msgr2=0x7f8e8406cae0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:23:49.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.442+0000 7f8e80ff9640 1 --2- 172.21.15.19:0/3808558828 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e8413a1c0 0x7f8e8406cae0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:49.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.442+0000 7f8e8890d640 1 --2- 172.21.15.19:0/3808558828 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8e70077980 0x7f8e70079e90 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c035000 tx=0x7f8e7c03b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:49.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.450+0000 7f8e8a370640 1 --2- 172.21.15.19:0/3808558828 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e841554d0 0x7f8e8406d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:49.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.450+0000 7f8e8a370640 1 -- 172.21.15.19:0/3808558828 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f8e840c58a0 con 0x7f8e841554d0 2024-08-31T12:23:49.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.450+0000 7f8e837fe640 1 --2- 172.21.15.19:0/3808558828 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e841554d0 0x7f8e8406d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:49.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.450+0000 7f8e837fe640 1 -- 172.21.15.19:0/3808558828 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8e84156e40 con 0x7f8e841554d0 2024-08-31T12:23:49.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.450+0000 7f8e837fe640 1 --2- 172.21.15.19:0/3808558828 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e841554d0 0x7f8e8406d020 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f8e78019000 tx=0x7f8e780046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:49.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.450+0000 7f8e80ff9640 1 -- 172.21.15.19:0/3808558828 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e78013d90 con 0x7f8e841554d0 2024-08-31T12:23:49.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.451+0000 7f8e80ff9640 1 -- 172.21.15.19:0/3808558828 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5045 (secure 0 0 0) 0x7f8e78013d90 con 0x7f8e841554d0 2024-08-31T12:23:49.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.451+0000 7f8e80ff9640 1 -- 172.21.15.19:0/3808558828 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e841554d0 msgr2=0x7f8e8406d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:49.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.451+0000 7f8e80ff9640 1 --2- 172.21.15.19:0/3808558828 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e841554d0 0x7f8e8406d020 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f8e78019000 tx=0x7f8e780046b0 comp rx=0 tx=0).stop 2024-08-31T12:23:49.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.451+0000 7f8e80ff9640 1 -- 172.21.15.19:0/3808558828 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e841554d0 msgr2=0x7f8e8406d020 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:23:49.454 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.451+0000 7f8e80ff9640 1 --2- 172.21.15.19:0/3808558828 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e841554d0 0x7f8e8406d020 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:49.454 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:23:49.454 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:23:49.454 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:23:49.454 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "leader", 2024-08-31T12:23:49.454 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 56, 2024-08-31T12:23:49.454 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 409441, 2024-08-31T12:23:49.454 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:23:49.454 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:23:49.455 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:23:49.455 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:23:49.455 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:23:49.455 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 15, 2024-08-31T12:23:49.455 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:23:49.455 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:23:49.455 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:23:49.455 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:23:49.455 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:23:49.455 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:23:49.455 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:23:49.455 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:23:49.455 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:23:49.456 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:23:49.456 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:23:49.456 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:23:49.456 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:23:49.456 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:23:49.456 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:23:49.456 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:23:49.456 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:23:49.456 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:23:49.456 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:23:49.456 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:23:49.456 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:23:49.456 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:23:49.456 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:23:49.457 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:23:49.457 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:23:49.457 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:23:49.457 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:23:49.457 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:23:49.457 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:49.457 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:49.457 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:23:49.457 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:23:49.457 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:23:49.457 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:23:49.457 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:23:49.457 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:23:49.457 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:23:49.458 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:23:49.458 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:23:49.458 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:23:49.458 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:23:49.458 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:23:49.458 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:23:49.458 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:23:49.458 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:23:49.458 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:23:49.458 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:23:49.458 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:23:49.458 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:23:49.458 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:23:49.459 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:23:49.459 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:23:49.459 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:23:49.459 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:23:49.459 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:23:49.459 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:23:49.459 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:23:49.459 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:23:49.459 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:23:49.459 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:23:49.459 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:49.459 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:23:49.459 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:49.459 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:23:49.459 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:23:49.460 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:23:49.460 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:23:49.460 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:49.460 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:23:49.460 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:23:49.460 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:49.460 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:49.460 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:49.460 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:23:49.461 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:23:49.461 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:49.461 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:49.461 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:49.461 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:49.461 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:23:49.461 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:23:49.461 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:23:49.461 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:23:49.461 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:23:49.461 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:49.461 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:49.461 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:23:49.461 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:23:49.462 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:23:49.462 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:23:49.462 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:49.462 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:23:49.462 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:23:49.462 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:49.462 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:49.462 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:49.462 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:23:49.462 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:23:49.462 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:49.462 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:49.462 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:49.462 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:49.462 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:23:49.463 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:23:49.463 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:23:49.463 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:23:49.463 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:23:49.463 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:49.463 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:49.463 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:23:49.463 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:23:49.463 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:23:49.463 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:23:49.463 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:49.463 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:23:49.463 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:23:49.464 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:49.464 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:49.464 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:49.464 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:23:49.464 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:23:49.464 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:49.464 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:49.464 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:49.464 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:49.464 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:23:49.464 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:23:49.464 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:23:49.464 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:23:49.465 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:23:49.465 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:49.465 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:49.465 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:49.465 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:23:49.465 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:23:49.465 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:49.465 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:23:49.465 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:23:49.465 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:23:49.465 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:49.465 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:23:49.465 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:23:49.465 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:49.465 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:23:49.466 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:23:49.466 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 6 2024-08-31T12:23:49.466 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:49.466 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:23:49.466 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:23:49.466 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:49.466 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:23:49.466 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:23:49.466 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 8 2024-08-31T12:23:49.466 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:49.466 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:23:49.466 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:23:49.466 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:49.466 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:23:49.466 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:23:49.467 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 6 2024-08-31T12:23:49.467 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:49.467 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:23:49.467 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:23:49.467 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:49.467 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:23:49.467 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:23:49.467 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:23:49.467 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:49.467 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:49.467 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:49.467 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:23:49.467 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:23:49.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.451+0000 7f8e8a370640 1 -- 172.21.15.19:0/3808558828 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8e70077980 msgr2=0x7f8e70079e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:49.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.451+0000 7f8e8a370640 1 --2- 172.21.15.19:0/3808558828 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8e70077980 0x7f8e70079e90 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c035000 tx=0x7f8e7c03b040 comp rx=0 tx=0).stop 2024-08-31T12:23:49.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.451+0000 7f8e8a370640 1 -- 172.21.15.19:0/3808558828 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e840719b0 msgr2=0x7f8e841f1a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:49.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.451+0000 7f8e8a370640 1 --2- 172.21.15.19:0/3808558828 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e840719b0 0x7f8e841f1a00 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f8e7400b440 tx=0x7f8e7400b900 comp rx=0 tx=0).stop 2024-08-31T12:23:49.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.451+0000 7f8e8890d640 1 -- 172.21.15.19:0/3808558828 reap_dead start 2024-08-31T12:23:49.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.451+0000 7f8e8890d640 1 -- 172.21.15.19:0/3808558828 reap_dead start 2024-08-31T12:23:49.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.451+0000 7f8e8a370640 1 -- 172.21.15.19:0/3808558828 shutdown_connections 2024-08-31T12:23:49.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.451+0000 7f8e8a370640 1 -- 172.21.15.19:0/3808558828 >> 172.21.15.19:0/3808558828 conn(0x7f8e840cd7d0 msgr2=0x7f8e84157360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:23:49.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.451+0000 7f8e8a370640 1 -- 172.21.15.19:0/3808558828 shutdown_connections 2024-08-31T12:23:49.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.451+0000 7f8e8a370640 1 -- 172.21.15.19:0/3808558828 wait complete. 2024-08-31T12:23:49.469 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:23:49.470 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-08-31T12:23:49.470 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-31T12:23:49.592 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.590+0000 7f8fc2653640 1 -- 172.21.15.19:0/604916827 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8fbc0c8a40 msgr2=0x7f8fbc0c9850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:49.592 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.590+0000 7f8fc2653640 1 --2- 172.21.15.19:0/604916827 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8fbc0c8a40 0x7f8fbc0c9850 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8fac005fd0 tx=0x7f8fac02a670 comp rx=0 tx=0).stop 2024-08-31T12:23:49.592 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.590+0000 7f8fc2653640 1 -- 172.21.15.19:0/604916827 shutdown_connections 2024-08-31T12:23:49.592 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.590+0000 7f8fc2653640 1 --2- 172.21.15.19:0/604916827 >> v2:172.21.15.43:3300/0 conn(0x7f8fbc06e0e0 0x7f8fbc155300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:49.593 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.590+0000 7f8fc2653640 1 --2- 172.21.15.19:0/604916827 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8fbc0c8a40 0x7f8fbc0c9850 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:49.593 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.590+0000 7f8fc2653640 1 -- 172.21.15.19:0/604916827 >> 172.21.15.19:0/604916827 conn(0x7f8fbc0690d0 msgr2=0x7f8fbc0d0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:23:49.593 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.590+0000 7f8fc2653640 1 -- 172.21.15.19:0/604916827 shutdown_connections 2024-08-31T12:23:49.593 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.590+0000 7f8fc2653640 1 -- 172.21.15.19:0/604916827 wait complete. 2024-08-31T12:23:49.594 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.592+0000 7f8fc2653640 1 Processor -- start 2024-08-31T12:23:49.594 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.592+0000 7f8fc2653640 1 -- start start 2024-08-31T12:23:49.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.594+0000 7f8fc2653640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8fbc06e0e0 0x7f8fbc1e6b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:49.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.594+0000 7f8fc2653640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8fbc0c8a40 0x7f8fbc0c5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:49.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.594+0000 7f8fc2653640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8fbc15b6e0 0x7f8fbc1f1c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:49.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.594+0000 7f8fc2653640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f8fbc0c8610 con 0x7f8fbc15b6e0 2024-08-31T12:23:49.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.594+0000 7f8fc2653640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8fbc0c8330 con 0x7f8fbc06e0e0 2024-08-31T12:23:49.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.594+0000 7f8fc2653640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f8fbc06d710 con 0x7f8fbc0c8a40 2024-08-31T12:23:49.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.594+0000 7f8fc0bf0640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8fbc15b6e0 0x7f8fbc1f1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:49.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.594+0000 7f8fbbfff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8fbc06e0e0 0x7f8fbc1e6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:49.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.594+0000 7f8fc0bf0640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8fbc15b6e0 0x7f8fbc1f1c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:43240/0 (socket says 172.21.15.19:43240) 2024-08-31T12:23:49.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.594+0000 7f8fc0bf0640 1 -- 172.21.15.19:0/3077129843 learned_addr learned my addr 172.21.15.19:0/3077129843 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:23:49.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.594+0000 7f8fbb7fe640 1 --2- 172.21.15.19:0/3077129843 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8fbc0c8a40 0x7f8fbc0c5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:49.598 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.594+0000 7f8fc0bf0640 1 -- 172.21.15.19:0/3077129843 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8fbc0c8a40 msgr2=0x7f8fbc0c5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:49.598 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.594+0000 7f8fc0bf0640 1 --2- 172.21.15.19:0/3077129843 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8fbc0c8a40 0x7f8fbc0c5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:49.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.594+0000 7f8fc0bf0640 1 -- 172.21.15.19:0/3077129843 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8fbc06e0e0 msgr2=0x7f8fbc1e6b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:49.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.594+0000 7f8fc0bf0640 1 --2- 172.21.15.19:0/3077129843 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8fbc06e0e0 0x7f8fbc1e6b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:49.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.594+0000 7f8fc0bf0640 1 -- 172.21.15.19:0/3077129843 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fbc0ce640 con 0x7f8fbc15b6e0 2024-08-31T12:23:49.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.594+0000 7f8fbb7fe640 1 --2- 172.21.15.19:0/3077129843 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8fbc0c8a40 0x7f8fbc0c5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:23:49.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.595+0000 7f8fc0bf0640 1 --2- 172.21.15.19:0/3077129843 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8fbc15b6e0 0x7f8fbc1f1c40 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f8fb400c760 tx=0x7f8fb400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:49.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.595+0000 7f8fb8ff9640 1 -- 172.21.15.19:0/3077129843 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fb4003b90 con 0x7f8fbc15b6e0 2024-08-31T12:23:49.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.595+0000 7f8fb8ff9640 1 -- 172.21.15.19:0/3077129843 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8fb4003d50 con 0x7f8fbc15b6e0 2024-08-31T12:23:49.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.595+0000 7f8fc2653640 1 -- 172.21.15.19:0/3077129843 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fbc1f2180 con 0x7f8fbc15b6e0 2024-08-31T12:23:49.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.595+0000 7f8fb8ff9640 1 -- 172.21.15.19:0/3077129843 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fb400aab0 con 0x7f8fbc15b6e0 2024-08-31T12:23:49.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.595+0000 7f8fc2653640 1 -- 172.21.15.19:0/3077129843 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fbc0d0670 con 0x7f8fbc15b6e0 2024-08-31T12:23:49.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.596+0000 7f8fc2653640 1 --2- 172.21.15.19:0/3077129843 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8fbc0d0930 0x7f8fbc0cf010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:49.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.596+0000 7f8fc2653640 1 -- 172.21.15.19:0/3077129843 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8fbc0d01a0 con 0x7f8fbc0d0930 2024-08-31T12:23:49.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.597+0000 7f8fbbfff640 1 --2- 172.21.15.19:0/3077129843 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8fbc0d0930 0x7f8fbc0cf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:49.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.597+0000 7f8fbbfff640 1 -- 172.21.15.19:0/3077129843 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8fbc0ce810 con 0x7f8fbc0d0930 2024-08-31T12:23:49.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.597+0000 7f8fbbfff640 1 --2- 172.21.15.19:0/3077129843 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8fbc0d0930 0x7f8fbc0cf010 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc0c7730 tx=0x7f8fac004530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:49.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.597+0000 7f8fb8ff9640 1 -- 172.21.15.19:0/3077129843 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fac040070 con 0x7f8fbc0d0930 2024-08-31T12:23:49.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.597+0000 7f8fb8ff9640 1 -- 172.21.15.19:0/3077129843 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f8fb400ac50 con 0x7f8fbc15b6e0 2024-08-31T12:23:49.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.598+0000 7f8fb8ff9640 1 --2- 172.21.15.19:0/3077129843 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8fa8077950 0x7f8fa8079e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:49.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.598+0000 7f8fb8ff9640 1 -- 172.21.15.19:0/3077129843 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f8fb4099370 con 0x7f8fbc15b6e0 2024-08-31T12:23:49.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.598+0000 7f8fbb7fe640 1 --2- 172.21.15.19:0/3077129843 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8fa8077950 0x7f8fa8079e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:49.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.599+0000 7f8fb8ff9640 1 -- 172.21.15.19:0/3077129843 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8fac003b40 con 0x7f8fbc0d0930 2024-08-31T12:23:49.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.599+0000 7f8fb8ff9640 1 -- 172.21.15.19:0/3077129843 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8fbc0d0930 msgr2=0x7f8fbc0cf010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:49.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.599+0000 7f8fb8ff9640 1 --2- 172.21.15.19:0/3077129843 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8fbc0d0930 0x7f8fbc0cf010 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc0c7730 tx=0x7f8fac004530 comp rx=0 tx=0).stop 2024-08-31T12:23:49.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.599+0000 7f8fb8ff9640 1 -- 172.21.15.19:0/3077129843 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8fbc0d0930 msgr2=0x7f8fbc0cf010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:23:49.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.599+0000 7f8fb8ff9640 1 --2- 172.21.15.19:0/3077129843 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8fbc0d0930 0x7f8fbc0cf010 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:49.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.600+0000 7f8fbb7fe640 1 --2- 172.21.15.19:0/3077129843 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8fa8077950 0x7f8fa8079e60 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc1f2460 tx=0x7f8fb0005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:49.609 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.606+0000 7f8fc2653640 1 --2- 172.21.15.19:0/3077129843 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8fbc153ee0 0x7f8fbc1542f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:49.609 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.606+0000 7f8fc2653640 1 -- 172.21.15.19:0/3077129843 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8fbc0c9850 con 0x7f8fbc153ee0 2024-08-31T12:23:49.609 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.607+0000 7f8fbbfff640 1 --2- 172.21.15.19:0/3077129843 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8fbc153ee0 0x7f8fbc1542f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:49.610 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.607+0000 7f8fbbfff640 1 -- 172.21.15.19:0/3077129843 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8fbc0ce0a0 con 0x7f8fbc153ee0 2024-08-31T12:23:49.610 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.608+0000 7f8fbbfff640 1 --2- 172.21.15.19:0/3077129843 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8fbc153ee0 0x7f8fbc1542f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8fac006010 tx=0x7f8fac009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:49.610 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.608+0000 7f8fb8ff9640 1 -- 172.21.15.19:0/3077129843 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fac035040 con 0x7f8fbc153ee0 2024-08-31T12:23:49.610 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.608+0000 7f8fb8ff9640 1 -- 172.21.15.19:0/3077129843 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4542 (secure 0 0 0) 0x7f8fac044990 con 0x7f8fbc153ee0 2024-08-31T12:23:49.611 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.608+0000 7f8fb8ff9640 1 -- 172.21.15.19:0/3077129843 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8fbc153ee0 msgr2=0x7f8fbc1542f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:49.611 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.608+0000 7f8fb8ff9640 1 --2- 172.21.15.19:0/3077129843 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8fbc153ee0 0x7f8fbc1542f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8fac006010 tx=0x7f8fac009260 comp rx=0 tx=0).stop 2024-08-31T12:23:49.611 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.608+0000 7f8fb8ff9640 1 -- 172.21.15.19:0/3077129843 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8fbc153ee0 msgr2=0x7f8fbc1542f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:23:49.611 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.608+0000 7f8fb8ff9640 1 --2- 172.21.15.19:0/3077129843 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8fbc153ee0 0x7f8fbc1542f0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:49.611 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:23:49.611 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:23:49.611 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:23:49.611 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:23:49.611 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 56, 2024-08-31T12:23:49.612 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 139082, 2024-08-31T12:23:49.612 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:23:49.612 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:23:49.612 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:23:49.612 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:23:49.612 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:23:49.612 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 15, 2024-08-31T12:23:49.612 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:23:49.612 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:23:49.612 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:23:49.612 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:23:49.612 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:23:49.612 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:23:49.613 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:23:49.613 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:23:49.613 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:23:49.613 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:23:49.613 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:23:49.613 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:23:49.613 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:23:49.613 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:23:49.613 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:23:49.613 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:23:49.613 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:23:49.613 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:23:49.613 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:23:49.614 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:23:49.614 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:23:49.614 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:23:49.614 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:23:49.614 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:23:49.614 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:23:49.614 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:23:49.614 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:23:49.614 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:23:49.614 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:49.614 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:49.614 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:23:49.614 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:23:49.614 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:23:49.615 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:23:49.615 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:23:49.615 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:23:49.615 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:23:49.615 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:23:49.615 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:23:49.615 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:23:49.615 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:23:49.615 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:23:49.615 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:23:49.615 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:23:49.615 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:23:49.615 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:23:49.616 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:23:49.616 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:23:49.616 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:23:49.616 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:23:49.616 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:23:49.616 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:23:49.616 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:23:49.616 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:23:49.616 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:23:49.616 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:23:49.616 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:23:49.616 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:23:49.616 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:23:49.616 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:23:49.617 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:49.617 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:23:49.617 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:49.617 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:23:49.617 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:23:49.617 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:23:49.617 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:23:49.617 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:49.617 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:23:49.617 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:23:49.617 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:49.617 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:49.617 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:49.618 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:23:49.618 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:23:49.618 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:49.618 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:49.618 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:49.618 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:49.618 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:23:49.618 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:23:49.618 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:23:49.618 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:23:49.618 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:23:49.618 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:49.618 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:49.618 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:23:49.619 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:23:49.619 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:23:49.619 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:23:49.619 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:49.619 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:23:49.619 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:23:49.619 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:49.619 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:49.619 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:49.619 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:23:49.619 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:23:49.619 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:49.619 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:49.620 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:49.620 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:49.620 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:23:49.620 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:23:49.620 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:23:49.620 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:23:49.620 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:23:49.620 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:49.620 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:49.621 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:23:49.621 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:23:49.621 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:23:49.621 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:23:49.621 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:49.621 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:23:49.621 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:23:49.621 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:49.621 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:49.621 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:49.621 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:23:49.621 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:23:49.621 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:49.622 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:49.622 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:49.622 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:49.622 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:23:49.622 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:23:49.622 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:23:49.622 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:23:49.622 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:23:49.622 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:49.622 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:49.622 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:49.622 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:23:49.622 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:23:49.622 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:49.623 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:23:49.623 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:23:49.623 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:23:49.623 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:49.623 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:23:49.623 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:23:49.623 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:49.623 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:23:49.623 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:23:49.623 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:23:49.623 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:49.623 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:49.623 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:49.624 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:23:49.624 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:23:49.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.609+0000 7f8fc2653640 1 -- 172.21.15.19:0/3077129843 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8fa8077950 msgr2=0x7f8fa8079e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:49.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.609+0000 7f8fc2653640 1 --2- 172.21.15.19:0/3077129843 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8fa8077950 0x7f8fa8079e60 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc1f2460 tx=0x7f8fb0005f90 comp rx=0 tx=0).stop 2024-08-31T12:23:49.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.609+0000 7f8fc2653640 1 -- 172.21.15.19:0/3077129843 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8fbc15b6e0 msgr2=0x7f8fbc1f1c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:49.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.609+0000 7f8fc0bf0640 1 -- 172.21.15.19:0/3077129843 reap_dead start 2024-08-31T12:23:49.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.609+0000 7f8fc2653640 1 --2- 172.21.15.19:0/3077129843 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8fbc15b6e0 0x7f8fbc1f1c40 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f8fb400c760 tx=0x7f8fb400cc20 comp rx=0 tx=0).stop 2024-08-31T12:23:49.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.609+0000 7f8fc2653640 1 -- 172.21.15.19:0/3077129843 shutdown_connections 2024-08-31T12:23:49.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.609+0000 7f8fc2653640 1 --2- 172.21.15.19:0/3077129843 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8fbc15b6e0 0x7f8fbc1f1c40 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:49.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.609+0000 7f8fc2653640 1 -- 172.21.15.19:0/3077129843 >> 172.21.15.19:0/3077129843 conn(0x7f8fbc0690d0 msgr2=0x7f8fbc1597a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:23:49.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.609+0000 7f8fc2653640 1 -- 172.21.15.19:0/3077129843 shutdown_connections 2024-08-31T12:23:49.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.609+0000 7f8fc2653640 1 -- 172.21.15.19:0/3077129843 wait complete. 2024-08-31T12:23:49.626 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:23:49.626 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-08-31T12:23:49.626 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-31T12:23:49.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.743+0000 7fb64e470640 1 -- 172.21.15.19:0/2347578416 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb6300041c0 msgr2=0x7fb630004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:49.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.743+0000 7fb64e470640 1 --2- 172.21.15.19:0/2347578416 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb6300041c0 0x7fb630004800 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fb63c006f40 tx=0x7fb630005830 comp rx=0 tx=0).stop 2024-08-31T12:23:49.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.743+0000 7fb64ca0d640 1 -- 172.21.15.19:0/2347578416 reap_dead start 2024-08-31T12:23:49.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.743+0000 7fb64e470640 1 -- 172.21.15.19:0/2347578416 shutdown_connections 2024-08-31T12:23:49.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.743+0000 7fb64e470640 1 -- 172.21.15.19:0/2347578416 >> 172.21.15.19:0/2347578416 conn(0x7fb64806cae0 msgr2=0x7fb64806cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:23:49.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.743+0000 7fb64e470640 1 -- 172.21.15.19:0/2347578416 shutdown_connections 2024-08-31T12:23:49.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.744+0000 7fb64e470640 1 -- 172.21.15.19:0/2347578416 wait complete. 2024-08-31T12:23:49.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.746+0000 7fb64e470640 1 Processor -- start 2024-08-31T12:23:49.748 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.746+0000 7fb64e470640 1 -- start start 2024-08-31T12:23:49.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.747+0000 7fb64e470640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb6300041c0 0x7fb6481423f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:49.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.747+0000 7fb64e470640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb6480c5510 0x7fb6480c5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:49.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.747+0000 7fb64e470640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb6480c5e60 0x7fb6481f1470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:49.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.747+0000 7fb647fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb6300041c0 0x7fb6481423f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:49.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.747+0000 7fb64e470640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fb6480c3e90 con 0x7fb6300041c0 2024-08-31T12:23:49.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.747+0000 7fb64e470640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb6480b2dc0 con 0x7fb6480c5e60 2024-08-31T12:23:49.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.747+0000 7fb64e470640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fb6480c4010 con 0x7fb6480c5510 2024-08-31T12:23:49.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.747+0000 7fb6477fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb6480c5510 0x7fb6480c5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:49.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.747+0000 7fb647fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb6300041c0 0x7fb6481423f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:43250/0 (socket says 172.21.15.19:43250) 2024-08-31T12:23:49.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.747+0000 7fb6477fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb6480c5510 0x7fb6480c5920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.19:33306/0 (socket says 172.21.15.19:33306) 2024-08-31T12:23:49.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.747+0000 7fb647fff640 1 -- 172.21.15.19:0/2336735385 learned_addr learned my addr 172.21.15.19:0/2336735385 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:23:49.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.748+0000 7fb64ca0d640 1 --2- 172.21.15.19:0/2336735385 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb6480c5e60 0x7fb6481f1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:49.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.748+0000 7fb647fff640 1 -- 172.21.15.19:0/2336735385 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb6480c5510 msgr2=0x7fb6480c5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:49.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.748+0000 7fb647fff640 1 --2- 172.21.15.19:0/2336735385 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb6480c5510 0x7fb6480c5920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:49.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.748+0000 7fb647fff640 1 -- 172.21.15.19:0/2336735385 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb6480c5e60 msgr2=0x7fb6481f1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:49.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.748+0000 7fb647fff640 1 --2- 172.21.15.19:0/2336735385 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb6480c5e60 0x7fb6481f1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:49.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.748+0000 7fb647fff640 1 -- 172.21.15.19:0/2336735385 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6480c4280 con 0x7fb6300041c0 2024-08-31T12:23:49.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.748+0000 7fb64ca0d640 1 --2- 172.21.15.19:0/2336735385 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb6480c5e60 0x7fb6481f1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:23:49.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.748+0000 7fb647fff640 1 --2- 172.21.15.19:0/2336735385 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb6300041c0 0x7fb6481423f0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fb638003e10 tx=0x7fb6380017f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:49.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.748+0000 7fb644ff9640 1 -- 172.21.15.19:0/2336735385 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb638011700 con 0x7fb6300041c0 2024-08-31T12:23:49.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.748+0000 7fb644ff9640 1 -- 172.21.15.19:0/2336735385 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb638016020 con 0x7fb6300041c0 2024-08-31T12:23:49.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.748+0000 7fb64e470640 1 -- 172.21.15.19:0/2336735385 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6480cb6c0 con 0x7fb6300041c0 2024-08-31T12:23:49.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.748+0000 7fb644ff9640 1 -- 172.21.15.19:0/2336735385 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb638011b60 con 0x7fb6300041c0 2024-08-31T12:23:49.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.748+0000 7fb64e470640 1 -- 172.21.15.19:0/2336735385 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6480cbc80 con 0x7fb6300041c0 2024-08-31T12:23:49.754 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.749+0000 7fb64e470640 1 --2- 172.21.15.19:0/2336735385 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb64813afa0 0x7fb6481f19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:49.754 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.749+0000 7fb64e470640 1 -- 172.21.15.19:0/2336735385 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fb6480c4540 con 0x7fb64813afa0 2024-08-31T12:23:49.754 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.750+0000 7fb6477fe640 1 --2- 172.21.15.19:0/2336735385 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb64813afa0 0x7fb6481f19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:49.754 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.750+0000 7fb6477fe640 1 -- 172.21.15.19:0/2336735385 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb64814e720 con 0x7fb64813afa0 2024-08-31T12:23:49.754 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.750+0000 7fb6477fe640 1 --2- 172.21.15.19:0/2336735385 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb64813afa0 0x7fb6481f19b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb648073300 tx=0x7fb63c006290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:49.754 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.750+0000 7fb644ff9640 1 -- 172.21.15.19:0/2336735385 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fb638011d00 con 0x7fb6300041c0 2024-08-31T12:23:49.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.751+0000 7fb644ff9640 1 --2- 172.21.15.19:0/2336735385 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb634077a50 0x7fb634079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:49.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.751+0000 7fb644ff9640 1 -- 172.21.15.19:0/2336735385 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fb63809a460 con 0x7fb6300041c0 2024-08-31T12:23:49.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.751+0000 7fb64ca0d640 1 --2- 172.21.15.19:0/2336735385 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb634077a50 0x7fb634079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:49.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.751+0000 7fb644ff9640 1 -- 172.21.15.19:0/2336735385 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb63c048070 con 0x7fb64813afa0 2024-08-31T12:23:49.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.751+0000 7fb644ff9640 1 -- 172.21.15.19:0/2336735385 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb63c0037f0 con 0x7fb64813afa0 2024-08-31T12:23:49.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.751+0000 7fb644ff9640 1 -- 172.21.15.19:0/2336735385 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb64813afa0 msgr2=0x7fb6481f19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:49.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.751+0000 7fb644ff9640 1 --2- 172.21.15.19:0/2336735385 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb64813afa0 0x7fb6481f19b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb648073300 tx=0x7fb63c006290 comp rx=0 tx=0).stop 2024-08-31T12:23:49.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.751+0000 7fb644ff9640 1 -- 172.21.15.19:0/2336735385 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb64813afa0 msgr2=0x7fb6481f19b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:23:49.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.751+0000 7fb644ff9640 1 --2- 172.21.15.19:0/2336735385 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb64813afa0 0x7fb6481f19b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:49.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.752+0000 7fb64ca0d640 1 --2- 172.21.15.19:0/2336735385 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb634077a50 0x7fb634079f60 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb648071d60 tx=0x7fb640009500 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:49.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.763+0000 7fb64e470640 1 --2- 172.21.15.19:0/2336735385 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb6481f1ef0 0x7fb64814c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:49.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.763+0000 7fb64e470640 1 -- 172.21.15.19:0/2336735385 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fb6481f2300 con 0x7fb6481f1ef0 2024-08-31T12:23:49.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.764+0000 7fb6477fe640 1 --2- 172.21.15.19:0/2336735385 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb6481f1ef0 0x7fb64814c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:49.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.764+0000 7fb6477fe640 1 -- 172.21.15.19:0/2336735385 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb6480c78a0 con 0x7fb6481f1ef0 2024-08-31T12:23:49.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.765+0000 7fb6477fe640 1 --2- 172.21.15.19:0/2336735385 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb6481f1ef0 0x7fb64814c1b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb6480c46c0 tx=0x7fb630005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:49.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.765+0000 7fb644ff9640 1 -- 172.21.15.19:0/2336735385 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb63c048040 con 0x7fb6481f1ef0 2024-08-31T12:23:49.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.765+0000 7fb644ff9640 1 -- 172.21.15.19:0/2336735385 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4541 (secure 0 0 0) 0x7fb63c04b6c0 con 0x7fb6481f1ef0 2024-08-31T12:23:49.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.765+0000 7fb644ff9640 1 -- 172.21.15.19:0/2336735385 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb6481f1ef0 msgr2=0x7fb64814c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:49.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.765+0000 7fb644ff9640 1 --2- 172.21.15.19:0/2336735385 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb6481f1ef0 0x7fb64814c1b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb6480c46c0 tx=0x7fb630005830 comp rx=0 tx=0).stop 2024-08-31T12:23:49.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.765+0000 7fb644ff9640 1 -- 172.21.15.19:0/2336735385 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb6481f1ef0 msgr2=0x7fb64814c1b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:23:49.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.765+0000 7fb644ff9640 1 --2- 172.21.15.19:0/2336735385 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb6481f1ef0 0x7fb64814c1b0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:49.768 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:23:49.768 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:23:49.768 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:23:49.768 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:23:49.768 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 56, 2024-08-31T12:23:49.768 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 17122, 2024-08-31T12:23:49.768 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:23:49.768 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:23:49.769 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:23:49.769 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:23:49.769 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:23:49.772 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 15, 2024-08-31T12:23:49.772 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:23:49.772 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:23:49.772 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:23:49.772 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:23:49.772 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:23:49.772 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:23:49.773 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:23:49.773 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:23:49.773 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:23:49.773 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:23:49.773 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:23:49.773 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:23:49.773 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:23:49.773 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:23:49.773 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:23:49.773 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:23:49.773 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:23:49.773 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:23:49.773 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:23:49.774 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:23:49.774 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:23:49.774 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:23:49.774 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:23:49.774 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:23:49.774 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:23:49.774 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:23:49.774 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:23:49.774 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:23:49.774 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:49.774 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:49.774 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:23:49.774 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:23:49.775 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:23:49.775 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:23:49.775 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:23:49.775 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:23:49.775 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:23:49.775 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:23:49.775 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:23:49.775 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:23:49.775 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:23:49.775 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:23:49.775 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:23:49.775 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:23:49.775 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:23:49.775 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:23:49.776 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:23:49.776 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:23:49.776 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:23:49.776 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:23:49.776 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:23:49.776 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:23:49.776 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:23:49.776 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:23:49.776 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:23:49.776 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:23:49.776 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:23:49.776 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:23:49.776 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:23:49.777 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:23:49.777 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:49.777 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:23:49.777 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:49.777 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:23:49.777 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:23:49.777 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:23:49.777 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:23:49.777 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:49.777 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:23:49.777 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:23:49.777 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:49.777 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:49.778 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:49.778 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:23:49.778 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:23:49.778 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:49.778 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:49.778 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:49.778 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:49.778 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:23:49.778 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:23:49.778 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:23:49.778 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:23:49.778 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:23:49.778 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:49.778 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:49.779 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:23:49.779 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:23:49.779 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:23:49.779 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:23:49.779 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:49.779 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:23:49.779 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:23:49.779 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:49.779 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:49.779 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:49.779 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:23:49.779 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:23:49.779 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:49.780 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:49.780 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:49.780 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:49.780 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:23:49.780 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:23:49.780 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:23:49.780 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:23:49.781 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:23:49.781 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:49.781 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:49.781 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:23:49.781 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:23:49.781 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:23:49.781 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:23:49.781 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:49.781 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:23:49.781 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:23:49.781 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:49.781 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:49.781 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:49.782 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:23:49.782 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:23:49.782 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:23:49.782 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:49.782 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:49.782 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:49.782 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:23:49.782 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:23:49.782 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:23:49.782 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:23:49.782 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:23:49.782 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:49.782 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:49.783 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:49.783 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:23:49.783 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:23:49.783 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:49.783 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:23:49.783 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:23:49.783 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:23:49.783 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:49.783 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:23:49.783 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:23:49.783 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:23:49.783 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:23:49.783 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:23:49.784 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:23:49.784 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:23:49.784 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:23:49.784 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:23:49.784 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:23:49.784 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:23:49.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.766+0000 7fb64e470640 1 -- 172.21.15.19:0/2336735385 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb634077a50 msgr2=0x7fb634079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:49.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.766+0000 7fb64e470640 1 --2- 172.21.15.19:0/2336735385 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb634077a50 0x7fb634079f60 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb648071d60 tx=0x7fb640009500 comp rx=0 tx=0).stop 2024-08-31T12:23:49.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.766+0000 7fb64e470640 1 -- 172.21.15.19:0/2336735385 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb6300041c0 msgr2=0x7fb6481423f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:49.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.766+0000 7fb64e470640 1 --2- 172.21.15.19:0/2336735385 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb6300041c0 0x7fb6481423f0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fb638003e10 tx=0x7fb6380017f0 comp rx=0 tx=0).stop 2024-08-31T12:23:49.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.766+0000 7fb64ca0d640 1 -- 172.21.15.19:0/2336735385 reap_dead start 2024-08-31T12:23:49.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.766+0000 7fb64ca0d640 1 -- 172.21.15.19:0/2336735385 reap_dead start 2024-08-31T12:23:49.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.766+0000 7fb64e470640 1 -- 172.21.15.19:0/2336735385 shutdown_connections 2024-08-31T12:23:49.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.766+0000 7fb64e470640 1 -- 172.21.15.19:0/2336735385 >> 172.21.15.19:0/2336735385 conn(0x7fb64806cae0 msgr2=0x7fb648073860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:23:49.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.766+0000 7fb64e470640 1 -- 172.21.15.19:0/2336735385 shutdown_connections 2024-08-31T12:23:49.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.766+0000 7fb64e470640 1 -- 172.21.15.19:0/2336735385 wait complete. 2024-08-31T12:23:49.786 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:23:49.786 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-08-31T12:23:49.787 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-08-31T12:23:49.787 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-08-31T12:23:49.787 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-08-31T12:23:49.787 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-08-31T12:23:49.787 INFO:teuthology.orchestra.run:waiting for 300 2024-08-31T12:23:49.801 INFO:tasks.ceph.mon.a:Stopped 2024-08-31T12:23:49.801 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-08-31T12:23:49.801 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:23:49.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.960+0000 7f93eb2ec640 1 -- 172.21.15.19:0/2284806656 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93e40c3a60 msgr2=0x7f93e41519a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:49.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.960+0000 7f93eb2ec640 1 --2- 172.21.15.19:0/2284806656 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93e40c3a60 0x7f93e41519a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f93e40cd600 tx=0x7f93e002b8b0 comp rx=0 tx=0).stop 2024-08-31T12:23:49.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.960+0000 7f93eb2ec640 1 -- 172.21.15.19:0/2284806656 shutdown_connections 2024-08-31T12:23:49.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.960+0000 7f93eb2ec640 1 --2- 172.21.15.19:0/2284806656 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f93e40c5430 0x7f93e40c5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:49.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.960+0000 7f93eb2ec640 1 --2- 172.21.15.19:0/2284806656 >> v2:172.21.15.19:3300/0 conn(0x7f93e40c76c0 0x7f93e40c7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:49.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.960+0000 7f93eb2ec640 1 --2- 172.21.15.19:0/2284806656 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93e40c3a60 0x7f93e41519a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:49.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.960+0000 7f93eb2ec640 1 -- 172.21.15.19:0/2284806656 >> 172.21.15.19:0/2284806656 conn(0x7f93e40cc770 msgr2=0x7f93e40ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:23:49.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.961+0000 7f93eb2ec640 1 -- 172.21.15.19:0/2284806656 shutdown_connections 2024-08-31T12:23:49.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.961+0000 7f93eb2ec640 1 -- 172.21.15.19:0/2284806656 wait complete. 2024-08-31T12:23:49.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.962+0000 7f93eb2ec640 1 Processor -- start 2024-08-31T12:23:49.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.962+0000 7f93eb2ec640 1 -- start start 2024-08-31T12:23:49.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.964+0000 7f93eb2ec640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f93e40c3a60 0x7f93e41f3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:49.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.964+0000 7f93eb2ec640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f93e40c5430 0x7f93e4147e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:49.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.964+0000 7f93eb2ec640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93e40c76c0 0x7f93e4148c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:49.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.964+0000 7f93eb2ec640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f93e4157b00 con 0x7f93e40c3a60 2024-08-31T12:23:49.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.964+0000 7f93eb2ec640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f93e4157820 con 0x7f93e40c76c0 2024-08-31T12:23:49.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.964+0000 7f93eb2ec640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f93e4157de0 con 0x7f93e40c5430 2024-08-31T12:23:49.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.964+0000 7f93e9088640 1 -- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f93e40c3a60 msgr2=0x7f93e41f3080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:3300/0 2024-08-31T12:23:49.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.964+0000 7f93e9088640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f93e40c3a60 0x7f93e41f3080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:23:49.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.964+0000 7f93e9889640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93e40c76c0 0x7f93e4148c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:49.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.964+0000 7f93e8887640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f93e40c5430 0x7f93e4147e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:49.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.964+0000 7f93e8887640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f93e40c5430 0x7f93e4147e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.19:33400/0 (socket says 172.21.15.19:33400) 2024-08-31T12:23:49.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.964+0000 7f93e9889640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93e40c76c0 0x7f93e4148c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:35732/0 (socket says 172.21.15.19:35732) 2024-08-31T12:23:49.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.964+0000 7f93e8887640 1 -- 172.21.15.19:0/4041052999 learned_addr learned my addr 172.21.15.19:0/4041052999 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:23:49.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.964+0000 7f93e8887640 1 -- 172.21.15.19:0/4041052999 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93e40c76c0 msgr2=0x7f93e4148c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:49.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.964+0000 7f93e8887640 1 --2- 172.21.15.19:0/4041052999 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93e40c76c0 0x7f93e4148c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:49.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.964+0000 7f93e8887640 1 -- 172.21.15.19:0/4041052999 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f93e40c3a60 msgr2=0x7f93e41f3080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:23:49.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.964+0000 7f93e8887640 1 --2- 172.21.15.19:0/4041052999 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f93e40c3a60 0x7f93e41f3080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:49.969 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.964+0000 7f93e8887640 1 -- 172.21.15.19:0/4041052999 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93e41f35c0 con 0x7f93e40c5430 2024-08-31T12:23:49.969 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.964+0000 7f93e9889640 1 --2- 172.21.15.19:0/4041052999 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93e40c76c0 0x7f93e4148c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:23:49.969 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.965+0000 7f93e8887640 1 --2- 172.21.15.19:0/4041052999 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f93e40c5430 0x7f93e4147e30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f93d400b4d0 tx=0x7f93d400b990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:49.969 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.965+0000 7f93d9ffb640 1 -- 172.21.15.19:0/4041052999 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d4014070 con 0x7f93e40c5430 2024-08-31T12:23:49.969 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.965+0000 7f93d9ffb640 1 -- 172.21.15.19:0/4041052999 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f93d4010980 con 0x7f93e40c5430 2024-08-31T12:23:49.969 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.965+0000 7f93d9ffb640 1 -- 172.21.15.19:0/4041052999 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d4010dd0 con 0x7f93e40c5430 2024-08-31T12:23:49.969 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.965+0000 7f93eb2ec640 1 -- 172.21.15.19:0/4041052999 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93e414b600 con 0x7f93e40c5430 2024-08-31T12:23:49.969 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.965+0000 7f93eb2ec640 1 -- 172.21.15.19:0/4041052999 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f93e414da90 con 0x7f93e40c5430 2024-08-31T12:23:49.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.966+0000 7f93eb2ec640 1 -- 172.21.15.19:0/4041052999 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93e414d530 con 0x7f93e40c5430 2024-08-31T12:23:49.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.967+0000 7f93d9ffb640 1 -- 172.21.15.19:0/4041052999 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f93d4017070 con 0x7f93e40c5430 2024-08-31T12:23:49.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.967+0000 7f93d9ffb640 1 --2- 172.21.15.19:0/4041052999 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f93c8077a00 0x7f93c8079f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:49.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.967+0000 7f93d9ffb640 1 -- 172.21.15.19:0/4041052999 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f93d4099400 con 0x7f93e40c5430 2024-08-31T12:23:49.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.967+0000 7f93e9088640 1 --2- 172.21.15.19:0/4041052999 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f93c8077a00 0x7f93c8079f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:49.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.968+0000 7f93e9088640 1 --2- 172.21.15.19:0/4041052999 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f93c8077a00 0x7f93c8079f10 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f93e0036000 tx=0x7f93e0004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:49.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:49.972+0000 7f93d9ffb640 1 -- 172.21.15.19:0/4041052999 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f93e414d530 con 0x7f93e40c5430 2024-08-31T12:23:50.282 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:50.280+0000 7f93eb2ec640 1 -- 172.21.15.19:0/4041052999 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f93e40c6390 con 0x7f93e40c5430 2024-08-31T12:23:50.283 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:50.281+0000 7f93d9ffb640 1 -- 172.21.15.19:0/4041052999 <== mon.2 v2:172.21.15.43:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f93d4061aa0 con 0x7f93e40c5430 2024-08-31T12:23:50.283 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":56,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:23:50.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:50.284+0000 7f93eb2ec640 1 -- 172.21.15.19:0/4041052999 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f93c8077a00 msgr2=0x7f93c8079f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:50.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:50.284+0000 7f93eb2ec640 1 --2- 172.21.15.19:0/4041052999 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f93c8077a00 0x7f93c8079f10 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f93e0036000 tx=0x7f93e0004730 comp rx=0 tx=0).stop 2024-08-31T12:23:50.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:50.284+0000 7f93eb2ec640 1 -- 172.21.15.19:0/4041052999 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f93e40c5430 msgr2=0x7f93e4147e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:50.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:50.284+0000 7f93eb2ec640 1 --2- 172.21.15.19:0/4041052999 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f93e40c5430 0x7f93e4147e30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f93d400b4d0 tx=0x7f93d400b990 comp rx=0 tx=0).stop 2024-08-31T12:23:50.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:50.284+0000 7f93eb2ec640 1 -- 172.21.15.19:0/4041052999 shutdown_connections 2024-08-31T12:23:50.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:50.284+0000 7f93eb2ec640 1 --2- 172.21.15.19:0/4041052999 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f93c8077a00 0x7f93c8079f10 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:50.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:50.284+0000 7f93eb2ec640 1 --2- 172.21.15.19:0/4041052999 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93e40c76c0 0x7f93e4148c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:50.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:50.284+0000 7f93eb2ec640 1 --2- 172.21.15.19:0/4041052999 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f93e40c5430 0x7f93e4147e30 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:50.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:50.284+0000 7f93eb2ec640 1 --2- 172.21.15.19:0/4041052999 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f93e40c3a60 0x7f93e41f3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:50.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:50.284+0000 7f93eb2ec640 1 -- 172.21.15.19:0/4041052999 >> 172.21.15.19:0/4041052999 conn(0x7f93e40cc770 msgr2=0x7f93e4154390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:23:50.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:50.284+0000 7f93eb2ec640 1 -- 172.21.15.19:0/4041052999 shutdown_connections 2024-08-31T12:23:50.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:50.284+0000 7f93eb2ec640 1 -- 172.21.15.19:0/4041052999 wait complete. 2024-08-31T12:23:53.299 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:23:53.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.417+0000 7fc4d5700640 1 -- 172.21.15.19:0/3189978646 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc4d00c6f90 msgr2=0x7fc4d00c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:53.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.417+0000 7fc4d5700640 1 --2- 172.21.15.19:0/3189978646 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc4d00c6f90 0x7fc4d00c73a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc4d00c65f0 tx=0x7fc4c402a5d0 comp rx=0 tx=0).stop 2024-08-31T12:23:53.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.417+0000 7fc4d5700640 1 -- 172.21.15.19:0/3189978646 shutdown_connections 2024-08-31T12:23:53.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.417+0000 7fc4d5700640 1 --2- 172.21.15.19:0/3189978646 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc4d00c6f90 0x7fc4d00c73a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:53.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.417+0000 7fc4d5700640 1 -- 172.21.15.19:0/3189978646 >> 172.21.15.19:0/3189978646 conn(0x7fc4d00cd7d0 msgr2=0x7fc4d00cdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:23:53.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.418+0000 7fc4d5700640 1 -- 172.21.15.19:0/3189978646 shutdown_connections 2024-08-31T12:23:53.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.418+0000 7fc4d5700640 1 -- 172.21.15.19:0/3189978646 wait complete. 2024-08-31T12:23:53.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.419+0000 7fc4d5700640 1 Processor -- start 2024-08-31T12:23:53.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.420+0000 7fc4d5700640 1 -- start start 2024-08-31T12:23:53.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.421+0000 7fc4d5700640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc4d00719b0 0x7fc4d0155e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:53.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.421+0000 7fc4d5700640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc4d00c6f90 0x7fc4d01574f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:53.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.421+0000 7fc4d5700640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc4d00c7dd0 0x7fc4d01540d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:53.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.421+0000 7fc4d5700640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fc4d0074c70 con 0x7fc4d00c6f90 2024-08-31T12:23:53.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.421+0000 7fc4d5700640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc4d00c5360 con 0x7fc4d00719b0 2024-08-31T12:23:53.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.421+0000 7fc4d5700640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fc4d0074df0 con 0x7fc4d00c7dd0 2024-08-31T12:23:53.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.421+0000 7fc4ce7fc640 1 -- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc4d00c6f90 msgr2=0x7fc4d01574f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:3300/0 2024-08-31T12:23:53.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.421+0000 7fc4ce7fc640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc4d00c6f90 0x7fc4d01574f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:23:53.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.421+0000 7fc4ceffd640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc4d00719b0 0x7fc4d0155e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:53.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.421+0000 7fc4ceffd640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc4d00719b0 0x7fc4d0155e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:35762/0 (socket says 172.21.15.19:35762) 2024-08-31T12:23:53.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.421+0000 7fc4ceffd640 1 -- 172.21.15.19:0/3085136620 learned_addr learned my addr 172.21.15.19:0/3085136620 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:23:53.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.421+0000 7fc4cf7fe640 1 --2- 172.21.15.19:0/3085136620 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc4d00c7dd0 0x7fc4d01540d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:53.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.421+0000 7fc4ceffd640 1 -- 172.21.15.19:0/3085136620 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc4d00c7dd0 msgr2=0x7fc4d01540d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:53.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.421+0000 7fc4ceffd640 1 --2- 172.21.15.19:0/3085136620 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc4d00c7dd0 0x7fc4d01540d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:53.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.421+0000 7fc4ceffd640 1 -- 172.21.15.19:0/3085136620 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc4d00c6f90 msgr2=0x7fc4d01574f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:23:53.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.421+0000 7fc4ceffd640 1 --2- 172.21.15.19:0/3085136620 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc4d00c6f90 0x7fc4d01574f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:53.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.421+0000 7fc4ceffd640 1 -- 172.21.15.19:0/3085136620 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4d0156570 con 0x7fc4d00719b0 2024-08-31T12:23:53.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.422+0000 7fc4ceffd640 1 --2- 172.21.15.19:0/3085136620 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc4d00719b0 0x7fc4d0155e60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc4c000b5b0 tx=0x7fc4c000ba70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:53.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.422+0000 7fc4b77fe640 1 -- 172.21.15.19:0/3085136620 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4c000f870 con 0x7fc4d00719b0 2024-08-31T12:23:53.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.422+0000 7fc4b77fe640 1 -- 172.21.15.19:0/3085136620 <== mon.1 v2:172.21.15.43:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc4c000fb20 con 0x7fc4d00719b0 2024-08-31T12:23:53.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.422+0000 7fc4d5700640 1 -- 172.21.15.19:0/3085136620 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4d01e0aa0 con 0x7fc4d00719b0 2024-08-31T12:23:53.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.422+0000 7fc4b77fe640 1 -- 172.21.15.19:0/3085136620 <== mon.1 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4c0017400 con 0x7fc4d00719b0 2024-08-31T12:23:53.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.422+0000 7fc4d5700640 1 -- 172.21.15.19:0/3085136620 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4d0156810 con 0x7fc4d00719b0 2024-08-31T12:23:53.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.423+0000 7fc4d5700640 1 -- 172.21.15.19:0/3085136620 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4d00c73a0 con 0x7fc4d00719b0 2024-08-31T12:23:53.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.424+0000 7fc4b77fe640 1 -- 172.21.15.19:0/3085136620 <== mon.1 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fc4c000d400 con 0x7fc4d00719b0 2024-08-31T12:23:53.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.424+0000 7fc4b77fe640 1 --2- 172.21.15.19:0/3085136620 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc4bc077a00 0x7fc4bc079f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:53.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.424+0000 7fc4b77fe640 1 -- 172.21.15.19:0/3085136620 <== mon.1 v2:172.21.15.43:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fc4c0099830 con 0x7fc4d00719b0 2024-08-31T12:23:53.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.424+0000 7fc4ce7fc640 1 --2- 172.21.15.19:0/3085136620 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc4bc077a00 0x7fc4bc079f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:53.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.425+0000 7fc4ce7fc640 1 --2- 172.21.15.19:0/3085136620 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc4bc077a00 0x7fc4bc079f10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4033000 tx=0x7fc4c4004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:53.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.429+0000 7fc4b77fe640 1 -- 172.21.15.19:0/3085136620 <== mon.1 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fc4c0013030 con 0x7fc4d00719b0 2024-08-31T12:23:53.744 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.742+0000 7fc4d5700640 1 -- 172.21.15.19:0/3085136620 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc4d00c8820 con 0x7fc4d00719b0 2024-08-31T12:23:53.744 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.742+0000 7fc4b77fe640 1 -- 172.21.15.19:0/3085136620 <== mon.1 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fc4c0062520 con 0x7fc4d00719b0 2024-08-31T12:23:53.745 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":56,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":19,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:23:53.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.745+0000 7fc4d5700640 1 -- 172.21.15.19:0/3085136620 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc4bc077a00 msgr2=0x7fc4bc079f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:53.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.745+0000 7fc4d5700640 1 --2- 172.21.15.19:0/3085136620 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc4bc077a00 0x7fc4bc079f10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4033000 tx=0x7fc4c4004730 comp rx=0 tx=0).stop 2024-08-31T12:23:53.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.745+0000 7fc4d5700640 1 -- 172.21.15.19:0/3085136620 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc4d00719b0 msgr2=0x7fc4d0155e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:53.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.745+0000 7fc4d5700640 1 --2- 172.21.15.19:0/3085136620 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc4d00719b0 0x7fc4d0155e60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc4c000b5b0 tx=0x7fc4c000ba70 comp rx=0 tx=0).stop 2024-08-31T12:23:53.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.745+0000 7fc4d5700640 1 -- 172.21.15.19:0/3085136620 shutdown_connections 2024-08-31T12:23:53.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.745+0000 7fc4d5700640 1 --2- 172.21.15.19:0/3085136620 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc4bc077a00 0x7fc4bc079f10 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:53.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.745+0000 7fc4d5700640 1 --2- 172.21.15.19:0/3085136620 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc4d00c7dd0 0x7fc4d01540d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:53.748 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.745+0000 7fc4d5700640 1 --2- 172.21.15.19:0/3085136620 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc4d00c6f90 0x7fc4d01574f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:53.748 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.745+0000 7fc4d5700640 1 --2- 172.21.15.19:0/3085136620 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc4d00719b0 0x7fc4d0155e60 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:53.748 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.745+0000 7fc4d5700640 1 -- 172.21.15.19:0/3085136620 >> 172.21.15.19:0/3085136620 conn(0x7fc4d00cd7d0 msgr2=0x7fc4d0157e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:23:53.748 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.745+0000 7fc4d5700640 1 -- 172.21.15.19:0/3085136620 shutdown_connections 2024-08-31T12:23:53.748 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:53.745+0000 7fc4d5700640 1 -- 172.21.15.19:0/3085136620 wait complete. 2024-08-31T12:23:56.759 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:23:56.883 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:56.881+0000 7fb145990640 1 -- 172.21.15.19:0/1560194874 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb14015fe10 msgr2=0x7fb140162230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:56.883 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:56.881+0000 7fb145990640 1 --2- 172.21.15.19:0/1560194874 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb14015fe10 0x7fb140162230 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb138005fd0 tx=0x7fb13802a7f0 comp rx=0 tx=0).stop 2024-08-31T12:23:56.883 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:56.881+0000 7fb145990640 1 -- 172.21.15.19:0/1560194874 shutdown_connections 2024-08-31T12:23:56.883 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:56.881+0000 7fb145990640 1 --2- 172.21.15.19:0/1560194874 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb14015fe10 0x7fb140162230 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:56.884 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:56.881+0000 7fb145990640 1 --2- 172.21.15.19:0/1560194874 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb14015ab90 0x7fb14015afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:56.884 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:56.881+0000 7fb145990640 1 -- 172.21.15.19:0/1560194874 >> 172.21.15.19:0/1560194874 conn(0x7fb1400710e0 msgr2=0x7fb140071520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:23:56.884 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:56.881+0000 7fb145990640 1 -- 172.21.15.19:0/1560194874 shutdown_connections 2024-08-31T12:23:56.884 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:56.881+0000 7fb145990640 1 -- 172.21.15.19:0/1560194874 wait complete. 2024-08-31T12:23:56.884 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:56.882+0000 7fb145990640 1 Processor -- start 2024-08-31T12:23:56.884 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:56.883+0000 7fb145990640 1 -- start start 2024-08-31T12:23:56.887 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:56.884+0000 7fb145990640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb140154af0 0x7fb1400c7250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:56.887 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:56.884+0000 7fb145990640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb14015ab90 0x7fb140158890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:56.887 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:56.884+0000 7fb145990640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb14015fe10 0x7fb14015a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:56.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:56.884+0000 7fb145990640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fb1400cbdd0 con 0x7fb14015fe10 2024-08-31T12:23:56.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:56.884+0000 7fb145990640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb1400c3930 con 0x7fb140154af0 2024-08-31T12:23:56.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:56.884+0000 7fb145990640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fb14015b4e0 con 0x7fb14015ab90 2024-08-31T12:23:56.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:56.884+0000 7fb13f7fe640 1 -- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb14015fe10 msgr2=0x7fb14015a110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:3300/0 2024-08-31T12:23:56.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:56.884+0000 7fb13f7fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb14015fe10 0x7fb14015a110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:23:56.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:56.884+0000 7fb13e7fc640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb14015ab90 0x7fb140158890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:56.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:56.884+0000 7fb13e7fc640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb14015ab90 0x7fb140158890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.19:33428/0 (socket says 172.21.15.19:33428) 2024-08-31T12:23:56.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:56.884+0000 7fb13e7fc640 1 -- 172.21.15.19:0/4175993754 learned_addr learned my addr 172.21.15.19:0/4175993754 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:23:56.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:56.884+0000 7fb13effd640 1 --2- 172.21.15.19:0/4175993754 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb140154af0 0x7fb1400c7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:56.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:56.884+0000 7fb13e7fc640 1 -- 172.21.15.19:0/4175993754 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb140154af0 msgr2=0x7fb1400c7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:56.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:56.884+0000 7fb13e7fc640 1 --2- 172.21.15.19:0/4175993754 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb140154af0 0x7fb1400c7250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:56.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:56.884+0000 7fb13e7fc640 1 -- 172.21.15.19:0/4175993754 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb14015fe10 msgr2=0x7fb14015a110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:23:56.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:56.884+0000 7fb13e7fc640 1 --2- 172.21.15.19:0/4175993754 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb14015fe10 0x7fb14015a110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:56.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:56.884+0000 7fb13e7fc640 1 -- 172.21.15.19:0/4175993754 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1401475e0 con 0x7fb14015ab90 2024-08-31T12:23:56.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:56.885+0000 7fb13e7fc640 1 --2- 172.21.15.19:0/4175993754 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb14015ab90 0x7fb140158890 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb13400e9f0 tx=0x7fb13400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:56.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:56.885+0000 7fb127fff640 1 -- 172.21.15.19:0/4175993754 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb13400ccc0 con 0x7fb14015ab90 2024-08-31T12:23:56.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:56.885+0000 7fb127fff640 1 -- 172.21.15.19:0/4175993754 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb13400ce60 con 0x7fb14015ab90 2024-08-31T12:23:56.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:56.885+0000 7fb127fff640 1 -- 172.21.15.19:0/4175993754 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb134004260 con 0x7fb14015ab90 2024-08-31T12:23:56.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:56.885+0000 7fb145990640 1 -- 172.21.15.19:0/4175993754 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1400c5c60 con 0x7fb14015ab90 2024-08-31T12:23:56.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:56.885+0000 7fb145990640 1 -- 172.21.15.19:0/4175993754 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb140159420 con 0x7fb14015ab90 2024-08-31T12:23:56.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:56.886+0000 7fb145990640 1 -- 172.21.15.19:0/4175993754 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb140154f00 con 0x7fb14015ab90 2024-08-31T12:23:56.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:56.887+0000 7fb127fff640 1 -- 172.21.15.19:0/4175993754 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fb1340035f0 con 0x7fb14015ab90 2024-08-31T12:23:56.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:56.887+0000 7fb127fff640 1 --2- 172.21.15.19:0/4175993754 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb12c077a00 0x7fb12c079f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:23:56.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:56.887+0000 7fb127fff640 1 -- 172.21.15.19:0/4175993754 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fb13409a030 con 0x7fb14015ab90 2024-08-31T12:23:56.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:56.888+0000 7fb13effd640 1 --2- 172.21.15.19:0/4175993754 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb12c077a00 0x7fb12c079f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:23:56.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:56.888+0000 7fb13effd640 1 --2- 172.21.15.19:0/4175993754 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb12c077a00 0x7fb12c079f10 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb130000fb0 tx=0x7fb130009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:23:56.894 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:56.892+0000 7fb127fff640 1 -- 172.21.15.19:0/4175993754 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fb134062d20 con 0x7fb14015ab90 2024-08-31T12:23:57.095 DEBUG:teuthology.orchestra.run.smithi019:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:23:57.098 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:23:57.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:57.168+0000 7fb145990640 1 -- 172.21.15.19:0/4175993754 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb14006a540 con 0x7fb14015ab90 2024-08-31T12:23:57.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:57.168+0000 7fb127fff640 1 -- 172.21.15.19:0/4175993754 <== mon.2 v2:172.21.15.43:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fb134062620 con 0x7fb14015ab90 2024-08-31T12:23:57.171 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":56,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":23,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:23:57.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:57.171+0000 7fb145990640 1 -- 172.21.15.19:0/4175993754 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb12c077a00 msgr2=0x7fb12c079f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:57.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:57.171+0000 7fb145990640 1 --2- 172.21.15.19:0/4175993754 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb12c077a00 0x7fb12c079f10 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb130000fb0 tx=0x7fb130009190 comp rx=0 tx=0).stop 2024-08-31T12:23:57.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:57.171+0000 7fb145990640 1 -- 172.21.15.19:0/4175993754 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb14015ab90 msgr2=0x7fb140158890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:23:57.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:57.171+0000 7fb145990640 1 --2- 172.21.15.19:0/4175993754 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb14015ab90 0x7fb140158890 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb13400e9f0 tx=0x7fb13400eeb0 comp rx=0 tx=0).stop 2024-08-31T12:23:57.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:57.171+0000 7fb145990640 1 -- 172.21.15.19:0/4175993754 shutdown_connections 2024-08-31T12:23:57.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:57.171+0000 7fb145990640 1 --2- 172.21.15.19:0/4175993754 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb12c077a00 0x7fb12c079f10 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:57.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:57.171+0000 7fb145990640 1 --2- 172.21.15.19:0/4175993754 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb14015fe10 0x7fb14015a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:57.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:57.171+0000 7fb145990640 1 --2- 172.21.15.19:0/4175993754 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb14015ab90 0x7fb140158890 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:57.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:57.171+0000 7fb145990640 1 --2- 172.21.15.19:0/4175993754 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb140154af0 0x7fb1400c7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:23:57.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:57.171+0000 7fb145990640 1 -- 172.21.15.19:0/4175993754 >> 172.21.15.19:0/4175993754 conn(0x7fb1400710e0 msgr2=0x7fb1400714f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:23:57.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:57.171+0000 7fb145990640 1 -- 172.21.15.19:0/4175993754 shutdown_connections 2024-08-31T12:23:57.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:23:57.171+0000 7fb145990640 1 -- 172.21.15.19:0/4175993754 wait complete. 2024-08-31T12:24:00.186 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:24:04.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.705+0000 7f1954825640 1 -- 172.21.15.19:0/2258006414 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19300036f0 msgr2=0x7f1930005ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:04.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.705+0000 7f1954825640 1 --2- 172.21.15.19:0/2258006414 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19300036f0 0x7f1930005ba0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1930003680 tx=0x7f194002c460 comp rx=0 tx=0).stop 2024-08-31T12:24:04.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.705+0000 7f194ed9d640 1 -- 172.21.15.19:0/2258006414 reap_dead start 2024-08-31T12:24:04.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.705+0000 7f1954825640 1 -- 172.21.15.19:0/2258006414 shutdown_connections 2024-08-31T12:24:04.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.705+0000 7f1954825640 1 --2- 172.21.15.19:0/2258006414 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19300036f0 0x7f1930005ba0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:04.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.705+0000 7f1954825640 1 -- 172.21.15.19:0/2258006414 >> 172.21.15.19:0/2258006414 conn(0x7f1950074810 msgr2=0x7f1950068aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:24:04.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.706+0000 7f1954825640 1 -- 172.21.15.19:0/2258006414 shutdown_connections 2024-08-31T12:24:04.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.706+0000 7f1954825640 1 -- 172.21.15.19:0/2258006414 wait complete. 2024-08-31T12:24:04.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.708+0000 7f1954825640 1 Processor -- start 2024-08-31T12:24:04.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.708+0000 7f1954825640 1 -- start start 2024-08-31T12:24:04.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.709+0000 7f1954825640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19300036f0 0x7f195015cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:04.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.709+0000 7f1954825640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1950159f90 0x7f19500c8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:04.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.709+0000 7f1954825640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f195015a3a0 0x7f19500c9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:04.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.709+0000 7f1954825640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f19501557e0 con 0x7f1950159f90 2024-08-31T12:24:04.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.709+0000 7f1954825640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1950155660 con 0x7f19300036f0 2024-08-31T12:24:04.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.709+0000 7f1954825640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f195006e1a0 con 0x7f195015a3a0 2024-08-31T12:24:04.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.709+0000 7f194dd9b640 1 -- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1950159f90 msgr2=0x7f19500c8de0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:3300/0 2024-08-31T12:24:04.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.709+0000 7f194dd9b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1950159f90 0x7f19500c8de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:24:04.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.709+0000 7f194e59c640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19300036f0 0x7f195015cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:04.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.710+0000 7f194ed9d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f195015a3a0 0x7f19500c9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:04.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.710+0000 7f194e59c640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19300036f0 0x7f195015cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:50968/0 (socket says 172.21.15.19:50968) 2024-08-31T12:24:04.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.710+0000 7f194e59c640 1 -- 172.21.15.19:0/3655243998 learned_addr learned my addr 172.21.15.19:0/3655243998 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:24:04.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.710+0000 7f194ed9d640 1 -- 172.21.15.19:0/3655243998 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19300036f0 msgr2=0x7f195015cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:04.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.710+0000 7f194ed9d640 1 --2- 172.21.15.19:0/3655243998 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19300036f0 0x7f195015cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:04.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.710+0000 7f194ed9d640 1 -- 172.21.15.19:0/3655243998 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1950159f90 msgr2=0x7f19500c8de0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:24:04.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.710+0000 7f194ed9d640 1 --2- 172.21.15.19:0/3655243998 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1950159f90 0x7f19500c8de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:04.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.710+0000 7f194ed9d640 1 -- 172.21.15.19:0/3655243998 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f195015d1c0 con 0x7f195015a3a0 2024-08-31T12:24:04.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.710+0000 7f194e59c640 1 --2- 172.21.15.19:0/3655243998 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19300036f0 0x7f195015cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:24:04.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.710+0000 7f194ed9d640 1 --2- 172.21.15.19:0/3655243998 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f195015a3a0 0x7f19500c9320 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f19500cb1f0 tx=0x7f194800cf00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:24:04.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.710+0000 7f193effd640 1 -- 172.21.15.19:0/3655243998 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1948017070 con 0x7f195015a3a0 2024-08-31T12:24:04.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.711+0000 7f193effd640 1 -- 172.21.15.19:0/3655243998 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f194800abd0 con 0x7f195015a3a0 2024-08-31T12:24:04.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.711+0000 7f193effd640 1 -- 172.21.15.19:0/3655243998 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1948007070 con 0x7f195015a3a0 2024-08-31T12:24:04.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.711+0000 7f1954825640 1 -- 172.21.15.19:0/3655243998 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19501f1a10 con 0x7f195015a3a0 2024-08-31T12:24:04.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.711+0000 7f1954825640 1 -- 172.21.15.19:0/3655243998 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f195015d9c0 con 0x7f195015a3a0 2024-08-31T12:24:04.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.712+0000 7f1954825640 1 -- 172.21.15.19:0/3655243998 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19500cb280 con 0x7f195015a3a0 2024-08-31T12:24:04.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.712+0000 7f193effd640 1 -- 172.21.15.19:0/3655243998 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f1948007210 con 0x7f195015a3a0 2024-08-31T12:24:04.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.713+0000 7f193effd640 1 --2- 172.21.15.19:0/3655243998 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1930078880 0x7f193007ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:04.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.713+0000 7f193effd640 1 -- 172.21.15.19:0/3655243998 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f1948098fe0 con 0x7f195015a3a0 2024-08-31T12:24:04.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.713+0000 7f194e59c640 1 --2- 172.21.15.19:0/3655243998 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1930078880 0x7f193007ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:04.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.714+0000 7f194e59c640 1 --2- 172.21.15.19:0/3655243998 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1930078880 0x7f193007ad90 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1940004610 tx=0x7f19400046c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:24:04.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.718+0000 7f193effd640 1 -- 172.21.15.19:0/3655243998 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f19480616b0 con 0x7f195015a3a0 2024-08-31T12:24:04.991 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.990+0000 7f1954825640 1 -- 172.21.15.19:0/3655243998 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1950158b40 con 0x7f195015a3a0 2024-08-31T12:24:04.992 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.991+0000 7f193effd640 1 -- 172.21.15.19:0/3655243998 <== mon.2 v2:172.21.15.43:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7f194806a1b0 con 0x7f195015a3a0 2024-08-31T12:24:04.992 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":58,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:24:04.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.993+0000 7f1954825640 1 -- 172.21.15.19:0/3655243998 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1930078880 msgr2=0x7f193007ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:04.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.993+0000 7f1954825640 1 --2- 172.21.15.19:0/3655243998 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1930078880 0x7f193007ad90 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1940004610 tx=0x7f19400046c0 comp rx=0 tx=0).stop 2024-08-31T12:24:04.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.993+0000 7f1954825640 1 -- 172.21.15.19:0/3655243998 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f195015a3a0 msgr2=0x7f19500c9320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:04.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.993+0000 7f1954825640 1 --2- 172.21.15.19:0/3655243998 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f195015a3a0 0x7f19500c9320 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f19500cb1f0 tx=0x7f194800cf00 comp rx=0 tx=0).stop 2024-08-31T12:24:04.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.993+0000 7f1954825640 1 -- 172.21.15.19:0/3655243998 shutdown_connections 2024-08-31T12:24:04.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.993+0000 7f1954825640 1 --2- 172.21.15.19:0/3655243998 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1930078880 0x7f193007ad90 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:04.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.993+0000 7f1954825640 1 --2- 172.21.15.19:0/3655243998 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f195015a3a0 0x7f19500c9320 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:04.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.993+0000 7f1954825640 1 --2- 172.21.15.19:0/3655243998 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1950159f90 0x7f19500c8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:04.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.993+0000 7f1954825640 1 --2- 172.21.15.19:0/3655243998 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f19300036f0 0x7f195015cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:04.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.993+0000 7f1954825640 1 -- 172.21.15.19:0/3655243998 >> 172.21.15.19:0/3655243998 conn(0x7f1950074810 msgr2=0x7f19500c56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:24:04.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.993+0000 7f1954825640 1 -- 172.21.15.19:0/3655243998 shutdown_connections 2024-08-31T12:24:04.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:04.993+0000 7f1954825640 1 -- 172.21.15.19:0/3655243998 wait complete. 2024-08-31T12:24:05.007 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-08-31T12:24:05.008 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-08-31T12:24:05.008 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-31T12:24:05.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.130+0000 7f97d5919640 1 -- 172.21.15.19:0/2314463375 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f97b8005350 msgr2=0x7f97b800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:05.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.130+0000 7f97d5919640 1 --2- 172.21.15.19:0/2314463375 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f97b8005350 0x7f97b800b9c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f97c800c640 tx=0x7f97b8005830 comp rx=0 tx=0).stop 2024-08-31T12:24:05.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.130+0000 7f97d5919640 1 -- 172.21.15.19:0/2314463375 shutdown_connections 2024-08-31T12:24:05.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.130+0000 7f97d5919640 1 --2- 172.21.15.19:0/2314463375 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f97b8005350 0x7f97b800b9c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:05.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.130+0000 7f97d5919640 1 -- 172.21.15.19:0/2314463375 >> 172.21.15.19:0/2314463375 conn(0x7f97d00cd7d0 msgr2=0x7f97d00cdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:24:05.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.130+0000 7f97d5919640 1 -- 172.21.15.19:0/2314463375 shutdown_connections 2024-08-31T12:24:05.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.130+0000 7f97d5919640 1 -- 172.21.15.19:0/2314463375 wait complete. 2024-08-31T12:24:05.133 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.132+0000 7f97d5919640 1 Processor -- start 2024-08-31T12:24:05.133 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.132+0000 7f97d5919640 1 -- start start 2024-08-31T12:24:05.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.133+0000 7f97d5919640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f97b8005350 0x7f97d01f3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:05.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.133+0000 7f97d5919640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f97d00b8f20 0x7f97d006ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:05.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.133+0000 7f97d5919640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f97d006d200 0x7f97d013fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:05.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.133+0000 7f97d5919640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f97d00c6920 con 0x7f97d00b8f20 2024-08-31T12:24:05.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.133+0000 7f97d5919640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f97d00c6c20 con 0x7f97d006d200 2024-08-31T12:24:05.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.133+0000 7f97d5919640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f97d015b2b0 con 0x7f97b8005350 2024-08-31T12:24:05.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.133+0000 7f97ce7fc640 1 -- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f97d00b8f20 msgr2=0x7f97d006ccc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:3300/0 2024-08-31T12:24:05.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.134+0000 7f97ce7fc640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f97d00b8f20 0x7f97d006ccc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:24:05.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.134+0000 7f97ceffd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f97b8005350 0x7f97d01f3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:05.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.134+0000 7f97ceffd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f97b8005350 0x7f97d01f3300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.19:56980/0 (socket says 172.21.15.19:56980) 2024-08-31T12:24:05.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.134+0000 7f97ceffd640 1 -- 172.21.15.19:0/3429541057 learned_addr learned my addr 172.21.15.19:0/3429541057 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:24:05.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.134+0000 7f97cf7fe640 1 --2- 172.21.15.19:0/3429541057 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f97d006d200 0x7f97d013fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:05.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.134+0000 7f97ceffd640 1 -- 172.21.15.19:0/3429541057 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f97d006d200 msgr2=0x7f97d013fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:05.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.134+0000 7f97ceffd640 1 --2- 172.21.15.19:0/3429541057 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f97d006d200 0x7f97d013fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:05.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.134+0000 7f97ceffd640 1 -- 172.21.15.19:0/3429541057 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f97d00b8f20 msgr2=0x7f97d006ccc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:24:05.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.134+0000 7f97ceffd640 1 --2- 172.21.15.19:0/3429541057 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f97d00b8f20 0x7f97d006ccc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:05.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.134+0000 7f97ceffd640 1 -- 172.21.15.19:0/3429541057 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97d006d610 con 0x7f97b8005350 2024-08-31T12:24:05.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.134+0000 7f97ceffd640 1 --2- 172.21.15.19:0/3429541057 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f97b8005350 0x7f97d01f3300 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f97c000b580 tx=0x7f97c000ba40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:24:05.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.135+0000 7f97b7fff640 1 -- 172.21.15.19:0/3429541057 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c0014070 con 0x7f97b8005350 2024-08-31T12:24:05.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.135+0000 7f97b7fff640 1 -- 172.21.15.19:0/3429541057 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f97c00109a0 con 0x7f97b8005350 2024-08-31T12:24:05.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.135+0000 7f97d5919640 1 -- 172.21.15.19:0/3429541057 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97d01f2820 con 0x7f97b8005350 2024-08-31T12:24:05.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.135+0000 7f97b7fff640 1 -- 172.21.15.19:0/3429541057 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c0010df0 con 0x7f97b8005350 2024-08-31T12:24:05.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.135+0000 7f97d5919640 1 -- 172.21.15.19:0/3429541057 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f97d0153b80 con 0x7f97b8005350 2024-08-31T12:24:05.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.136+0000 7f97d5919640 1 --2- 172.21.15.19:0/3429541057 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f97d0154020 0x7f97d01401c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:05.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.136+0000 7f97d5919640 1 -- 172.21.15.19:0/3429541057 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f97d0154480 con 0x7f97d0154020 2024-08-31T12:24:05.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.136+0000 7f97ce7fc640 1 --2- 172.21.15.19:0/3429541057 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f97d0154020 0x7f97d01401c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:05.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.136+0000 7f97ce7fc640 1 -- 172.21.15.19:0/3429541057 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f97d0152600 con 0x7f97d0154020 2024-08-31T12:24:05.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.137+0000 7f97b7fff640 1 -- 172.21.15.19:0/3429541057 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f97c0017070 con 0x7f97b8005350 2024-08-31T12:24:05.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.137+0000 7f97ce7fc640 1 --2- 172.21.15.19:0/3429541057 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f97d0154020 0x7f97d01401c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f97d0155690 tx=0x7f97c400e9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:24:05.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.137+0000 7f97b7fff640 1 --2- 172.21.15.19:0/3429541057 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f97bc077a30 0x7f97bc079f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:05.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.137+0000 7f97b7fff640 1 -- 172.21.15.19:0/3429541057 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f97c00994c0 con 0x7f97b8005350 2024-08-31T12:24:05.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.137+0000 7f97cf7fe640 1 --2- 172.21.15.19:0/3429541057 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f97bc077a30 0x7f97bc079f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:05.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.137+0000 7f97b7fff640 1 -- 172.21.15.19:0/3429541057 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c400ca20 con 0x7f97d0154020 2024-08-31T12:24:05.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.138+0000 7f97b7fff640 1 -- 172.21.15.19:0/3429541057 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f97c400cbc0 con 0x7f97d0154020 2024-08-31T12:24:05.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.138+0000 7f97b7fff640 1 -- 172.21.15.19:0/3429541057 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f97d0154020 msgr2=0x7f97d01401c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:05.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.138+0000 7f97b7fff640 1 --2- 172.21.15.19:0/3429541057 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f97d0154020 0x7f97d01401c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f97d0155690 tx=0x7f97c400e9f0 comp rx=0 tx=0).stop 2024-08-31T12:24:05.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.138+0000 7f97b7fff640 1 -- 172.21.15.19:0/3429541057 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f97d0154020 msgr2=0x7f97d01401c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:24:05.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.138+0000 7f97b7fff640 1 --2- 172.21.15.19:0/3429541057 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f97d0154020 0x7f97d01401c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:05.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.138+0000 7f97cf7fe640 1 --2- 172.21.15.19:0/3429541057 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f97bc077a30 0x7f97bc079f40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f97b800c650 tx=0x7f97c8005040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:24:05.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.144+0000 7f97d5919640 1 --2- 172.21.15.19:0/3429541057 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f97d01594b0 0x7f97d01598c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:05.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.144+0000 7f97d5919640 1 -- 172.21.15.19:0/3429541057 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f97d00c3ff0 con 0x7f97d01594b0 2024-08-31T12:24:05.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.144+0000 7f97ce7fc640 1 --2- 172.21.15.19:0/3429541057 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f97d01594b0 0x7f97d01598c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:05.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.144+0000 7f97ce7fc640 1 -- 172.21.15.19:0/3429541057 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f97d00c90d0 con 0x7f97d01594b0 2024-08-31T12:24:05.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.145+0000 7f97ce7fc640 1 --2- 172.21.15.19:0/3429541057 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f97d01594b0 0x7f97d01598c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f97c4018000 tx=0x7f97c40048a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:24:05.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.145+0000 7f97b7fff640 1 -- 172.21.15.19:0/3429541057 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c4004020 con 0x7f97d01594b0 2024-08-31T12:24:05.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.145+0000 7f97b7fff640 1 -- 172.21.15.19:0/3429541057 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5033 (secure 0 0 0) 0x7f97c4018640 con 0x7f97d01594b0 2024-08-31T12:24:05.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.145+0000 7f97b7fff640 1 -- 172.21.15.19:0/3429541057 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f97d01594b0 msgr2=0x7f97d01598c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:05.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.145+0000 7f97b7fff640 1 --2- 172.21.15.19:0/3429541057 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f97d01594b0 0x7f97d01598c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f97c4018000 tx=0x7f97c40048a0 comp rx=0 tx=0).stop 2024-08-31T12:24:05.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.145+0000 7f97b7fff640 1 -- 172.21.15.19:0/3429541057 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f97d01594b0 msgr2=0x7f97d01598c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:24:05.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.145+0000 7f97b7fff640 1 --2- 172.21.15.19:0/3429541057 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f97d01594b0 0x7f97d01598c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:05.147 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:24:05.147 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:24:05.147 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:24:05.147 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "leader", 2024-08-31T12:24:05.147 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 58, 2024-08-31T12:24:05.147 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 154618, 2024-08-31T12:24:05.147 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:24:05.148 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:24:05.148 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:24:05.148 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:24:05.148 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 1, 2024-08-31T12:24:05.148 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:24:05.148 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:24:05.148 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:24:05.148 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:24:05.148 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:24:05.148 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:24:05.148 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:24:05.148 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:24:05.148 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:24:05.149 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:24:05.149 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:24:05.149 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:24:05.149 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:24:05.149 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:24:05.149 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:24:05.149 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:24:05.149 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:24:05.149 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:24:05.149 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:24:05.149 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:24:05.149 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:24:05.149 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:24:05.149 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:24:05.150 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:24:05.150 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:24:05.150 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:24:05.150 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:24:05.150 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:24:05.150 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:24:05.150 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:05.150 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:24:05.150 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:24:05.150 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:24:05.150 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:24:05.150 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:24:05.150 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:24:05.151 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:24:05.151 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:24:05.151 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:24:05.151 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:24:05.151 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:24:05.151 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:24:05.151 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:24:05.151 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:24:05.151 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:24:05.151 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:24:05.151 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:24:05.151 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:24:05.151 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:24:05.151 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:24:05.151 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:24:05.152 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:24:05.152 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:24:05.152 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:24:05.152 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:24:05.152 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:24:05.152 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:24:05.152 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:24:05.152 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:24:05.152 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:24:05.152 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:05.152 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:24:05.152 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:05.152 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:24:05.153 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:24:05.153 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:24:05.153 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:24:05.153 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:05.153 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:24:05.153 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:24:05.153 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:24:05.153 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:05.153 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:05.153 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:24:05.153 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:24:05.153 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:24:05.153 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:24:05.153 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:24:05.154 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:05.154 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:24:05.154 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:24:05.154 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:24:05.154 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:24:05.154 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:24:05.154 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:05.154 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:05.154 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:24:05.154 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:24:05.154 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:24:05.154 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:24:05.154 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:05.154 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:24:05.155 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:24:05.155 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:24:05.155 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:05.155 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:05.155 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:24:05.155 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:24:05.155 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:24:05.155 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:24:05.155 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:24:05.155 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:05.155 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:24:05.155 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:24:05.155 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:24:05.155 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:24:05.156 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:24:05.156 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:05.156 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:05.156 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:24:05.156 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:24:05.156 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:24:05.156 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:24:05.156 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:05.156 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:24:05.156 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:24:05.156 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:24:05.156 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:05.156 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:05.156 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:24:05.157 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:24:05.157 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:24:05.157 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:24:05.157 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:24:05.157 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:05.157 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:24:05.157 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:24:05.157 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:24:05.157 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:24:05.157 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:24:05.157 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:24:05.157 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:24:05.157 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:05.157 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:24:05.158 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:24:05.158 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:05.158 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:24:05.158 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:24:05.158 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:24:05.158 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:24:05.158 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:24:05.158 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:24:05.158 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:05.158 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:24:05.158 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:24:05.158 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 3 2024-08-31T12:24:05.158 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:24:05.158 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:24:05.159 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:24:05.159 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:05.159 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:24:05.159 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:24:05.159 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 4 2024-08-31T12:24:05.159 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:24:05.159 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:24:05.159 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:24:05.159 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:05.159 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:24:05.159 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:24:05.159 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 6 2024-08-31T12:24:05.159 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:24:05.159 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:24:05.159 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:24:05.160 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:05.160 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:24:05.160 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:24:05.160 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:24:05.160 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:24:05.160 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:24:05.160 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:05.160 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:24:05.160 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:24:05.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.146+0000 7f97d5919640 1 -- 172.21.15.19:0/3429541057 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f97bc077a30 msgr2=0x7f97bc079f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:05.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.146+0000 7f97d5919640 1 --2- 172.21.15.19:0/3429541057 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f97bc077a30 0x7f97bc079f40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f97b800c650 tx=0x7f97c8005040 comp rx=0 tx=0).stop 2024-08-31T12:24:05.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.146+0000 7f97d5919640 1 -- 172.21.15.19:0/3429541057 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f97b8005350 msgr2=0x7f97d01f3300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:05.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.146+0000 7f97d5919640 1 --2- 172.21.15.19:0/3429541057 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f97b8005350 0x7f97d01f3300 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f97c000b580 tx=0x7f97c000ba40 comp rx=0 tx=0).stop 2024-08-31T12:24:05.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.146+0000 7f97cf7fe640 1 -- 172.21.15.19:0/3429541057 reap_dead start 2024-08-31T12:24:05.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.146+0000 7f97cf7fe640 1 -- 172.21.15.19:0/3429541057 reap_dead start 2024-08-31T12:24:05.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.146+0000 7f97d5919640 1 -- 172.21.15.19:0/3429541057 shutdown_connections 2024-08-31T12:24:05.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.146+0000 7f97d5919640 1 -- 172.21.15.19:0/3429541057 >> 172.21.15.19:0/3429541057 conn(0x7f97d00cd7d0 msgr2=0x7f97d0063b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:24:05.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.146+0000 7f97d5919640 1 -- 172.21.15.19:0/3429541057 shutdown_connections 2024-08-31T12:24:05.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.146+0000 7f97d5919640 1 -- 172.21.15.19:0/3429541057 wait complete. 2024-08-31T12:24:05.162 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-08-31T12:24:05.162 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-08-31T12:24:05.162 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-31T12:24:05.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.288+0000 7facac794640 1 -- 172.21.15.19:0/468131600 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac90004800 msgr2=0x7fac90004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:05.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.288+0000 7facac794640 1 --2- 172.21.15.19:0/468131600 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac90004800 0x7fac90004c10 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fac940010a0 tx=0x7fac90005830 comp rx=0 tx=0).stop 2024-08-31T12:24:05.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.288+0000 7facac794640 1 -- 172.21.15.19:0/468131600 shutdown_connections 2024-08-31T12:24:05.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.288+0000 7facac794640 1 --2- 172.21.15.19:0/468131600 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac90004800 0x7fac90004c10 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:05.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.288+0000 7facac794640 1 -- 172.21.15.19:0/468131600 >> 172.21.15.19:0/468131600 conn(0x7faca40690d0 msgr2=0x7faca40d0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:24:05.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.288+0000 7facac794640 1 -- 172.21.15.19:0/468131600 shutdown_connections 2024-08-31T12:24:05.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.288+0000 7facac794640 1 -- 172.21.15.19:0/468131600 wait complete. 2024-08-31T12:24:05.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.290+0000 7facac794640 1 Processor -- start 2024-08-31T12:24:05.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.290+0000 7facac794640 1 -- start start 2024-08-31T12:24:05.294 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.291+0000 7facac794640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fac90004800 0x7faca40c11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:05.294 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.291+0000 7facac794640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faca41e4040 0x7faca41e11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:05.294 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.291+0000 7facac794640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faca414cf60 0x7faca40bf810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:05.294 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.291+0000 7facac794640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7faca415b250 con 0x7fac90004800 2024-08-31T12:24:05.294 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.291+0000 7facac794640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7faca415af70 con 0x7faca41e4040 2024-08-31T12:24:05.294 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.291+0000 7facac794640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7faca40c8610 con 0x7faca414cf60 2024-08-31T12:24:05.294 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.291+0000 7facaa530640 1 -- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fac90004800 msgr2=0x7faca40c11c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:3300/0 2024-08-31T12:24:05.294 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.291+0000 7facaa530640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fac90004800 0x7faca40c11c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:24:05.295 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.292+0000 7faca9d2f640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faca41e4040 0x7faca41e11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:05.295 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.292+0000 7faca9d2f640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faca41e4040 0x7faca41e11c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:51022/0 (socket says 172.21.15.19:51022) 2024-08-31T12:24:05.295 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.292+0000 7faca9d2f640 1 -- 172.21.15.19:0/3537396310 learned_addr learned my addr 172.21.15.19:0/3537396310 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:24:05.295 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.292+0000 7facaad31640 1 --2- 172.21.15.19:0/3537396310 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faca414cf60 0x7faca40bf810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:05.295 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.292+0000 7faca9d2f640 1 -- 172.21.15.19:0/3537396310 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faca414cf60 msgr2=0x7faca40bf810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:05.295 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.292+0000 7faca9d2f640 1 --2- 172.21.15.19:0/3537396310 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faca414cf60 0x7faca40bf810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:05.295 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.292+0000 7faca9d2f640 1 -- 172.21.15.19:0/3537396310 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fac90004800 msgr2=0x7faca40c11c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:24:05.295 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.292+0000 7faca9d2f640 1 --2- 172.21.15.19:0/3537396310 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fac90004800 0x7faca40c11c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:05.295 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.292+0000 7faca9d2f640 1 -- 172.21.15.19:0/3537396310 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faca40cfb00 con 0x7faca41e4040 2024-08-31T12:24:05.295 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.292+0000 7faca9d2f640 1 --2- 172.21.15.19:0/3537396310 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faca41e4040 0x7faca41e11c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fac94009ab0 tx=0x7fac94030570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:24:05.295 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.292+0000 7fac9affd640 1 -- 172.21.15.19:0/3537396310 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac94042070 con 0x7faca41e4040 2024-08-31T12:24:05.295 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.292+0000 7fac9affd640 1 -- 172.21.15.19:0/3537396310 <== mon.1 v2:172.21.15.43:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fac94030750 con 0x7faca41e4040 2024-08-31T12:24:05.295 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.292+0000 7fac9affd640 1 -- 172.21.15.19:0/3537396310 <== mon.1 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac94037040 con 0x7faca41e4040 2024-08-31T12:24:05.295 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.293+0000 7facac794640 1 -- 172.21.15.19:0/3537396310 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faca40cfde0 con 0x7faca41e4040 2024-08-31T12:24:05.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.293+0000 7facac794640 1 -- 172.21.15.19:0/3537396310 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faca40c09c0 con 0x7faca41e4040 2024-08-31T12:24:05.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.293+0000 7facac794640 1 --2- 172.21.15.19:0/3537396310 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faca414d370 0x7faca40bfd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:05.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.293+0000 7facac794640 1 -- 172.21.15.19:0/3537396310 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7faca41e52e0 con 0x7faca414d370 2024-08-31T12:24:05.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.294+0000 7facaa530640 1 --2- 172.21.15.19:0/3537396310 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faca414d370 0x7faca40bfd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:05.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.294+0000 7facaa530640 1 -- 172.21.15.19:0/3537396310 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faca40c9a10 con 0x7faca414d370 2024-08-31T12:24:05.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.294+0000 7facaa530640 1 --2- 172.21.15.19:0/3537396310 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faca414d370 0x7faca40bfd50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7faca406f0b0 tx=0x7faca000ed80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:24:05.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.294+0000 7fac9affd640 1 -- 172.21.15.19:0/3537396310 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faca00095c0 con 0x7faca414d370 2024-08-31T12:24:05.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.295+0000 7fac9affd640 1 -- 172.21.15.19:0/3537396310 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7faca000cc80 con 0x7faca414d370 2024-08-31T12:24:05.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.295+0000 7fac9affd640 1 -- 172.21.15.19:0/3537396310 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faca414d370 msgr2=0x7faca40bfd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:05.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.295+0000 7fac9affd640 1 --2- 172.21.15.19:0/3537396310 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faca414d370 0x7faca40bfd50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7faca406f0b0 tx=0x7faca000ed80 comp rx=0 tx=0).stop 2024-08-31T12:24:05.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.295+0000 7fac9affd640 1 -- 172.21.15.19:0/3537396310 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faca414d370 msgr2=0x7faca40bfd50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:24:05.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.295+0000 7fac9affd640 1 --2- 172.21.15.19:0/3537396310 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faca414d370 0x7faca40bfd50 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:05.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.295+0000 7fac9affd640 1 -- 172.21.15.19:0/3537396310 <== mon.1 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fac94030b80 con 0x7faca41e4040 2024-08-31T12:24:05.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.295+0000 7fac9affd640 1 --2- 172.21.15.19:0/3537396310 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fac880774c0 0x7fac880799d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:05.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.295+0000 7fac9affd640 1 -- 172.21.15.19:0/3537396310 <== mon.1 v2:172.21.15.43:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fac940b9550 con 0x7faca41e4040 2024-08-31T12:24:05.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.296+0000 7facaa530640 1 --2- 172.21.15.19:0/3537396310 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fac880774c0 0x7fac880799d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:05.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.296+0000 7facaa530640 1 --2- 172.21.15.19:0/3537396310 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fac880774c0 0x7fac880799d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7faca0019000 tx=0x7faca0014000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:24:05.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.302+0000 7facac794640 1 --2- 172.21.15.19:0/3537396310 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faca41f4670 0x7faca41f4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:05.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.302+0000 7facac794640 1 -- 172.21.15.19:0/3537396310 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7faca40c0b90 con 0x7faca41f4670 2024-08-31T12:24:05.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.303+0000 7facaad31640 1 --2- 172.21.15.19:0/3537396310 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faca41f4670 0x7faca41f4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:05.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.303+0000 7facaad31640 1 -- 172.21.15.19:0/3537396310 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7faca40cf820 con 0x7faca41f4670 2024-08-31T12:24:05.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.303+0000 7facaad31640 1 --2- 172.21.15.19:0/3537396310 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faca41f4670 0x7faca41f4a80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7faca40d1620 tx=0x7fac9c00ff70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:24:05.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.303+0000 7fac9affd640 1 -- 172.21.15.19:0/3537396310 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac9c003b80 con 0x7faca41f4670 2024-08-31T12:24:05.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.303+0000 7fac9affd640 1 -- 172.21.15.19:0/3537396310 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5030 (secure 0 0 0) 0x7fac9c003b80 con 0x7faca41f4670 2024-08-31T12:24:05.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.303+0000 7fac9affd640 1 -- 172.21.15.19:0/3537396310 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faca41f4670 msgr2=0x7faca41f4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:05.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.303+0000 7fac9affd640 1 --2- 172.21.15.19:0/3537396310 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faca41f4670 0x7faca41f4a80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7faca40d1620 tx=0x7fac9c00ff70 comp rx=0 tx=0).stop 2024-08-31T12:24:05.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.303+0000 7fac9affd640 1 -- 172.21.15.19:0/3537396310 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faca41f4670 msgr2=0x7faca41f4a80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:24:05.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.303+0000 7fac9affd640 1 --2- 172.21.15.19:0/3537396310 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faca41f4670 0x7faca41f4a80 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:05.305 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:24:05.305 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:24:05.306 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:24:05.306 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:24:05.306 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 58, 2024-08-31T12:24:05.306 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 32660, 2024-08-31T12:24:05.306 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:24:05.306 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:24:05.306 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:24:05.306 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:24:05.306 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 1, 2024-08-31T12:24:05.306 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:24:05.306 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:24:05.306 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:24:05.306 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:24:05.307 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:24:05.307 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:24:05.307 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:24:05.307 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:24:05.307 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:24:05.307 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:24:05.307 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:24:05.307 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:24:05.307 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:24:05.307 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:24:05.307 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:24:05.307 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:24:05.307 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:24:05.308 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:24:05.308 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:24:05.308 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:24:05.308 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:24:05.308 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:24:05.308 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:24:05.308 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:24:05.308 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:24:05.308 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:24:05.308 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:24:05.308 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:24:05.308 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:24:05.308 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:05.309 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:24:05.309 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:24:05.309 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:24:05.309 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:24:05.309 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:24:05.309 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:24:05.309 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:24:05.309 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:24:05.309 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:24:05.309 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:24:05.309 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:24:05.309 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:24:05.309 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:24:05.309 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:24:05.310 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:24:05.310 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:24:05.310 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:24:05.310 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:24:05.310 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:24:05.310 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:24:05.310 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:24:05.310 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:24:05.310 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:24:05.310 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:24:05.310 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:24:05.310 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:24:05.310 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:24:05.310 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:24:05.310 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:24:05.311 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:24:05.311 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:05.311 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:24:05.311 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:05.311 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:24:05.311 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:24:05.311 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:24:05.311 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:24:05.311 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:05.311 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:24:05.311 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:24:05.311 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:24:05.311 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:05.312 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:05.312 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:24:05.312 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:24:05.312 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:24:05.312 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:24:05.312 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:24:05.312 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:05.312 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:24:05.312 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:24:05.312 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:24:05.312 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:24:05.312 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:24:05.312 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:05.312 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:05.313 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:24:05.313 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:24:05.313 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:24:05.313 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:24:05.313 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:05.313 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:24:05.313 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:24:05.313 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:24:05.313 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:05.313 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:05.313 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:24:05.313 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:24:05.313 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:24:05.314 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:24:05.314 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:24:05.314 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:05.314 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:24:05.314 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:24:05.314 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:24:05.314 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:24:05.314 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:24:05.314 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:05.314 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:05.314 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:24:05.314 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:24:05.314 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:24:05.314 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:24:05.315 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:05.315 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:24:05.315 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:24:05.315 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:24:05.315 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:05.315 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:05.315 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:24:05.315 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:24:05.315 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:24:05.315 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:24:05.315 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:24:05.315 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:05.315 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:24:05.315 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:24:05.316 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:24:05.316 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:24:05.316 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:24:05.316 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:24:05.316 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:24:05.316 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:05.316 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:24:05.316 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:24:05.316 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:05.316 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:24:05.316 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:24:05.316 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:24:05.316 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:24:05.316 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:24:05.317 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:24:05.317 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:05.317 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:24:05.317 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:24:05.317 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 3 2024-08-31T12:24:05.317 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:24:05.317 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:24:05.317 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:24:05.317 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:05.317 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:24:05.317 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:24:05.317 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 4 2024-08-31T12:24:05.317 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:24:05.318 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:24:05.318 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:24:05.318 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:05.318 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:24:05.318 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:24:05.318 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:24:05.318 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:24:05.318 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:24:05.318 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:24:05.318 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:05.318 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:24:05.318 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:24:05.318 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:24:05.318 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:24:05.319 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:24:05.319 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:05.319 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:24:05.319 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:24:05.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.304+0000 7facac794640 1 -- 172.21.15.19:0/3537396310 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fac880774c0 msgr2=0x7fac880799d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:05.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.304+0000 7facac794640 1 --2- 172.21.15.19:0/3537396310 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fac880774c0 0x7fac880799d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7faca0019000 tx=0x7faca0014000 comp rx=0 tx=0).stop 2024-08-31T12:24:05.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.304+0000 7facac794640 1 -- 172.21.15.19:0/3537396310 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faca41e4040 msgr2=0x7faca41e11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:05.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.304+0000 7facaad31640 1 -- 172.21.15.19:0/3537396310 reap_dead start 2024-08-31T12:24:05.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.304+0000 7facac794640 1 --2- 172.21.15.19:0/3537396310 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faca41e4040 0x7faca41e11c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fac94009ab0 tx=0x7fac94030570 comp rx=0 tx=0).stop 2024-08-31T12:24:05.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.304+0000 7facac794640 1 -- 172.21.15.19:0/3537396310 shutdown_connections 2024-08-31T12:24:05.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.304+0000 7facac794640 1 --2- 172.21.15.19:0/3537396310 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faca41e4040 0x7faca41e11c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:05.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.304+0000 7facac794640 1 -- 172.21.15.19:0/3537396310 >> 172.21.15.19:0/3537396310 conn(0x7faca40690d0 msgr2=0x7faca40c3050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:24:05.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.304+0000 7facac794640 1 -- 172.21.15.19:0/3537396310 shutdown_connections 2024-08-31T12:24:05.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:05.304+0000 7facac794640 1 -- 172.21.15.19:0/3537396310 wait complete. 2024-08-31T12:24:05.321 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-08-31T12:24:05.321 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-08-31T12:24:25.323 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-08-31T12:24:25.323 INFO:tasks.ceph.mon.a:Restarting daemon 2024-08-31T12:24:25.323 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-08-31T12:24:25.326 INFO:tasks.ceph.mon.a:Started 2024-08-31T12:24:25.326 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-08-31T12:24:25.326 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:24:25.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.459+0000 7fbbaf7fe640 1 -- 172.21.15.19:0/253129878 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb9c031bf0 con 0x7fbbb0157040 2024-08-31T12:24:25.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.459+0000 7fbbb6f57640 1 -- 172.21.15.19:0/253129878 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbbb0157040 msgr2=0x7fbbb0157450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:25.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.459+0000 7fbbb6f57640 1 --2- 172.21.15.19:0/253129878 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbbb0157040 0x7fbbb0157450 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbbb006d9d0 tx=0x7fbb9c02a4a0 comp rx=0 tx=0).stop 2024-08-31T12:24:25.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.459+0000 7fbbb6f57640 1 -- 172.21.15.19:0/253129878 shutdown_connections 2024-08-31T12:24:25.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.459+0000 7fbbb6f57640 1 --2- 172.21.15.19:0/253129878 >> v2:172.21.15.43:3300/0 conn(0x7fbbb0070710 0x7fbbb0070b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:25.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.459+0000 7fbbb6f57640 1 --2- 172.21.15.19:0/253129878 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbbb0157040 0x7fbbb0157450 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:25.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.460+0000 7fbbb6f57640 1 --2- 172.21.15.19:0/253129878 >> v2:172.21.15.19:3300/0 conn(0x7fbbb00c4d00 0x7fbbb0151250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:25.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.460+0000 7fbbb6f57640 1 -- 172.21.15.19:0/253129878 >> 172.21.15.19:0/253129878 conn(0x7fbbb006d110 msgr2=0x7fbbb00cc950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:24:25.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.460+0000 7fbbb6f57640 1 -- 172.21.15.19:0/253129878 shutdown_connections 2024-08-31T12:24:25.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.460+0000 7fbbb6f57640 1 -- 172.21.15.19:0/253129878 wait complete. 2024-08-31T12:24:25.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.462+0000 7fbbb6f57640 1 Processor -- start 2024-08-31T12:24:25.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.462+0000 7fbbb6f57640 1 -- start start 2024-08-31T12:24:25.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.463+0000 7fbbb6f57640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbbb0070710 0x7fbbb01f0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:25.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.463+0000 7fbbb6f57640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbbb00c4d00 0x7fbbb0154890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:25.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.463+0000 7fbbb6f57640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbbb0157040 0x7fbbb0148d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:25.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.463+0000 7fbbb6f57640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fbbb0074330 con 0x7fbbb00c4d00 2024-08-31T12:24:25.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.463+0000 7fbbb6f57640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fbbb0074920 con 0x7fbbb0157040 2024-08-31T12:24:25.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.463+0000 7fbbb6f57640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fbbb0072f30 con 0x7fbbb0070710 2024-08-31T12:24:25.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.463+0000 7fbbaffff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbbb00c4d00 0x7fbbb0154890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:25.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.463+0000 7fbbb4cf3640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbbb0070710 0x7fbbb01f0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:25.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.463+0000 7fbbaffff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbbb00c4d00 0x7fbbb0154890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:47640/0 (socket says 172.21.15.19:47640) 2024-08-31T12:24:25.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.463+0000 7fbbaffff640 1 -- 172.21.15.19:0/430592524 learned_addr learned my addr 172.21.15.19:0/430592524 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:24:25.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.463+0000 7fbbb54f4640 1 --2- 172.21.15.19:0/430592524 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbbb0157040 0x7fbbb0148d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:25.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.464+0000 7fbbaffff640 1 -- 172.21.15.19:0/430592524 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbbb00c4d00 msgr2=0x7fbbb0154890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-08-31T12:24:25.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.464+0000 7fbbaffff640 1 -- 172.21.15.19:0/430592524 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbbb00c4d00 msgr2=0x7fbbb0154890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-31T12:24:25.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.464+0000 7fbbaffff640 1 --2- 172.21.15.19:0/430592524 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbbb00c4d00 0x7fbbb0154890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-31T12:24:25.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.464+0000 7fbbaffff640 1 --2- 172.21.15.19:0/430592524 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbbb00c4d00 0x7fbbb0154890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:24:25.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.464+0000 7fbbb4cf3640 1 -- 172.21.15.19:0/430592524 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbbb0157040 msgr2=0x7fbbb0148d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:25.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.464+0000 7fbbb4cf3640 1 --2- 172.21.15.19:0/430592524 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbbb0157040 0x7fbbb0148d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:25.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.464+0000 7fbbb4cf3640 1 -- 172.21.15.19:0/430592524 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbbb00c4d00 msgr2=0x7fbbb0154890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:24:25.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.464+0000 7fbbb4cf3640 1 --2- 172.21.15.19:0/430592524 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbbb00c4d00 0x7fbbb0154890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:25.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.464+0000 7fbbb4cf3640 1 -- 172.21.15.19:0/430592524 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbb01edb70 con 0x7fbbb0070710 2024-08-31T12:24:25.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.464+0000 7fbbb4cf3640 1 --2- 172.21.15.19:0/430592524 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbbb0070710 0x7fbbb01f0270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbba4008b60 tx=0x7fbba4008f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:24:25.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.464+0000 7fbbad7fa640 1 -- 172.21.15.19:0/430592524 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbba401f070 con 0x7fbbb0070710 2024-08-31T12:24:25.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.464+0000 7fbbad7fa640 1 -- 172.21.15.19:0/430592524 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbba40116c0 con 0x7fbbb0070710 2024-08-31T12:24:25.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.464+0000 7fbbad7fa640 1 -- 172.21.15.19:0/430592524 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbba4011b10 con 0x7fbbb0070710 2024-08-31T12:24:25.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.464+0000 7fbbb6f57640 1 -- 172.21.15.19:0/430592524 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbb014fe70 con 0x7fbbb0070710 2024-08-31T12:24:25.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.464+0000 7fbbb6f57640 1 -- 172.21.15.19:0/430592524 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbb01ecdb0 con 0x7fbbb0070710 2024-08-31T12:24:25.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.465+0000 7fbbb6f57640 1 -- 172.21.15.19:0/430592524 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbb00c35b0 con 0x7fbbb0070710 2024-08-31T12:24:25.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.466+0000 7fbbad7fa640 1 -- 172.21.15.19:0/430592524 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fbba4022070 con 0x7fbbb0070710 2024-08-31T12:24:25.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.467+0000 7fbbad7fa640 1 --2- 172.21.15.19:0/430592524 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fbb94077a80 0x7fbb94079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:25.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.467+0000 7fbbad7fa640 1 -- 172.21.15.19:0/430592524 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fbba409d0d0 con 0x7fbbb0070710 2024-08-31T12:24:25.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.467+0000 7fbbaffff640 1 --2- 172.21.15.19:0/430592524 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fbb94077a80 0x7fbb94079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:25.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.468+0000 7fbbaffff640 1 --2- 172.21.15.19:0/430592524 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fbb94077a80 0x7fbb94079f90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c005eb0 tx=0x7fbb9c004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:24:25.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.472+0000 7fbbad7fa640 1 -- 172.21.15.19:0/430592524 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fbba4065cb0 con 0x7fbbb0070710 2024-08-31T12:24:25.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.776+0000 7fbbb6f57640 1 -- 172.21.15.19:0/430592524 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fbbb00c8d70 con 0x7fbbb0070710 2024-08-31T12:24:25.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.777+0000 7fbbad7fa640 1 -- 172.21.15.19:0/430592524 <== mon.2 v2:172.21.15.43:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7fbba40655b0 con 0x7fbbb0070710 2024-08-31T12:24:25.778 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":58,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:24:25.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.779+0000 7fbbb6f57640 1 -- 172.21.15.19:0/430592524 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fbb94077a80 msgr2=0x7fbb94079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:25.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.779+0000 7fbbb6f57640 1 --2- 172.21.15.19:0/430592524 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fbb94077a80 0x7fbb94079f90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c005eb0 tx=0x7fbb9c004730 comp rx=0 tx=0).stop 2024-08-31T12:24:25.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.779+0000 7fbbb6f57640 1 -- 172.21.15.19:0/430592524 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbbb0070710 msgr2=0x7fbbb01f0270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:25.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.779+0000 7fbbb6f57640 1 --2- 172.21.15.19:0/430592524 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbbb0070710 0x7fbbb01f0270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbba4008b60 tx=0x7fbba4008f60 comp rx=0 tx=0).stop 2024-08-31T12:24:25.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.779+0000 7fbbb6f57640 1 -- 172.21.15.19:0/430592524 shutdown_connections 2024-08-31T12:24:25.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.779+0000 7fbbb6f57640 1 --2- 172.21.15.19:0/430592524 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fbb94077a80 0x7fbb94079f90 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:25.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.779+0000 7fbbb6f57640 1 --2- 172.21.15.19:0/430592524 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbbb0157040 0x7fbbb0148d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:25.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.779+0000 7fbbb6f57640 1 --2- 172.21.15.19:0/430592524 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbbb00c4d00 0x7fbbb0154890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:25.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.779+0000 7fbbb6f57640 1 --2- 172.21.15.19:0/430592524 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbbb0070710 0x7fbbb01f0270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:25.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.779+0000 7fbbb6f57640 1 -- 172.21.15.19:0/430592524 >> 172.21.15.19:0/430592524 conn(0x7fbbb006d110 msgr2=0x7fbbb00cc760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:24:25.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.779+0000 7fbbb6f57640 1 -- 172.21.15.19:0/430592524 shutdown_connections 2024-08-31T12:24:25.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:25.779+0000 7fbbb6f57640 1 -- 172.21.15.19:0/430592524 wait complete. 2024-08-31T12:24:27.162 DEBUG:teuthology.orchestra.run.smithi019:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:24:27.165 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:24:28.794 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:24:28.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:28.911+0000 7f9bd14ab640 1 -- 172.21.15.19:0/1179017747 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9bcc071b40 msgr2=0x7f9bcc0caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:28.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:28.911+0000 7f9bd14ab640 1 --2- 172.21.15.19:0/1179017747 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9bcc071b40 0x7f9bcc0caa20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0009860 tx=0x7f9bc002a620 comp rx=0 tx=0).stop 2024-08-31T12:24:28.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:28.911+0000 7f9bd14ab640 1 -- 172.21.15.19:0/1179017747 shutdown_connections 2024-08-31T12:24:28.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:28.911+0000 7f9bd14ab640 1 --2- 172.21.15.19:0/1179017747 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9bcc071b40 0x7f9bcc0caa20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:28.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:28.911+0000 7f9bd14ab640 1 -- 172.21.15.19:0/1179017747 >> 172.21.15.19:0/1179017747 conn(0x7f9bcc072c80 msgr2=0x7f9bcc073090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:24:28.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:28.912+0000 7f9bd14ab640 1 -- 172.21.15.19:0/1179017747 shutdown_connections 2024-08-31T12:24:28.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:28.912+0000 7f9bd14ab640 1 -- 172.21.15.19:0/1179017747 wait complete. 2024-08-31T12:24:28.914 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:28.914+0000 7f9bd14ab640 1 Processor -- start 2024-08-31T12:24:28.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:28.914+0000 7f9bd14ab640 1 -- start start 2024-08-31T12:24:28.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:28.915+0000 7f9bd14ab640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9bcc071b40 0x7f9bcc0ccd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:28.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:28.915+0000 7f9bd14ab640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9bcc153230 0x7f9bcc0ce560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:28.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:28.915+0000 7f9bd14ab640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9bcc159470 0x7f9bcc1f2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:28.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:28.915+0000 7f9bd14ab640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f9bcc072360 con 0x7f9bcc071b40 2024-08-31T12:24:28.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:28.915+0000 7f9bd14ab640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9bcc0721e0 con 0x7f9bcc159470 2024-08-31T12:24:28.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:28.915+0000 7f9bd14ab640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f9bcc0711e0 con 0x7f9bcc153230 2024-08-31T12:24:28.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:28.915+0000 7f9bcaffd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9bcc071b40 0x7f9bcc0ccd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:28.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:28.915+0000 7f9bcaffd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9bcc071b40 0x7f9bcc0ccd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:34694/0 (socket says 172.21.15.19:34694) 2024-08-31T12:24:28.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:28.915+0000 7f9bcaffd640 1 -- 172.21.15.19:0/2294951905 learned_addr learned my addr 172.21.15.19:0/2294951905 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:24:28.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:28.915+0000 7f9bca7fc640 1 --2- 172.21.15.19:0/2294951905 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9bcc153230 0x7f9bcc0ce560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:28.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:28.915+0000 7f9bcb7fe640 1 --2- 172.21.15.19:0/2294951905 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9bcc159470 0x7f9bcc1f2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:28.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:28.915+0000 7f9bcaffd640 1 -- 172.21.15.19:0/2294951905 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9bcc071b40 msgr2=0x7f9bcc0ccd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-08-31T12:24:28.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:28.915+0000 7f9bcaffd640 1 -- 172.21.15.19:0/2294951905 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9bcc071b40 msgr2=0x7f9bcc0ccd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-31T12:24:28.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:28.915+0000 7f9bcaffd640 1 --2- 172.21.15.19:0/2294951905 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9bcc071b40 0x7f9bcc0ccd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-31T12:24:28.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:28.916+0000 7f9bcaffd640 1 --2- 172.21.15.19:0/2294951905 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9bcc071b40 0x7f9bcc0ccd40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:24:28.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:28.916+0000 7f9bca7fc640 1 -- 172.21.15.19:0/2294951905 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9bcc159470 msgr2=0x7f9bcc1f2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:28.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:28.916+0000 7f9bca7fc640 1 --2- 172.21.15.19:0/2294951905 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9bcc159470 0x7f9bcc1f2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:28.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:28.916+0000 7f9bca7fc640 1 -- 172.21.15.19:0/2294951905 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9bcc071b40 msgr2=0x7f9bcc0ccd40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:24:28.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:28.916+0000 7f9bca7fc640 1 --2- 172.21.15.19:0/2294951905 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9bcc071b40 0x7f9bcc0ccd40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:28.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:28.916+0000 7f9bca7fc640 1 -- 172.21.15.19:0/2294951905 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bcc0cd280 con 0x7f9bcc153230 2024-08-31T12:24:28.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:28.916+0000 7f9bca7fc640 1 --2- 172.21.15.19:0/2294951905 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9bcc153230 0x7f9bcc0ce560 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0031000 tx=0x7f9bc0033ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:24:28.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:28.916+0000 7f9bb37fe640 1 -- 172.21.15.19:0/2294951905 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc0031b50 con 0x7f9bcc153230 2024-08-31T12:24:28.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:28.916+0000 7f9bb37fe640 1 -- 172.21.15.19:0/2294951905 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9bc0031de0 con 0x7f9bcc153230 2024-08-31T12:24:28.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:28.916+0000 7f9bd14ab640 1 -- 172.21.15.19:0/2294951905 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bcc0cc580 con 0x7f9bcc153230 2024-08-31T12:24:28.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:28.916+0000 7f9bb37fe640 1 -- 172.21.15.19:0/2294951905 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc0005280 con 0x7f9bcc153230 2024-08-31T12:24:28.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:28.916+0000 7f9bd14ab640 1 -- 172.21.15.19:0/2294951905 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bcc1f05c0 con 0x7f9bcc153230 2024-08-31T12:24:28.920 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:28.917+0000 7f9bd14ab640 1 -- 172.21.15.19:0/2294951905 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bcc0cf090 con 0x7f9bcc153230 2024-08-31T12:24:28.920 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:28.918+0000 7f9bb37fe640 1 -- 172.21.15.19:0/2294951905 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f9bc0005420 con 0x7f9bcc153230 2024-08-31T12:24:28.920 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:28.919+0000 7f9bb37fe640 1 --2- 172.21.15.19:0/2294951905 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9bb8077a50 0x7f9bb8079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:28.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:28.919+0000 7f9bb37fe640 1 -- 172.21.15.19:0/2294951905 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f9bc00bab00 con 0x7f9bcc153230 2024-08-31T12:24:28.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:28.919+0000 7f9bcaffd640 1 --2- 172.21.15.19:0/2294951905 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9bb8077a50 0x7f9bb8079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:28.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:28.919+0000 7f9bcaffd640 1 --2- 172.21.15.19:0/2294951905 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9bb8077a50 0x7f9bb8079f60 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc007d70 tx=0x7f9bbc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:24:28.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:28.924+0000 7f9bb37fe640 1 -- 172.21.15.19:0/2294951905 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f9bc003f030 con 0x7f9bcc153230 2024-08-31T12:24:29.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:29.218+0000 7f9bd14ab640 1 -- 172.21.15.19:0/2294951905 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9bcc0ca110 con 0x7f9bcc153230 2024-08-31T12:24:29.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:29.219+0000 7f9bb37fe640 1 -- 172.21.15.19:0/2294951905 <== mon.2 v2:172.21.15.43:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f9bc00bae20 con 0x7f9bcc153230 2024-08-31T12:24:29.220 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":58,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":25,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:24:29.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:29.221+0000 7f9bd14ab640 1 -- 172.21.15.19:0/2294951905 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9bb8077a50 msgr2=0x7f9bb8079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:29.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:29.221+0000 7f9bd14ab640 1 --2- 172.21.15.19:0/2294951905 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9bb8077a50 0x7f9bb8079f60 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc007d70 tx=0x7f9bbc009040 comp rx=0 tx=0).stop 2024-08-31T12:24:29.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:29.221+0000 7f9bd14ab640 1 -- 172.21.15.19:0/2294951905 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9bcc153230 msgr2=0x7f9bcc0ce560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:29.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:29.221+0000 7f9bd14ab640 1 --2- 172.21.15.19:0/2294951905 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9bcc153230 0x7f9bcc0ce560 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0031000 tx=0x7f9bc0033ba0 comp rx=0 tx=0).stop 2024-08-31T12:24:29.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:29.221+0000 7f9bd14ab640 1 -- 172.21.15.19:0/2294951905 shutdown_connections 2024-08-31T12:24:29.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:29.221+0000 7f9bd14ab640 1 --2- 172.21.15.19:0/2294951905 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9bb8077a50 0x7f9bb8079f60 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:29.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:29.221+0000 7f9bd14ab640 1 --2- 172.21.15.19:0/2294951905 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9bcc159470 0x7f9bcc1f2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:29.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:29.221+0000 7f9bd14ab640 1 --2- 172.21.15.19:0/2294951905 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9bcc153230 0x7f9bcc0ce560 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:29.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:29.221+0000 7f9bd14ab640 1 --2- 172.21.15.19:0/2294951905 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9bcc071b40 0x7f9bcc0ccd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:29.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:29.221+0000 7f9bd14ab640 1 -- 172.21.15.19:0/2294951905 >> 172.21.15.19:0/2294951905 conn(0x7f9bcc072c80 msgr2=0x7f9bcc14df90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:24:29.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:29.221+0000 7f9bd14ab640 1 -- 172.21.15.19:0/2294951905 shutdown_connections 2024-08-31T12:24:29.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:29.221+0000 7f9bd14ab640 1 -- 172.21.15.19:0/2294951905 wait complete. 2024-08-31T12:24:32.235 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:24:50.892 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:50.890+0000 7fc14879f640 1 -- 172.21.15.19:0/2305233627 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc12c0048c0 msgr2=0x7fc12c004d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:50.893 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:50.891+0000 7fc14879f640 1 --2- 172.21.15.19:0/2305233627 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc12c0048c0 0x7fc12c004d90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc124010d60 tx=0x7fc12c005830 comp rx=0 tx=0).stop 2024-08-31T12:24:50.893 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:50.891+0000 7fc14879f640 1 -- 172.21.15.19:0/2305233627 shutdown_connections 2024-08-31T12:24:50.893 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:50.891+0000 7fc14879f640 1 --2- 172.21.15.19:0/2305233627 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc12c0048c0 0x7fc12c004d90 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:50.893 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:50.891+0000 7fc14879f640 1 -- 172.21.15.19:0/2305233627 >> 172.21.15.19:0/2305233627 conn(0x7fc140073d40 msgr2=0x7fc1400ca000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:24:50.893 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:50.891+0000 7fc14879f640 1 -- 172.21.15.19:0/2305233627 shutdown_connections 2024-08-31T12:24:50.893 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:50.891+0000 7fc14879f640 1 -- 172.21.15.19:0/2305233627 wait complete. 2024-08-31T12:24:50.894 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:50.893+0000 7fc14879f640 1 Processor -- start 2024-08-31T12:24:50.894 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:50.893+0000 7fc14879f640 1 -- start start 2024-08-31T12:24:50.896 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:50.894+0000 7fc14879f640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc12c0048c0 0x7fc1401f1a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:50.896 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:50.894+0000 7fc14879f640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc14014c880 0x7fc140147460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:50.897 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:50.894+0000 7fc14879f640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc14014dac0 0x7fc14014af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:50.897 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:50.894+0000 7fc14879f640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fc1400c84c0 con 0x7fc14014dac0 2024-08-31T12:24:50.897 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:50.894+0000 7fc14879f640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc1400caed0 con 0x7fc12c0048c0 2024-08-31T12:24:50.897 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:50.894+0000 7fc14879f640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fc1400c8640 con 0x7fc14014c880 2024-08-31T12:24:50.897 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:50.894+0000 7fc146d3c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc14014dac0 0x7fc14014af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:50.897 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:50.894+0000 7fc146d3c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc14014dac0 0x7fc14014af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:52254/0 (socket says 172.21.15.19:52254) 2024-08-31T12:24:50.897 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:50.894+0000 7fc146d3c640 1 -- 172.21.15.19:0/2523127493 learned_addr learned my addr 172.21.15.19:0/2523127493 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:24:50.897 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:50.894+0000 7fc145d3a640 1 --2- 172.21.15.19:0/2523127493 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc14014c880 0x7fc140147460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:50.897 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:50.894+0000 7fc14653b640 1 --2- 172.21.15.19:0/2523127493 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc12c0048c0 0x7fc1401f1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:50.897 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:50.895+0000 7fc145d3a640 1 -- 172.21.15.19:0/2523127493 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc12c0048c0 msgr2=0x7fc1401f1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:50.897 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:50.895+0000 7fc145d3a640 1 --2- 172.21.15.19:0/2523127493 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc12c0048c0 0x7fc1401f1a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:50.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:50.895+0000 7fc145d3a640 1 -- 172.21.15.19:0/2523127493 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc14014dac0 msgr2=0x7fc14014af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:50.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:50.895+0000 7fc145d3a640 1 --2- 172.21.15.19:0/2523127493 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc14014dac0 0x7fc14014af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:50.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:50.895+0000 7fc145d3a640 1 -- 172.21.15.19:0/2523127493 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1401f0ff0 con 0x7fc14014c880 2024-08-31T12:24:50.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:50.895+0000 7fc146d3c640 1 --2- 172.21.15.19:0/2523127493 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc14014dac0 0x7fc14014af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:24:50.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:50.895+0000 7fc145d3a640 1 --2- 172.21.15.19:0/2523127493 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc14014c880 0x7fc140147460 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc130034000 tx=0x7fc130005af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:24:50.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:50.895+0000 7fc136ffd640 1 -- 172.21.15.19:0/2523127493 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc130039050 con 0x7fc14014c880 2024-08-31T12:24:50.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:50.895+0000 7fc136ffd640 1 -- 172.21.15.19:0/2523127493 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc13003a040 con 0x7fc14014c880 2024-08-31T12:24:50.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:50.895+0000 7fc14879f640 1 -- 172.21.15.19:0/2523127493 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc140152a00 con 0x7fc14014c880 2024-08-31T12:24:50.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:50.895+0000 7fc136ffd640 1 -- 172.21.15.19:0/2523127493 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1300406b0 con 0x7fc14014c880 2024-08-31T12:24:50.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:50.895+0000 7fc14879f640 1 -- 172.21.15.19:0/2523127493 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1401f4670 con 0x7fc14014c880 2024-08-31T12:24:50.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:50.896+0000 7fc14879f640 1 -- 172.21.15.19:0/2523127493 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1400ca8a0 con 0x7fc14014c880 2024-08-31T12:24:50.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:50.897+0000 7fc136ffd640 1 -- 172.21.15.19:0/2523127493 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fc130047020 con 0x7fc14014c880 2024-08-31T12:24:50.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:50.897+0000 7fc136ffd640 1 --2- 172.21.15.19:0/2523127493 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc12407a250 0x7fc12407c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:50.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:50.897+0000 7fc136ffd640 1 -- 172.21.15.19:0/2523127493 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fc1300bc550 con 0x7fc14014c880 2024-08-31T12:24:50.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:50.898+0000 7fc14653b640 1 --2- 172.21.15.19:0/2523127493 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc12407a250 0x7fc12407c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:50.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:50.898+0000 7fc14653b640 1 --2- 172.21.15.19:0/2523127493 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc12407a250 0x7fc12407c760 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fc13c001eb0 tx=0x7fc13c016040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:24:50.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:50.903+0000 7fc136ffd640 1 -- 172.21.15.19:0/2523127493 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fc13004e020 con 0x7fc14014c880 2024-08-31T12:24:51.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.159+0000 7fc14879f640 1 -- 172.21.15.19:0/2523127493 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc1400c8b20 con 0x7fc14014c880 2024-08-31T12:24:51.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.160+0000 7fc136ffd640 1 -- 172.21.15.19:0/2523127493 <== mon.2 v2:172.21.15.43:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fc1300bae20 con 0x7fc14014c880 2024-08-31T12:24:51.161 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":74,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:24:51.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.163+0000 7fc14879f640 1 -- 172.21.15.19:0/2523127493 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc12407a250 msgr2=0x7fc12407c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:51.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.163+0000 7fc14879f640 1 --2- 172.21.15.19:0/2523127493 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc12407a250 0x7fc12407c760 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fc13c001eb0 tx=0x7fc13c016040 comp rx=0 tx=0).stop 2024-08-31T12:24:51.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.163+0000 7fc14879f640 1 -- 172.21.15.19:0/2523127493 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc14014c880 msgr2=0x7fc140147460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:51.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.163+0000 7fc14879f640 1 --2- 172.21.15.19:0/2523127493 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc14014c880 0x7fc140147460 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc130034000 tx=0x7fc130005af0 comp rx=0 tx=0).stop 2024-08-31T12:24:51.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.163+0000 7fc14879f640 1 -- 172.21.15.19:0/2523127493 shutdown_connections 2024-08-31T12:24:51.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.163+0000 7fc14879f640 1 --2- 172.21.15.19:0/2523127493 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc12407a250 0x7fc12407c760 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:51.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.163+0000 7fc14879f640 1 --2- 172.21.15.19:0/2523127493 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc14014dac0 0x7fc14014af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:51.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.163+0000 7fc14879f640 1 --2- 172.21.15.19:0/2523127493 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc14014c880 0x7fc140147460 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:51.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.163+0000 7fc14879f640 1 --2- 172.21.15.19:0/2523127493 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc12c0048c0 0x7fc1401f1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:51.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.163+0000 7fc14879f640 1 -- 172.21.15.19:0/2523127493 >> 172.21.15.19:0/2523127493 conn(0x7fc140073d40 msgr2=0x7fc1401481b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:24:51.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.163+0000 7fc14879f640 1 -- 172.21.15.19:0/2523127493 shutdown_connections 2024-08-31T12:24:51.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.163+0000 7fc14879f640 1 -- 172.21.15.19:0/2523127493 wait complete. 2024-08-31T12:24:51.175 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-08-31T12:24:51.175 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-08-31T12:24:51.175 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-31T12:24:51.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.289+0000 7f00ca407640 1 -- 172.21.15.19:0/940909062 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f00c415b0a0 msgr2=0x7f00c406d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:51.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.289+0000 7f00ca407640 1 --2- 172.21.15.19:0/940909062 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f00c415b0a0 0x7f00c406d8e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f00c406f120 tx=0x7f00b802a530 comp rx=0 tx=0).stop 2024-08-31T12:24:51.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.289+0000 7f00ca407640 1 -- 172.21.15.19:0/940909062 shutdown_connections 2024-08-31T12:24:51.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.289+0000 7f00ca407640 1 --2- 172.21.15.19:0/940909062 >> v2:172.21.15.43:3300/0 conn(0x7f00c406eb00 0x7f00c406de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:51.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.289+0000 7f00ca407640 1 --2- 172.21.15.19:0/940909062 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f00c415b0a0 0x7f00c406d8e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:51.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.289+0000 7f00ca407640 1 -- 172.21.15.19:0/940909062 >> 172.21.15.19:0/940909062 conn(0x7f00c4069260 msgr2=0x7f00c40d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:24:51.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.289+0000 7f00ca407640 1 -- 172.21.15.19:0/940909062 shutdown_connections 2024-08-31T12:24:51.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.289+0000 7f00ca407640 1 -- 172.21.15.19:0/940909062 wait complete. 2024-08-31T12:24:51.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.291+0000 7f00ca407640 1 Processor -- start 2024-08-31T12:24:51.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.291+0000 7f00ca407640 1 -- start start 2024-08-31T12:24:51.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.292+0000 7f00ca407640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f00c406eb00 0x7f00c41f4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:51.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.292+0000 7f00ca407640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f00c40cc820 0x7f00c415a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:51.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.292+0000 7f00ca407640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f00c415b0a0 0x7f00c4151b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:51.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.292+0000 7f00ca407640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f00c40ccf90 con 0x7f00c40cc820 2024-08-31T12:24:51.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.292+0000 7f00ca407640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f00c40b4a40 con 0x7f00c406eb00 2024-08-31T12:24:51.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.292+0000 7f00ca407640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f00c40b5670 con 0x7f00c415b0a0 2024-08-31T12:24:51.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.292+0000 7f00c37fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f00c40cc820 0x7f00c415a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:51.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.292+0000 7f00c37fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f00c40cc820 0x7f00c415a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:52272/0 (socket says 172.21.15.19:52272) 2024-08-31T12:24:51.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.292+0000 7f00c37fe640 1 -- 172.21.15.19:0/2990764557 learned_addr learned my addr 172.21.15.19:0/2990764557 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:24:51.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.292+0000 7f00c89a4640 1 --2- 172.21.15.19:0/2990764557 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f00c415b0a0 0x7f00c4151b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:51.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.292+0000 7f00c3fff640 1 --2- 172.21.15.19:0/2990764557 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f00c406eb00 0x7f00c41f4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:51.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.293+0000 7f00c37fe640 1 -- 172.21.15.19:0/2990764557 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f00c415b0a0 msgr2=0x7f00c4151b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:51.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.293+0000 7f00c37fe640 1 --2- 172.21.15.19:0/2990764557 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f00c415b0a0 0x7f00c4151b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:51.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.293+0000 7f00c37fe640 1 -- 172.21.15.19:0/2990764557 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f00c406eb00 msgr2=0x7f00c41f4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:51.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.293+0000 7f00c37fe640 1 --2- 172.21.15.19:0/2990764557 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f00c406eb00 0x7f00c41f4f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:51.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.293+0000 7f00c37fe640 1 -- 172.21.15.19:0/2990764557 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00c4158670 con 0x7f00c40cc820 2024-08-31T12:24:51.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.293+0000 7f00c89a4640 1 --2- 172.21.15.19:0/2990764557 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f00c415b0a0 0x7f00c4151b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-31T12:24:51.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.293+0000 7f00c3fff640 1 --2- 172.21.15.19:0/2990764557 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f00c406eb00 0x7f00c41f4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-31T12:24:51.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.293+0000 7f00c37fe640 1 --2- 172.21.15.19:0/2990764557 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f00c40cc820 0x7f00c415a2a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f00b80098f0 tx=0x7f00b8033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:24:51.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.293+0000 7f00c0ff9640 1 -- 172.21.15.19:0/2990764557 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b8005c90 con 0x7f00c40cc820 2024-08-31T12:24:51.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.293+0000 7f00c0ff9640 1 -- 172.21.15.19:0/2990764557 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f00b8031e50 con 0x7f00c40cc820 2024-08-31T12:24:51.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.293+0000 7f00ca407640 1 -- 172.21.15.19:0/2990764557 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00c41f44c0 con 0x7f00c40cc820 2024-08-31T12:24:51.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.293+0000 7f00c0ff9640 1 -- 172.21.15.19:0/2990764557 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b8005260 con 0x7f00c40cc820 2024-08-31T12:24:51.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.293+0000 7f00ca407640 1 -- 172.21.15.19:0/2990764557 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00c41f4760 con 0x7f00c40cc820 2024-08-31T12:24:51.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.294+0000 7f00ca407640 1 --2- 172.21.15.19:0/2990764557 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f00c4152080 0x7f00c41f0f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:51.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.294+0000 7f00ca407640 1 -- 172.21.15.19:0/2990764557 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f00c4156f60 con 0x7f00c4152080 2024-08-31T12:24:51.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.294+0000 7f00c3fff640 1 --2- 172.21.15.19:0/2990764557 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f00c4152080 0x7f00c41f0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:51.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.294+0000 7f00c3fff640 1 -- 172.21.15.19:0/2990764557 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f00c41e4700 con 0x7f00c4152080 2024-08-31T12:24:51.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.294+0000 7f00c3fff640 1 --2- 172.21.15.19:0/2990764557 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f00c4152080 0x7f00c41f0f10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f00b400d4c0 tx=0x7f00b400d980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:24:51.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.295+0000 7f00c0ff9640 1 -- 172.21.15.19:0/2990764557 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b4005480 con 0x7f00c4152080 2024-08-31T12:24:51.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.295+0000 7f00c0ff9640 1 -- 172.21.15.19:0/2990764557 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f00b8005400 con 0x7f00c40cc820 2024-08-31T12:24:51.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.295+0000 7f00c0ff9640 1 --2- 172.21.15.19:0/2990764557 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f00b0077af0 0x7f00b007a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:51.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.295+0000 7f00c0ff9640 1 -- 172.21.15.19:0/2990764557 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f00b80ba880 con 0x7f00c40cc820 2024-08-31T12:24:51.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.295+0000 7f00c0ff9640 1 -- 172.21.15.19:0/2990764557 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f00b4017070 con 0x7f00c4152080 2024-08-31T12:24:51.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.295+0000 7f00c0ff9640 1 -- 172.21.15.19:0/2990764557 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f00c4152080 msgr2=0x7f00c41f0f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:51.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.295+0000 7f00c0ff9640 1 --2- 172.21.15.19:0/2990764557 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f00c4152080 0x7f00c41f0f10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f00b400d4c0 tx=0x7f00b400d980 comp rx=0 tx=0).stop 2024-08-31T12:24:51.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.295+0000 7f00c0ff9640 1 -- 172.21.15.19:0/2990764557 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f00c4152080 msgr2=0x7f00c41f0f10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:24:51.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.295+0000 7f00c0ff9640 1 --2- 172.21.15.19:0/2990764557 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f00c4152080 0x7f00c41f0f10 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f00b400d4c0 tx=0x7f00b400d980 comp rx=0 tx=0).stop 2024-08-31T12:24:51.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.295+0000 7f00c89a4640 1 --2- 172.21.15.19:0/2990764557 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f00b0077af0 0x7f00b007a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:51.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.296+0000 7f00c89a4640 1 --2- 172.21.15.19:0/2990764557 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f00b0077af0 0x7f00b007a000 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f00c415f670 tx=0x7f00bc009500 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:24:51.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.302+0000 7f00ca407640 1 --2- 172.21.15.19:0/2990764557 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f00c41f1450 0x7f00c41f1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:51.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.302+0000 7f00ca407640 1 -- 172.21.15.19:0/2990764557 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f00c4152490 con 0x7f00c41f1450 2024-08-31T12:24:51.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.302+0000 7f00c3fff640 1 --2- 172.21.15.19:0/2990764557 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f00c41f1450 0x7f00c41f1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:51.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.302+0000 7f00c3fff640 1 -- 172.21.15.19:0/2990764557 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f00c41f1da0 con 0x7f00c41f1450 2024-08-31T12:24:51.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.302+0000 7f00c3fff640 1 --2- 172.21.15.19:0/2990764557 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f00c41f1450 0x7f00c41f1860 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f00b400d4c0 tx=0x7f00b4001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:24:51.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.302+0000 7f00c0ff9640 1 -- 172.21.15.19:0/2990764557 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b401c030 con 0x7f00c41f1450 2024-08-31T12:24:51.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.302+0000 7f00c0ff9640 1 -- 172.21.15.19:0/2990764557 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4542 (secure 0 0 0) 0x7f00b4025450 con 0x7f00c41f1450 2024-08-31T12:24:51.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.302+0000 7f00c0ff9640 1 -- 172.21.15.19:0/2990764557 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f00c41f1450 msgr2=0x7f00c41f1860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:51.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.302+0000 7f00c0ff9640 1 --2- 172.21.15.19:0/2990764557 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f00c41f1450 0x7f00c41f1860 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f00b400d4c0 tx=0x7f00b4001080 comp rx=0 tx=0).stop 2024-08-31T12:24:51.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.303+0000 7f00c0ff9640 1 -- 172.21.15.19:0/2990764557 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f00c41f1450 msgr2=0x7f00c41f1860 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:24:51.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.303+0000 7f00c0ff9640 1 --2- 172.21.15.19:0/2990764557 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f00c41f1450 0x7f00c41f1860 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:51.305 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:24:51.305 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:24:51.305 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:24:51.305 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "leader", 2024-08-31T12:24:51.305 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 74, 2024-08-31T12:24:51.305 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 25856, 2024-08-31T12:24:51.305 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:24:51.305 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:24:51.305 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:24:51.305 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:24:51.305 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:24:51.305 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 0, 2024-08-31T12:24:51.306 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:24:51.306 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:24:51.306 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:24:51.306 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:24:51.306 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:24:51.306 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:24:51.306 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:24:51.306 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:24:51.306 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:24:51.306 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:24:51.306 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:24:51.306 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:24:51.306 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:24:51.306 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:24:51.307 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:24:51.307 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:24:51.307 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:24:51.307 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:24:51.307 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:24:51.307 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:24:51.307 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:24:51.307 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:24:51.307 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:24:51.307 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:24:51.307 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:24:51.307 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:24:51.307 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:24:51.307 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:24:51.308 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:24:51.308 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:51.308 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:24:51.308 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:24:51.308 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:24:51.308 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:24:51.308 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:24:51.308 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:24:51.308 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:24:51.308 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:24:51.308 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:24:51.308 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:24:51.308 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:24:51.309 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:24:51.309 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:24:51.309 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:24:51.309 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:24:51.309 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:24:51.309 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:24:51.309 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:24:51.309 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:24:51.309 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:24:51.309 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:24:51.309 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:24:51.309 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:24:51.309 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:24:51.309 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:24:51.310 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:24:51.310 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:24:51.310 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:24:51.310 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:24:51.310 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:24:51.310 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:51.310 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:24:51.310 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:51.310 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:24:51.310 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:24:51.310 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:24:51.310 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:24:51.310 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:51.310 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:24:51.311 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:24:51.311 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:24:51.311 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:51.311 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:51.311 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:24:51.311 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:24:51.311 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:24:51.311 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:24:51.311 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:24:51.311 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:51.311 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:24:51.311 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:24:51.311 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:24:51.312 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:24:51.312 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:24:51.312 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:51.312 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:51.312 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:24:51.312 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:24:51.312 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:24:51.312 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:24:51.312 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:51.312 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:24:51.312 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:24:51.312 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:24:51.312 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:51.312 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:51.313 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:24:51.313 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:24:51.313 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:24:51.313 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:24:51.313 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:24:51.313 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:51.313 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:24:51.313 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:24:51.313 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:24:51.313 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:24:51.313 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:24:51.313 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:51.313 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:51.314 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:24:51.314 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:24:51.314 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:24:51.314 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:24:51.314 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:51.314 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:24:51.314 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:24:51.314 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:24:51.314 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:51.314 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:51.314 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:24:51.314 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:24:51.314 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:24:51.315 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:24:51.315 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:24:51.315 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:51.315 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:24:51.315 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:24:51.315 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:24:51.315 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:24:51.315 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:24:51.315 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:24:51.315 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:24:51.315 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:51.315 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:24:51.315 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:24:51.315 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:51.315 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:24:51.315 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:24:51.316 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:24:51.316 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:24:51.316 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:24:51.316 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:24:51.316 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:51.316 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:24:51.316 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:24:51.316 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:24:51.316 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:24:51.316 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:24:51.316 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:51.316 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:24:51.316 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:24:51.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.303+0000 7f00ca407640 1 -- 172.21.15.19:0/2990764557 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f00b0077af0 msgr2=0x7f00b007a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:51.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.303+0000 7f00ca407640 1 --2- 172.21.15.19:0/2990764557 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f00b0077af0 0x7f00b007a000 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f00c415f670 tx=0x7f00bc009500 comp rx=0 tx=0).stop 2024-08-31T12:24:51.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.303+0000 7f00ca407640 1 -- 172.21.15.19:0/2990764557 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f00c40cc820 msgr2=0x7f00c415a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:51.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.303+0000 7f00ca407640 1 --2- 172.21.15.19:0/2990764557 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f00c40cc820 0x7f00c415a2a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f00b80098f0 tx=0x7f00b8033c20 comp rx=0 tx=0).stop 2024-08-31T12:24:51.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.303+0000 7f00c89a4640 1 -- 172.21.15.19:0/2990764557 reap_dead start 2024-08-31T12:24:51.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.303+0000 7f00c89a4640 1 -- 172.21.15.19:0/2990764557 reap_dead start 2024-08-31T12:24:51.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.303+0000 7f00ca407640 1 -- 172.21.15.19:0/2990764557 shutdown_connections 2024-08-31T12:24:51.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.303+0000 7f00ca407640 1 -- 172.21.15.19:0/2990764557 >> 172.21.15.19:0/2990764557 conn(0x7f00c4069260 msgr2=0x7f00c40c8ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:24:51.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.303+0000 7f00ca407640 1 -- 172.21.15.19:0/2990764557 shutdown_connections 2024-08-31T12:24:51.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.303+0000 7f00ca407640 1 -- 172.21.15.19:0/2990764557 wait complete. 2024-08-31T12:24:51.319 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:24:51.319 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-08-31T12:24:51.319 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-31T12:24:51.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.432+0000 7f69c6e07640 1 -- 172.21.15.19:0/283692512 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f69c0155300 msgr2=0x7f69c006d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:51.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.432+0000 7f69c6e07640 1 --2- 172.21.15.19:0/283692512 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f69c0155300 0x7f69c006d000 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f69c015b0f0 tx=0x7f69b802aa90 comp rx=0 tx=0).stop 2024-08-31T12:24:51.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.433+0000 7f69c6e07640 1 -- 172.21.15.19:0/283692512 shutdown_connections 2024-08-31T12:24:51.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.433+0000 7f69c6e07640 1 --2- 172.21.15.19:0/283692512 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f69c0155300 0x7f69c006d000 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:51.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.433+0000 7f69c6e07640 1 --2- 172.21.15.19:0/283692512 >> v2:172.21.15.43:3300/0 conn(0x7f69c00c8a40 0x7f69c00c9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:51.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.433+0000 7f69c6e07640 1 -- 172.21.15.19:0/283692512 >> 172.21.15.19:0/283692512 conn(0x7f69c00690d0 msgr2=0x7f69c00d0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:24:51.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.433+0000 7f69c6e07640 1 -- 172.21.15.19:0/283692512 shutdown_connections 2024-08-31T12:24:51.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.433+0000 7f69c6e07640 1 -- 172.21.15.19:0/283692512 wait complete. 2024-08-31T12:24:51.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.434+0000 7f69c6e07640 1 Processor -- start 2024-08-31T12:24:51.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.434+0000 7f69c6e07640 1 -- start start 2024-08-31T12:24:51.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.436+0000 7f69c6e07640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69c006e0e0 0x7f69c01541d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:51.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.436+0000 7f69c6e07640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f69c00c8a40 0x7f69c01f4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:51.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.436+0000 7f69c6e07640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f69c0155300 0x7f69c01e50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:51.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.436+0000 7f69c6e07640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f69c00c8330 con 0x7f69c00c8a40 2024-08-31T12:24:51.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.436+0000 7f69c6e07640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f69c015b250 con 0x7f69c006e0e0 2024-08-31T12:24:51.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.436+0000 7f69c6e07640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f69c00c8610 con 0x7f69c0155300 2024-08-31T12:24:51.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.436+0000 7f69bffff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f69c00c8a40 0x7f69c01f4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:51.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.436+0000 7f69bffff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f69c00c8a40 0x7f69c01f4f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:52308/0 (socket says 172.21.15.19:52308) 2024-08-31T12:24:51.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.436+0000 7f69bffff640 1 -- 172.21.15.19:0/383267868 learned_addr learned my addr 172.21.15.19:0/383267868 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:24:51.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.436+0000 7f69c53a4640 1 --2- 172.21.15.19:0/383267868 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f69c0155300 0x7f69c01e50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:51.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.436+0000 7f69c4ba3640 1 --2- 172.21.15.19:0/383267868 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69c006e0e0 0x7f69c01541d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:51.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.436+0000 7f69bffff640 1 -- 172.21.15.19:0/383267868 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f69c0155300 msgr2=0x7f69c01e50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:51.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.436+0000 7f69bffff640 1 --2- 172.21.15.19:0/383267868 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f69c0155300 0x7f69c01e50a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:51.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.436+0000 7f69bffff640 1 -- 172.21.15.19:0/383267868 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69c006e0e0 msgr2=0x7f69c01541d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:51.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.436+0000 7f69bffff640 1 --2- 172.21.15.19:0/383267868 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69c006e0e0 0x7f69c01541d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:51.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.436+0000 7f69bffff640 1 -- 172.21.15.19:0/383267868 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69c00c4a60 con 0x7f69c00c8a40 2024-08-31T12:24:51.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.436+0000 7f69c4ba3640 1 --2- 172.21.15.19:0/383267868 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69c006e0e0 0x7f69c01541d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-31T12:24:51.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.436+0000 7f69bffff640 1 --2- 172.21.15.19:0/383267868 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f69c00c8a40 0x7f69c01f4f10 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f69b400b420 tx=0x7f69b400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:24:51.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.437+0000 7f69bd7fa640 1 -- 172.21.15.19:0/383267868 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b4011830 con 0x7f69c00c8a40 2024-08-31T12:24:51.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.437+0000 7f69bd7fa640 1 -- 172.21.15.19:0/383267868 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f69b4016070 con 0x7f69c00c8a40 2024-08-31T12:24:51.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.437+0000 7f69c6e07640 1 -- 172.21.15.19:0/383267868 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69c00c4d40 con 0x7f69c00c8a40 2024-08-31T12:24:51.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.437+0000 7f69bd7fa640 1 -- 172.21.15.19:0/383267868 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b4011d60 con 0x7f69c00c8a40 2024-08-31T12:24:51.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.437+0000 7f69c6e07640 1 -- 172.21.15.19:0/383267868 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69c00c5300 con 0x7f69c00c8a40 2024-08-31T12:24:51.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.438+0000 7f69c6e07640 1 --2- 172.21.15.19:0/383267868 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69c01e55e0 0x7f69c00c2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:51.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.438+0000 7f69c6e07640 1 -- 172.21.15.19:0/383267868 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f69c01e43c0 con 0x7f69c01e55e0 2024-08-31T12:24:51.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.438+0000 7f69c4ba3640 1 --2- 172.21.15.19:0/383267868 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69c01e55e0 0x7f69c00c2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:51.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.438+0000 7f69c4ba3640 1 -- 172.21.15.19:0/383267868 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f69c00d1220 con 0x7f69c01e55e0 2024-08-31T12:24:51.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.438+0000 7f69bd7fa640 1 -- 172.21.15.19:0/383267868 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f69b401b460 con 0x7f69c00c8a40 2024-08-31T12:24:51.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.438+0000 7f69c4ba3640 1 --2- 172.21.15.19:0/383267868 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69c01e55e0 0x7f69c00c2860 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f69b000e6b0 tx=0x7f69b000eb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:24:51.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.439+0000 7f69bd7fa640 1 --2- 172.21.15.19:0/383267868 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f69ac077ab0 0x7f69ac079fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:51.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.439+0000 7f69bd7fa640 1 -- 172.21.15.19:0/383267868 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f69b4099290 con 0x7f69c00c8a40 2024-08-31T12:24:51.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.439+0000 7f69c53a4640 1 --2- 172.21.15.19:0/383267868 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f69ac077ab0 0x7f69ac079fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:51.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.439+0000 7f69bd7fa640 1 -- 172.21.15.19:0/383267868 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b000cac0 con 0x7f69c01e55e0 2024-08-31T12:24:51.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.439+0000 7f69bd7fa640 1 -- 172.21.15.19:0/383267868 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f69b000cc60 con 0x7f69c01e55e0 2024-08-31T12:24:51.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.439+0000 7f69bd7fa640 1 -- 172.21.15.19:0/383267868 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69c01e55e0 msgr2=0x7f69c00c2860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:51.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.439+0000 7f69bd7fa640 1 --2- 172.21.15.19:0/383267868 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69c01e55e0 0x7f69c00c2860 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f69b000e6b0 tx=0x7f69b000eb70 comp rx=0 tx=0).stop 2024-08-31T12:24:51.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.439+0000 7f69bd7fa640 1 -- 172.21.15.19:0/383267868 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69c01e55e0 msgr2=0x7f69c00c2860 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:24:51.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.439+0000 7f69bd7fa640 1 --2- 172.21.15.19:0/383267868 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69c01e55e0 0x7f69c00c2860 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:51.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.440+0000 7f69c53a4640 1 --2- 172.21.15.19:0/383267868 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f69ac077ab0 0x7f69ac079fc0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f69b8035000 tx=0x7f69b8038040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:24:51.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.445+0000 7f69c6e07640 1 --2- 172.21.15.19:0/383267868 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69c00c2da0 0x7f69c00cf590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:51.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.445+0000 7f69c6e07640 1 -- 172.21.15.19:0/383267868 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f69c015a210 con 0x7f69c00c2da0 2024-08-31T12:24:51.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.446+0000 7f69c4ba3640 1 --2- 172.21.15.19:0/383267868 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69c00c2da0 0x7f69c00cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:51.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.446+0000 7f69c4ba3640 1 -- 172.21.15.19:0/383267868 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f69c00c9850 con 0x7f69c00c2da0 2024-08-31T12:24:51.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.446+0000 7f69c4ba3640 1 --2- 172.21.15.19:0/383267868 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69c00c2da0 0x7f69c00cf590 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f69b0019000 tx=0x7f69b00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:24:51.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.446+0000 7f69bd7fa640 1 -- 172.21.15.19:0/383267868 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b001fcb0 con 0x7f69c00c2da0 2024-08-31T12:24:51.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.447+0000 7f69bd7fa640 1 -- 172.21.15.19:0/383267868 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5042 (secure 0 0 0) 0x7f69b001a480 con 0x7f69c00c2da0 2024-08-31T12:24:51.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.447+0000 7f69bd7fa640 1 -- 172.21.15.19:0/383267868 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69c00c2da0 msgr2=0x7f69c00cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:51.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.447+0000 7f69bd7fa640 1 --2- 172.21.15.19:0/383267868 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69c00c2da0 0x7f69c00cf590 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f69b0019000 tx=0x7f69b00046c0 comp rx=0 tx=0).stop 2024-08-31T12:24:51.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.447+0000 7f69bd7fa640 1 -- 172.21.15.19:0/383267868 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69c00c2da0 msgr2=0x7f69c00cf590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:24:51.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.447+0000 7f69bd7fa640 1 --2- 172.21.15.19:0/383267868 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69c00c2da0 0x7f69c00cf590 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:51.449 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:24:51.449 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:24:51.449 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:24:51.449 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:24:51.449 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 74, 2024-08-31T12:24:51.449 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 200920, 2024-08-31T12:24:51.449 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:24:51.449 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:24:51.449 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:24:51.449 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:24:51.449 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:24:51.450 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 0, 2024-08-31T12:24:51.450 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:24:51.450 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:24:51.450 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:24:51.450 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:24:51.450 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:24:51.450 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:24:51.450 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:24:51.450 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:24:51.450 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:24:51.450 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:24:51.450 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:24:51.450 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:24:51.450 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:24:51.451 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:24:51.451 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:24:51.451 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:24:51.451 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:24:51.451 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:24:51.451 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:24:51.451 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:24:51.451 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:24:51.451 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:24:51.451 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:24:51.451 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:24:51.451 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:24:51.451 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:24:51.452 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:24:51.452 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:24:51.452 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:24:51.452 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:51.452 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:24:51.452 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:24:51.452 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:24:51.452 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:24:51.452 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:24:51.452 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:24:51.452 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:24:51.452 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:24:51.452 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:24:51.453 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:24:51.453 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:24:51.453 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:24:51.453 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:24:51.453 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:24:51.453 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:24:51.453 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:24:51.453 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:24:51.453 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:24:51.453 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:24:51.453 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:24:51.453 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:24:51.453 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:24:51.453 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:24:51.454 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:24:51.454 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:24:51.454 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:24:51.454 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:24:51.454 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:24:51.454 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:24:51.454 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:24:51.454 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:51.454 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:24:51.454 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:51.454 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:24:51.454 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:24:51.454 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:24:51.455 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:24:51.455 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:51.455 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:24:51.455 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:24:51.455 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:24:51.455 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:51.455 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:51.455 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:24:51.455 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:24:51.455 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:24:51.455 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:24:51.455 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:24:51.455 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:51.456 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:24:51.456 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:24:51.456 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:24:51.456 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:24:51.456 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:24:51.456 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:51.456 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:51.456 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:24:51.456 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:24:51.456 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:24:51.456 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:24:51.456 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:51.456 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:24:51.456 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:24:51.457 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:24:51.457 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:51.457 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:51.457 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:24:51.457 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:24:51.457 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:24:51.457 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:24:51.457 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:24:51.457 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:51.457 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:24:51.457 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:24:51.457 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:24:51.457 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:24:51.458 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:24:51.458 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:51.458 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:51.458 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:24:51.458 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:24:51.458 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:24:51.458 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:24:51.458 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:51.458 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:24:51.458 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:24:51.458 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:24:51.458 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:51.459 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:51.459 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:24:51.459 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:24:51.459 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:24:51.459 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:24:51.459 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:24:51.459 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:51.459 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:24:51.459 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:24:51.459 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:24:51.459 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:24:51.459 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:24:51.459 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:24:51.460 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:24:51.460 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:51.460 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:24:51.460 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:24:51.460 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:51.460 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:24:51.460 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:24:51.460 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:24:51.460 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:24:51.460 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:24:51.460 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:24:51.460 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:51.461 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:24:51.461 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:24:51.461 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 3 2024-08-31T12:24:51.461 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:24:51.461 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:24:51.461 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:24:51.461 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:51.461 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:24:51.461 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:24:51.461 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 4 2024-08-31T12:24:51.461 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:24:51.461 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:24:51.461 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:24:51.461 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:51.462 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:24:51.462 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:24:51.462 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 6 2024-08-31T12:24:51.462 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:24:51.462 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:24:51.462 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:24:51.462 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:51.462 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:24:51.462 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:24:51.462 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:24:51.462 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:24:51.462 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:24:51.462 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:51.463 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:24:51.463 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:24:51.464 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.447+0000 7f69c6e07640 1 -- 172.21.15.19:0/383267868 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f69ac077ab0 msgr2=0x7f69ac079fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:51.464 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.447+0000 7f69c6e07640 1 --2- 172.21.15.19:0/383267868 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f69ac077ab0 0x7f69ac079fc0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f69b8035000 tx=0x7f69b8038040 comp rx=0 tx=0).stop 2024-08-31T12:24:51.464 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.447+0000 7f69c6e07640 1 -- 172.21.15.19:0/383267868 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f69c00c8a40 msgr2=0x7f69c01f4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:51.464 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.447+0000 7f69c6e07640 1 --2- 172.21.15.19:0/383267868 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f69c00c8a40 0x7f69c01f4f10 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f69b400b420 tx=0x7f69b400b8e0 comp rx=0 tx=0).stop 2024-08-31T12:24:51.464 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.447+0000 7f69c53a4640 1 -- 172.21.15.19:0/383267868 reap_dead start 2024-08-31T12:24:51.464 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.447+0000 7f69c53a4640 1 -- 172.21.15.19:0/383267868 reap_dead start 2024-08-31T12:24:51.464 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.447+0000 7f69c6e07640 1 -- 172.21.15.19:0/383267868 shutdown_connections 2024-08-31T12:24:51.464 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.447+0000 7f69c6e07640 1 -- 172.21.15.19:0/383267868 >> 172.21.15.19:0/383267868 conn(0x7f69c00690d0 msgr2=0x7f69c006d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:24:51.465 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.447+0000 7f69c6e07640 1 -- 172.21.15.19:0/383267868 shutdown_connections 2024-08-31T12:24:51.465 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.447+0000 7f69c6e07640 1 -- 172.21.15.19:0/383267868 wait complete. 2024-08-31T12:24:51.465 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:24:51.465 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-08-31T12:24:51.465 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-31T12:24:51.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.578+0000 7f389f500640 1 -- 172.21.15.19:0/3012504559 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f38980ca770 msgr2=0x7f38980d0cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:51.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.578+0000 7f389f500640 1 --2- 172.21.15.19:0/3012504559 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f38980ca770 0x7f38980d0cd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3888009830 tx=0x7f388802a5a0 comp rx=0 tx=0).stop 2024-08-31T12:24:51.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.579+0000 7f389f500640 1 -- 172.21.15.19:0/3012504559 shutdown_connections 2024-08-31T12:24:51.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.579+0000 7f389f500640 1 --2- 172.21.15.19:0/3012504559 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f38980ca770 0x7f38980d0cd0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:51.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.579+0000 7f389f500640 1 --2- 172.21.15.19:0/3012504559 >> v2:172.21.15.43:3300/0 conn(0x7f38980cc620 0x7f38980cdab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:51.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.579+0000 7f389f500640 1 -- 172.21.15.19:0/3012504559 >> 172.21.15.19:0/3012504559 conn(0x7f389806e6e0 msgr2=0x7f389806eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:24:51.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.579+0000 7f389f500640 1 -- 172.21.15.19:0/3012504559 shutdown_connections 2024-08-31T12:24:51.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.579+0000 7f389f500640 1 -- 172.21.15.19:0/3012504559 wait complete. 2024-08-31T12:24:51.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.580+0000 7f389f500640 1 Processor -- start 2024-08-31T12:24:51.582 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.580+0000 7f389f500640 1 -- start start 2024-08-31T12:24:51.584 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.582+0000 7f389f500640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f38980ca770 0x7f3898152430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:51.584 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.582+0000 7f389f500640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f38980cc620 0x7f389814f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:51.584 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.582+0000 7f389f500640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38980d1210 0x7f38980c6660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:51.584 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.582+0000 7f389f500640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f389815d090 con 0x7f38980cc620 2024-08-31T12:24:51.584 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.582+0000 7f389f500640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f389815cdb0 con 0x7f38980d1210 2024-08-31T12:24:51.584 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.582+0000 7f389f500640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f38980cc320 con 0x7f38980ca770 2024-08-31T12:24:51.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.582+0000 7f389ca9b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f38980cc620 0x7f389814f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:51.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.582+0000 7f389ca9b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f38980cc620 0x7f389814f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:52324/0 (socket says 172.21.15.19:52324) 2024-08-31T12:24:51.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.582+0000 7f389ca9b640 1 -- 172.21.15.19:0/3399727868 learned_addr learned my addr 172.21.15.19:0/3399727868 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:24:51.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.582+0000 7f389d29c640 1 --2- 172.21.15.19:0/3399727868 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f38980ca770 0x7f3898152430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:51.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.582+0000 7f389da9d640 1 --2- 172.21.15.19:0/3399727868 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38980d1210 0x7f38980c6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:51.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.582+0000 7f389ca9b640 1 -- 172.21.15.19:0/3399727868 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f38980ca770 msgr2=0x7f3898152430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:51.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.582+0000 7f389ca9b640 1 --2- 172.21.15.19:0/3399727868 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f38980ca770 0x7f3898152430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:51.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.582+0000 7f389ca9b640 1 -- 172.21.15.19:0/3399727868 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38980d1210 msgr2=0x7f38980c6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:51.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.582+0000 7f389ca9b640 1 --2- 172.21.15.19:0/3399727868 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38980d1210 0x7f38980c6660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:51.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.582+0000 7f389ca9b640 1 -- 172.21.15.19:0/3399727868 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3898152b80 con 0x7f38980cc620 2024-08-31T12:24:51.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.582+0000 7f389da9d640 1 --2- 172.21.15.19:0/3399727868 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38980d1210 0x7f38980c6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:24:51.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.582+0000 7f389ca9b640 1 --2- 172.21.15.19:0/3399727868 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f38980cc620 0x7f389814f6e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3888009950 tx=0x7f3888033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:24:51.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.583+0000 7f388dffb640 1 -- 172.21.15.19:0/3399727868 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3888005c90 con 0x7f38980cc620 2024-08-31T12:24:51.586 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.583+0000 7f388dffb640 1 -- 172.21.15.19:0/3399727868 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3888031e50 con 0x7f38980cc620 2024-08-31T12:24:51.586 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.583+0000 7f389f500640 1 -- 172.21.15.19:0/3399727868 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3898151670 con 0x7f38980cc620 2024-08-31T12:24:51.586 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.583+0000 7f388dffb640 1 -- 172.21.15.19:0/3399727868 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3888005260 con 0x7f38980cc620 2024-08-31T12:24:51.586 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.583+0000 7f389f500640 1 -- 172.21.15.19:0/3399727868 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3898151c30 con 0x7f38980cc620 2024-08-31T12:24:51.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.584+0000 7f389f500640 1 --2- 172.21.15.19:0/3399727868 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f38980cb2c0 0x7f3898158320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:51.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.584+0000 7f389f500640 1 -- 172.21.15.19:0/3399727868 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f3898152970 con 0x7f38980cb2c0 2024-08-31T12:24:51.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.584+0000 7f389d29c640 1 --2- 172.21.15.19:0/3399727868 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f38980cb2c0 0x7f3898158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:51.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.584+0000 7f389d29c640 1 -- 172.21.15.19:0/3399727868 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f38981f4c60 con 0x7f38980cb2c0 2024-08-31T12:24:51.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.584+0000 7f388dffb640 1 -- 172.21.15.19:0/3399727868 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f3888005400 con 0x7f38980cc620 2024-08-31T12:24:51.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.585+0000 7f389d29c640 1 --2- 172.21.15.19:0/3399727868 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f38980cb2c0 0x7f3898158320 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3894011780 tx=0x7f3894011c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:24:51.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.585+0000 7f388dffb640 1 --2- 172.21.15.19:0/3399727868 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f387c077a80 0x7f387c079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:51.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.585+0000 7f388dffb640 1 -- 172.21.15.19:0/3399727868 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f38880ba860 con 0x7f38980cc620 2024-08-31T12:24:51.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.585+0000 7f389da9d640 1 --2- 172.21.15.19:0/3399727868 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f387c077a80 0x7f387c079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:51.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.585+0000 7f388dffb640 1 -- 172.21.15.19:0/3399727868 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389401f070 con 0x7f38980cb2c0 2024-08-31T12:24:51.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.585+0000 7f388dffb640 1 -- 172.21.15.19:0/3399727868 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3894011e90 con 0x7f38980cb2c0 2024-08-31T12:24:51.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.585+0000 7f388dffb640 1 -- 172.21.15.19:0/3399727868 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f38980cb2c0 msgr2=0x7f3898158320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:51.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.585+0000 7f388dffb640 1 --2- 172.21.15.19:0/3399727868 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f38980cb2c0 0x7f3898158320 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3894011780 tx=0x7f3894011c40 comp rx=0 tx=0).stop 2024-08-31T12:24:51.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.585+0000 7f388dffb640 1 -- 172.21.15.19:0/3399727868 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f38980cb2c0 msgr2=0x7f3898158320 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:24:51.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.585+0000 7f388dffb640 1 --2- 172.21.15.19:0/3399727868 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f38980cb2c0 0x7f3898158320 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:51.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.586+0000 7f389da9d640 1 --2- 172.21.15.19:0/3399727868 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f387c077a80 0x7f387c079f90 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3890007f20 tx=0x7f3890009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:24:51.592 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.591+0000 7f389f500640 1 --2- 172.21.15.19:0/3399727868 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f38980c70a0 0x7f3898158860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:51.593 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.591+0000 7f389f500640 1 -- 172.21.15.19:0/3399727868 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f389814d4d0 con 0x7f38980c70a0 2024-08-31T12:24:51.593 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.592+0000 7f389d29c640 1 --2- 172.21.15.19:0/3399727868 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f38980c70a0 0x7f3898158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:51.593 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.592+0000 7f389d29c640 1 -- 172.21.15.19:0/3399727868 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f38980cab80 con 0x7f38980c70a0 2024-08-31T12:24:51.593 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.592+0000 7f389d29c640 1 --2- 172.21.15.19:0/3399727868 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f38980c70a0 0x7f3898158860 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3894011780 tx=0x7f38940046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:24:51.593 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.592+0000 7f388dffb640 1 -- 172.21.15.19:0/3399727868 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389401f030 con 0x7f38980c70a0 2024-08-31T12:24:51.594 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.593+0000 7f388dffb640 1 -- 172.21.15.19:0/3399727868 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5041 (secure 0 0 0) 0x7f3894022c10 con 0x7f38980c70a0 2024-08-31T12:24:51.594 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.593+0000 7f388dffb640 1 -- 172.21.15.19:0/3399727868 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f38980c70a0 msgr2=0x7f3898158860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:51.594 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.593+0000 7f388dffb640 1 --2- 172.21.15.19:0/3399727868 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f38980c70a0 0x7f3898158860 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3894011780 tx=0x7f38940046c0 comp rx=0 tx=0).stop 2024-08-31T12:24:51.594 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.593+0000 7f388dffb640 1 -- 172.21.15.19:0/3399727868 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f38980c70a0 msgr2=0x7f3898158860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:24:51.594 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.593+0000 7f388dffb640 1 --2- 172.21.15.19:0/3399727868 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f38980c70a0 0x7f3898158860 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:51.595 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:24:51.595 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:24:51.595 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:24:51.595 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:24:51.595 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 74, 2024-08-31T12:24:51.595 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 78949, 2024-08-31T12:24:51.595 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:24:51.595 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:24:51.595 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:24:51.595 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:24:51.595 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:24:51.596 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 0, 2024-08-31T12:24:51.596 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:24:51.596 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:24:51.596 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:24:51.596 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:24:51.596 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:24:51.596 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:24:51.596 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:24:51.596 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:24:51.596 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:24:51.596 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:24:51.596 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:24:51.596 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:24:51.597 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:24:51.597 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:24:51.597 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:24:51.597 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:24:51.597 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:24:51.597 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:24:51.597 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:24:51.597 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:24:51.597 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:24:51.597 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:24:51.597 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:24:51.597 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:24:51.598 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:24:51.598 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:24:51.598 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:24:51.598 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:24:51.598 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:24:51.598 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:51.598 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:24:51.598 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:24:51.598 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:24:51.598 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:24:51.598 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:24:51.598 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:24:51.598 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:24:51.598 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:24:51.599 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:24:51.599 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:24:51.599 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:24:51.599 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:24:51.599 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:24:51.599 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:24:51.599 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:24:51.599 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:24:51.599 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:24:51.599 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:24:51.599 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:24:51.599 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:24:51.599 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:24:51.600 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:24:51.600 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:24:51.600 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:24:51.600 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:24:51.600 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:24:51.600 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:24:51.600 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:24:51.600 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:24:51.600 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:24:51.600 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:51.600 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:24:51.600 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:51.600 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:24:51.601 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:24:51.601 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:24:51.601 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:24:51.601 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:51.601 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:24:51.601 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:24:51.601 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:24:51.601 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:51.601 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:51.601 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:24:51.601 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:24:51.601 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:24:51.601 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:24:51.601 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:24:51.601 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:51.602 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:24:51.602 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:24:51.602 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:24:51.602 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:24:51.602 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:24:51.602 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:51.602 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:51.602 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:24:51.602 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:24:51.602 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:24:51.602 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:24:51.602 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:51.602 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:24:51.602 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:24:51.603 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:24:51.603 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:51.603 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:51.603 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:24:51.603 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:24:51.603 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:24:51.603 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:24:51.603 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:24:51.603 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:51.603 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:24:51.603 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:24:51.603 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:24:51.603 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:24:51.603 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:24:51.604 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:51.604 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:51.604 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:24:51.604 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:24:51.604 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:24:51.604 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:24:51.604 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:51.604 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:24:51.604 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:24:51.604 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:24:51.604 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:51.604 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:51.604 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:24:51.605 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:24:51.605 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:24:51.605 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:24:51.605 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:24:51.605 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:51.605 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:24:51.605 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:24:51.605 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:24:51.605 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:24:51.605 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:24:51.605 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:24:51.605 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:24:51.605 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:51.605 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:24:51.605 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:24:51.606 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:51.606 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:24:51.606 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:24:51.606 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:24:51.606 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:24:51.606 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:24:51.606 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:24:51.606 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:51.606 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:24:51.606 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:24:51.606 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 3 2024-08-31T12:24:51.606 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:24:51.607 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:24:51.607 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:24:51.607 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:51.607 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:24:51.607 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:24:51.607 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 4 2024-08-31T12:24:51.607 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:24:51.607 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:24:51.607 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:24:51.607 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:51.607 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:24:51.607 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:24:51.607 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:24:51.607 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:24:51.607 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:24:51.608 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:24:51.608 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:24:51.608 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:24:51.608 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:24:51.608 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:24:51.608 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:24:51.608 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:24:51.608 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:24:51.608 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:24:51.608 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:24:51.609 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.593+0000 7f389f500640 1 -- 172.21.15.19:0/3399727868 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f387c077a80 msgr2=0x7f387c079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:51.609 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.593+0000 7f389f500640 1 --2- 172.21.15.19:0/3399727868 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f387c077a80 0x7f387c079f90 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3890007f20 tx=0x7f3890009040 comp rx=0 tx=0).stop 2024-08-31T12:24:51.609 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.593+0000 7f389f500640 1 -- 172.21.15.19:0/3399727868 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f38980cc620 msgr2=0x7f389814f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:51.610 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.593+0000 7f389f500640 1 --2- 172.21.15.19:0/3399727868 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f38980cc620 0x7f389814f6e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3888009950 tx=0x7f3888033c20 comp rx=0 tx=0).stop 2024-08-31T12:24:51.610 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.593+0000 7f389da9d640 1 -- 172.21.15.19:0/3399727868 reap_dead start 2024-08-31T12:24:51.610 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.593+0000 7f389da9d640 1 -- 172.21.15.19:0/3399727868 reap_dead start 2024-08-31T12:24:51.610 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.593+0000 7f389f500640 1 -- 172.21.15.19:0/3399727868 shutdown_connections 2024-08-31T12:24:51.610 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.593+0000 7f389f500640 1 -- 172.21.15.19:0/3399727868 >> 172.21.15.19:0/3399727868 conn(0x7f389806e6e0 msgr2=0x7f38980cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:24:51.610 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.593+0000 7f389f500640 1 -- 172.21.15.19:0/3399727868 shutdown_connections 2024-08-31T12:24:51.610 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.593+0000 7f389f500640 1 -- 172.21.15.19:0/3399727868 wait complete. 2024-08-31T12:24:51.610 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:24:51.610 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-08-31T12:24:51.610 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-08-31T12:24:51.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.723+0000 7f1647d2b640 1 -- 172.21.15.19:0/941282038 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1640159960 msgr2=0x7f1640070990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:51.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.723+0000 7f1647d2b640 1 --2- 172.21.15.19:0/941282038 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1640159960 0x7f1640070990 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f164015c650 tx=0x7f163002a4b0 comp rx=0 tx=0).stop 2024-08-31T12:24:51.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.723+0000 7f1647d2b640 1 -- 172.21.15.19:0/941282038 shutdown_connections 2024-08-31T12:24:51.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.723+0000 7f1647d2b640 1 --2- 172.21.15.19:0/941282038 >> v2:172.21.15.43:3300/0 conn(0x7f1640070ed0 0x7f16400ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:51.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.723+0000 7f1647d2b640 1 --2- 172.21.15.19:0/941282038 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1640159960 0x7f1640070990 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:51.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.723+0000 7f1647d2b640 1 -- 172.21.15.19:0/941282038 >> 172.21.15.19:0/941282038 conn(0x7f1640068aa0 msgr2=0x7f1640068eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:24:51.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.723+0000 7f1647d2b640 1 -- 172.21.15.19:0/941282038 shutdown_connections 2024-08-31T12:24:51.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.723+0000 7f1647d2b640 1 -- 172.21.15.19:0/941282038 wait complete. 2024-08-31T12:24:51.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.724+0000 7f1647d2b640 1 Processor -- start 2024-08-31T12:24:51.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.724+0000 7f1647d2b640 1 -- start start 2024-08-31T12:24:51.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.726+0000 7f1647d2b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1640070ed0 0x7f16400cd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:51.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.726+0000 7f1647d2b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f16400cbce0 0x7f1640149310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:51.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.726+0000 7f1647d2b640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1640159960 0x7f16400b3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:51.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.726+0000 7f1647d2b640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f16400c5790 con 0x7f16400cbce0 2024-08-31T12:24:51.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.726+0000 7f1647d2b640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f164015cce0 con 0x7f1640159960 2024-08-31T12:24:51.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.726+0000 7f1647d2b640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f1640159d70 con 0x7f1640070ed0 2024-08-31T12:24:51.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.726+0000 7f16452c6640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f16400cbce0 0x7f1640149310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:51.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.726+0000 7f16452c6640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f16400cbce0 0x7f1640149310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:52344/0 (socket says 172.21.15.19:52344) 2024-08-31T12:24:51.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.726+0000 7f16452c6640 1 -- 172.21.15.19:0/3911342761 learned_addr learned my addr 172.21.15.19:0/3911342761 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:24:51.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.726+0000 7f1645ac7640 1 --2- 172.21.15.19:0/3911342761 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1640070ed0 0x7f16400cd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:51.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.726+0000 7f16462c8640 1 --2- 172.21.15.19:0/3911342761 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1640159960 0x7f16400b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:51.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.726+0000 7f16452c6640 1 -- 172.21.15.19:0/3911342761 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1640070ed0 msgr2=0x7f16400cd7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:51.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.726+0000 7f16452c6640 1 --2- 172.21.15.19:0/3911342761 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1640070ed0 0x7f16400cd7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:51.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.726+0000 7f16452c6640 1 -- 172.21.15.19:0/3911342761 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1640159960 msgr2=0x7f16400b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:51.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.726+0000 7f16452c6640 1 --2- 172.21.15.19:0/3911342761 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1640159960 0x7f16400b3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:51.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.726+0000 7f16452c6640 1 -- 172.21.15.19:0/3911342761 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16400cdcf0 con 0x7f16400cbce0 2024-08-31T12:24:51.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.726+0000 7f16452c6640 1 --2- 172.21.15.19:0/3911342761 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f16400cbce0 0x7f1640149310 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1630034040 tx=0x7f16300039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:24:51.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.726+0000 7f16367fc640 1 -- 172.21.15.19:0/3911342761 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f163003e030 con 0x7f16400cbce0 2024-08-31T12:24:51.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.726+0000 7f16367fc640 1 -- 172.21.15.19:0/3911342761 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f16300047a0 con 0x7f16400cbce0 2024-08-31T12:24:51.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.727+0000 7f1647d2b640 1 -- 172.21.15.19:0/3911342761 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16400cee80 con 0x7f16400cbce0 2024-08-31T12:24:51.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.727+0000 7f16367fc640 1 -- 172.21.15.19:0/3911342761 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1630035040 con 0x7f16400cbce0 2024-08-31T12:24:51.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.727+0000 7f1647d2b640 1 -- 172.21.15.19:0/3911342761 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16401ef680 con 0x7f16400cbce0 2024-08-31T12:24:51.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.728+0000 7f1647d2b640 1 -- 172.21.15.19:0/3911342761 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16400c91b0 con 0x7f16400cbce0 2024-08-31T12:24:51.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.728+0000 7f16367fc640 1 -- 172.21.15.19:0/3911342761 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f1630041030 con 0x7f16400cbce0 2024-08-31T12:24:51.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.729+0000 7f16367fc640 1 --2- 172.21.15.19:0/3911342761 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1624077a50 0x7f1624079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:24:51.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.729+0000 7f16367fc640 1 -- 172.21.15.19:0/3911342761 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f16300b9d50 con 0x7f16400cbce0 2024-08-31T12:24:51.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.729+0000 7f1645ac7640 1 --2- 172.21.15.19:0/3911342761 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1624077a50 0x7f1624079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:24:51.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.729+0000 7f1645ac7640 1 --2- 172.21.15.19:0/3911342761 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1624077a50 0x7f1624079f60 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f163c006fd0 tx=0x7f163c009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:24:51.734 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:51.733+0000 7f16367fc640 1 -- 172.21.15.19:0/3911342761 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f16300bd030 con 0x7f16400cbce0 2024-08-31T12:24:52.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:52.013+0000 7f1647d2b640 1 -- 172.21.15.19:0/3911342761 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f1640159420 con 0x7f16400cbce0 2024-08-31T12:24:52.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:52.014+0000 7f16367fc640 1 -- 172.21.15.19:0/3911342761 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f1630082480 con 0x7f16400cbce0 2024-08-31T12:24:52.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:52.017+0000 7f1647d2b640 1 -- 172.21.15.19:0/3911342761 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1624077a50 msgr2=0x7f1624079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:52.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:52.017+0000 7f1647d2b640 1 --2- 172.21.15.19:0/3911342761 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1624077a50 0x7f1624079f60 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f163c006fd0 tx=0x7f163c009060 comp rx=0 tx=0).stop 2024-08-31T12:24:52.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:52.017+0000 7f1647d2b640 1 -- 172.21.15.19:0/3911342761 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f16400cbce0 msgr2=0x7f1640149310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:24:52.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:52.017+0000 7f1647d2b640 1 --2- 172.21.15.19:0/3911342761 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f16400cbce0 0x7f1640149310 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1630034040 tx=0x7f16300039e0 comp rx=0 tx=0).stop 2024-08-31T12:24:52.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:52.017+0000 7f1647d2b640 1 -- 172.21.15.19:0/3911342761 shutdown_connections 2024-08-31T12:24:52.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:52.017+0000 7f1647d2b640 1 --2- 172.21.15.19:0/3911342761 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1624077a50 0x7f1624079f60 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:52.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:52.017+0000 7f1647d2b640 1 --2- 172.21.15.19:0/3911342761 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1640159960 0x7f16400b3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:52.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:52.017+0000 7f1647d2b640 1 --2- 172.21.15.19:0/3911342761 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f16400cbce0 0x7f1640149310 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:52.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:52.017+0000 7f1647d2b640 1 --2- 172.21.15.19:0/3911342761 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1640070ed0 0x7f16400cd7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:24:52.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:52.017+0000 7f1647d2b640 1 -- 172.21.15.19:0/3911342761 >> 172.21.15.19:0/3911342761 conn(0x7f1640068aa0 msgr2=0x7f1640072930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:24:52.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:52.017+0000 7f1647d2b640 1 -- 172.21.15.19:0/3911342761 shutdown_connections 2024-08-31T12:24:52.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:24:52.017+0000 7f1647d2b640 1 -- 172.21.15.19:0/3911342761 wait complete. 2024-08-31T12:24:52.031 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2024-08-31T12:24:57.512 DEBUG:teuthology.orchestra.run.smithi019:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:24:57.515 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:25:02.032 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-08-31T12:25:02.032 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:25:02.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.158+0000 7f5e309c6640 1 -- 172.21.15.19:0/1220340605 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5e2c0c3a60 msgr2=0x7f5e2c0c6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:02.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.158+0000 7f5e309c6640 1 --2- 172.21.15.19:0/1220340605 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5e2c0c3a60 0x7f5e2c0c6390 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c0c50c0 tx=0x7f5e1c02a5d0 comp rx=0 tx=0).stop 2024-08-31T12:25:02.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.158+0000 7f5e309c6640 1 -- 172.21.15.19:0/1220340605 shutdown_connections 2024-08-31T12:25:02.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.158+0000 7f5e309c6640 1 --2- 172.21.15.19:0/1220340605 >> v2:172.21.15.43:3300/0 conn(0x7f5e2c151840 0x7f5e2c0c7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:02.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.158+0000 7f5e309c6640 1 --2- 172.21.15.19:0/1220340605 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5e2c0c3a60 0x7f5e2c0c6390 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:02.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.158+0000 7f5e309c6640 1 -- 172.21.15.19:0/1220340605 >> 172.21.15.19:0/1220340605 conn(0x7f5e2c0cc770 msgr2=0x7f5e2c0ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:02.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.158+0000 7f5e309c6640 1 -- 172.21.15.19:0/1220340605 shutdown_connections 2024-08-31T12:25:02.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.158+0000 7f5e309c6640 1 -- 172.21.15.19:0/1220340605 wait complete. 2024-08-31T12:25:02.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.160+0000 7f5e309c6640 1 Processor -- start 2024-08-31T12:25:02.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.160+0000 7f5e309c6640 1 -- start start 2024-08-31T12:25:02.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.162+0000 7f5e309c6640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5e2c0c3a60 0x7f5e2c1f55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:02.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.162+0000 7f5e309c6640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5e2c151840 0x7f5e2c1f1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:02.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.162+0000 7f5e309c6640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5e2c151c50 0x7f5e2c1508c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:02.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.162+0000 7f5e309c6640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f5e2c070b20 con 0x7f5e2c151c50 2024-08-31T12:25:02.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.162+0000 7f5e309c6640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5e2c070840 con 0x7f5e2c0c3a60 2024-08-31T12:25:02.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.162+0000 7f5e309c6640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f5e2c071400 con 0x7f5e2c151840 2024-08-31T12:25:02.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.162+0000 7f5e2ad9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5e2c151c50 0x7f5e2c1508c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:02.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.162+0000 7f5e2ad9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5e2c151c50 0x7f5e2c1508c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:35442/0 (socket says 172.21.15.19:35442) 2024-08-31T12:25:02.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.162+0000 7f5e2a59c640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5e2c0c3a60 0x7f5e2c1f55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:02.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.162+0000 7f5e2ad9d640 1 -- 172.21.15.19:0/1111314486 learned_addr learned my addr 172.21.15.19:0/1111314486 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:25:02.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.162+0000 7f5e29d9b640 1 --2- 172.21.15.19:0/1111314486 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5e2c151840 0x7f5e2c1f1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:02.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.162+0000 7f5e2ad9d640 1 -- 172.21.15.19:0/1111314486 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5e2c151840 msgr2=0x7f5e2c1f1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:02.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.162+0000 7f5e2ad9d640 1 --2- 172.21.15.19:0/1111314486 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5e2c151840 0x7f5e2c1f1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:02.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.162+0000 7f5e2ad9d640 1 -- 172.21.15.19:0/1111314486 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5e2c0c3a60 msgr2=0x7f5e2c1f55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:02.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.162+0000 7f5e2ad9d640 1 --2- 172.21.15.19:0/1111314486 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5e2c0c3a60 0x7f5e2c1f55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:02.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.162+0000 7f5e2ad9d640 1 -- 172.21.15.19:0/1111314486 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e2c1f24e0 con 0x7f5e2c151c50 2024-08-31T12:25:02.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.162+0000 7f5e2ad9d640 1 --2- 172.21.15.19:0/1111314486 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5e2c151c50 0x7f5e2c1508c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5e2400cb10 tx=0x7f5e2400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:02.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.163+0000 7f5e1affd640 1 -- 172.21.15.19:0/1111314486 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e24016020 con 0x7f5e2c151c50 2024-08-31T12:25:02.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.163+0000 7f5e1affd640 1 -- 172.21.15.19:0/1111314486 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5e2400ad60 con 0x7f5e2c151c50 2024-08-31T12:25:02.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.163+0000 7f5e1affd640 1 -- 172.21.15.19:0/1111314486 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e24012680 con 0x7f5e2c151c50 2024-08-31T12:25:02.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.163+0000 7f5e309c6640 1 -- 172.21.15.19:0/1111314486 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e2c14d730 con 0x7f5e2c151c50 2024-08-31T12:25:02.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.163+0000 7f5e309c6640 1 -- 172.21.15.19:0/1111314486 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e2c1f4df0 con 0x7f5e2c151c50 2024-08-31T12:25:02.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.164+0000 7f5e309c6640 1 -- 172.21.15.19:0/1111314486 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e2c14d4a0 con 0x7f5e2c151c50 2024-08-31T12:25:02.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.165+0000 7f5e1affd640 1 -- 172.21.15.19:0/1111314486 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f5e24019020 con 0x7f5e2c151c50 2024-08-31T12:25:02.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.165+0000 7f5e1affd640 1 --2- 172.21.15.19:0/1111314486 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5e0c077a50 0x7f5e0c079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:02.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.166+0000 7f5e1affd640 1 -- 172.21.15.19:0/1111314486 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f5e24099bf0 con 0x7f5e2c151c50 2024-08-31T12:25:02.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.166+0000 7f5e2a59c640 1 --2- 172.21.15.19:0/1111314486 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5e0c077a50 0x7f5e0c079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:02.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.166+0000 7f5e2a59c640 1 --2- 172.21.15.19:0/1111314486 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5e0c077a50 0x7f5e0c079f60 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c034000 tx=0x7f5e1c004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:02.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.171+0000 7f5e1affd640 1 -- 172.21.15.19:0/1111314486 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f5e24020020 con 0x7f5e2c151c50 2024-08-31T12:25:02.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.493+0000 7f5e309c6640 1 -- 172.21.15.19:0/1111314486 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5e2c0740d0 con 0x7f5e2c151c50 2024-08-31T12:25:02.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.494+0000 7f5e1affd640 1 -- 172.21.15.19:0/1111314486 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f5e2c1f24e0 con 0x7f5e2c151c50 2024-08-31T12:25:02.495 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":74,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:25:02.498 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.496+0000 7f5e309c6640 1 -- 172.21.15.19:0/1111314486 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5e0c077a50 msgr2=0x7f5e0c079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:02.498 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.496+0000 7f5e309c6640 1 --2- 172.21.15.19:0/1111314486 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5e0c077a50 0x7f5e0c079f60 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c034000 tx=0x7f5e1c004730 comp rx=0 tx=0).stop 2024-08-31T12:25:02.498 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.496+0000 7f5e309c6640 1 -- 172.21.15.19:0/1111314486 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5e2c151c50 msgr2=0x7f5e2c1508c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:02.498 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.496+0000 7f5e309c6640 1 --2- 172.21.15.19:0/1111314486 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5e2c151c50 0x7f5e2c1508c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5e2400cb10 tx=0x7f5e2400a060 comp rx=0 tx=0).stop 2024-08-31T12:25:02.498 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.496+0000 7f5e309c6640 1 -- 172.21.15.19:0/1111314486 shutdown_connections 2024-08-31T12:25:02.498 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.496+0000 7f5e309c6640 1 --2- 172.21.15.19:0/1111314486 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f5e0c077a50 0x7f5e0c079f60 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:02.498 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.496+0000 7f5e309c6640 1 --2- 172.21.15.19:0/1111314486 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5e2c151c50 0x7f5e2c1508c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:02.498 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.496+0000 7f5e309c6640 1 --2- 172.21.15.19:0/1111314486 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5e2c151840 0x7f5e2c1f1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:02.498 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.496+0000 7f5e309c6640 1 --2- 172.21.15.19:0/1111314486 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5e2c0c3a60 0x7f5e2c1f55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:02.498 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.496+0000 7f5e309c6640 1 -- 172.21.15.19:0/1111314486 >> 172.21.15.19:0/1111314486 conn(0x7f5e2c0cc770 msgr2=0x7f5e2c0c7a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:02.498 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.496+0000 7f5e309c6640 1 -- 172.21.15.19:0/1111314486 shutdown_connections 2024-08-31T12:25:02.498 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.496+0000 7f5e309c6640 1 -- 172.21.15.19:0/1111314486 wait complete. 2024-08-31T12:25:02.509 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-08-31T12:25:02.509 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-08-31T12:25:02.509 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-08-31T12:25:02.509 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-31T12:25:02.636 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.635+0000 7f7e573c6640 1 -- 172.21.15.19:0/1410391418 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7e3c003c10 msgr2=0x7f7e3c004060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:02.636 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.635+0000 7f7e573c6640 1 --2- 172.21.15.19:0/1410391418 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7e3c003c10 0x7f7e3c004060 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c009d20 tx=0x7f7e3c005830 comp rx=0 tx=0).stop 2024-08-31T12:25:02.636 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.635+0000 7f7e55963640 1 -- 172.21.15.19:0/1410391418 reap_dead start 2024-08-31T12:25:02.636 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.635+0000 7f7e573c6640 1 -- 172.21.15.19:0/1410391418 shutdown_connections 2024-08-31T12:25:02.637 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.635+0000 7f7e573c6640 1 -- 172.21.15.19:0/1410391418 >> 172.21.15.19:0/1410391418 conn(0x7f7e500728b0 msgr2=0x7f7e50072cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:02.637 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.635+0000 7f7e573c6640 1 -- 172.21.15.19:0/1410391418 shutdown_connections 2024-08-31T12:25:02.637 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.635+0000 7f7e573c6640 1 -- 172.21.15.19:0/1410391418 wait complete. 2024-08-31T12:25:02.638 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.637+0000 7f7e573c6640 1 Processor -- start 2024-08-31T12:25:02.638 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.637+0000 7f7e573c6640 1 -- start start 2024-08-31T12:25:02.640 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.638+0000 7f7e573c6640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7e3c003c10 0x7f7e501efc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:02.640 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.638+0000 7f7e573c6640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7e50156a50 0x7f7e5014e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:02.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.638+0000 7f7e573c6640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e501e3460 0x7f7e501e3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:02.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.638+0000 7f7e573c6640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f7e501535a0 con 0x7f7e3c003c10 2024-08-31T12:25:02.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.638+0000 7f7e573c6640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7e500c4db0 con 0x7f7e501e3460 2024-08-31T12:25:02.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.638+0000 7f7e573c6640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f7e500cb750 con 0x7f7e50156a50 2024-08-31T12:25:02.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.638+0000 7f7e55162640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7e3c003c10 0x7f7e501efc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:02.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.638+0000 7f7e55162640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7e3c003c10 0x7f7e501efc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:35456/0 (socket says 172.21.15.19:35456) 2024-08-31T12:25:02.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.638+0000 7f7e55162640 1 -- 172.21.15.19:0/799202208 learned_addr learned my addr 172.21.15.19:0/799202208 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:25:02.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.638+0000 7f7e55963640 1 --2- 172.21.15.19:0/799202208 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e501e3460 0x7f7e501e3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:02.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.638+0000 7f7e54961640 1 --2- 172.21.15.19:0/799202208 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7e50156a50 0x7f7e5014e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:02.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.639+0000 7f7e55162640 1 -- 172.21.15.19:0/799202208 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7e50156a50 msgr2=0x7f7e5014e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:02.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.639+0000 7f7e55162640 1 --2- 172.21.15.19:0/799202208 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7e50156a50 0x7f7e5014e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:02.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.639+0000 7f7e55162640 1 -- 172.21.15.19:0/799202208 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e501e3460 msgr2=0x7f7e501e3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:02.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.639+0000 7f7e55162640 1 --2- 172.21.15.19:0/799202208 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e501e3460 0x7f7e501e3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:02.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.639+0000 7f7e55162640 1 -- 172.21.15.19:0/799202208 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e501f0140 con 0x7f7e3c003c10 2024-08-31T12:25:02.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.639+0000 7f7e54961640 1 --2- 172.21.15.19:0/799202208 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7e50156a50 0x7f7e5014e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:25:02.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.639+0000 7f7e55162640 1 --2- 172.21.15.19:0/799202208 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7e3c003c10 0x7f7e501efc00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c009c40 tx=0x7f7e4c005d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:02.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.639+0000 7f7e45ffb640 1 -- 172.21.15.19:0/799202208 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c041020 con 0x7f7e3c003c10 2024-08-31T12:25:02.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.639+0000 7f7e45ffb640 1 -- 172.21.15.19:0/799202208 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7e4c02ba30 con 0x7f7e3c003c10 2024-08-31T12:25:02.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.639+0000 7f7e573c6640 1 -- 172.21.15.19:0/799202208 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e501ef700 con 0x7f7e3c003c10 2024-08-31T12:25:02.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.639+0000 7f7e45ffb640 1 -- 172.21.15.19:0/799202208 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c0054c0 con 0x7f7e3c003c10 2024-08-31T12:25:02.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.639+0000 7f7e573c6640 1 -- 172.21.15.19:0/799202208 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e5015b170 con 0x7f7e3c003c10 2024-08-31T12:25:02.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.640+0000 7f7e573c6640 1 --2- 172.21.15.19:0/799202208 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7e501f0310 0x7f7e50148a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:02.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.640+0000 7f7e573c6640 1 -- 172.21.15.19:0/799202208 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f7e501e42e0 con 0x7f7e501f0310 2024-08-31T12:25:02.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.641+0000 7f7e54961640 1 --2- 172.21.15.19:0/799202208 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7e501f0310 0x7f7e50148a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:02.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.641+0000 7f7e54961640 1 -- 172.21.15.19:0/799202208 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7e500c54c0 con 0x7f7e501f0310 2024-08-31T12:25:02.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.641+0000 7f7e54961640 1 --2- 172.21.15.19:0/799202208 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7e501f0310 0x7f7e50148a60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7e500c8cf0 tx=0x7f7e400039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:02.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.641+0000 7f7e45ffb640 1 -- 172.21.15.19:0/799202208 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e40012910 con 0x7f7e501f0310 2024-08-31T12:25:02.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.642+0000 7f7e45ffb640 1 -- 172.21.15.19:0/799202208 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f7e4c044020 con 0x7f7e3c003c10 2024-08-31T12:25:02.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.642+0000 7f7e45ffb640 1 --2- 172.21.15.19:0/799202208 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7e34077950 0x7f7e34079e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:02.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.642+0000 7f7e45ffb640 1 -- 172.21.15.19:0/799202208 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f7e4c0ba460 con 0x7f7e3c003c10 2024-08-31T12:25:02.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.642+0000 7f7e55963640 1 --2- 172.21.15.19:0/799202208 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7e34077950 0x7f7e34079e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:02.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.642+0000 7f7e45ffb640 1 -- 172.21.15.19:0/799202208 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7e40016030 con 0x7f7e501f0310 2024-08-31T12:25:02.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.642+0000 7f7e45ffb640 1 -- 172.21.15.19:0/799202208 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7e501f0310 msgr2=0x7f7e50148a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:02.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.642+0000 7f7e45ffb640 1 --2- 172.21.15.19:0/799202208 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7e501f0310 0x7f7e50148a60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7e500c8cf0 tx=0x7f7e400039c0 comp rx=0 tx=0).stop 2024-08-31T12:25:02.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.642+0000 7f7e45ffb640 1 -- 172.21.15.19:0/799202208 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7e501f0310 msgr2=0x7f7e50148a60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:25:02.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.642+0000 7f7e45ffb640 1 --2- 172.21.15.19:0/799202208 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7e501f0310 0x7f7e50148a60 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:02.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.643+0000 7f7e55963640 1 --2- 172.21.15.19:0/799202208 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7e34077950 0x7f7e34079e60 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c003b40 tx=0x7f7e48006e90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:02.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.651+0000 7f7e573c6640 1 --2- 172.21.15.19:0/799202208 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7e50148fa0 0x7f7e50159510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:02.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.651+0000 7f7e573c6640 1 -- 172.21.15.19:0/799202208 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f7e501493b0 con 0x7f7e50148fa0 2024-08-31T12:25:02.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.651+0000 7f7e54961640 1 --2- 172.21.15.19:0/799202208 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7e50148fa0 0x7f7e50159510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:02.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.651+0000 7f7e54961640 1 -- 172.21.15.19:0/799202208 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7e500735b0 con 0x7f7e50148fa0 2024-08-31T12:25:02.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.651+0000 7f7e54961640 1 --2- 172.21.15.19:0/799202208 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7e50148fa0 0x7f7e50159510 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7e40019000 tx=0x7f7e40006720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:02.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.651+0000 7f7e45ffb640 1 -- 172.21.15.19:0/799202208 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e40027600 con 0x7f7e50148fa0 2024-08-31T12:25:02.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.652+0000 7f7e45ffb640 1 -- 172.21.15.19:0/799202208 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4543 (secure 0 0 0) 0x7f7e40011440 con 0x7f7e50148fa0 2024-08-31T12:25:02.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.652+0000 7f7e45ffb640 1 -- 172.21.15.19:0/799202208 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7e50148fa0 msgr2=0x7f7e50159510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:02.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.652+0000 7f7e45ffb640 1 --2- 172.21.15.19:0/799202208 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7e50148fa0 0x7f7e50159510 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7e40019000 tx=0x7f7e40006720 comp rx=0 tx=0).stop 2024-08-31T12:25:02.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.652+0000 7f7e45ffb640 1 -- 172.21.15.19:0/799202208 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7e50148fa0 msgr2=0x7f7e50159510 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:25:02.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.652+0000 7f7e45ffb640 1 --2- 172.21.15.19:0/799202208 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7e50148fa0 0x7f7e50159510 secure :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7e40019000 tx=0x7f7e40006720 comp rx=0 tx=0).stop 2024-08-31T12:25:02.654 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:25:02.654 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:25:02.654 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:25:02.654 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "leader", 2024-08-31T12:25:02.654 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 74, 2024-08-31T12:25:02.654 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 37205, 2024-08-31T12:25:02.654 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:25:02.654 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:25:02.654 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:25:02.654 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:25:02.654 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:02.654 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 11, 2024-08-31T12:25:02.655 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:25:02.655 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:25:02.655 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:25:02.655 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:25:02.655 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:25:02.655 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:25:02.655 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:25:02.655 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:25:02.655 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:25:02.655 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:25:02.655 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:25:02.655 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:25:02.655 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:25:02.656 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:25:02.656 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:02.656 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:25:02.656 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:25:02.656 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:25:02.656 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:25:02.656 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:25:02.656 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:25:02.656 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:25:02.656 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:25:02.656 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:25:02.656 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:25:02.657 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:25:02.657 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:25:02.657 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:25:02.657 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:02.657 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:02.657 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:25:02.657 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:25:02.657 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:25:02.657 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:25:02.657 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:25:02.657 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:25:02.657 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:25:02.657 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:25:02.658 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:25:02.658 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:25:02.658 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:25:02.658 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:25:02.658 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:25:02.658 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:25:02.658 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:25:02.658 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:25:02.658 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:25:02.658 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:25:02.658 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:25:02.658 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:25:02.658 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:25:02.658 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:25:02.659 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:25:02.659 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:25:02.659 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:25:02.659 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:25:02.659 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:25:02.659 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:25:02.659 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:02.659 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:25:02.659 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:02.659 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:25:02.659 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:02.659 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:25:02.659 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:25:02.659 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:25:02.660 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:25:02.660 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:02.660 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:25:02.660 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:25:02.660 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:02.660 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:02.660 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:02.660 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:25:02.660 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:25:02.660 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:02.660 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:02.660 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:02.660 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:02.661 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:25:02.661 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:25:02.661 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:25:02.661 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:25:02.661 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:25:02.661 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:02.661 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:02.661 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:25:02.661 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:25:02.661 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:25:02.661 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:25:02.661 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:02.661 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:25:02.662 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:25:02.662 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:02.662 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:02.662 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:02.662 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:25:02.662 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:25:02.662 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:02.662 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:02.662 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:02.662 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:02.662 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:25:02.662 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:25:02.662 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:25:02.662 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:25:02.663 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:25:02.663 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:02.663 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:02.663 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:25:02.663 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:25:02.663 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:25:02.663 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:25:02.663 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:02.663 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:25:02.663 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:25:02.663 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:02.663 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:02.663 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:02.664 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:25:02.664 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:25:02.664 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:02.664 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:02.664 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:02.664 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:02.664 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:25:02.664 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:25:02.664 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:25:02.664 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:25:02.664 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:25:02.664 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:02.664 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:02.665 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:02.665 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:25:02.665 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:25:02.665 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:02.665 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:02.665 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:02.665 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:25:02.665 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:02.665 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:02.665 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:25:02.665 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:02.665 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:02.665 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:02.666 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:25:02.666 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:02.666 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:02.666 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:02.666 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:25:02.666 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:25:02.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.652+0000 7f7e573c6640 1 -- 172.21.15.19:0/799202208 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7e34077950 msgr2=0x7f7e34079e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:02.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.652+0000 7f7e573c6640 1 --2- 172.21.15.19:0/799202208 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7e34077950 0x7f7e34079e60 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c003b40 tx=0x7f7e48006e90 comp rx=0 tx=0).stop 2024-08-31T12:25:02.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.652+0000 7f7e573c6640 1 -- 172.21.15.19:0/799202208 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7e3c003c10 msgr2=0x7f7e501efc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:02.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.652+0000 7f7e573c6640 1 --2- 172.21.15.19:0/799202208 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7e3c003c10 0x7f7e501efc00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c009c40 tx=0x7f7e4c005d70 comp rx=0 tx=0).stop 2024-08-31T12:25:02.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.652+0000 7f7e55963640 1 -- 172.21.15.19:0/799202208 reap_dead start 2024-08-31T12:25:02.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.652+0000 7f7e573c6640 1 -- 172.21.15.19:0/799202208 shutdown_connections 2024-08-31T12:25:02.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.652+0000 7f7e573c6640 1 --2- 172.21.15.19:0/799202208 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7e3c003c10 0x7f7e501efc00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:02.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.652+0000 7f7e573c6640 1 -- 172.21.15.19:0/799202208 >> 172.21.15.19:0/799202208 conn(0x7f7e500728b0 msgr2=0x7f7e501e4920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:02.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.652+0000 7f7e573c6640 1 -- 172.21.15.19:0/799202208 shutdown_connections 2024-08-31T12:25:02.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.652+0000 7f7e573c6640 1 -- 172.21.15.19:0/799202208 wait complete. 2024-08-31T12:25:02.668 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:25:02.668 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-08-31T12:25:02.668 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-31T12:25:02.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.806+0000 7f376b724640 1 -- 172.21.15.19:0/3799561711 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3764153190 msgr2=0x7f3764159820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:02.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.806+0000 7f376b724640 1 --2- 172.21.15.19:0/3799561711 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3764153190 0x7f3764159820 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3760005fd0 tx=0x7f376002a670 comp rx=0 tx=0).stop 2024-08-31T12:25:02.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.806+0000 7f376b724640 1 -- 172.21.15.19:0/3799561711 shutdown_connections 2024-08-31T12:25:02.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.806+0000 7f376b724640 1 --2- 172.21.15.19:0/3799561711 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f37640c9030 0x7f37640cb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:02.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.806+0000 7f376b724640 1 --2- 172.21.15.19:0/3799561711 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3764153190 0x7f3764159820 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:02.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.806+0000 7f376b724640 1 -- 172.21.15.19:0/3799561711 >> 172.21.15.19:0/3799561711 conn(0x7f37640728b0 msgr2=0x7f3764072cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:02.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.806+0000 7f376b724640 1 -- 172.21.15.19:0/3799561711 shutdown_connections 2024-08-31T12:25:02.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.806+0000 7f376b724640 1 -- 172.21.15.19:0/3799561711 wait complete. 2024-08-31T12:25:02.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.808+0000 7f376b724640 1 Processor -- start 2024-08-31T12:25:02.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.808+0000 7f376b724640 1 -- start start 2024-08-31T12:25:02.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.809+0000 7f376b724640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f37640c9030 0x7f376414a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:02.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.809+0000 7f376b724640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f37640c9440 0x7f376414b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:02.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.809+0000 7f376b724640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3764153190 0x7f37641470b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:02.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.809+0000 7f376b724640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f37641535a0 con 0x7f3764153190 2024-08-31T12:25:02.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.809+0000 7f376b724640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f37640c4db0 con 0x7f37640c9440 2024-08-31T12:25:02.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.809+0000 7f376b724640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f376415cdc0 con 0x7f37640c9030 2024-08-31T12:25:02.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.809+0000 7f3769cc1640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3764153190 0x7f37641470b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:02.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.809+0000 7f3769cc1640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3764153190 0x7f37641470b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:35504/0 (socket says 172.21.15.19:35504) 2024-08-31T12:25:02.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.810+0000 7f3769cc1640 1 -- 172.21.15.19:0/1231046629 learned_addr learned my addr 172.21.15.19:0/1231046629 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:25:02.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.810+0000 7f37694c0640 1 --2- 172.21.15.19:0/1231046629 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f37640c9030 0x7f376414a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:02.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.810+0000 7f3768cbf640 1 --2- 172.21.15.19:0/1231046629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f37640c9440 0x7f376414b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:02.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.810+0000 7f3769cc1640 1 -- 172.21.15.19:0/1231046629 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f37640c9030 msgr2=0x7f376414a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:02.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.810+0000 7f3769cc1640 1 --2- 172.21.15.19:0/1231046629 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f37640c9030 0x7f376414a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:02.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.810+0000 7f3769cc1640 1 -- 172.21.15.19:0/1231046629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f37640c9440 msgr2=0x7f376414b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:02.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.810+0000 7f3769cc1640 1 --2- 172.21.15.19:0/1231046629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f37640c9440 0x7f376414b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:02.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.810+0000 7f3769cc1640 1 -- 172.21.15.19:0/1231046629 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f376414ab20 con 0x7f3764153190 2024-08-31T12:25:02.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.810+0000 7f3769cc1640 1 --2- 172.21.15.19:0/1231046629 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3764153190 0x7f37641470b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f375c00c730 tx=0x7f375c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:02.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.810+0000 7f3759ffb640 1 -- 172.21.15.19:0/1231046629 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f375c003b70 con 0x7f3764153190 2024-08-31T12:25:02.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.810+0000 7f3759ffb640 1 -- 172.21.15.19:0/1231046629 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f375c003d30 con 0x7f3764153190 2024-08-31T12:25:02.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.810+0000 7f376b724640 1 -- 172.21.15.19:0/1231046629 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f376415a770 con 0x7f3764153190 2024-08-31T12:25:02.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.811+0000 7f3759ffb640 1 -- 172.21.15.19:0/1231046629 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f375c0052d0 con 0x7f3764153190 2024-08-31T12:25:02.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.811+0000 7f376b724640 1 -- 172.21.15.19:0/1231046629 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f376415a010 con 0x7f3764153190 2024-08-31T12:25:02.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.812+0000 7f376b724640 1 --2- 172.21.15.19:0/1231046629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3764149fd0 0x7f3764145100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:02.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.812+0000 7f376b724640 1 -- 172.21.15.19:0/1231046629 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f37641f2bd0 con 0x7f3764149fd0 2024-08-31T12:25:02.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.812+0000 7f37694c0640 1 --2- 172.21.15.19:0/1231046629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3764149fd0 0x7f3764145100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:02.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.812+0000 7f37694c0640 1 -- 172.21.15.19:0/1231046629 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3764145d40 con 0x7f3764149fd0 2024-08-31T12:25:02.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.813+0000 7f37694c0640 1 --2- 172.21.15.19:0/1231046629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3764149fd0 0x7f3764145100 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3764074d30 tx=0x7f37600045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:02.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.813+0000 7f3759ffb640 1 -- 172.21.15.19:0/1231046629 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3760040070 con 0x7f3764149fd0 2024-08-31T12:25:02.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.813+0000 7f3759ffb640 1 -- 172.21.15.19:0/1231046629 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f375c005680 con 0x7f3764153190 2024-08-31T12:25:02.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.813+0000 7f3759ffb640 1 --2- 172.21.15.19:0/1231046629 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3748077980 0x7f3748079e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:02.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.814+0000 7f3759ffb640 1 -- 172.21.15.19:0/1231046629 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f375c0995c0 con 0x7f3764153190 2024-08-31T12:25:02.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.814+0000 7f3759ffb640 1 -- 172.21.15.19:0/1231046629 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3760003a80 con 0x7f3764149fd0 2024-08-31T12:25:02.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.814+0000 7f3759ffb640 1 -- 172.21.15.19:0/1231046629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3764149fd0 msgr2=0x7f3764145100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:02.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.814+0000 7f3759ffb640 1 --2- 172.21.15.19:0/1231046629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3764149fd0 0x7f3764145100 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3764074d30 tx=0x7f37600045e0 comp rx=0 tx=0).stop 2024-08-31T12:25:02.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.814+0000 7f3759ffb640 1 -- 172.21.15.19:0/1231046629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3764149fd0 msgr2=0x7f3764145100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:25:02.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.814+0000 7f3759ffb640 1 --2- 172.21.15.19:0/1231046629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3764149fd0 0x7f3764145100 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:02.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.814+0000 7f3768cbf640 1 --2- 172.21.15.19:0/1231046629 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3748077980 0x7f3748079e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:02.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.814+0000 7f3768cbf640 1 --2- 172.21.15.19:0/1231046629 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3748077980 0x7f3748079e90 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f376414d550 tx=0x7f3754005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:02.822 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.821+0000 7f376b724640 1 --2- 172.21.15.19:0/1231046629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f376414cdc0 0x7f3764145640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:02.822 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.821+0000 7f376b724640 1 -- 172.21.15.19:0/1231046629 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f37640caba0 con 0x7f376414cdc0 2024-08-31T12:25:02.822 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.821+0000 7f37694c0640 1 --2- 172.21.15.19:0/1231046629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f376414cdc0 0x7f3764145640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:02.823 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.822+0000 7f37694c0640 1 -- 172.21.15.19:0/1231046629 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3764145f90 con 0x7f376414cdc0 2024-08-31T12:25:02.823 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.822+0000 7f37694c0640 1 --2- 172.21.15.19:0/1231046629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f376414cdc0 0x7f3764145640 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f376414a3e0 tx=0x7f3760009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:02.823 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.822+0000 7f3759ffb640 1 -- 172.21.15.19:0/1231046629 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3760035040 con 0x7f376414cdc0 2024-08-31T12:25:02.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.822+0000 7f3759ffb640 1 -- 172.21.15.19:0/1231046629 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5043 (secure 0 0 0) 0x7f376003c950 con 0x7f376414cdc0 2024-08-31T12:25:02.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.822+0000 7f3759ffb640 1 -- 172.21.15.19:0/1231046629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f376414cdc0 msgr2=0x7f3764145640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:02.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.822+0000 7f3759ffb640 1 --2- 172.21.15.19:0/1231046629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f376414cdc0 0x7f3764145640 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f376414a3e0 tx=0x7f3760009260 comp rx=0 tx=0).stop 2024-08-31T12:25:02.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.822+0000 7f3759ffb640 1 -- 172.21.15.19:0/1231046629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f376414cdc0 msgr2=0x7f3764145640 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:25:02.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.822+0000 7f3759ffb640 1 --2- 172.21.15.19:0/1231046629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f376414cdc0 0x7f3764145640 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:02.824 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:25:02.824 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:25:02.824 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:25:02.824 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:25:02.825 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 74, 2024-08-31T12:25:02.825 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 212295, 2024-08-31T12:25:02.825 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:25:02.825 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:25:02.825 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:25:02.825 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:25:02.825 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:02.825 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 11, 2024-08-31T12:25:02.825 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:25:02.825 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:25:02.825 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:25:02.825 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:25:02.826 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:25:02.826 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:25:02.826 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:25:02.826 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:25:02.826 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:25:02.826 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:25:02.826 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:25:02.826 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:25:02.826 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:25:02.826 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:25:02.826 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:02.826 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:25:02.826 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:25:02.826 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:25:02.827 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:25:02.827 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:25:02.827 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:25:02.827 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:25:02.827 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:25:02.827 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:25:02.827 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:25:02.827 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:25:02.827 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:25:02.827 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:25:02.827 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:02.827 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:02.827 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:25:02.828 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:25:02.828 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:25:02.828 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:25:02.828 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:25:02.828 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:25:02.828 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:25:02.828 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:25:02.828 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:25:02.828 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:25:02.828 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:25:02.828 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:25:02.828 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:25:02.828 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:25:02.829 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:25:02.829 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:25:02.829 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:25:02.829 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:25:02.829 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:25:02.829 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:25:02.829 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:25:02.829 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:25:02.829 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:25:02.829 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:25:02.829 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:25:02.829 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:25:02.829 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:25:02.829 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:25:02.830 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:02.830 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:25:02.830 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:02.830 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:25:02.830 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:02.830 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:25:02.830 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:25:02.830 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:25:02.830 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:25:02.830 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:02.830 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:25:02.830 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:25:02.831 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:02.831 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:02.831 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:02.831 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:25:02.831 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:25:02.831 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:02.831 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:02.831 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:02.831 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:02.831 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:25:02.831 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:25:02.831 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:25:02.831 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:25:02.832 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:25:02.832 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:02.832 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:02.832 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:25:02.832 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:25:02.832 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:25:02.832 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:25:02.832 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:02.832 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:25:02.832 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:25:02.832 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:02.832 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:02.832 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:02.833 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:25:02.833 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:25:02.833 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:02.833 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:02.833 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:02.833 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:02.833 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:25:02.833 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:25:02.833 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:25:02.833 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:25:02.833 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:25:02.833 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:02.834 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:02.834 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:25:02.834 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:25:02.834 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:25:02.834 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:25:02.834 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:02.834 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:25:02.834 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:25:02.834 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:02.834 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:02.834 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:02.834 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:25:02.834 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:25:02.835 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:02.835 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:02.835 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:02.835 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:02.835 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:25:02.835 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:25:02.835 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:25:02.835 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:25:02.835 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:25:02.835 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:02.835 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:02.835 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:02.835 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:25:02.836 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:25:02.836 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:02.836 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:02.836 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:02.836 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:25:02.836 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:02.836 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:02.836 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:25:02.836 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:02.836 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:02.836 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:02.836 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 3 2024-08-31T12:25:02.837 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:02.837 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:02.837 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:25:02.837 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:02.837 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:02.837 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:02.837 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 4 2024-08-31T12:25:02.837 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:02.837 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:02.837 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:25:02.837 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:02.837 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:02.837 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:02.837 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 6 2024-08-31T12:25:02.838 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:02.838 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:02.838 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:25:02.838 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:02.838 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:02.838 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:02.838 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:25:02.838 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:02.838 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:02.838 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:02.838 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:25:02.838 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:25:02.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.823+0000 7f376b724640 1 -- 172.21.15.19:0/1231046629 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3748077980 msgr2=0x7f3748079e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:02.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.823+0000 7f376b724640 1 --2- 172.21.15.19:0/1231046629 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3748077980 0x7f3748079e90 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f376414d550 tx=0x7f3754005f90 comp rx=0 tx=0).stop 2024-08-31T12:25:02.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.823+0000 7f376b724640 1 -- 172.21.15.19:0/1231046629 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3764153190 msgr2=0x7f37641470b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:02.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.823+0000 7f376b724640 1 --2- 172.21.15.19:0/1231046629 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3764153190 0x7f37641470b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f375c00c730 tx=0x7f375c00cbf0 comp rx=0 tx=0).stop 2024-08-31T12:25:02.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.823+0000 7f3769cc1640 1 -- 172.21.15.19:0/1231046629 reap_dead start 2024-08-31T12:25:02.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.823+0000 7f3769cc1640 1 -- 172.21.15.19:0/1231046629 reap_dead start 2024-08-31T12:25:02.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.823+0000 7f376b724640 1 -- 172.21.15.19:0/1231046629 shutdown_connections 2024-08-31T12:25:02.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.823+0000 7f376b724640 1 -- 172.21.15.19:0/1231046629 >> 172.21.15.19:0/1231046629 conn(0x7f37640728b0 msgr2=0x7f3764150060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:02.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.823+0000 7f376b724640 1 -- 172.21.15.19:0/1231046629 shutdown_connections 2024-08-31T12:25:02.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.823+0000 7f376b724640 1 -- 172.21.15.19:0/1231046629 wait complete. 2024-08-31T12:25:02.841 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:25:02.841 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-08-31T12:25:02.841 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-31T12:25:02.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.965+0000 7f3264e61640 1 -- 172.21.15.19:0/2764537541 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3248004db0 msgr2=0x7f3248005280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:02.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.965+0000 7f3264e61640 1 --2- 172.21.15.19:0/2764537541 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3248004db0 0x7f3248005280 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3258009ed0 tx=0x7f3248005830 comp rx=0 tx=0).stop 2024-08-31T12:25:02.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.965+0000 7f325ed9d640 1 -- 172.21.15.19:0/2764537541 reap_dead start 2024-08-31T12:25:02.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.965+0000 7f3264e61640 1 -- 172.21.15.19:0/2764537541 shutdown_connections 2024-08-31T12:25:02.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.965+0000 7f3264e61640 1 -- 172.21.15.19:0/2764537541 >> 172.21.15.19:0/2764537541 conn(0x7f326006cae0 msgr2=0x7f326006cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:02.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.966+0000 7f3264e61640 1 -- 172.21.15.19:0/2764537541 shutdown_connections 2024-08-31T12:25:02.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.966+0000 7f3264e61640 1 -- 172.21.15.19:0/2764537541 wait complete. 2024-08-31T12:25:02.969 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.968+0000 7f3264e61640 1 Processor -- start 2024-08-31T12:25:02.969 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.968+0000 7f3264e61640 1 -- start start 2024-08-31T12:25:02.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.969+0000 7f3264e61640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3248004db0 0x7f32600c5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:02.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.969+0000 7f3264e61640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f32600c4c30 0x7f32600c5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:02.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.969+0000 7f3264e61640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3260156d60 0x7f3260157170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:02.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.969+0000 7f3264e61640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f32600c3e90 con 0x7f32600c4c30 2024-08-31T12:25:02.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.969+0000 7f3264e61640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f32600b2dc0 con 0x7f3260156d60 2024-08-31T12:25:02.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.969+0000 7f3264e61640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f32600c4010 con 0x7f3248004db0 2024-08-31T12:25:02.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.969+0000 7f325dd9b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f32600c4c30 0x7f32600c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:02.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.969+0000 7f325dd9b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f32600c4c30 0x7f32600c5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:35528/0 (socket says 172.21.15.19:35528) 2024-08-31T12:25:02.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.969+0000 7f325dd9b640 1 -- 172.21.15.19:0/114869016 learned_addr learned my addr 172.21.15.19:0/114869016 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:25:02.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.969+0000 7f325e59c640 1 --2- 172.21.15.19:0/114869016 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3248004db0 0x7f32600c5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:02.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.969+0000 7f325ed9d640 1 --2- 172.21.15.19:0/114869016 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3260156d60 0x7f3260157170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:02.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.970+0000 7f325dd9b640 1 -- 172.21.15.19:0/114869016 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3248004db0 msgr2=0x7f32600c5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:02.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.970+0000 7f325dd9b640 1 --2- 172.21.15.19:0/114869016 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3248004db0 0x7f32600c5bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:02.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.970+0000 7f325dd9b640 1 -- 172.21.15.19:0/114869016 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3260156d60 msgr2=0x7f3260157170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:02.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.970+0000 7f325dd9b640 1 --2- 172.21.15.19:0/114869016 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3260156d60 0x7f3260157170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:02.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.970+0000 7f325dd9b640 1 -- 172.21.15.19:0/114869016 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f326014a6f0 con 0x7f32600c4c30 2024-08-31T12:25:02.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.970+0000 7f325dd9b640 1 --2- 172.21.15.19:0/114869016 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f32600c4c30 0x7f32600c5040 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3254006680 tx=0x7f32540066b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:02.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.970+0000 7f324effd640 1 -- 172.21.15.19:0/114869016 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3254015070 con 0x7f32600c4c30 2024-08-31T12:25:02.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.970+0000 7f324effd640 1 -- 172.21.15.19:0/114869016 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3254011840 con 0x7f32600c4c30 2024-08-31T12:25:02.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.970+0000 7f3264e61640 1 -- 172.21.15.19:0/114869016 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f326014a9b0 con 0x7f32600c4c30 2024-08-31T12:25:02.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.970+0000 7f324effd640 1 -- 172.21.15.19:0/114869016 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3254011c70 con 0x7f32600c4c30 2024-08-31T12:25:02.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.970+0000 7f3264e61640 1 -- 172.21.15.19:0/114869016 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32600c4280 con 0x7f32600c4c30 2024-08-31T12:25:02.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.971+0000 7f3264e61640 1 --2- 172.21.15.19:0/114869016 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3260148a50 0x7f3260148e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:02.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.971+0000 7f3264e61640 1 -- 172.21.15.19:0/114869016 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f32600cac10 con 0x7f3260148a50 2024-08-31T12:25:02.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.971+0000 7f325e59c640 1 --2- 172.21.15.19:0/114869016 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3260148a50 0x7f3260148e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:02.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.972+0000 7f325e59c640 1 -- 172.21.15.19:0/114869016 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f32600ccae0 con 0x7f3260148a50 2024-08-31T12:25:02.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.972+0000 7f325e59c640 1 --2- 172.21.15.19:0/114869016 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3260148a50 0x7f3260148e60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f32500065e0 tx=0x7f32500017f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:02.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.972+0000 7f324effd640 1 -- 172.21.15.19:0/114869016 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32500129d0 con 0x7f3260148a50 2024-08-31T12:25:02.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.972+0000 7f324effd640 1 -- 172.21.15.19:0/114869016 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f3254018070 con 0x7f32600c4c30 2024-08-31T12:25:02.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.973+0000 7f324effd640 1 --2- 172.21.15.19:0/114869016 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3240077950 0x7f3240079e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:02.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.973+0000 7f324effd640 1 -- 172.21.15.19:0/114869016 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3250012b70 con 0x7f3260148a50 2024-08-31T12:25:02.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.973+0000 7f324effd640 1 -- 172.21.15.19:0/114869016 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3260148a50 msgr2=0x7f3260148e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:02.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.973+0000 7f324effd640 1 --2- 172.21.15.19:0/114869016 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3260148a50 0x7f3260148e60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f32500065e0 tx=0x7f32500017f0 comp rx=0 tx=0).stop 2024-08-31T12:25:02.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.973+0000 7f324effd640 1 -- 172.21.15.19:0/114869016 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3260148a50 msgr2=0x7f3260148e60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:25:02.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.973+0000 7f324effd640 1 --2- 172.21.15.19:0/114869016 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3260148a50 0x7f3260148e60 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:02.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.973+0000 7f324effd640 1 -- 172.21.15.19:0/114869016 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f325409a2e0 con 0x7f32600c4c30 2024-08-31T12:25:02.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.973+0000 7f325ed9d640 1 --2- 172.21.15.19:0/114869016 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3240077950 0x7f3240079e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:02.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.974+0000 7f325ed9d640 1 --2- 172.21.15.19:0/114869016 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3240077950 0x7f3240079e60 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3258035000 tx=0x7f3258004a40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:02.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.980+0000 7f3264e61640 1 --2- 172.21.15.19:0/114869016 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f32601493a0 0x7f3260155be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:02.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.980+0000 7f3264e61640 1 -- 172.21.15.19:0/114869016 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f32600cadb0 con 0x7f32601493a0 2024-08-31T12:25:02.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.980+0000 7f325e59c640 1 --2- 172.21.15.19:0/114869016 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f32601493a0 0x7f3260155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:02.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.981+0000 7f325e59c640 1 -- 172.21.15.19:0/114869016 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f326014e720 con 0x7f32601493a0 2024-08-31T12:25:02.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.981+0000 7f325e59c640 1 --2- 172.21.15.19:0/114869016 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f32601493a0 0x7f3260155be0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f32500065e0 tx=0x7f3250001340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:02.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.981+0000 7f324effd640 1 -- 172.21.15.19:0/114869016 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f325000f810 con 0x7f32601493a0 2024-08-31T12:25:02.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.981+0000 7f324effd640 1 -- 172.21.15.19:0/114869016 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5042 (secure 0 0 0) 0x7f325000f810 con 0x7f32601493a0 2024-08-31T12:25:02.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.981+0000 7f324effd640 1 -- 172.21.15.19:0/114869016 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f32601493a0 msgr2=0x7f3260155be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:02.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.981+0000 7f324effd640 1 --2- 172.21.15.19:0/114869016 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f32601493a0 0x7f3260155be0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f32500065e0 tx=0x7f3250001340 comp rx=0 tx=0).stop 2024-08-31T12:25:02.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.981+0000 7f324effd640 1 -- 172.21.15.19:0/114869016 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f32601493a0 msgr2=0x7f3260155be0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:25:02.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.981+0000 7f324effd640 1 --2- 172.21.15.19:0/114869016 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f32601493a0 0x7f3260155be0 secure :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f32500065e0 tx=0x7f3250001340 comp rx=0 tx=0).stop 2024-08-31T12:25:02.983 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:25:02.983 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:25:02.983 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:25:02.983 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:25:02.983 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 74, 2024-08-31T12:25:02.983 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 90337, 2024-08-31T12:25:02.984 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:25:02.984 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:25:02.984 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:25:02.984 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:25:02.984 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:02.984 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 12, 2024-08-31T12:25:02.984 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:25:02.984 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:25:02.984 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:25:02.984 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:25:02.984 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:25:02.984 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:25:02.984 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:25:02.985 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:25:02.985 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:25:02.985 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:25:02.985 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:25:02.985 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:25:02.985 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:25:02.985 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:25:02.985 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:02.985 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:25:02.985 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:25:02.985 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:25:02.985 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:25:02.986 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:25:02.986 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:25:02.986 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:25:02.986 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:25:02.986 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:25:02.986 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:25:02.986 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:25:02.986 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:25:02.986 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:25:02.986 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:02.986 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:02.986 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:25:02.987 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:25:02.987 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:25:02.987 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:25:02.987 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:25:02.987 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:25:02.987 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:25:02.987 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:25:02.987 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:25:02.987 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:25:02.987 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:25:02.987 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:25:02.987 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:25:02.988 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:25:02.988 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:25:02.988 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:25:02.988 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:25:02.988 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:25:02.988 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:25:02.988 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:25:02.988 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:25:02.988 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:25:02.988 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:25:02.988 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:25:02.988 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:25:02.988 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:25:02.989 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:25:02.989 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:25:02.989 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:02.989 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:25:02.989 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:02.989 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:25:02.989 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:02.989 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:25:02.989 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:25:02.989 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:25:02.990 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:25:02.990 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:02.990 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:25:02.990 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:25:02.990 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:02.990 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:02.990 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:02.990 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:25:02.990 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:25:02.990 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:02.990 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:02.990 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:02.990 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:02.991 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:25:02.991 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:25:02.991 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:25:02.991 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:25:02.991 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:25:02.991 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:02.991 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:02.991 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:25:02.991 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:25:02.991 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:25:02.991 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:25:02.991 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:02.991 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:25:02.992 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:25:02.992 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:02.992 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:02.992 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:02.992 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:25:02.992 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:25:02.992 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:02.992 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:02.992 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:02.992 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:02.992 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:25:02.992 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:25:02.992 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:25:02.993 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:25:02.993 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:25:02.993 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:02.993 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:02.993 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:25:02.993 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:25:02.993 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:25:02.993 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:25:02.993 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:02.993 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:25:02.993 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:25:02.993 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:02.993 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:02.994 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:02.994 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:25:02.994 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:25:02.994 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:02.994 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:02.994 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:02.994 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:02.994 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:25:02.994 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:25:02.994 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:25:02.994 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:25:02.994 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:25:02.994 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:02.995 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:02.995 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:02.995 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:25:02.995 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:25:02.995 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:02.995 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:02.995 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:02.995 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:25:02.995 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:02.995 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:02.995 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:25:02.995 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:02.995 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:02.996 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:02.996 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 3 2024-08-31T12:25:02.996 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:02.996 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:02.996 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:25:02.996 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:02.996 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:02.996 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:02.996 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 4 2024-08-31T12:25:02.996 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:02.996 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:02.996 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:25:02.996 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:02.996 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:02.997 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:02.997 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:25:02.997 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:02.997 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:02.997 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:25:02.997 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:02.997 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:02.997 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:02.997 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:25:02.997 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:02.997 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:02.997 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:02.997 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:25:02.998 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:25:02.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.982+0000 7f3264e61640 1 -- 172.21.15.19:0/114869016 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3240077950 msgr2=0x7f3240079e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:02.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.982+0000 7f3264e61640 1 --2- 172.21.15.19:0/114869016 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3240077950 0x7f3240079e60 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3258035000 tx=0x7f3258004a40 comp rx=0 tx=0).stop 2024-08-31T12:25:02.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.982+0000 7f3264e61640 1 -- 172.21.15.19:0/114869016 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f32600c4c30 msgr2=0x7f32600c5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:02.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.982+0000 7f3264e61640 1 --2- 172.21.15.19:0/114869016 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f32600c4c30 0x7f32600c5040 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3254006680 tx=0x7f32540066b0 comp rx=0 tx=0).stop 2024-08-31T12:25:02.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.982+0000 7f325ed9d640 1 -- 172.21.15.19:0/114869016 reap_dead start 2024-08-31T12:25:02.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.982+0000 7f325ed9d640 1 -- 172.21.15.19:0/114869016 reap_dead start 2024-08-31T12:25:02.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.982+0000 7f3264e61640 1 -- 172.21.15.19:0/114869016 shutdown_connections 2024-08-31T12:25:02.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.982+0000 7f3264e61640 1 -- 172.21.15.19:0/114869016 >> 172.21.15.19:0/114869016 conn(0x7f326006cae0 msgr2=0x7f3260142940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:03.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.982+0000 7f3264e61640 1 -- 172.21.15.19:0/114869016 shutdown_connections 2024-08-31T12:25:03.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:02.982+0000 7f3264e61640 1 -- 172.21.15.19:0/114869016 wait complete. 2024-08-31T12:25:03.000 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:25:03.000 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-08-31T12:25:03.000 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-08-31T12:25:03.000 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-08-31T12:25:03.000 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-08-31T12:25:03.001 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-08-31T12:25:03.001 INFO:teuthology.orchestra.run:waiting for 300 2024-08-31T12:25:03.016 INFO:tasks.ceph.mon.b:Stopped 2024-08-31T12:25:03.016 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-08-31T12:25:03.016 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:25:03.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.146+0000 7f490632b640 1 -- 172.21.15.19:0/2215540253 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f490008c620 msgr2=0x7f48e80039e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:03.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.146+0000 7f490632b640 1 --2- 172.21.15.19:0/2215540253 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f490008c620 0x7f48e80039e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f48f000c290 tx=0x7f48e8005830 comp rx=0 tx=0).stop 2024-08-31T12:25:03.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.146+0000 7f49048c8640 1 -- 172.21.15.19:0/2215540253 reap_dead start 2024-08-31T12:25:03.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.146+0000 7f490632b640 1 -- 172.21.15.19:0/2215540253 shutdown_connections 2024-08-31T12:25:03.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.146+0000 7f490632b640 1 -- 172.21.15.19:0/2215540253 >> 172.21.15.19:0/2215540253 conn(0x7f49000cc770 msgr2=0x7f49000ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:03.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.146+0000 7f490632b640 1 -- 172.21.15.19:0/2215540253 shutdown_connections 2024-08-31T12:25:03.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.146+0000 7f490632b640 1 -- 172.21.15.19:0/2215540253 wait complete. 2024-08-31T12:25:03.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.148+0000 7f490632b640 1 Processor -- start 2024-08-31T12:25:03.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.148+0000 7f490632b640 1 -- start start 2024-08-31T12:25:03.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.149+0000 7f490632b640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f490008c620 0x7f49001fe120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:03.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.149+0000 7f490632b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f49000cc0a0 0x7f49001636e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:03.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.149+0000 7f490632b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f49001592a0 0x7f49001596b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:03.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.149+0000 7f490632b640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f49000737b0 con 0x7f49000cc0a0 2024-08-31T12:25:03.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.149+0000 7f490632b640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4900070560 con 0x7f490008c620 2024-08-31T12:25:03.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.149+0000 7f490632b640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f4900073930 con 0x7f49001592a0 2024-08-31T12:25:03.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.149+0000 7f48fffff640 1 -- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f490008c620 msgr2=0x7f49001fe120 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3300/0 2024-08-31T12:25:03.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.149+0000 7f48fffff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f490008c620 0x7f49001fe120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:25:03.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.149+0000 7f48ff7fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f49000cc0a0 0x7f49001636e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:03.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.149+0000 7f48ff7fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f49000cc0a0 0x7f49001636e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:35566/0 (socket says 172.21.15.19:35566) 2024-08-31T12:25:03.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.149+0000 7f48ff7fe640 1 -- 172.21.15.19:0/3038459964 learned_addr learned my addr 172.21.15.19:0/3038459964 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:25:03.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.149+0000 7f49048c8640 1 --2- 172.21.15.19:0/3038459964 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f49001592a0 0x7f49001596b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:03.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.150+0000 7f48ff7fe640 1 -- 172.21.15.19:0/3038459964 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f49001592a0 msgr2=0x7f49001596b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:03.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.150+0000 7f48ff7fe640 1 --2- 172.21.15.19:0/3038459964 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f49001592a0 0x7f49001596b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:03.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.150+0000 7f48ff7fe640 1 -- 172.21.15.19:0/3038459964 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f490008c620 msgr2=0x7f49001fe120 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:25:03.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.150+0000 7f48ff7fe640 1 --2- 172.21.15.19:0/3038459964 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f490008c620 0x7f49001fe120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:03.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.150+0000 7f48ff7fe640 1 -- 172.21.15.19:0/3038459964 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49001fe660 con 0x7f49000cc0a0 2024-08-31T12:25:03.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.150+0000 7f48ff7fe640 1 --2- 172.21.15.19:0/3038459964 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f49000cc0a0 0x7f49001636e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f48f4011b10 tx=0x7f48f4009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:03.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.150+0000 7f48fcff9640 1 -- 172.21.15.19:0/3038459964 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48f4004460 con 0x7f49000cc0a0 2024-08-31T12:25:03.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.150+0000 7f48fcff9640 1 -- 172.21.15.19:0/3038459964 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f48f400f9d0 con 0x7f49000cc0a0 2024-08-31T12:25:03.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.150+0000 7f48fcff9640 1 -- 172.21.15.19:0/3038459964 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48f400fde0 con 0x7f49000cc0a0 2024-08-31T12:25:03.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.150+0000 7f490632b640 1 -- 172.21.15.19:0/3038459964 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f490015dc70 con 0x7f49000cc0a0 2024-08-31T12:25:03.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.150+0000 7f490632b640 1 -- 172.21.15.19:0/3038459964 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f490015e230 con 0x7f49000cc0a0 2024-08-31T12:25:03.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.151+0000 7f490632b640 1 -- 172.21.15.19:0/3038459964 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49000c7fe0 con 0x7f49000cc0a0 2024-08-31T12:25:03.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.152+0000 7f48fcff9640 1 -- 172.21.15.19:0/3038459964 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f48f4005020 con 0x7f49000cc0a0 2024-08-31T12:25:03.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.153+0000 7f48fcff9640 1 --2- 172.21.15.19:0/3038459964 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f48ec077a30 0x7f48ec079f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:03.155 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.153+0000 7f48fcff9640 1 -- 172.21.15.19:0/3038459964 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f48f409a0f0 con 0x7f49000cc0a0 2024-08-31T12:25:03.155 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.153+0000 7f48fffff640 1 --2- 172.21.15.19:0/3038459964 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f48ec077a30 0x7f48ec079f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:03.155 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.154+0000 7f48fffff640 1 --2- 172.21.15.19:0/3038459964 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f48ec077a30 0x7f48ec079f40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f48f0033000 tx=0x7f4900168a40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:03.158 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.158+0000 7f48fcff9640 1 -- 172.21.15.19:0/3038459964 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f48f4062820 con 0x7f49000cc0a0 2024-08-31T12:25:03.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.462+0000 7f490632b640 1 -- 172.21.15.19:0/3038459964 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4900074130 con 0x7f49000cc0a0 2024-08-31T12:25:03.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.462+0000 7f48fcff9640 1 -- 172.21.15.19:0/3038459964 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f48f406b320 con 0x7f49000cc0a0 2024-08-31T12:25:03.464 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":74,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:25:03.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.464+0000 7f490632b640 1 -- 172.21.15.19:0/3038459964 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f48ec077a30 msgr2=0x7f48ec079f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:03.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.464+0000 7f490632b640 1 --2- 172.21.15.19:0/3038459964 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f48ec077a30 0x7f48ec079f40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f48f0033000 tx=0x7f4900168a40 comp rx=0 tx=0).stop 2024-08-31T12:25:03.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.464+0000 7f490632b640 1 -- 172.21.15.19:0/3038459964 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f49000cc0a0 msgr2=0x7f49001636e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:03.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.464+0000 7f490632b640 1 --2- 172.21.15.19:0/3038459964 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f49000cc0a0 0x7f49001636e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f48f4011b10 tx=0x7f48f4009e20 comp rx=0 tx=0).stop 2024-08-31T12:25:03.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.465+0000 7f490632b640 1 -- 172.21.15.19:0/3038459964 shutdown_connections 2024-08-31T12:25:03.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.465+0000 7f490632b640 1 --2- 172.21.15.19:0/3038459964 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f48ec077a30 0x7f48ec079f40 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:03.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.465+0000 7f490632b640 1 --2- 172.21.15.19:0/3038459964 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f49001592a0 0x7f49001596b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:03.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.465+0000 7f490632b640 1 --2- 172.21.15.19:0/3038459964 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f49000cc0a0 0x7f49001636e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:03.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.465+0000 7f490632b640 1 --2- 172.21.15.19:0/3038459964 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f490008c620 0x7f49001fe120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:03.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.465+0000 7f490632b640 1 -- 172.21.15.19:0/3038459964 >> 172.21.15.19:0/3038459964 conn(0x7f49000cc770 msgr2=0x7f490006d190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:03.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.465+0000 7f490632b640 1 -- 172.21.15.19:0/3038459964 shutdown_connections 2024-08-31T12:25:03.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:03.465+0000 7f490632b640 1 -- 172.21.15.19:0/3038459964 wait complete. 2024-08-31T12:25:06.480 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:25:06.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.600+0000 7f1c54cce640 1 -- 172.21.15.19:0/2746906908 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1c38003f60 msgr2=0x7f1c38004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:06.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.600+0000 7f1c54cce640 1 --2- 172.21.15.19:0/2746906908 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1c38003f60 0x7f1c38004800 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1c500cc4e0 tx=0x7f1c38005830 comp rx=0 tx=0).stop 2024-08-31T12:25:06.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.600+0000 7f1c4ed9d640 1 -- 172.21.15.19:0/2746906908 reap_dead start 2024-08-31T12:25:06.603 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.600+0000 7f1c54cce640 1 -- 172.21.15.19:0/2746906908 shutdown_connections 2024-08-31T12:25:06.603 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.600+0000 7f1c54cce640 1 -- 172.21.15.19:0/2746906908 >> 172.21.15.19:0/2746906908 conn(0x7f1c50072c80 msgr2=0x7f1c50073090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:06.603 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.600+0000 7f1c54cce640 1 -- 172.21.15.19:0/2746906908 shutdown_connections 2024-08-31T12:25:06.603 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.600+0000 7f1c54cce640 1 -- 172.21.15.19:0/2746906908 wait complete. 2024-08-31T12:25:06.604 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.602+0000 7f1c54cce640 1 Processor -- start 2024-08-31T12:25:06.604 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.602+0000 7f1c54cce640 1 -- start start 2024-08-31T12:25:06.606 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.603+0000 7f1c54cce640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1c38003f60 0x7f1c5015b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:06.607 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.603+0000 7f1c54cce640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1c5006dbf0 0x7f1c500ce7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:06.607 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.603+0000 7f1c54cce640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c501f19b0 0x7f1c500ced10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:06.607 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.603+0000 7f1c54cce640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f1c500721e0 con 0x7f1c5006dbf0 2024-08-31T12:25:06.607 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.603+0000 7f1c54cce640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1c500cae30 con 0x7f1c501f19b0 2024-08-31T12:25:06.607 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.603+0000 7f1c54cce640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f1c50072360 con 0x7f1c38003f60 2024-08-31T12:25:06.607 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.604+0000 7f1c4dd9b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1c5006dbf0 0x7f1c500ce7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:06.607 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.604+0000 7f1c4ed9d640 1 -- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c501f19b0 msgr2=0x7f1c500ced10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3300/0 2024-08-31T12:25:06.607 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.604+0000 7f1c4ed9d640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c501f19b0 0x7f1c500ced10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:25:06.607 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.604+0000 7f1c4dd9b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1c5006dbf0 0x7f1c500ce7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:35590/0 (socket says 172.21.15.19:35590) 2024-08-31T12:25:06.607 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.604+0000 7f1c4dd9b640 1 -- 172.21.15.19:0/2607603331 learned_addr learned my addr 172.21.15.19:0/2607603331 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:25:06.607 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.604+0000 7f1c4e59c640 1 --2- 172.21.15.19:0/2607603331 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1c38003f60 0x7f1c5015b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:06.608 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.604+0000 7f1c4dd9b640 1 -- 172.21.15.19:0/2607603331 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1c38003f60 msgr2=0x7f1c5015b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:06.608 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.604+0000 7f1c4dd9b640 1 --2- 172.21.15.19:0/2607603331 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1c38003f60 0x7f1c5015b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:06.608 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.604+0000 7f1c4dd9b640 1 -- 172.21.15.19:0/2607603331 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c501f19b0 msgr2=0x7f1c500ced10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:25:06.608 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.604+0000 7f1c4dd9b640 1 --2- 172.21.15.19:0/2607603331 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c501f19b0 0x7f1c500ced10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:06.608 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.604+0000 7f1c4dd9b640 1 -- 172.21.15.19:0/2607603331 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c5006f540 con 0x7f1c5006dbf0 2024-08-31T12:25:06.608 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.604+0000 7f1c4dd9b640 1 --2- 172.21.15.19:0/2607603331 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1c5006dbf0 0x7f1c500ce7d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1c44009a70 tx=0x7f1c44032e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:06.608 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.604+0000 7f1c3effd640 1 -- 172.21.15.19:0/2607603331 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c4403f030 con 0x7f1c5006dbf0 2024-08-31T12:25:06.608 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.604+0000 7f1c3effd640 1 -- 172.21.15.19:0/2607603331 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1c4403b500 con 0x7f1c5006dbf0 2024-08-31T12:25:06.608 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.604+0000 7f1c54cce640 1 -- 172.21.15.19:0/2607603331 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c500b02c0 con 0x7f1c5006dbf0 2024-08-31T12:25:06.608 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.604+0000 7f1c3effd640 1 -- 172.21.15.19:0/2607603331 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c44037040 con 0x7f1c5006dbf0 2024-08-31T12:25:06.608 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.605+0000 7f1c54cce640 1 -- 172.21.15.19:0/2607603331 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c500b0880 con 0x7f1c5006dbf0 2024-08-31T12:25:06.609 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.606+0000 7f1c54cce640 1 -- 172.21.15.19:0/2607603331 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c5006f2b0 con 0x7f1c5006dbf0 2024-08-31T12:25:06.609 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.606+0000 7f1c3effd640 1 -- 172.21.15.19:0/2607603331 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f1c44042070 con 0x7f1c5006dbf0 2024-08-31T12:25:06.609 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.607+0000 7f1c3effd640 1 --2- 172.21.15.19:0/2607603331 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1c30077a30 0x7f1c30079f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:06.609 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.607+0000 7f1c3effd640 1 -- 172.21.15.19:0/2607603331 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f1c440b9340 con 0x7f1c5006dbf0 2024-08-31T12:25:06.609 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.607+0000 7f1c4e59c640 1 --2- 172.21.15.19:0/2607603331 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1c30077a30 0x7f1c30079f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:06.610 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.608+0000 7f1c4e59c640 1 --2- 172.21.15.19:0/2607603331 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1c30077a30 0x7f1c30079f40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f1c40006c50 tx=0x7f1c40006720 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:06.613 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.611+0000 7f1c3effd640 1 -- 172.21.15.19:0/2607603331 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f1c44049070 con 0x7f1c5006dbf0 2024-08-31T12:25:06.896 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.893+0000 7f1c54cce640 1 -- 172.21.15.19:0/2607603331 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1c50070ad0 con 0x7f1c5006dbf0 2024-08-31T12:25:06.896 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.894+0000 7f1c3effd640 1 -- 172.21.15.19:0/2607603331 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f1c44082030 con 0x7f1c5006dbf0 2024-08-31T12:25:06.896 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":74,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:25:06.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.896+0000 7f1c54cce640 1 -- 172.21.15.19:0/2607603331 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1c30077a30 msgr2=0x7f1c30079f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:06.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.896+0000 7f1c54cce640 1 --2- 172.21.15.19:0/2607603331 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1c30077a30 0x7f1c30079f40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f1c40006c50 tx=0x7f1c40006720 comp rx=0 tx=0).stop 2024-08-31T12:25:06.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.897+0000 7f1c54cce640 1 -- 172.21.15.19:0/2607603331 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1c5006dbf0 msgr2=0x7f1c500ce7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:06.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.897+0000 7f1c54cce640 1 --2- 172.21.15.19:0/2607603331 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1c5006dbf0 0x7f1c500ce7d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1c44009a70 tx=0x7f1c44032e40 comp rx=0 tx=0).stop 2024-08-31T12:25:06.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.897+0000 7f1c54cce640 1 -- 172.21.15.19:0/2607603331 shutdown_connections 2024-08-31T12:25:06.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.897+0000 7f1c54cce640 1 --2- 172.21.15.19:0/2607603331 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1c30077a30 0x7f1c30079f40 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:06.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.897+0000 7f1c54cce640 1 --2- 172.21.15.19:0/2607603331 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c501f19b0 0x7f1c500ced10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:06.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.897+0000 7f1c54cce640 1 --2- 172.21.15.19:0/2607603331 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1c5006dbf0 0x7f1c500ce7d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:06.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.897+0000 7f1c54cce640 1 --2- 172.21.15.19:0/2607603331 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1c38003f60 0x7f1c5015b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:06.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.897+0000 7f1c54cce640 1 -- 172.21.15.19:0/2607603331 >> 172.21.15.19:0/2607603331 conn(0x7f1c50072c80 msgr2=0x7f1c5014efd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:06.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.897+0000 7f1c54cce640 1 -- 172.21.15.19:0/2607603331 shutdown_connections 2024-08-31T12:25:06.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:06.897+0000 7f1c54cce640 1 -- 172.21.15.19:0/2607603331 wait complete. 2024-08-31T12:25:09.911 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:25:10.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.060+0000 7f6948c0e640 1 -- 172.21.15.19:0/1171090244 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f694414e970 msgr2=0x7f69440c8340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:10.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.060+0000 7f6948c0e640 1 --2- 172.21.15.19:0/1171090244 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f694414e970 0x7f69440c8340 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f69440c7410 tx=0x7f693402a580 comp rx=0 tx=0).stop 2024-08-31T12:25:10.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.060+0000 7f6948c0e640 1 -- 172.21.15.19:0/1171090244 shutdown_connections 2024-08-31T12:25:10.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.060+0000 7f6948c0e640 1 --2- 172.21.15.19:0/1171090244 >> v2:172.21.15.43:3300/0 conn(0x7f6944160900 0x7f6944162d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:10.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.060+0000 7f6948c0e640 1 --2- 172.21.15.19:0/1171090244 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f694414e970 0x7f69440c8340 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:10.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.060+0000 7f6948c0e640 1 -- 172.21.15.19:0/1171090244 >> 172.21.15.19:0/1171090244 conn(0x7f69440ccda0 msgr2=0x7f69440cd1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:10.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.060+0000 7f6948c0e640 1 -- 172.21.15.19:0/1171090244 shutdown_connections 2024-08-31T12:25:10.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.060+0000 7f6948c0e640 1 -- 172.21.15.19:0/1171090244 wait complete. 2024-08-31T12:25:10.064 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.062+0000 7f6948c0e640 1 Processor -- start 2024-08-31T12:25:10.064 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.062+0000 7f6948c0e640 1 -- start start 2024-08-31T12:25:10.067 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.063+0000 7f6948c0e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f69440c3470 0x7f69440733f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:10.067 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.063+0000 7f6948c0e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f694414e970 0x7f694413f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:10.067 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.063+0000 7f6948c0e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6944160900 0x7f694413fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:10.067 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.063+0000 7f6948c0e640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f6944158660 con 0x7f694414e970 2024-08-31T12:25:10.067 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.063+0000 7f6948c0e640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6944158140 con 0x7f6944160900 2024-08-31T12:25:10.067 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.063+0000 7f6948c0e640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f69440c2fe0 con 0x7f69440c3470 2024-08-31T12:25:10.067 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.063+0000 7f6941d9b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f694414e970 0x7f694413f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:10.067 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.063+0000 7f6942d9d640 1 -- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6944160900 msgr2=0x7f694413fbe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3300/0 2024-08-31T12:25:10.067 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.063+0000 7f6941d9b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f694414e970 0x7f694413f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:48818/0 (socket says 172.21.15.19:48818) 2024-08-31T12:25:10.067 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.064+0000 7f6941d9b640 1 -- 172.21.15.19:0/722912103 learned_addr learned my addr 172.21.15.19:0/722912103 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:25:10.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.064+0000 7f6942d9d640 1 --2- 172.21.15.19:0/722912103 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6944160900 0x7f694413fbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:25:10.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.064+0000 7f694259c640 1 --2- 172.21.15.19:0/722912103 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f69440c3470 0x7f69440733f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:10.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.064+0000 7f6941d9b640 1 -- 172.21.15.19:0/722912103 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f69440c3470 msgr2=0x7f69440733f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:10.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.064+0000 7f6941d9b640 1 --2- 172.21.15.19:0/722912103 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f69440c3470 0x7f69440733f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:10.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.064+0000 7f6941d9b640 1 -- 172.21.15.19:0/722912103 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6944160900 msgr2=0x7f694413fbe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:25:10.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.064+0000 7f6941d9b640 1 --2- 172.21.15.19:0/722912103 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6944160900 0x7f694413fbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:10.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.064+0000 7f6941d9b640 1 -- 172.21.15.19:0/722912103 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6944073ed0 con 0x7f694414e970 2024-08-31T12:25:10.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.064+0000 7f6941d9b640 1 --2- 172.21.15.19:0/722912103 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f694414e970 0x7f694413f6a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f693800b580 tx=0x7f693800ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:10.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.064+0000 7f6932ffd640 1 -- 172.21.15.19:0/722912103 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6938014070 con 0x7f694414e970 2024-08-31T12:25:10.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.064+0000 7f6932ffd640 1 -- 172.21.15.19:0/722912103 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f69380109a0 con 0x7f694414e970 2024-08-31T12:25:10.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.064+0000 7f6948c0e640 1 -- 172.21.15.19:0/722912103 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6944160220 con 0x7f694414e970 2024-08-31T12:25:10.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.064+0000 7f6932ffd640 1 -- 172.21.15.19:0/722912103 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6938010df0 con 0x7f694414e970 2024-08-31T12:25:10.069 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.064+0000 7f6948c0e640 1 -- 172.21.15.19:0/722912103 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69440cc910 con 0x7f694414e970 2024-08-31T12:25:10.069 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.065+0000 7f6948c0e640 1 -- 172.21.15.19:0/722912103 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6944156f30 con 0x7f694414e970 2024-08-31T12:25:10.069 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.066+0000 7f6932ffd640 1 -- 172.21.15.19:0/722912103 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f6938017070 con 0x7f694414e970 2024-08-31T12:25:10.070 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.067+0000 7f6932ffd640 1 --2- 172.21.15.19:0/722912103 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6924077a00 0x7f6924079f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:10.070 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.067+0000 7f6932ffd640 1 -- 172.21.15.19:0/722912103 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f6938099680 con 0x7f694414e970 2024-08-31T12:25:10.070 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.067+0000 7f694259c640 1 --2- 172.21.15.19:0/722912103 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6924077a00 0x7f6924079f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:10.070 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.067+0000 7f694259c640 1 --2- 172.21.15.19:0/722912103 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6924077a00 0x7f6924079f10 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f6934033000 tx=0x7f6934004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:10.074 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.072+0000 7f6932ffd640 1 -- 172.21.15.19:0/722912103 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f6938062370 con 0x7f694414e970 2024-08-31T12:25:10.356 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.354+0000 7f6948c0e640 1 -- 172.21.15.19:0/722912103 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f694414ed80 con 0x7f694414e970 2024-08-31T12:25:10.356 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.354+0000 7f6932ffd640 1 -- 172.21.15.19:0/722912103 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f6938061c70 con 0x7f694414e970 2024-08-31T12:25:10.356 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":74,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":19,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:25:10.359 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.357+0000 7f6948c0e640 1 -- 172.21.15.19:0/722912103 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6924077a00 msgr2=0x7f6924079f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:10.359 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.357+0000 7f6948c0e640 1 --2- 172.21.15.19:0/722912103 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6924077a00 0x7f6924079f10 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f6934033000 tx=0x7f6934004730 comp rx=0 tx=0).stop 2024-08-31T12:25:10.359 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.357+0000 7f6948c0e640 1 -- 172.21.15.19:0/722912103 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f694414e970 msgr2=0x7f694413f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:10.359 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.357+0000 7f6948c0e640 1 --2- 172.21.15.19:0/722912103 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f694414e970 0x7f694413f6a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f693800b580 tx=0x7f693800ba40 comp rx=0 tx=0).stop 2024-08-31T12:25:10.359 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.357+0000 7f6948c0e640 1 -- 172.21.15.19:0/722912103 shutdown_connections 2024-08-31T12:25:10.359 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.357+0000 7f6948c0e640 1 --2- 172.21.15.19:0/722912103 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6924077a00 0x7f6924079f10 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:10.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.357+0000 7f6948c0e640 1 --2- 172.21.15.19:0/722912103 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6944160900 0x7f694413fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:10.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.357+0000 7f6948c0e640 1 --2- 172.21.15.19:0/722912103 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f694414e970 0x7f694413f6a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:10.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.357+0000 7f6948c0e640 1 --2- 172.21.15.19:0/722912103 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f69440c3470 0x7f69440733f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:10.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.357+0000 7f6948c0e640 1 -- 172.21.15.19:0/722912103 >> 172.21.15.19:0/722912103 conn(0x7f69440ccda0 msgr2=0x7f694406cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:10.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.357+0000 7f6948c0e640 1 -- 172.21.15.19:0/722912103 shutdown_connections 2024-08-31T12:25:10.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:10.357+0000 7f6948c0e640 1 -- 172.21.15.19:0/722912103 wait complete. 2024-08-31T12:25:13.372 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:25:17.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:17.922+0000 7febd434a640 1 -- 172.21.15.19:0/276294773 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7febb8005350 msgr2=0x7febb800ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:17.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:17.922+0000 7febd434a640 1 --2- 172.21.15.19:0/276294773 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7febb8005350 0x7febb800ba00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7febcc069f30 tx=0x7febb8005830 comp rx=0 tx=0).stop 2024-08-31T12:25:17.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:17.922+0000 7febd434a640 1 -- 172.21.15.19:0/276294773 shutdown_connections 2024-08-31T12:25:17.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:17.922+0000 7febd434a640 1 --2- 172.21.15.19:0/276294773 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7febb8005350 0x7febb800ba00 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:17.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:17.922+0000 7febd434a640 1 --2- 172.21.15.19:0/276294773 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7febb8004800 0x7febb8004c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:17.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:17.922+0000 7febd434a640 1 --2- 172.21.15.19:0/276294773 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7febb80039b0 0x7febb8003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:17.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:17.922+0000 7febd434a640 1 -- 172.21.15.19:0/276294773 >> 172.21.15.19:0/276294773 conn(0x7febcc0cd7d0 msgr2=0x7febcc0cdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:17.926 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:17.922+0000 7febd434a640 1 -- 172.21.15.19:0/276294773 shutdown_connections 2024-08-31T12:25:17.926 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:17.922+0000 7febd434a640 1 -- 172.21.15.19:0/276294773 wait complete. 2024-08-31T12:25:17.927 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:17.925+0000 7febd434a640 1 Processor -- start 2024-08-31T12:25:17.927 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:17.925+0000 7febd434a640 1 -- start start 2024-08-31T12:25:17.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:17.926+0000 7febd434a640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7febb80039b0 0x7febcc134f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:17.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:17.926+0000 7febd434a640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7febb8004800 0x7febcc151170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:17.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:17.926+0000 7febd434a640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7febb8005350 0x7febcc13b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:17.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:17.926+0000 7febd434a640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7febcc0c5360 con 0x7febb80039b0 2024-08-31T12:25:17.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:17.926+0000 7febd434a640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7febcc0c6a70 con 0x7febb8005350 2024-08-31T12:25:17.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:17.926+0000 7febd434a640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7febcc1404e0 con 0x7febb8004800 2024-08-31T12:25:17.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:17.927+0000 7febd28e7640 1 -- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7febb8005350 msgr2=0x7febcc13b000 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3300/0 2024-08-31T12:25:17.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:17.927+0000 7febd28e7640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7febb8005350 0x7febcc13b000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:25:17.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:17.927+0000 7febd20e6640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7febb80039b0 0x7febcc134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:17.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:17.927+0000 7febd20e6640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7febb80039b0 0x7febcc134f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:48860/0 (socket says 172.21.15.19:48860) 2024-08-31T12:25:17.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:17.927+0000 7febd20e6640 1 -- 172.21.15.19:0/848243279 learned_addr learned my addr 172.21.15.19:0/848243279 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:25:17.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:17.927+0000 7febd18e5640 1 --2- 172.21.15.19:0/848243279 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7febb8004800 0x7febcc151170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:17.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:17.927+0000 7febd20e6640 1 -- 172.21.15.19:0/848243279 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7febb8004800 msgr2=0x7febcc151170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:17.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:17.927+0000 7febd20e6640 1 --2- 172.21.15.19:0/848243279 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7febb8004800 0x7febcc151170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:17.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:17.927+0000 7febd20e6640 1 -- 172.21.15.19:0/848243279 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7febb8005350 msgr2=0x7febcc13b000 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:25:17.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:17.927+0000 7febd20e6640 1 --2- 172.21.15.19:0/848243279 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7febb8005350 0x7febcc13b000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:17.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:17.927+0000 7febd20e6640 1 -- 172.21.15.19:0/848243279 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febcc13c070 con 0x7febb80039b0 2024-08-31T12:25:17.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:17.927+0000 7febd20e6640 1 --2- 172.21.15.19:0/848243279 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7febb80039b0 0x7febcc134f70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7febc800b7a0 tx=0x7febc800bc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:17.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:17.927+0000 7febc2ffd640 1 -- 172.21.15.19:0/848243279 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febc8019070 con 0x7febb80039b0 2024-08-31T12:25:17.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:17.927+0000 7febc2ffd640 1 -- 172.21.15.19:0/848243279 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7febc800c990 con 0x7febb80039b0 2024-08-31T12:25:17.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:17.927+0000 7febd434a640 1 -- 172.21.15.19:0/848243279 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febcc13c350 con 0x7febb80039b0 2024-08-31T12:25:17.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:17.927+0000 7febc2ffd640 1 -- 172.21.15.19:0/848243279 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febc800cde0 con 0x7febb80039b0 2024-08-31T12:25:17.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:17.928+0000 7febd434a640 1 -- 172.21.15.19:0/848243279 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febcc134770 con 0x7febb80039b0 2024-08-31T12:25:17.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:17.929+0000 7febd434a640 1 -- 172.21.15.19:0/848243279 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febcc0c7a30 con 0x7febb80039b0 2024-08-31T12:25:17.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:17.929+0000 7febc2ffd640 1 -- 172.21.15.19:0/848243279 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7febc801c070 con 0x7febb80039b0 2024-08-31T12:25:17.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:17.930+0000 7febc2ffd640 1 --2- 172.21.15.19:0/848243279 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7febb0078250 0x7febb007a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:17.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:17.930+0000 7febc2ffd640 1 -- 172.21.15.19:0/848243279 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7febc8098800 con 0x7febb80039b0 2024-08-31T12:25:17.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:17.930+0000 7febd18e5640 1 --2- 172.21.15.19:0/848243279 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7febb0078250 0x7febb007a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:17.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:17.931+0000 7febd18e5640 1 --2- 172.21.15.19:0/848243279 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7febb0078250 0x7febb007a760 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7febcc139f20 tx=0x7febbc015040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:17.937 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:17.935+0000 7febc2ffd640 1 -- 172.21.15.19:0/848243279 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7febc8051cd0 con 0x7febb80039b0 2024-08-31T12:25:18.234 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.232+0000 7febd434a640 1 -- 172.21.15.19:0/848243279 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7febcc139a20 con 0x7febb80039b0 2024-08-31T12:25:18.235 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.232+0000 7febc2ffd640 1 -- 172.21.15.19:0/848243279 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7febc8060f80 con 0x7febb80039b0 2024-08-31T12:25:18.235 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":78,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:25:18.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.235+0000 7febd434a640 1 -- 172.21.15.19:0/848243279 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7febb0078250 msgr2=0x7febb007a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:18.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.235+0000 7febd434a640 1 --2- 172.21.15.19:0/848243279 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7febb0078250 0x7febb007a760 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7febcc139f20 tx=0x7febbc015040 comp rx=0 tx=0).stop 2024-08-31T12:25:18.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.235+0000 7febd434a640 1 -- 172.21.15.19:0/848243279 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7febb80039b0 msgr2=0x7febcc134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:18.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.235+0000 7febd434a640 1 --2- 172.21.15.19:0/848243279 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7febb80039b0 0x7febcc134f70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7febc800b7a0 tx=0x7febc800bc60 comp rx=0 tx=0).stop 2024-08-31T12:25:18.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.235+0000 7febd434a640 1 -- 172.21.15.19:0/848243279 shutdown_connections 2024-08-31T12:25:18.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.235+0000 7febd434a640 1 --2- 172.21.15.19:0/848243279 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7febb0078250 0x7febb007a760 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:18.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.235+0000 7febd434a640 1 --2- 172.21.15.19:0/848243279 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7febb8005350 0x7febcc13b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:18.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.235+0000 7febd434a640 1 --2- 172.21.15.19:0/848243279 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7febb8004800 0x7febcc151170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:18.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.235+0000 7febd434a640 1 --2- 172.21.15.19:0/848243279 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7febb80039b0 0x7febcc134f70 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:18.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.235+0000 7febd434a640 1 -- 172.21.15.19:0/848243279 >> 172.21.15.19:0/848243279 conn(0x7febcc0cd7d0 msgr2=0x7febcc0cdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:18.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.235+0000 7febd434a640 1 -- 172.21.15.19:0/848243279 shutdown_connections 2024-08-31T12:25:18.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.235+0000 7febd434a640 1 -- 172.21.15.19:0/848243279 wait complete. 2024-08-31T12:25:18.250 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-08-31T12:25:18.251 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-08-31T12:25:18.251 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-31T12:25:18.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.412+0000 7f2668c7e640 1 -- 172.21.15.19:0/589494012 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f26640c8a40 msgr2=0x7f26640c9850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:18.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.412+0000 7f2668c7e640 1 --2- 172.21.15.19:0/589494012 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f26640c8a40 0x7f26640c9850 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2654009830 tx=0x7f265402a610 comp rx=0 tx=0).stop 2024-08-31T12:25:18.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.412+0000 7f2668c7e640 1 -- 172.21.15.19:0/589494012 shutdown_connections 2024-08-31T12:25:18.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.412+0000 7f2668c7e640 1 --2- 172.21.15.19:0/589494012 >> v2:172.21.15.43:3300/0 conn(0x7f266406e0e0 0x7f2664155300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:18.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.412+0000 7f2668c7e640 1 --2- 172.21.15.19:0/589494012 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f26640c8a40 0x7f26640c9850 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:18.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.412+0000 7f2668c7e640 1 -- 172.21.15.19:0/589494012 >> 172.21.15.19:0/589494012 conn(0x7f26640690d0 msgr2=0x7f26640d0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:18.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.412+0000 7f2668c7e640 1 -- 172.21.15.19:0/589494012 shutdown_connections 2024-08-31T12:25:18.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.412+0000 7f2668c7e640 1 -- 172.21.15.19:0/589494012 wait complete. 2024-08-31T12:25:18.416 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.414+0000 7f2668c7e640 1 Processor -- start 2024-08-31T12:25:18.416 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.414+0000 7f2668c7e640 1 -- start start 2024-08-31T12:25:18.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.415+0000 7f2668c7e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f266406e0e0 0x7f266415c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:18.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.415+0000 7f2668c7e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f26640c8a40 0x7f26641f4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:18.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.415+0000 7f2668c7e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f266415b6e0 0x7f26641f5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:18.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.415+0000 7f2668c7e640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f26640c8330 con 0x7f266415b6e0 2024-08-31T12:25:18.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.415+0000 7f2668c7e640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f266415b250 con 0x7f26640c8a40 2024-08-31T12:25:18.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.415+0000 7f2668c7e640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f26640c8610 con 0x7f266406e0e0 2024-08-31T12:25:18.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.415+0000 7f265bfff640 1 -- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f26640c8a40 msgr2=0x7f26641f4f40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3300/0 2024-08-31T12:25:18.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.415+0000 7f2662d9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f266415b6e0 0x7f26641f5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:18.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.415+0000 7f265bfff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f26640c8a40 0x7f26641f4f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:25:18.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.415+0000 7f2662d9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f266415b6e0 0x7f26641f5480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:34720/0 (socket says 172.21.15.19:34720) 2024-08-31T12:25:18.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.415+0000 7f2662d9d640 1 -- 172.21.15.19:0/2772032621 learned_addr learned my addr 172.21.15.19:0/2772032621 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:25:18.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.415+0000 7f266259c640 1 --2- 172.21.15.19:0/2772032621 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f266406e0e0 0x7f266415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:18.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.416+0000 7f2662d9d640 1 -- 172.21.15.19:0/2772032621 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f266406e0e0 msgr2=0x7f266415c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:18.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.416+0000 7f2662d9d640 1 --2- 172.21.15.19:0/2772032621 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f266406e0e0 0x7f266415c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:18.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.416+0000 7f2662d9d640 1 -- 172.21.15.19:0/2772032621 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f26640c8a40 msgr2=0x7f26641f4f40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:25:18.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.416+0000 7f2662d9d640 1 --2- 172.21.15.19:0/2772032621 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f26640c8a40 0x7f26641f4f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:18.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.416+0000 7f2662d9d640 1 -- 172.21.15.19:0/2772032621 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f266415dac0 con 0x7f266415b6e0 2024-08-31T12:25:18.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.416+0000 7f2662d9d640 1 --2- 172.21.15.19:0/2772032621 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f266415b6e0 0x7f26641f5480 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f265c00ca80 tx=0x7f265c00cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:18.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.416+0000 7f265affd640 1 -- 172.21.15.19:0/2772032621 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f265c016020 con 0x7f266415b6e0 2024-08-31T12:25:18.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.416+0000 7f265affd640 1 -- 172.21.15.19:0/2772032621 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f265c00ad20 con 0x7f266415b6e0 2024-08-31T12:25:18.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.416+0000 7f265affd640 1 -- 172.21.15.19:0/2772032621 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f265c012680 con 0x7f266415b6e0 2024-08-31T12:25:18.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.416+0000 7f2668c7e640 1 -- 172.21.15.19:0/2772032621 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f266415dda0 con 0x7f266415b6e0 2024-08-31T12:25:18.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.416+0000 7f2668c7e640 1 -- 172.21.15.19:0/2772032621 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26641e6070 con 0x7f266415b6e0 2024-08-31T12:25:18.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.417+0000 7f2668c7e640 1 --2- 172.21.15.19:0/2772032621 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f266415cb40 0x7f26640c6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:18.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.417+0000 7f2668c7e640 1 -- 172.21.15.19:0/2772032621 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f266415cfd0 con 0x7f266415cb40 2024-08-31T12:25:18.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.417+0000 7f266259c640 1 --2- 172.21.15.19:0/2772032621 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f266415cb40 0x7f26640c6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:18.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.418+0000 7f266259c640 1 -- 172.21.15.19:0/2772032621 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2664155300 con 0x7f266415cb40 2024-08-31T12:25:18.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.418+0000 7f266259c640 1 --2- 172.21.15.19:0/2772032621 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f266415cb40 0x7f26640c6b70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f266408dc00 tx=0x7f2654003960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:18.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.418+0000 7f265affd640 1 -- 172.21.15.19:0/2772032621 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2654005020 con 0x7f266415cb40 2024-08-31T12:25:18.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.418+0000 7f265affd640 1 -- 172.21.15.19:0/2772032621 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f265c019020 con 0x7f266415b6e0 2024-08-31T12:25:18.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.419+0000 7f265affd640 1 --2- 172.21.15.19:0/2772032621 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2644077950 0x7f2644079e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:18.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.419+0000 7f265affd640 1 -- 172.21.15.19:0/2772032621 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f265c099b50 con 0x7f266415b6e0 2024-08-31T12:25:18.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.419+0000 7f265bfff640 1 --2- 172.21.15.19:0/2772032621 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2644077950 0x7f2644079e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:18.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.419+0000 7f265affd640 1 -- 172.21.15.19:0/2772032621 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2654040030 con 0x7f266415cb40 2024-08-31T12:25:18.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.419+0000 7f265affd640 1 -- 172.21.15.19:0/2772032621 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f266415cb40 msgr2=0x7f26640c6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:18.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.419+0000 7f265affd640 1 --2- 172.21.15.19:0/2772032621 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f266415cb40 0x7f26640c6b70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f266408dc00 tx=0x7f2654003960 comp rx=0 tx=0).stop 2024-08-31T12:25:18.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.419+0000 7f265affd640 1 -- 172.21.15.19:0/2772032621 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f266415cb40 msgr2=0x7f26640c6b70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:25:18.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.419+0000 7f265affd640 1 --2- 172.21.15.19:0/2772032621 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f266415cb40 0x7f26640c6b70 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:18.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.419+0000 7f265bfff640 1 --2- 172.21.15.19:0/2772032621 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2644077950 0x7f2644079e60 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f264c006fd0 tx=0x7f264c006d60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:18.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.426+0000 7f2668c7e640 1 --2- 172.21.15.19:0/2772032621 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f26640c70b0 0x7f2664150950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:18.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.426+0000 7f2668c7e640 1 -- 172.21.15.19:0/2772032621 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f266415e880 con 0x7f26640c70b0 2024-08-31T12:25:18.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.426+0000 7f266259c640 1 --2- 172.21.15.19:0/2772032621 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f26640c70b0 0x7f2664150950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:18.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.426+0000 7f266259c640 1 -- 172.21.15.19:0/2772032621 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f26640c74c0 con 0x7f26640c70b0 2024-08-31T12:25:18.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.426+0000 7f266259c640 1 --2- 172.21.15.19:0/2772032621 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f26640c70b0 0x7f2664150950 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2654009780 tx=0x7f2654005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:18.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.427+0000 7f265affd640 1 -- 172.21.15.19:0/2772032621 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2654045470 con 0x7f26640c70b0 2024-08-31T12:25:18.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.427+0000 7f265affd640 1 -- 172.21.15.19:0/2772032621 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5032 (secure 0 0 0) 0x7f2654045470 con 0x7f26640c70b0 2024-08-31T12:25:18.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.427+0000 7f265affd640 1 -- 172.21.15.19:0/2772032621 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f26640c70b0 msgr2=0x7f2664150950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:18.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.427+0000 7f265affd640 1 --2- 172.21.15.19:0/2772032621 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f26640c70b0 0x7f2664150950 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2654009780 tx=0x7f2654005f90 comp rx=0 tx=0).stop 2024-08-31T12:25:18.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.427+0000 7f265affd640 1 -- 172.21.15.19:0/2772032621 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f26640c70b0 msgr2=0x7f2664150950 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:25:18.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.427+0000 7f265affd640 1 --2- 172.21.15.19:0/2772032621 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f26640c70b0 0x7f2664150950 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2654009780 tx=0x7f2654005f90 comp rx=0 tx=0).stop 2024-08-31T12:25:18.430 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:25:18.430 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:25:18.430 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:25:18.430 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "leader", 2024-08-31T12:25:18.430 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 78, 2024-08-31T12:25:18.430 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 52982, 2024-08-31T12:25:18.430 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:25:18.431 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:25:18.431 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:25:18.431 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:18.431 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 0, 2024-08-31T12:25:18.431 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:25:18.431 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:25:18.431 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:25:18.431 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:25:18.431 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:25:18.431 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:25:18.431 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:25:18.431 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:25:18.431 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:25:18.431 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:25:18.432 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:25:18.432 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:25:18.432 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:25:18.432 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:25:18.432 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:18.432 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:25:18.432 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:25:18.432 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:25:18.432 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:25:18.432 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:25:18.432 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:25:18.432 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:25:18.432 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:25:18.433 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:25:18.433 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:25:18.433 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:25:18.433 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:25:18.433 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:25:18.433 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:18.433 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:18.433 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:25:18.433 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:25:18.433 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:25:18.433 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:25:18.433 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:25:18.433 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:25:18.434 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:25:18.434 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:25:18.434 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:25:18.434 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:25:18.434 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:25:18.434 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:25:18.434 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:25:18.434 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:25:18.434 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:25:18.434 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:25:18.434 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:25:18.434 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:25:18.434 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:25:18.434 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:25:18.434 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:25:18.435 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:25:18.435 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:25:18.435 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:25:18.435 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:25:18.435 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:25:18.435 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:25:18.435 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:25:18.435 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:18.435 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:25:18.435 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:18.435 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:25:18.435 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:18.435 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:25:18.435 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:25:18.436 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:25:18.436 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:25:18.436 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:18.436 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:25:18.436 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:25:18.436 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:18.436 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:18.436 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:18.436 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:25:18.436 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:25:18.436 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:18.436 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:18.436 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:18.436 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:18.437 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:25:18.437 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:25:18.437 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:25:18.437 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:25:18.437 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:25:18.437 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:18.437 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:18.437 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:25:18.437 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:25:18.437 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:25:18.437 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:25:18.437 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:18.437 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:25:18.438 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:25:18.438 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:18.438 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:18.438 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:18.438 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:25:18.438 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:25:18.438 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:18.438 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:18.438 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:18.438 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:18.438 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:25:18.438 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:25:18.438 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:25:18.439 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:25:18.439 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:25:18.439 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:18.439 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:18.439 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:25:18.439 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:25:18.439 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:25:18.439 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:25:18.439 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:18.439 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:25:18.439 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:25:18.439 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:18.440 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:18.440 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:18.440 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:25:18.440 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:25:18.440 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:18.440 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:18.440 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:18.440 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:18.440 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:25:18.440 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:25:18.440 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:25:18.440 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:25:18.440 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:25:18.440 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:18.441 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:18.441 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:18.441 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:25:18.441 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:25:18.441 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:18.441 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:18.441 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:18.441 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:25:18.441 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:18.441 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:18.441 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:25:18.441 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:18.441 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:18.441 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:18.442 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 3 2024-08-31T12:25:18.442 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:18.442 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:18.442 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:25:18.442 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:18.442 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:18.442 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:18.442 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 3 2024-08-31T12:25:18.442 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:18.442 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:18.442 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:25:18.442 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:18.442 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:18.442 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:18.442 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 5 2024-08-31T12:25:18.443 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:18.443 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:18.443 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:25:18.443 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:18.443 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:18.443 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:18.443 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:25:18.443 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:18.443 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:18.443 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:18.443 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:25:18.443 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:25:18.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.427+0000 7f2668c7e640 1 -- 172.21.15.19:0/2772032621 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2644077950 msgr2=0x7f2644079e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:18.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.427+0000 7f2668c7e640 1 --2- 172.21.15.19:0/2772032621 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2644077950 0x7f2644079e60 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f264c006fd0 tx=0x7f264c006d60 comp rx=0 tx=0).stop 2024-08-31T12:25:18.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.427+0000 7f2668c7e640 1 -- 172.21.15.19:0/2772032621 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f266415b6e0 msgr2=0x7f26641f5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:18.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.427+0000 7f2668c7e640 1 --2- 172.21.15.19:0/2772032621 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f266415b6e0 0x7f26641f5480 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f265c00ca80 tx=0x7f265c00cf40 comp rx=0 tx=0).stop 2024-08-31T12:25:18.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.427+0000 7f2662d9d640 1 -- 172.21.15.19:0/2772032621 reap_dead start 2024-08-31T12:25:18.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.427+0000 7f2662d9d640 1 -- 172.21.15.19:0/2772032621 reap_dead start 2024-08-31T12:25:18.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.427+0000 7f2668c7e640 1 -- 172.21.15.19:0/2772032621 shutdown_connections 2024-08-31T12:25:18.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.427+0000 7f2668c7e640 1 -- 172.21.15.19:0/2772032621 >> 172.21.15.19:0/2772032621 conn(0x7f26640690d0 msgr2=0x7f2664159030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:18.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.427+0000 7f2668c7e640 1 -- 172.21.15.19:0/2772032621 shutdown_connections 2024-08-31T12:25:18.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.427+0000 7f2668c7e640 1 -- 172.21.15.19:0/2772032621 wait complete. 2024-08-31T12:25:18.445 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-08-31T12:25:18.446 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-08-31T12:25:18.446 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-31T12:25:18.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.572+0000 7fa9953c6640 1 -- 172.21.15.19:0/2605910488 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa9900cc400 msgr2=0x7fa9900cc810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:18.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.572+0000 7fa9953c6640 1 --2- 172.21.15.19:0/2605910488 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa9900cc400 0x7fa9900cc810 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa988005fd0 tx=0x7fa98802aa90 comp rx=0 tx=0).stop 2024-08-31T12:25:18.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.572+0000 7fa9953c6640 1 -- 172.21.15.19:0/2605910488 shutdown_connections 2024-08-31T12:25:18.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.572+0000 7fa9953c6640 1 --2- 172.21.15.19:0/2605910488 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa9900cc400 0x7fa9900cc810 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:18.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.572+0000 7fa9953c6640 1 --2- 172.21.15.19:0/2605910488 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa9900c5370 0x7fa99006ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:18.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.572+0000 7fa9953c6640 1 -- 172.21.15.19:0/2605910488 >> 172.21.15.19:0/2605910488 conn(0x7fa9900ca670 msgr2=0x7fa9900cd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:18.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.572+0000 7fa9953c6640 1 -- 172.21.15.19:0/2605910488 shutdown_connections 2024-08-31T12:25:18.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.572+0000 7fa9953c6640 1 -- 172.21.15.19:0/2605910488 wait complete. 2024-08-31T12:25:18.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.574+0000 7fa9953c6640 1 Processor -- start 2024-08-31T12:25:18.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.575+0000 7fa9953c6640 1 -- start start 2024-08-31T12:25:18.579 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.576+0000 7fa9953c6640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa9900c5370 0x7fa9901e3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:18.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.576+0000 7fa9953c6640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa9900cb2b0 0x7fa9901dea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:18.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.576+0000 7fa9953c6640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa9900cc400 0x7fa990147160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:18.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.576+0000 7fa9953c6640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fa9900c72e0 con 0x7fa9900c5370 2024-08-31T12:25:18.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.576+0000 7fa98effd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa9900c5370 0x7fa9901e3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:18.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.576+0000 7fa98effd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa9900c5370 0x7fa9901e3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:34754/0 (socket says 172.21.15.19:34754) 2024-08-31T12:25:18.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.576+0000 7fa98f7fe640 1 -- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa9900cc400 msgr2=0x7fa990147160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3300/0 2024-08-31T12:25:18.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.576+0000 7fa98effd640 1 -- 172.21.15.19:0/683290127 learned_addr learned my addr 172.21.15.19:0/683290127 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:25:18.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.576+0000 7fa9953c6640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa9900ccd50 con 0x7fa9900cc400 2024-08-31T12:25:18.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.576+0000 7fa9953c6640 1 -- 172.21.15.19:0/683290127 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fa9900cb8c0 con 0x7fa9900cb2b0 2024-08-31T12:25:18.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.576+0000 7fa98f7fe640 1 --2- 172.21.15.19:0/683290127 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa9900cc400 0x7fa990147160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:25:18.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.576+0000 7fa987fff640 1 --2- 172.21.15.19:0/683290127 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa9900cb2b0 0x7fa9901dea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:18.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.577+0000 7fa98effd640 1 -- 172.21.15.19:0/683290127 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa9900cb2b0 msgr2=0x7fa9901dea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:18.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.577+0000 7fa98effd640 1 --2- 172.21.15.19:0/683290127 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa9900cb2b0 0x7fa9901dea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:18.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.577+0000 7fa98effd640 1 -- 172.21.15.19:0/683290127 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa9900cc400 msgr2=0x7fa990147160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:25:18.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.577+0000 7fa98effd640 1 --2- 172.21.15.19:0/683290127 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa9900cc400 0x7fa990147160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:18.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.577+0000 7fa98effd640 1 -- 172.21.15.19:0/683290127 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9901defd0 con 0x7fa9900c5370 2024-08-31T12:25:18.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.577+0000 7fa98effd640 1 --2- 172.21.15.19:0/683290127 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa9900c5370 0x7fa9901e3910 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa98000e7c0 tx=0x7fa98000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:18.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.577+0000 7fa9877fe640 1 -- 172.21.15.19:0/683290127 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa98000ccc0 con 0x7fa9900c5370 2024-08-31T12:25:18.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.577+0000 7fa9877fe640 1 -- 172.21.15.19:0/683290127 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa980010430 con 0x7fa9900c5370 2024-08-31T12:25:18.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.577+0000 7fa9953c6640 1 -- 172.21.15.19:0/683290127 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9901e3fa0 con 0x7fa9900c5370 2024-08-31T12:25:18.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.577+0000 7fa9877fe640 1 -- 172.21.15.19:0/683290127 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa980010860 con 0x7fa9900c5370 2024-08-31T12:25:18.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.577+0000 7fa9953c6640 1 -- 172.21.15.19:0/683290127 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9901e4580 con 0x7fa9900c5370 2024-08-31T12:25:18.583 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.578+0000 7fa9953c6640 1 --2- 172.21.15.19:0/683290127 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa9901e0670 0x7fa9901e4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:18.583 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.578+0000 7fa9953c6640 1 -- 172.21.15.19:0/683290127 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fa9901e4c90 con 0x7fa9901e0670 2024-08-31T12:25:18.583 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.579+0000 7fa987fff640 1 --2- 172.21.15.19:0/683290127 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa9901e0670 0x7fa9901e4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:18.583 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.579+0000 7fa987fff640 1 -- 172.21.15.19:0/683290127 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa9900641b0 con 0x7fa9901e0670 2024-08-31T12:25:18.583 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.579+0000 7fa9877fe640 1 -- 172.21.15.19:0/683290127 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fa980010a00 con 0x7fa9900c5370 2024-08-31T12:25:18.583 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.579+0000 7fa987fff640 1 --2- 172.21.15.19:0/683290127 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa9901e0670 0x7fa9901e4750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa9900cd310 tx=0x7fa97800eb50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:18.583 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.580+0000 7fa9877fe640 1 --2- 172.21.15.19:0/683290127 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa970077a30 0x7fa970079f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:18.583 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.580+0000 7fa9877fe640 1 -- 172.21.15.19:0/683290127 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fa98009a0b0 con 0x7fa9900c5370 2024-08-31T12:25:18.583 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.580+0000 7fa98f7fe640 1 --2- 172.21.15.19:0/683290127 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa970077a30 0x7fa970079f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:18.583 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.580+0000 7fa9877fe640 1 -- 172.21.15.19:0/683290127 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa978017070 con 0x7fa9901e0670 2024-08-31T12:25:18.583 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.580+0000 7fa9877fe640 1 -- 172.21.15.19:0/683290127 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa97800c6f0 con 0x7fa9901e0670 2024-08-31T12:25:18.584 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.580+0000 7fa9877fe640 1 -- 172.21.15.19:0/683290127 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa9901e0670 msgr2=0x7fa9901e4750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:18.584 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.580+0000 7fa9877fe640 1 --2- 172.21.15.19:0/683290127 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa9901e0670 0x7fa9901e4750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa9900cd310 tx=0x7fa97800eb50 comp rx=0 tx=0).stop 2024-08-31T12:25:18.584 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.580+0000 7fa9877fe640 1 -- 172.21.15.19:0/683290127 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa9901e0670 msgr2=0x7fa9901e4750 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:25:18.584 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.580+0000 7fa9877fe640 1 --2- 172.21.15.19:0/683290127 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa9901e0670 0x7fa9901e4750 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:18.584 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.581+0000 7fa98f7fe640 1 --2- 172.21.15.19:0/683290127 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa970077a30 0x7fa970079f40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa988035000 tx=0x7fa988038040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:18.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.587+0000 7fa9953c6640 1 --2- 172.21.15.19:0/683290127 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa9901df1a0 0x7fa990147fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:18.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.587+0000 7fa9953c6640 1 -- 172.21.15.19:0/683290127 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fa99006e760 con 0x7fa9901df1a0 2024-08-31T12:25:18.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.587+0000 7fa987fff640 1 --2- 172.21.15.19:0/683290127 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa9901df1a0 0x7fa990147fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:18.590 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.588+0000 7fa987fff640 1 -- 172.21.15.19:0/683290127 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa9900cbfd0 con 0x7fa9901df1a0 2024-08-31T12:25:18.590 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.588+0000 7fa987fff640 1 --2- 172.21.15.19:0/683290127 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa9901df1a0 0x7fa990147fd0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa978019000 tx=0x7fa9780047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:18.590 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.588+0000 7fa9877fe640 1 -- 172.21.15.19:0/683290127 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa978017070 con 0x7fa9901df1a0 2024-08-31T12:25:18.591 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.589+0000 7fa9877fe640 1 -- 172.21.15.19:0/683290127 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5031 (secure 0 0 0) 0x7fa9780064d0 con 0x7fa9901df1a0 2024-08-31T12:25:18.591 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.589+0000 7fa9877fe640 1 -- 172.21.15.19:0/683290127 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa9901df1a0 msgr2=0x7fa990147fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:18.591 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.589+0000 7fa9877fe640 1 --2- 172.21.15.19:0/683290127 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa9901df1a0 0x7fa990147fd0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa978019000 tx=0x7fa9780047b0 comp rx=0 tx=0).stop 2024-08-31T12:25:18.591 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.589+0000 7fa9877fe640 1 -- 172.21.15.19:0/683290127 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa9901df1a0 msgr2=0x7fa990147fd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:25:18.591 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.589+0000 7fa9877fe640 1 --2- 172.21.15.19:0/683290127 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa9901df1a0 0x7fa990147fd0 secure :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa978019000 tx=0x7fa9780047b0 comp rx=0 tx=0).stop 2024-08-31T12:25:18.592 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:25:18.592 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:25:18.592 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:25:18.592 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:25:18.592 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 78, 2024-08-31T12:25:18.592 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 105946, 2024-08-31T12:25:18.592 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:25:18.592 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:25:18.592 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:25:18.592 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:18.592 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 0, 2024-08-31T12:25:18.592 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:25:18.592 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:25:18.593 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:25:18.593 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:25:18.593 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:25:18.593 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:25:18.593 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:25:18.593 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:25:18.593 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:25:18.593 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:25:18.593 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:25:18.593 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:25:18.593 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:25:18.593 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:25:18.593 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:18.594 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:25:18.594 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:25:18.594 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:25:18.594 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:25:18.594 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:25:18.594 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:25:18.594 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:25:18.594 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:25:18.594 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:25:18.594 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:25:18.594 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:25:18.594 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:25:18.594 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:25:18.595 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:18.595 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:18.595 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:25:18.595 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:25:18.595 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:25:18.595 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:25:18.595 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:25:18.595 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:25:18.595 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:25:18.595 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:25:18.595 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:25:18.595 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:25:18.595 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:25:18.596 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:25:18.596 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:25:18.596 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:25:18.596 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:25:18.596 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:25:18.596 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:25:18.596 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:25:18.596 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:25:18.596 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:25:18.596 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:25:18.596 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:25:18.596 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:25:18.596 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:25:18.596 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:25:18.597 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:25:18.597 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:25:18.597 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:25:18.597 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:18.597 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:25:18.597 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:18.597 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:25:18.597 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:18.597 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:25:18.597 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:25:18.597 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:25:18.597 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:25:18.597 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:18.597 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:25:18.598 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:25:18.598 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:18.598 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:18.598 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:18.598 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:25:18.598 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:25:18.598 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:18.598 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:18.598 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:18.598 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:18.598 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:25:18.598 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:25:18.598 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:25:18.599 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:25:18.599 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:25:18.599 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:18.599 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:18.599 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:25:18.599 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:25:18.599 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:25:18.599 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:25:18.599 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:18.599 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:25:18.599 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:25:18.599 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:18.599 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:18.599 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:18.600 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:25:18.600 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:25:18.600 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:18.600 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:18.600 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:18.600 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:18.600 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:25:18.600 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:25:18.600 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:25:18.600 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:25:18.600 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:25:18.600 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:18.600 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:18.600 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:25:18.601 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:25:18.601 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:25:18.601 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:25:18.601 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:18.601 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:25:18.601 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:25:18.601 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:18.601 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:18.601 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:18.601 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:25:18.601 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:25:18.601 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:18.601 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:18.602 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:18.602 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:18.602 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:25:18.602 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:25:18.602 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:25:18.602 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:25:18.602 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:25:18.602 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:18.602 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:18.602 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:18.602 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:25:18.602 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:25:18.602 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:18.603 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:18.603 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:18.603 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:25:18.603 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:18.603 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:18.603 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:25:18.603 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:18.603 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:18.603 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:18.603 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 3 2024-08-31T12:25:18.603 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:18.603 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:18.603 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:25:18.603 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:18.604 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:18.604 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:18.604 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 5 2024-08-31T12:25:18.604 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:18.604 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:18.604 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:25:18.604 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:18.604 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:18.604 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:18.604 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 3 2024-08-31T12:25:18.604 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:18.604 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:18.604 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:25:18.604 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:18.605 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:18.605 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:18.605 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:25:18.605 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:18.605 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:18.605 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:18.605 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:25:18.605 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:25:18.606 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.589+0000 7fa9953c6640 1 -- 172.21.15.19:0/683290127 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa970077a30 msgr2=0x7fa970079f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:18.606 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.589+0000 7fa9953c6640 1 --2- 172.21.15.19:0/683290127 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa970077a30 0x7fa970079f40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa988035000 tx=0x7fa988038040 comp rx=0 tx=0).stop 2024-08-31T12:25:18.606 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.589+0000 7fa9953c6640 1 -- 172.21.15.19:0/683290127 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa9900c5370 msgr2=0x7fa9901e3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:18.607 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.589+0000 7fa9953c6640 1 --2- 172.21.15.19:0/683290127 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa9900c5370 0x7fa9901e3910 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa98000e7c0 tx=0x7fa98000ec80 comp rx=0 tx=0).stop 2024-08-31T12:25:18.607 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.589+0000 7fa98f7fe640 1 -- 172.21.15.19:0/683290127 reap_dead start 2024-08-31T12:25:18.607 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.589+0000 7fa9953c6640 1 -- 172.21.15.19:0/683290127 shutdown_connections 2024-08-31T12:25:18.607 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.589+0000 7fa9953c6640 1 --2- 172.21.15.19:0/683290127 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa9900c5370 0x7fa9901e3910 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:18.607 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.589+0000 7fa9953c6640 1 -- 172.21.15.19:0/683290127 >> 172.21.15.19:0/683290127 conn(0x7fa9900ca670 msgr2=0x7fa99014a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:18.607 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.589+0000 7fa9953c6640 1 -- 172.21.15.19:0/683290127 shutdown_connections 2024-08-31T12:25:18.607 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:18.589+0000 7fa9953c6640 1 -- 172.21.15.19:0/683290127 wait complete. 2024-08-31T12:25:18.607 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-08-31T12:25:18.607 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-08-31T12:25:27.576 DEBUG:teuthology.orchestra.run.smithi019:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:25:27.579 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:25:38.608 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-08-31T12:25:38.611 INFO:tasks.ceph.mon.b:Restarting daemon 2024-08-31T12:25:38.612 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-08-31T12:25:38.614 INFO:tasks.ceph.mon.b:Started 2024-08-31T12:25:38.614 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-08-31T12:25:38.614 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:25:38.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:38.762+0000 7f7cd3a25640 1 -- 172.21.15.19:0/1556389280 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ccc0c8050 msgr2=0x7f7ccc0caad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:38.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:38.762+0000 7f7cd3a25640 1 --2- 172.21.15.19:0/1556389280 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ccc0c8050 0x7f7ccc0caad0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7cc8009830 tx=0x7f7cc802a510 comp rx=0 tx=0).stop 2024-08-31T12:25:38.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:38.762+0000 7f7cd3a25640 1 -- 172.21.15.19:0/1556389280 shutdown_connections 2024-08-31T12:25:38.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:38.762+0000 7f7cd3a25640 1 --2- 172.21.15.19:0/1556389280 >> v2:172.21.15.43:3300/0 conn(0x7f7ccc06e1e0 0x7f7ccc158170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:38.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:38.762+0000 7f7cd3a25640 1 --2- 172.21.15.19:0/1556389280 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ccc0c8050 0x7f7ccc0caad0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:38.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:38.762+0000 7f7cd3a25640 1 -- 172.21.15.19:0/1556389280 >> 172.21.15.19:0/1556389280 conn(0x7f7ccc0cdcb0 msgr2=0x7f7ccc0ca150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:38.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:38.762+0000 7f7cd3a25640 1 -- 172.21.15.19:0/1556389280 shutdown_connections 2024-08-31T12:25:38.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:38.762+0000 7f7cd3a25640 1 -- 172.21.15.19:0/1556389280 wait complete. 2024-08-31T12:25:38.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:38.764+0000 7f7cd3a25640 1 Processor -- start 2024-08-31T12:25:38.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:38.764+0000 7f7cd3a25640 1 -- start start 2024-08-31T12:25:38.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:38.766+0000 7f7cd3a25640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ccc06ddd0 0x7f7ccc159f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:38.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:38.766+0000 7f7cd3a25640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ccc06e1e0 0x7f7ccc153950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:38.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:38.766+0000 7f7cd3a25640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7ccc0c8050 0x7f7ccc0cccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:38.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:38.766+0000 7f7cd3a25640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f7ccc0c7a90 con 0x7f7ccc0c8050 2024-08-31T12:25:38.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:38.766+0000 7f7cd3a25640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7ccc0c7910 con 0x7f7ccc06ddd0 2024-08-31T12:25:38.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:38.766+0000 7f7cd3a25640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f7ccc152100 con 0x7f7ccc06e1e0 2024-08-31T12:25:38.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:38.766+0000 7f7cd1fc2640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7ccc0c8050 0x7f7ccc0cccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:38.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:38.766+0000 7f7cd1fc2640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7ccc0c8050 0x7f7ccc0cccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:48610/0 (socket says 172.21.15.19:48610) 2024-08-31T12:25:38.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:38.766+0000 7f7cd1fc2640 1 -- 172.21.15.19:0/1711160667 learned_addr learned my addr 172.21.15.19:0/1711160667 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:25:38.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:38.766+0000 7f7cd0fc0640 1 --2- 172.21.15.19:0/1711160667 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ccc06e1e0 0x7f7ccc153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:38.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:38.766+0000 7f7cd17c1640 1 --2- 172.21.15.19:0/1711160667 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ccc06ddd0 0x7f7ccc159f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:38.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:38.767+0000 7f7cd1fc2640 1 -- 172.21.15.19:0/1711160667 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ccc06e1e0 msgr2=0x7f7ccc153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:38.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:38.767+0000 7f7cd1fc2640 1 --2- 172.21.15.19:0/1711160667 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ccc06e1e0 0x7f7ccc153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:38.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:38.767+0000 7f7cd1fc2640 1 -- 172.21.15.19:0/1711160667 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ccc06ddd0 msgr2=0x7f7ccc159f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:38.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:38.767+0000 7f7cd1fc2640 1 --2- 172.21.15.19:0/1711160667 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ccc06ddd0 0x7f7ccc159f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:38.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:38.767+0000 7f7cd1fc2640 1 -- 172.21.15.19:0/1711160667 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ccc153e90 con 0x7f7ccc0c8050 2024-08-31T12:25:38.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:38.767+0000 7f7cd0fc0640 1 --2- 172.21.15.19:0/1711160667 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ccc06e1e0 0x7f7ccc153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:25:38.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:38.767+0000 7f7cd1fc2640 1 --2- 172.21.15.19:0/1711160667 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7ccc0c8050 0x7f7ccc0cccc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7cc400e990 tx=0x7f7cc400ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:38.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:38.767+0000 7f7cc1ffb640 1 -- 172.21.15.19:0/1711160667 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc4004d20 con 0x7f7ccc0c8050 2024-08-31T12:25:38.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:38.767+0000 7f7cc1ffb640 1 -- 172.21.15.19:0/1711160667 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7cc400cb00 con 0x7f7ccc0c8050 2024-08-31T12:25:38.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:38.767+0000 7f7cd3a25640 1 -- 172.21.15.19:0/1711160667 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ccc159040 con 0x7f7ccc0c8050 2024-08-31T12:25:38.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:38.767+0000 7f7cc1ffb640 1 -- 172.21.15.19:0/1711160667 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc4005020 con 0x7f7ccc0c8050 2024-08-31T12:25:38.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:38.768+0000 7f7cd3a25640 1 -- 172.21.15.19:0/1711160667 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ccc0ca890 con 0x7f7ccc0c8050 2024-08-31T12:25:38.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:38.769+0000 7f7cd3a25640 1 -- 172.21.15.19:0/1711160667 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ccc0c68c0 con 0x7f7ccc0c8050 2024-08-31T12:25:38.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:38.770+0000 7f7cc1ffb640 1 -- 172.21.15.19:0/1711160667 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f7cc40051c0 con 0x7f7ccc0c8050 2024-08-31T12:25:38.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:38.770+0000 7f7cc1ffb640 1 --2- 172.21.15.19:0/1711160667 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7cb0077a80 0x7f7cb0079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:38.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:38.770+0000 7f7cc1ffb640 1 -- 172.21.15.19:0/1711160667 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f7cc4099f10 con 0x7f7ccc0c8050 2024-08-31T12:25:38.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:38.770+0000 7f7cd17c1640 1 --2- 172.21.15.19:0/1711160667 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7cb0077a80 0x7f7cb0079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:38.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:38.771+0000 7f7cd17c1640 1 --2- 172.21.15.19:0/1711160667 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7cb0077a80 0x7f7cb0079f90 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7cc8034000 tx=0x7f7cc8004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:38.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:38.776+0000 7f7cc1ffb640 1 -- 172.21.15.19:0/1711160667 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f7cc4016030 con 0x7f7ccc0c8050 2024-08-31T12:25:39.090 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:39.088+0000 7f7cd3a25640 1 -- 172.21.15.19:0/1711160667 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7ccc06fdb0 con 0x7f7ccc0c8050 2024-08-31T12:25:39.090 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:39.088+0000 7f7cc1ffb640 1 -- 172.21.15.19:0/1711160667 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f7cc4062c90 con 0x7f7ccc0c8050 2024-08-31T12:25:39.091 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":78,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:25:39.094 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:39.091+0000 7f7cd3a25640 1 -- 172.21.15.19:0/1711160667 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7cb0077a80 msgr2=0x7f7cb0079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:39.094 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:39.091+0000 7f7cd3a25640 1 --2- 172.21.15.19:0/1711160667 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7cb0077a80 0x7f7cb0079f90 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7cc8034000 tx=0x7f7cc8004730 comp rx=0 tx=0).stop 2024-08-31T12:25:39.094 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:39.091+0000 7f7cd3a25640 1 -- 172.21.15.19:0/1711160667 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7ccc0c8050 msgr2=0x7f7ccc0cccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:39.094 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:39.091+0000 7f7cd3a25640 1 --2- 172.21.15.19:0/1711160667 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7ccc0c8050 0x7f7ccc0cccc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7cc400e990 tx=0x7f7cc400ee50 comp rx=0 tx=0).stop 2024-08-31T12:25:39.094 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:39.091+0000 7f7cd3a25640 1 -- 172.21.15.19:0/1711160667 shutdown_connections 2024-08-31T12:25:39.094 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:39.091+0000 7f7cd3a25640 1 --2- 172.21.15.19:0/1711160667 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7cb0077a80 0x7f7cb0079f90 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:39.094 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:39.091+0000 7f7cd3a25640 1 --2- 172.21.15.19:0/1711160667 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7ccc0c8050 0x7f7ccc0cccc0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:39.094 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:39.091+0000 7f7cd3a25640 1 --2- 172.21.15.19:0/1711160667 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ccc06e1e0 0x7f7ccc153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:39.094 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:39.091+0000 7f7cd3a25640 1 --2- 172.21.15.19:0/1711160667 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ccc06ddd0 0x7f7ccc159f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:39.094 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:39.091+0000 7f7cd3a25640 1 -- 172.21.15.19:0/1711160667 >> 172.21.15.19:0/1711160667 conn(0x7f7ccc0cdcb0 msgr2=0x7f7ccc14ffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:39.094 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:39.091+0000 7f7cd3a25640 1 -- 172.21.15.19:0/1711160667 shutdown_connections 2024-08-31T12:25:39.095 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:39.091+0000 7f7cd3a25640 1 -- 172.21.15.19:0/1711160667 wait complete. 2024-08-31T12:25:42.106 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:25:42.227 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.225+0000 7f36ce227640 1 -- 172.21.15.19:0/799855562 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f36c80c7f00 msgr2=0x7f36c80c53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:42.227 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.225+0000 7f36ce227640 1 --2- 172.21.15.19:0/799855562 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f36c80c7f00 0x7f36c80c53c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f36b8009820 tx=0x7f36b802a510 comp rx=0 tx=0).stop 2024-08-31T12:25:42.227 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.225+0000 7f36ce227640 1 -- 172.21.15.19:0/799855562 shutdown_connections 2024-08-31T12:25:42.228 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.225+0000 7f36ce227640 1 --2- 172.21.15.19:0/799855562 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f36c80c7f00 0x7f36c80c53c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:42.228 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.225+0000 7f36ce227640 1 -- 172.21.15.19:0/799855562 >> 172.21.15.19:0/799855562 conn(0x7f36c80ca630 msgr2=0x7f36c80cd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:42.228 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.225+0000 7f36ce227640 1 -- 172.21.15.19:0/799855562 shutdown_connections 2024-08-31T12:25:42.228 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.225+0000 7f36ce227640 1 -- 172.21.15.19:0/799855562 wait complete. 2024-08-31T12:25:42.229 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.227+0000 7f36ce227640 1 Processor -- start 2024-08-31T12:25:42.229 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.227+0000 7f36ce227640 1 -- start start 2024-08-31T12:25:42.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.229+0000 7f36ce227640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f36c80c6dd0 0x7f36c81ef740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:42.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.229+0000 7f36ce227640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f36c80c7f00 0x7f36c81f2c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:42.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.229+0000 7f36ce227640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f36c8151ff0 0x7f36c8147e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:42.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.229+0000 7f36ce227640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f36c8152400 con 0x7f36c80c6dd0 2024-08-31T12:25:42.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.229+0000 7f36ce227640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f36c80ce240 con 0x7f36c80c7f00 2024-08-31T12:25:42.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.229+0000 7f36ce227640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f36c8152580 con 0x7f36c8151ff0 2024-08-31T12:25:42.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.229+0000 7f36c77fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f36c80c6dd0 0x7f36c81ef740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:42.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.229+0000 7f36c77fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f36c80c6dd0 0x7f36c81ef740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:48638/0 (socket says 172.21.15.19:48638) 2024-08-31T12:25:42.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.229+0000 7f36c77fe640 1 -- 172.21.15.19:0/3890446215 learned_addr learned my addr 172.21.15.19:0/3890446215 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:25:42.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.229+0000 7f36c6ffd640 1 --2- 172.21.15.19:0/3890446215 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f36c80c7f00 0x7f36c81f2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:42.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.229+0000 7f36c7fff640 1 --2- 172.21.15.19:0/3890446215 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f36c8151ff0 0x7f36c8147e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:42.234 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.229+0000 7f36c6ffd640 1 -- 172.21.15.19:0/3890446215 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f36c80c7f00 msgr2=0x7f36c81f2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-08-31T12:25:42.234 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.229+0000 7f36c6ffd640 1 -- 172.21.15.19:0/3890446215 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f36c80c7f00 msgr2=0x7f36c81f2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-31T12:25:42.234 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.229+0000 7f36c6ffd640 1 --2- 172.21.15.19:0/3890446215 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f36c80c7f00 0x7f36c81f2c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-31T12:25:42.234 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.229+0000 7f36c6ffd640 1 --2- 172.21.15.19:0/3890446215 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f36c80c7f00 0x7f36c81f2c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:25:42.234 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.229+0000 7f36c77fe640 1 -- 172.21.15.19:0/3890446215 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f36c8151ff0 msgr2=0x7f36c8147e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:42.234 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.229+0000 7f36c77fe640 1 --2- 172.21.15.19:0/3890446215 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f36c8151ff0 0x7f36c8147e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:42.234 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.229+0000 7f36c77fe640 1 -- 172.21.15.19:0/3890446215 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f36c80c7f00 msgr2=0x7f36c81f2c60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:25:42.234 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.229+0000 7f36c77fe640 1 --2- 172.21.15.19:0/3890446215 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f36c80c7f00 0x7f36c81f2c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:42.234 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.229+0000 7f36c77fe640 1 -- 172.21.15.19:0/3890446215 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36c8148350 con 0x7f36c80c6dd0 2024-08-31T12:25:42.234 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.229+0000 7f36c77fe640 1 --2- 172.21.15.19:0/3890446215 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f36c80c6dd0 0x7f36c81ef740 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f36b8009910 tx=0x7f36b8005270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:42.234 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.230+0000 7f36affff640 1 -- 172.21.15.19:0/3890446215 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36b8031b50 con 0x7f36c80c6dd0 2024-08-31T12:25:42.235 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.230+0000 7f36affff640 1 -- 172.21.15.19:0/3890446215 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f36b8031de0 con 0x7f36c80c6dd0 2024-08-31T12:25:42.235 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.230+0000 7f36affff640 1 -- 172.21.15.19:0/3890446215 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36b8004a00 con 0x7f36c80c6dd0 2024-08-31T12:25:42.235 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.230+0000 7f36ce227640 1 -- 172.21.15.19:0/3890446215 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36c8151520 con 0x7f36c80c6dd0 2024-08-31T12:25:42.235 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.230+0000 7f36ce227640 1 -- 172.21.15.19:0/3890446215 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36c81f3920 con 0x7f36c80c6dd0 2024-08-31T12:25:42.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.231+0000 7f36ce227640 1 -- 172.21.15.19:0/3890446215 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36c80c9c40 con 0x7f36c80c6dd0 2024-08-31T12:25:42.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.231+0000 7f36affff640 1 -- 172.21.15.19:0/3890446215 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f36b8004ba0 con 0x7f36c80c6dd0 2024-08-31T12:25:42.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.232+0000 7f36affff640 1 --2- 172.21.15.19:0/3890446215 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f36b4077a80 0x7f36b4079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:42.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.232+0000 7f36affff640 1 -- 172.21.15.19:0/3890446215 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f36b80bac90 con 0x7f36c80c6dd0 2024-08-31T12:25:42.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.232+0000 7f36c6ffd640 1 --2- 172.21.15.19:0/3890446215 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f36b4077a80 0x7f36b4079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:42.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.233+0000 7f36c6ffd640 1 --2- 172.21.15.19:0/3890446215 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f36b4077a80 0x7f36b4079f90 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f36c8151840 tx=0x7f36bc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:42.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.238+0000 7f36affff640 1 -- 172.21.15.19:0/3890446215 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f36b803f030 con 0x7f36c80c6dd0 2024-08-31T12:25:42.515 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.513+0000 7f36ce227640 1 -- 172.21.15.19:0/3890446215 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f36c8155850 con 0x7f36c80c6dd0 2024-08-31T12:25:42.515 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.513+0000 7f36affff640 1 -- 172.21.15.19:0/3890446215 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f36b80838e0 con 0x7f36c80c6dd0 2024-08-31T12:25:42.515 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":78,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:25:42.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.516+0000 7f36ce227640 1 -- 172.21.15.19:0/3890446215 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f36b4077a80 msgr2=0x7f36b4079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:42.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.516+0000 7f36ce227640 1 --2- 172.21.15.19:0/3890446215 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f36b4077a80 0x7f36b4079f90 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f36c8151840 tx=0x7f36bc009040 comp rx=0 tx=0).stop 2024-08-31T12:25:42.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.516+0000 7f36ce227640 1 -- 172.21.15.19:0/3890446215 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f36c80c6dd0 msgr2=0x7f36c81ef740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:42.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.516+0000 7f36ce227640 1 --2- 172.21.15.19:0/3890446215 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f36c80c6dd0 0x7f36c81ef740 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f36b8009910 tx=0x7f36b8005270 comp rx=0 tx=0).stop 2024-08-31T12:25:42.519 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.516+0000 7f36ce227640 1 -- 172.21.15.19:0/3890446215 shutdown_connections 2024-08-31T12:25:42.519 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.516+0000 7f36ce227640 1 --2- 172.21.15.19:0/3890446215 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f36b4077a80 0x7f36b4079f90 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:42.519 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.516+0000 7f36ce227640 1 --2- 172.21.15.19:0/3890446215 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f36c8151ff0 0x7f36c8147e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:42.519 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.516+0000 7f36ce227640 1 --2- 172.21.15.19:0/3890446215 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f36c80c7f00 0x7f36c81f2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:42.519 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.516+0000 7f36ce227640 1 --2- 172.21.15.19:0/3890446215 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f36c80c6dd0 0x7f36c81ef740 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:42.519 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.516+0000 7f36ce227640 1 -- 172.21.15.19:0/3890446215 >> 172.21.15.19:0/3890446215 conn(0x7f36c80ca630 msgr2=0x7f36c814eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:42.519 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.516+0000 7f36ce227640 1 -- 172.21.15.19:0/3890446215 shutdown_connections 2024-08-31T12:25:42.519 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:42.516+0000 7f36ce227640 1 -- 172.21.15.19:0/3890446215 wait complete. 2024-08-31T12:25:45.531 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:25:45.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.645+0000 7f6e76362640 1 -- 172.21.15.19:0/3651493603 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6e700cc070 msgr2=0x7f6e700cc480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:45.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.645+0000 7f6e76362640 1 --2- 172.21.15.19:0/3651493603 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6e700cc070 0x7f6e700cc480 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6e64009830 tx=0x7f6e6402a5a0 comp rx=0 tx=0).stop 2024-08-31T12:25:45.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.645+0000 7f6e76362640 1 -- 172.21.15.19:0/3651493603 shutdown_connections 2024-08-31T12:25:45.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.645+0000 7f6e76362640 1 --2- 172.21.15.19:0/3651493603 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6e700cc070 0x7f6e700cc480 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:45.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.645+0000 7f6e76362640 1 --2- 172.21.15.19:0/3651493603 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6e700ccbb0 0x7f6e700cb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:45.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.645+0000 7f6e76362640 1 -- 172.21.15.19:0/3651493603 >> 172.21.15.19:0/3651493603 conn(0x7f6e70074810 msgr2=0x7f6e70068aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:45.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.645+0000 7f6e76362640 1 -- 172.21.15.19:0/3651493603 shutdown_connections 2024-08-31T12:25:45.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.645+0000 7f6e76362640 1 -- 172.21.15.19:0/3651493603 wait complete. 2024-08-31T12:25:45.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.647+0000 7f6e76362640 1 Processor -- start 2024-08-31T12:25:45.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.647+0000 7f6e76362640 1 -- start start 2024-08-31T12:25:45.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.648+0000 7f6e76362640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6e700cc070 0x7f6e70148210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:45.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.649+0000 7f6e76362640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6e700ccbb0 0x7f6e701f21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:45.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.649+0000 7f6e76362640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e7015b120 0x7f6e7015d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:45.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.649+0000 7f6e76362640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f6e701557e0 con 0x7f6e700ccbb0 2024-08-31T12:25:45.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.649+0000 7f6e76362640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6e70155660 con 0x7f6e7015b120 2024-08-31T12:25:45.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.649+0000 7f6e76362640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f6e7006e1a0 con 0x7f6e700cc070 2024-08-31T12:25:45.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.649+0000 7f6e6f7fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6e700ccbb0 0x7f6e701f21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:45.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.649+0000 7f6e6f7fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6e700ccbb0 0x7f6e701f21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:48646/0 (socket says 172.21.15.19:48646) 2024-08-31T12:25:45.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.649+0000 7f6e6f7fe640 1 -- 172.21.15.19:0/2359398397 learned_addr learned my addr 172.21.15.19:0/2359398397 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:25:45.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.649+0000 7f6e6ffff640 1 --2- 172.21.15.19:0/2359398397 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6e700cc070 0x7f6e70148210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:45.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.649+0000 7f6e748ff640 1 --2- 172.21.15.19:0/2359398397 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e7015b120 0x7f6e7015d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:45.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.649+0000 7f6e748ff640 1 -- 172.21.15.19:0/2359398397 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e7015b120 msgr2=0x7f6e7015d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-08-31T12:25:45.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.649+0000 7f6e748ff640 1 -- 172.21.15.19:0/2359398397 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e7015b120 msgr2=0x7f6e7015d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-31T12:25:45.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.649+0000 7f6e748ff640 1 --2- 172.21.15.19:0/2359398397 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e7015b120 0x7f6e7015d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-31T12:25:45.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.649+0000 7f6e748ff640 1 --2- 172.21.15.19:0/2359398397 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e7015b120 0x7f6e7015d430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:25:45.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.649+0000 7f6e6f7fe640 1 -- 172.21.15.19:0/2359398397 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6e700cc070 msgr2=0x7f6e70148210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:45.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.649+0000 7f6e6f7fe640 1 --2- 172.21.15.19:0/2359398397 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6e700cc070 0x7f6e70148210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:45.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.649+0000 7f6e6f7fe640 1 -- 172.21.15.19:0/2359398397 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e7015b120 msgr2=0x7f6e7015d430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:25:45.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.649+0000 7f6e6f7fe640 1 --2- 172.21.15.19:0/2359398397 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e7015b120 0x7f6e7015d430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:45.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.649+0000 7f6e6f7fe640 1 -- 172.21.15.19:0/2359398397 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e70148750 con 0x7f6e700ccbb0 2024-08-31T12:25:45.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.649+0000 7f6e6ffff640 1 --2- 172.21.15.19:0/2359398397 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6e700cc070 0x7f6e70148210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:25:45.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.649+0000 7f6e6f7fe640 1 --2- 172.21.15.19:0/2359398397 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6e700ccbb0 0x7f6e701f21b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6e64009950 tx=0x7f6e64033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:45.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.650+0000 7f6e6cff9640 1 -- 172.21.15.19:0/2359398397 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e64031d60 con 0x7f6e700ccbb0 2024-08-31T12:25:45.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.650+0000 7f6e6cff9640 1 -- 172.21.15.19:0/2359398397 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6e64033e10 con 0x7f6e700ccbb0 2024-08-31T12:25:45.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.650+0000 7f6e76362640 1 -- 172.21.15.19:0/2359398397 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e701f26f0 con 0x7f6e700ccbb0 2024-08-31T12:25:45.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.650+0000 7f6e6cff9640 1 -- 172.21.15.19:0/2359398397 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e64005260 con 0x7f6e700ccbb0 2024-08-31T12:25:45.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.650+0000 7f6e76362640 1 -- 172.21.15.19:0/2359398397 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e70147730 con 0x7f6e700ccbb0 2024-08-31T12:25:45.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.651+0000 7f6e76362640 1 -- 172.21.15.19:0/2359398397 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e70147520 con 0x7f6e700ccbb0 2024-08-31T12:25:45.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.652+0000 7f6e6cff9640 1 -- 172.21.15.19:0/2359398397 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f6e64005400 con 0x7f6e700ccbb0 2024-08-31T12:25:45.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.652+0000 7f6e6cff9640 1 --2- 172.21.15.19:0/2359398397 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6e5c077a80 0x7f6e5c079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:45.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.652+0000 7f6e6cff9640 1 -- 172.21.15.19:0/2359398397 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f6e640ba780 con 0x7f6e700ccbb0 2024-08-31T12:25:45.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.652+0000 7f6e6ffff640 1 --2- 172.21.15.19:0/2359398397 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6e5c077a80 0x7f6e5c079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:45.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.653+0000 7f6e6ffff640 1 --2- 172.21.15.19:0/2359398397 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6e5c077a80 0x7f6e5c079f90 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f6e60005fd0 tx=0x7f6e6000b2e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:45.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.657+0000 7f6e6cff9640 1 -- 172.21.15.19:0/2359398397 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f6e6403f030 con 0x7f6e700ccbb0 2024-08-31T12:25:45.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.981+0000 7f6e76362640 1 -- 172.21.15.19:0/2359398397 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6e700cb280 con 0x7f6e700ccbb0 2024-08-31T12:25:45.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.981+0000 7f6e6cff9640 1 -- 172.21.15.19:0/2359398397 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f6e64005d00 con 0x7f6e700ccbb0 2024-08-31T12:25:45.983 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:25:45.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.983+0000 7f6e76362640 1 -- 172.21.15.19:0/2359398397 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6e5c077a80 msgr2=0x7f6e5c079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:45.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.983+0000 7f6e76362640 1 --2- 172.21.15.19:0/2359398397 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6e5c077a80 0x7f6e5c079f90 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f6e60005fd0 tx=0x7f6e6000b2e0 comp rx=0 tx=0).stop 2024-08-31T12:25:45.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.983+0000 7f6e76362640 1 -- 172.21.15.19:0/2359398397 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6e700ccbb0 msgr2=0x7f6e701f21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:45.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.983+0000 7f6e76362640 1 --2- 172.21.15.19:0/2359398397 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6e700ccbb0 0x7f6e701f21b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6e64009950 tx=0x7f6e64033c20 comp rx=0 tx=0).stop 2024-08-31T12:25:45.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.983+0000 7f6e76362640 1 -- 172.21.15.19:0/2359398397 shutdown_connections 2024-08-31T12:25:45.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.983+0000 7f6e76362640 1 --2- 172.21.15.19:0/2359398397 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6e5c077a80 0x7f6e5c079f90 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:45.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.983+0000 7f6e76362640 1 --2- 172.21.15.19:0/2359398397 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e7015b120 0x7f6e7015d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:45.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.983+0000 7f6e76362640 1 --2- 172.21.15.19:0/2359398397 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6e700ccbb0 0x7f6e701f21b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:45.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.983+0000 7f6e76362640 1 --2- 172.21.15.19:0/2359398397 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6e700cc070 0x7f6e70148210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:45.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.983+0000 7f6e76362640 1 -- 172.21.15.19:0/2359398397 >> 172.21.15.19:0/2359398397 conn(0x7f6e70074810 msgr2=0x7f6e700c56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:45.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.984+0000 7f6e76362640 1 -- 172.21.15.19:0/2359398397 shutdown_connections 2024-08-31T12:25:45.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:45.984+0000 7f6e76362640 1 -- 172.21.15.19:0/2359398397 wait complete. 2024-08-31T12:25:45.998 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-08-31T12:25:45.999 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-08-31T12:25:45.999 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-31T12:25:46.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.129+0000 7fb1022de640 1 -- 172.21.15.19:0/979869067 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb0fc06e0e0 msgr2=0x7fb0fc155300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:46.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.129+0000 7fb1022de640 1 --2- 172.21.15.19:0/979869067 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb0fc06e0e0 0x7fb0fc155300 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc0cb2b0 tx=0x7fb0e402a5d0 comp rx=0 tx=0).stop 2024-08-31T12:25:46.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.129+0000 7fb1022de640 1 -- 172.21.15.19:0/979869067 shutdown_connections 2024-08-31T12:25:46.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.129+0000 7fb1022de640 1 --2- 172.21.15.19:0/979869067 >> v2:172.21.15.43:3300/0 conn(0x7fb0fc15b6e0 0x7fb0fc15baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:46.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.129+0000 7fb1022de640 1 --2- 172.21.15.19:0/979869067 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb0fc06e0e0 0x7fb0fc155300 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:46.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.129+0000 7fb1022de640 1 --2- 172.21.15.19:0/979869067 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb0fc0c8a40 0x7fb0fc0c9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:46.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.129+0000 7fb1022de640 1 -- 172.21.15.19:0/979869067 >> 172.21.15.19:0/979869067 conn(0x7fb0fc0690d0 msgr2=0x7fb0fc0d0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:46.133 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.130+0000 7fb1022de640 1 -- 172.21.15.19:0/979869067 shutdown_connections 2024-08-31T12:25:46.133 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.130+0000 7fb1022de640 1 -- 172.21.15.19:0/979869067 wait complete. 2024-08-31T12:25:46.133 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.132+0000 7fb1022de640 1 Processor -- start 2024-08-31T12:25:46.134 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.132+0000 7fb1022de640 1 -- start start 2024-08-31T12:25:46.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.133+0000 7fb1022de640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb0fc06e0e0 0x7fb0fc1561b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:46.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.133+0000 7fb1022de640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb0fc0c8a40 0x7fb0fc0c61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:46.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.133+0000 7fb1022de640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb0fc15b6e0 0x7fb0fc0c2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:46.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.133+0000 7fb1022de640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fb0fc0c8330 con 0x7fb0fc06e0e0 2024-08-31T12:25:46.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.133+0000 7fb1022de640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb0fc0c8050 con 0x7fb0fc0c8a40 2024-08-31T12:25:46.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.133+0000 7fb1022de640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fb0fc0c8610 con 0x7fb0fc15b6e0 2024-08-31T12:25:46.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.133+0000 7fb0fbfff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb0fc06e0e0 0x7fb0fc1561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:46.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.133+0000 7fb0fbfff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb0fc06e0e0 0x7fb0fc1561b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:48672/0 (socket says 172.21.15.19:48672) 2024-08-31T12:25:46.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.133+0000 7fb0fbfff640 1 -- 172.21.15.19:0/842862489 learned_addr learned my addr 172.21.15.19:0/842862489 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:25:46.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.133+0000 7fb10087b640 1 --2- 172.21.15.19:0/842862489 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb0fc15b6e0 0x7fb0fc0c2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:46.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.133+0000 7fb0f3fff640 1 --2- 172.21.15.19:0/842862489 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb0fc0c8a40 0x7fb0fc0c61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:46.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.134+0000 7fb0f3fff640 1 -- 172.21.15.19:0/842862489 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb0fc0c8a40 msgr2=0x7fb0fc0c61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-08-31T12:25:46.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.134+0000 7fb0f3fff640 1 -- 172.21.15.19:0/842862489 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb0fc0c8a40 msgr2=0x7fb0fc0c61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-31T12:25:46.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.134+0000 7fb0f3fff640 1 --2- 172.21.15.19:0/842862489 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb0fc0c8a40 0x7fb0fc0c61d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-31T12:25:46.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.134+0000 7fb0f3fff640 1 --2- 172.21.15.19:0/842862489 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb0fc0c8a40 0x7fb0fc0c61d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:25:46.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.134+0000 7fb0fbfff640 1 -- 172.21.15.19:0/842862489 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb0fc15b6e0 msgr2=0x7fb0fc0c2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:46.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.134+0000 7fb0fbfff640 1 --2- 172.21.15.19:0/842862489 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb0fc15b6e0 0x7fb0fc0c2870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:46.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.134+0000 7fb0fbfff640 1 -- 172.21.15.19:0/842862489 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb0fc0c8a40 msgr2=0x7fb0fc0c61d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:25:46.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.134+0000 7fb0fbfff640 1 --2- 172.21.15.19:0/842862489 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb0fc0c8a40 0x7fb0fc0c61d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:46.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.134+0000 7fb0fbfff640 1 -- 172.21.15.19:0/842862489 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0fc156b90 con 0x7fb0fc06e0e0 2024-08-31T12:25:46.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.134+0000 7fb0fbfff640 1 --2- 172.21.15.19:0/842862489 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb0fc06e0e0 0x7fb0fc1561b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec00d660 tx=0x7fb0ec00db20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:46.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.134+0000 7fb0f97fa640 1 -- 172.21.15.19:0/842862489 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0ec017070 con 0x7fb0fc06e0e0 2024-08-31T12:25:46.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.134+0000 7fb0f97fa640 1 -- 172.21.15.19:0/842862489 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb0ec005310 con 0x7fb0fc06e0e0 2024-08-31T12:25:46.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.134+0000 7fb0f97fa640 1 -- 172.21.15.19:0/842862489 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0ec009820 con 0x7fb0fc06e0e0 2024-08-31T12:25:46.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.134+0000 7fb1022de640 1 -- 172.21.15.19:0/842862489 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0fc0c2db0 con 0x7fb0fc06e0e0 2024-08-31T12:25:46.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.134+0000 7fb1022de640 1 -- 172.21.15.19:0/842862489 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0fc0c4780 con 0x7fb0fc06e0e0 2024-08-31T12:25:46.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.135+0000 7fb1022de640 1 --2- 172.21.15.19:0/842862489 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb0fc0c5260 0x7fb0fc0c49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:46.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.135+0000 7fb1022de640 1 -- 172.21.15.19:0/842862489 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fb0fc0c4f00 con 0x7fb0fc0c5260 2024-08-31T12:25:46.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.135+0000 7fb0f3fff640 1 --2- 172.21.15.19:0/842862489 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb0fc0c5260 0x7fb0fc0c49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:46.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.136+0000 7fb0f3fff640 1 -- 172.21.15.19:0/842862489 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb0fc0c9850 con 0x7fb0fc0c5260 2024-08-31T12:25:46.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.136+0000 7fb0f3fff640 1 --2- 172.21.15.19:0/842862489 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb0fc0c5260 0x7fb0fc0c49c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc0d1040 tx=0x7fb0e40042c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:46.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.136+0000 7fb0f97fa640 1 -- 172.21.15.19:0/842862489 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e403f030 con 0x7fb0fc0c5260 2024-08-31T12:25:46.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.136+0000 7fb0f97fa640 1 -- 172.21.15.19:0/842862489 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fb0ec0099c0 con 0x7fb0fc06e0e0 2024-08-31T12:25:46.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.137+0000 7fb0f97fa640 1 --2- 172.21.15.19:0/842862489 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb0dc077980 0x7fb0dc079e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:46.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.137+0000 7fb0f97fa640 1 -- 172.21.15.19:0/842862489 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fb0ec099760 con 0x7fb0fc06e0e0 2024-08-31T12:25:46.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.137+0000 7fb10087b640 1 --2- 172.21.15.19:0/842862489 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb0dc077980 0x7fb0dc079e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:46.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.137+0000 7fb0f97fa640 1 -- 172.21.15.19:0/842862489 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb0e40044d0 con 0x7fb0fc0c5260 2024-08-31T12:25:46.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.137+0000 7fb0f97fa640 1 -- 172.21.15.19:0/842862489 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb0fc0c5260 msgr2=0x7fb0fc0c49c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:46.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.137+0000 7fb0f97fa640 1 --2- 172.21.15.19:0/842862489 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb0fc0c5260 0x7fb0fc0c49c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc0d1040 tx=0x7fb0e40042c0 comp rx=0 tx=0).stop 2024-08-31T12:25:46.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.137+0000 7fb0f97fa640 1 -- 172.21.15.19:0/842862489 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb0fc0c5260 msgr2=0x7fb0fc0c49c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:25:46.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.137+0000 7fb0f97fa640 1 --2- 172.21.15.19:0/842862489 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb0fc0c5260 0x7fb0fc0c49c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:46.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.138+0000 7fb10087b640 1 --2- 172.21.15.19:0/842862489 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb0dc077980 0x7fb0dc079e90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc1f2480 tx=0x7fb0f4009560 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:46.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.145+0000 7fb1022de640 1 --2- 172.21.15.19:0/842862489 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb0fc1f13a0 0x7fb0fc0cbec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:46.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.145+0000 7fb1022de640 1 -- 172.21.15.19:0/842862489 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fb0fc0c6710 con 0x7fb0fc1f13a0 2024-08-31T12:25:46.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.145+0000 7fb0f3fff640 1 --2- 172.21.15.19:0/842862489 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb0fc1f13a0 0x7fb0fc0cbec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:46.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.145+0000 7fb0f3fff640 1 -- 172.21.15.19:0/842862489 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb0fc0ccf20 con 0x7fb0fc1f13a0 2024-08-31T12:25:46.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.145+0000 7fb0f3fff640 1 --2- 172.21.15.19:0/842862489 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb0fc1f13a0 0x7fb0fc0cbec0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc051d70 tx=0x7fb0e4009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:46.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.145+0000 7fb0f97fa640 1 -- 172.21.15.19:0/842862489 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e4035040 con 0x7fb0fc1f13a0 2024-08-31T12:25:46.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.146+0000 7fb0f97fa640 1 -- 172.21.15.19:0/842862489 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5043 (secure 0 0 0) 0x7fb0e403b460 con 0x7fb0fc1f13a0 2024-08-31T12:25:46.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.146+0000 7fb0f97fa640 1 -- 172.21.15.19:0/842862489 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb0fc1f13a0 msgr2=0x7fb0fc0cbec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:46.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.146+0000 7fb0f97fa640 1 --2- 172.21.15.19:0/842862489 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb0fc1f13a0 0x7fb0fc0cbec0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc051d70 tx=0x7fb0e4009260 comp rx=0 tx=0).stop 2024-08-31T12:25:46.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.146+0000 7fb0f97fa640 1 -- 172.21.15.19:0/842862489 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb0fc1f13a0 msgr2=0x7fb0fc0cbec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:25:46.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.146+0000 7fb0f97fa640 1 --2- 172.21.15.19:0/842862489 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb0fc1f13a0 0x7fb0fc0cbec0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:46.149 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:25:46.149 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:25:46.149 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:25:46.149 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "leader", 2024-08-31T12:25:46.149 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 80, 2024-08-31T12:25:46.149 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 80701, 2024-08-31T12:25:46.149 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:25:46.149 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:25:46.149 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:25:46.149 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:25:46.150 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:46.150 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 1, 2024-08-31T12:25:46.150 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:25:46.150 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:25:46.150 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:25:46.150 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:25:46.150 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:25:46.150 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:25:46.150 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:25:46.150 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:25:46.150 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:25:46.150 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:25:46.150 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:25:46.150 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:25:46.151 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:25:46.151 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:25:46.151 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:46.151 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:25:46.151 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:25:46.151 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:25:46.151 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:25:46.151 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:25:46.151 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:25:46.151 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:25:46.151 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:25:46.151 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:25:46.151 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:25:46.152 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:25:46.152 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:25:46.152 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:25:46.152 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:46.152 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:46.152 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:25:46.152 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:25:46.152 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:25:46.152 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:25:46.152 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:25:46.152 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:25:46.152 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:25:46.153 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:25:46.153 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:25:46.153 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:25:46.153 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:25:46.153 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:25:46.153 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:25:46.153 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:25:46.153 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:25:46.153 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:25:46.153 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:25:46.153 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:25:46.153 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:25:46.153 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:25:46.154 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:25:46.154 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:25:46.154 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:25:46.154 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:25:46.154 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:25:46.154 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:25:46.154 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:25:46.154 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:25:46.154 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:46.154 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:25:46.154 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:46.154 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:25:46.154 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:46.154 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:25:46.155 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:25:46.155 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:25:46.155 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:25:46.155 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:46.155 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:25:46.155 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:25:46.155 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:46.155 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:46.155 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:46.155 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:25:46.155 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:25:46.155 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:46.155 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:46.155 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:46.155 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:46.156 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:25:46.156 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:25:46.156 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:25:46.156 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:25:46.156 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:25:46.156 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:46.156 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:46.156 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:25:46.156 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:25:46.156 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:25:46.156 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:25:46.156 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:46.156 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:25:46.156 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:25:46.157 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:46.157 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:46.157 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:46.157 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:25:46.157 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:25:46.157 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:46.157 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:46.157 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:46.157 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:46.157 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:25:46.157 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:25:46.157 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:25:46.157 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:25:46.157 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:25:46.158 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:46.158 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:46.158 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:25:46.158 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:25:46.158 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:25:46.158 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:25:46.158 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:46.158 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:25:46.158 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:25:46.158 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:46.158 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:46.158 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:46.158 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:25:46.158 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:25:46.159 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:46.159 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:46.159 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:46.159 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:46.159 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:25:46.159 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:25:46.159 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:25:46.159 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:25:46.159 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:25:46.159 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:46.159 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:46.159 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:46.159 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:25:46.159 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:25:46.160 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:46.160 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:46.160 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:46.160 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:25:46.160 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:46.160 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:46.160 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:25:46.160 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:46.160 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:46.160 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:46.160 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 3 2024-08-31T12:25:46.160 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:46.160 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:46.160 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:25:46.161 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:46.161 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:46.161 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:46.161 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 3 2024-08-31T12:25:46.161 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:46.161 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:46.161 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:25:46.161 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:46.161 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:46.161 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:46.161 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 5 2024-08-31T12:25:46.161 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:46.161 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:46.162 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:25:46.162 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:46.162 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:46.162 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:46.162 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:25:46.162 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:46.162 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:46.162 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:46.162 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:25:46.162 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:25:46.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.146+0000 7fb1022de640 1 -- 172.21.15.19:0/842862489 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb0dc077980 msgr2=0x7fb0dc079e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:46.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.146+0000 7fb1022de640 1 --2- 172.21.15.19:0/842862489 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb0dc077980 0x7fb0dc079e90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc1f2480 tx=0x7fb0f4009560 comp rx=0 tx=0).stop 2024-08-31T12:25:46.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.146+0000 7fb1022de640 1 -- 172.21.15.19:0/842862489 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb0fc06e0e0 msgr2=0x7fb0fc1561b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:46.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.146+0000 7fb1022de640 1 --2- 172.21.15.19:0/842862489 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb0fc06e0e0 0x7fb0fc1561b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec00d660 tx=0x7fb0ec00db20 comp rx=0 tx=0).stop 2024-08-31T12:25:46.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.146+0000 7fb10087b640 1 -- 172.21.15.19:0/842862489 reap_dead start 2024-08-31T12:25:46.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.146+0000 7fb10087b640 1 -- 172.21.15.19:0/842862489 reap_dead start 2024-08-31T12:25:46.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.146+0000 7fb1022de640 1 -- 172.21.15.19:0/842862489 shutdown_connections 2024-08-31T12:25:46.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.146+0000 7fb1022de640 1 -- 172.21.15.19:0/842862489 >> 172.21.15.19:0/842862489 conn(0x7fb0fc0690d0 msgr2=0x7fb0fc159f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:46.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.146+0000 7fb1022de640 1 -- 172.21.15.19:0/842862489 shutdown_connections 2024-08-31T12:25:46.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.146+0000 7fb1022de640 1 -- 172.21.15.19:0/842862489 wait complete. 2024-08-31T12:25:46.164 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:25:46.164 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-08-31T12:25:46.165 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-31T12:25:46.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.301+0000 7fa1db13a640 1 -- 172.21.15.19:0/2635141340 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa1c4005300 msgr2=0x7fa1c400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:46.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.301+0000 7fa1db13a640 1 --2- 172.21.15.19:0/2635141340 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa1c4005300 0x7fa1c400b9c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc001d90 tx=0x7fa1c4005830 comp rx=0 tx=0).stop 2024-08-31T12:25:46.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.301+0000 7fa1db13a640 1 -- 172.21.15.19:0/2635141340 shutdown_connections 2024-08-31T12:25:46.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.301+0000 7fa1db13a640 1 --2- 172.21.15.19:0/2635141340 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa1c4005300 0x7fa1c400b9c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:46.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.301+0000 7fa1db13a640 1 -- 172.21.15.19:0/2635141340 >> 172.21.15.19:0/2635141340 conn(0x7fa1d40cdbb0 msgr2=0x7fa1d40c9f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:46.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.301+0000 7fa1db13a640 1 -- 172.21.15.19:0/2635141340 shutdown_connections 2024-08-31T12:25:46.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.301+0000 7fa1db13a640 1 -- 172.21.15.19:0/2635141340 wait complete. 2024-08-31T12:25:46.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.303+0000 7fa1db13a640 1 Processor -- start 2024-08-31T12:25:46.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.303+0000 7fa1db13a640 1 -- start start 2024-08-31T12:25:46.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.304+0000 7fa1db13a640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa1c4005300 0x7fa1d41ed0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:46.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.304+0000 7fa1db13a640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa1d414cb10 0x7fa1d4142cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:46.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.304+0000 7fa1db13a640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1d414cf20 0x7fa1d41431f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:46.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.304+0000 7fa1db13a640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d40c74d0 con 0x7fa1c4005300 2024-08-31T12:25:46.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.304+0000 7fa1db13a640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d40c66e0 con 0x7fa1d414cf20 2024-08-31T12:25:46.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.304+0000 7fa1db13a640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fa1d40cadb0 con 0x7fa1d414cb10 2024-08-31T12:25:46.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.305+0000 7fa1d8ed6640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa1c4005300 0x7fa1d41ed0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:46.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.305+0000 7fa1d8ed6640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa1c4005300 0x7fa1d41ed0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:48698/0 (socket says 172.21.15.19:48698) 2024-08-31T12:25:46.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.305+0000 7fa1d8ed6640 1 -- 172.21.15.19:0/2042073010 learned_addr learned my addr 172.21.15.19:0/2042073010 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:25:46.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.305+0000 7fa1d3fff640 1 --2- 172.21.15.19:0/2042073010 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa1d414cb10 0x7fa1d4142cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:46.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.305+0000 7fa1d96d7640 1 --2- 172.21.15.19:0/2042073010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1d414cf20 0x7fa1d41431f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:46.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.305+0000 7fa1d8ed6640 1 -- 172.21.15.19:0/2042073010 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa1d414cb10 msgr2=0x7fa1d4142cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:46.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.305+0000 7fa1d8ed6640 1 --2- 172.21.15.19:0/2042073010 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa1d414cb10 0x7fa1d4142cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:46.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.305+0000 7fa1d8ed6640 1 -- 172.21.15.19:0/2042073010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1d414cf20 msgr2=0x7fa1d41431f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:46.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.305+0000 7fa1d8ed6640 1 --2- 172.21.15.19:0/2042073010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1d414cf20 0x7fa1d41431f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:46.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.305+0000 7fa1d8ed6640 1 -- 172.21.15.19:0/2042073010 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1d41ed630 con 0x7fa1c4005300 2024-08-31T12:25:46.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.305+0000 7fa1d8ed6640 1 --2- 172.21.15.19:0/2042073010 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa1c4005300 0x7fa1d41ed0f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa1c80066e0 tx=0x7fa1c80039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:46.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.305+0000 7fa1d17fa640 1 -- 172.21.15.19:0/2042073010 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c8011700 con 0x7fa1c4005300 2024-08-31T12:25:46.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.305+0000 7fa1d17fa640 1 -- 172.21.15.19:0/2042073010 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa1c8016020 con 0x7fa1c4005300 2024-08-31T12:25:46.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.305+0000 7fa1d17fa640 1 -- 172.21.15.19:0/2042073010 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c8011b60 con 0x7fa1c4005300 2024-08-31T12:25:46.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.305+0000 7fa1db13a640 1 -- 172.21.15.19:0/2042073010 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1d40cbf30 con 0x7fa1c4005300 2024-08-31T12:25:46.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.306+0000 7fa1db13a640 1 -- 172.21.15.19:0/2042073010 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1d414a240 con 0x7fa1c4005300 2024-08-31T12:25:46.311 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.307+0000 7fa1db13a640 1 --2- 172.21.15.19:0/2042073010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1d414a480 0x7fa1d414aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:46.311 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.307+0000 7fa1db13a640 1 -- 172.21.15.19:0/2042073010 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d40c6fc0 con 0x7fa1d414a480 2024-08-31T12:25:46.311 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.307+0000 7fa1d3fff640 1 --2- 172.21.15.19:0/2042073010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1d414a480 0x7fa1d414aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:46.311 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.307+0000 7fa1d3fff640 1 -- 172.21.15.19:0/2042073010 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa1d4149d10 con 0x7fa1d414a480 2024-08-31T12:25:46.311 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.307+0000 7fa1d3fff640 1 --2- 172.21.15.19:0/2042073010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1d414a480 0x7fa1d414aed0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa1c0005fd0 tx=0x7fa1c000bf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:46.311 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.307+0000 7fa1d17fa640 1 -- 172.21.15.19:0/2042073010 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fa1c8011d00 con 0x7fa1c4005300 2024-08-31T12:25:46.311 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.308+0000 7fa1d17fa640 1 --2- 172.21.15.19:0/2042073010 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa1b8077a50 0x7fa1b8079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:46.311 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.308+0000 7fa1d17fa640 1 -- 172.21.15.19:0/2042073010 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fa1c8099db0 con 0x7fa1c4005300 2024-08-31T12:25:46.311 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.308+0000 7fa1d17fa640 1 -- 172.21.15.19:0/2042073010 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c0011890 con 0x7fa1d414a480 2024-08-31T12:25:46.311 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.308+0000 7fa1d17fa640 1 -- 172.21.15.19:0/2042073010 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa1c0011a30 con 0x7fa1d414a480 2024-08-31T12:25:46.311 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.308+0000 7fa1d17fa640 1 -- 172.21.15.19:0/2042073010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1d414a480 msgr2=0x7fa1d414aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:46.311 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.308+0000 7fa1d17fa640 1 --2- 172.21.15.19:0/2042073010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1d414a480 0x7fa1d414aed0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa1c0005fd0 tx=0x7fa1c000bf90 comp rx=0 tx=0).stop 2024-08-31T12:25:46.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.308+0000 7fa1d96d7640 1 --2- 172.21.15.19:0/2042073010 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa1b8077a50 0x7fa1b8079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:46.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.308+0000 7fa1d17fa640 1 -- 172.21.15.19:0/2042073010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1d414a480 msgr2=0x7fa1d414aed0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:25:46.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.308+0000 7fa1d17fa640 1 --2- 172.21.15.19:0/2042073010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1d414a480 0x7fa1d414aed0 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa1c0005fd0 tx=0x7fa1c000bf90 comp rx=0 tx=0).stop 2024-08-31T12:25:46.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.309+0000 7fa1d96d7640 1 --2- 172.21.15.19:0/2042073010 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa1b8077a50 0x7fa1b8079f60 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa1c400c3f0 tx=0x7fa1cc001510 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:46.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.315+0000 7fa1db13a640 1 --2- 172.21.15.19:0/2042073010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1d414b410 0x7fa1d4145920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:46.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.315+0000 7fa1db13a640 1 -- 172.21.15.19:0/2042073010 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d4151d90 con 0x7fa1d414b410 2024-08-31T12:25:46.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.316+0000 7fa1d3fff640 1 --2- 172.21.15.19:0/2042073010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1d414b410 0x7fa1d4145920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:46.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.316+0000 7fa1d3fff640 1 -- 172.21.15.19:0/2042073010 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa1d406e3d0 con 0x7fa1d414b410 2024-08-31T12:25:46.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.316+0000 7fa1d3fff640 1 --2- 172.21.15.19:0/2042073010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1d414b410 0x7fa1d4145920 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa1c0005fd0 tx=0x7fa1c0004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:46.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.317+0000 7fa1d17fa640 1 -- 172.21.15.19:0/2042073010 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c0005520 con 0x7fa1d414b410 2024-08-31T12:25:46.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.317+0000 7fa1d17fa640 1 -- 172.21.15.19:0/2042073010 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4539 (secure 0 0 0) 0x7fa1c0005520 con 0x7fa1d414b410 2024-08-31T12:25:46.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.317+0000 7fa1d17fa640 1 -- 172.21.15.19:0/2042073010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1d414b410 msgr2=0x7fa1d4145920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:46.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.317+0000 7fa1d17fa640 1 --2- 172.21.15.19:0/2042073010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1d414b410 0x7fa1d4145920 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa1c0005fd0 tx=0x7fa1c0004730 comp rx=0 tx=0).stop 2024-08-31T12:25:46.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.317+0000 7fa1d17fa640 1 -- 172.21.15.19:0/2042073010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1d414b410 msgr2=0x7fa1d4145920 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:25:46.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.317+0000 7fa1d17fa640 1 --2- 172.21.15.19:0/2042073010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1d414b410 0x7fa1d4145920 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa1c0005fd0 tx=0x7fa1c0004730 comp rx=0 tx=0).stop 2024-08-31T12:25:46.320 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:25:46.320 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:25:46.320 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:25:46.320 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:25:46.320 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 80, 2024-08-31T12:25:46.320 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 7604, 2024-08-31T12:25:46.320 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:25:46.320 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:25:46.320 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:25:46.320 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:25:46.320 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:46.321 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 2, 2024-08-31T12:25:46.321 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:25:46.321 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:25:46.321 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:25:46.321 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:25:46.321 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:25:46.321 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:25:46.321 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:25:46.321 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:25:46.321 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:25:46.321 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:25:46.321 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:25:46.321 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:25:46.321 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:25:46.322 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:25:46.322 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:46.322 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:25:46.322 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:25:46.322 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:25:46.322 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:25:46.322 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:25:46.322 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:25:46.322 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:25:46.322 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:25:46.322 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:25:46.322 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:25:46.322 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:25:46.322 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:25:46.323 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:25:46.323 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:46.323 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:46.323 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:25:46.323 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:25:46.323 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:25:46.323 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:25:46.323 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:25:46.323 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:25:46.323 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:25:46.323 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:25:46.323 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:25:46.323 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:25:46.323 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:25:46.324 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:25:46.324 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:25:46.324 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:25:46.324 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:25:46.324 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:25:46.324 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:25:46.324 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:25:46.324 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:25:46.324 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:25:46.324 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:25:46.324 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:25:46.324 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:25:46.324 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:25:46.324 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:25:46.325 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:25:46.325 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:25:46.325 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:25:46.325 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:46.325 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:25:46.325 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:46.325 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:25:46.325 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:46.325 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:25:46.325 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:25:46.325 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:25:46.325 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:25:46.325 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:46.325 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:25:46.326 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:25:46.326 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:46.326 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:46.326 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:46.326 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:25:46.326 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:25:46.326 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:46.326 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:46.326 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:46.326 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:46.326 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:25:46.326 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:25:46.326 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:25:46.327 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:25:46.327 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:25:46.327 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:46.327 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:46.327 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:25:46.327 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:25:46.327 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:25:46.327 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:25:46.327 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:46.327 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:25:46.327 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:25:46.327 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:46.327 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:46.327 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:46.328 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:25:46.328 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:25:46.328 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:46.328 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:46.328 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:46.328 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:46.328 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:25:46.328 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:25:46.328 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:25:46.328 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:25:46.328 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:25:46.328 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:46.328 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:46.328 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:25:46.329 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:25:46.329 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:25:46.329 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:25:46.329 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:46.329 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:25:46.329 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:25:46.329 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:46.329 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:46.329 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:46.329 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:25:46.329 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:25:46.329 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:46.329 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:46.329 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:46.330 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:46.330 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:25:46.330 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:25:46.330 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:25:46.330 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:25:46.330 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:25:46.330 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:46.330 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:46.330 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:46.330 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:25:46.330 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:25:46.330 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:46.330 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:46.330 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:46.331 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:25:46.331 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:46.331 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:46.331 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:25:46.331 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:46.331 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:46.331 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:46.331 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:25:46.331 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:46.331 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:46.331 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:46.331 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:25:46.331 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:25:46.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.317+0000 7fa1db13a640 1 -- 172.21.15.19:0/2042073010 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa1b8077a50 msgr2=0x7fa1b8079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:46.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.317+0000 7fa1db13a640 1 --2- 172.21.15.19:0/2042073010 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fa1b8077a50 0x7fa1b8079f60 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa1c400c3f0 tx=0x7fa1cc001510 comp rx=0 tx=0).stop 2024-08-31T12:25:46.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.317+0000 7fa1db13a640 1 -- 172.21.15.19:0/2042073010 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa1c4005300 msgr2=0x7fa1d41ed0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:46.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.317+0000 7fa1d96d7640 1 -- 172.21.15.19:0/2042073010 reap_dead start 2024-08-31T12:25:46.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.317+0000 7fa1db13a640 1 --2- 172.21.15.19:0/2042073010 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa1c4005300 0x7fa1d41ed0f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa1c80066e0 tx=0x7fa1c80039c0 comp rx=0 tx=0).stop 2024-08-31T12:25:46.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.317+0000 7fa1db13a640 1 -- 172.21.15.19:0/2042073010 shutdown_connections 2024-08-31T12:25:46.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.317+0000 7fa1db13a640 1 --2- 172.21.15.19:0/2042073010 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa1c4005300 0x7fa1d41ed0f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:46.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.317+0000 7fa1db13a640 1 -- 172.21.15.19:0/2042073010 >> 172.21.15.19:0/2042073010 conn(0x7fa1d40cdbb0 msgr2=0x7fa1d414f510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:46.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.317+0000 7fa1db13a640 1 -- 172.21.15.19:0/2042073010 shutdown_connections 2024-08-31T12:25:46.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.317+0000 7fa1db13a640 1 -- 172.21.15.19:0/2042073010 wait complete. 2024-08-31T12:25:46.334 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:25:46.334 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-08-31T12:25:46.334 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-31T12:25:46.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.455+0000 7f51bec39640 1 -- 172.21.15.19:0/175698438 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f51b80c9850 msgr2=0x7f51b8155300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:46.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.455+0000 7f51bec39640 1 --2- 172.21.15.19:0/175698438 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f51b80c9850 0x7f51b8155300 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f51b0009be0 tx=0x7f51b002a940 comp rx=0 tx=0).stop 2024-08-31T12:25:46.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.455+0000 7f51bec39640 1 -- 172.21.15.19:0/175698438 shutdown_connections 2024-08-31T12:25:46.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.455+0000 7f51bec39640 1 --2- 172.21.15.19:0/175698438 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f51b80c9850 0x7f51b8155300 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:46.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.455+0000 7f51bec39640 1 -- 172.21.15.19:0/175698438 >> 172.21.15.19:0/175698438 conn(0x7f51b80690d0 msgr2=0x7f51b80d0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:46.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.455+0000 7f51bec39640 1 -- 172.21.15.19:0/175698438 shutdown_connections 2024-08-31T12:25:46.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.455+0000 7f51bec39640 1 -- 172.21.15.19:0/175698438 wait complete. 2024-08-31T12:25:46.459 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.457+0000 7f51bec39640 1 Processor -- start 2024-08-31T12:25:46.459 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.457+0000 7f51bec39640 1 -- start start 2024-08-31T12:25:46.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.459+0000 7f51bec39640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f51b806e0e0 0x7f51b80c6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:46.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.459+0000 7f51bec39640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f51b80c8a40 0x7f51b8150d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:46.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.459+0000 7f51bec39640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f51b80c9850 0x7f51b81f1470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:46.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.459+0000 7f51bec39640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f51b815b250 con 0x7f51b80c9850 2024-08-31T12:25:46.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.459+0000 7f51bec39640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f51b815af70 con 0x7f51b80c8a40 2024-08-31T12:25:46.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.459+0000 7f51bec39640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f51b80c8610 con 0x7f51b806e0e0 2024-08-31T12:25:46.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.459+0000 7f51bd1d6640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f51b80c9850 0x7f51b81f1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:46.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.459+0000 7f51bd1d6640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f51b80c9850 0x7f51b81f1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:48710/0 (socket says 172.21.15.19:48710) 2024-08-31T12:25:46.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.459+0000 7f51bd1d6640 1 -- 172.21.15.19:0/115333131 learned_addr learned my addr 172.21.15.19:0/115333131 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:25:46.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.459+0000 7f51b7fff640 1 --2- 172.21.15.19:0/115333131 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f51b80c8a40 0x7f51b8150d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:46.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.459+0000 7f51bc9d5640 1 --2- 172.21.15.19:0/115333131 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f51b806e0e0 0x7f51b80c6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:46.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.460+0000 7f51bd1d6640 1 -- 172.21.15.19:0/115333131 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f51b806e0e0 msgr2=0x7f51b80c6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:46.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.460+0000 7f51bd1d6640 1 --2- 172.21.15.19:0/115333131 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f51b806e0e0 0x7f51b80c6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:46.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.460+0000 7f51bd1d6640 1 -- 172.21.15.19:0/115333131 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f51b80c8a40 msgr2=0x7f51b8150d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:46.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.460+0000 7f51bd1d6640 1 --2- 172.21.15.19:0/115333131 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f51b80c8a40 0x7f51b8150d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:46.464 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.460+0000 7f51bd1d6640 1 -- 172.21.15.19:0/115333131 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51b81f19b0 con 0x7f51b80c9850 2024-08-31T12:25:46.464 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.460+0000 7f51b7fff640 1 --2- 172.21.15.19:0/115333131 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f51b80c8a40 0x7f51b8150d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:25:46.464 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.460+0000 7f51bd1d6640 1 --2- 172.21.15.19:0/115333131 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f51b80c9850 0x7f51b81f1470 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f51b0005fd0 tx=0x7f51b0004660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:46.464 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.460+0000 7f51b57fa640 1 -- 172.21.15.19:0/115333131 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b0003e60 con 0x7f51b80c9850 2024-08-31T12:25:46.464 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.460+0000 7f51b57fa640 1 -- 172.21.15.19:0/115333131 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f51b0043070 con 0x7f51b80c9850 2024-08-31T12:25:46.464 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.460+0000 7f51bec39640 1 -- 172.21.15.19:0/115333131 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51b814e130 con 0x7f51b80c9850 2024-08-31T12:25:46.464 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.460+0000 7f51b57fa640 1 -- 172.21.15.19:0/115333131 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b0031d60 con 0x7f51b80c9850 2024-08-31T12:25:46.464 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.460+0000 7f51bec39640 1 -- 172.21.15.19:0/115333131 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51b814e6f0 con 0x7f51b80c9850 2024-08-31T12:25:46.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.461+0000 7f51bec39640 1 --2- 172.21.15.19:0/115333131 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f51b81f1b80 0x7f51b814e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:46.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.461+0000 7f51bec39640 1 -- 172.21.15.19:0/115333131 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f51b80c75b0 con 0x7f51b81f1b80 2024-08-31T12:25:46.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.461+0000 7f51bc9d5640 1 --2- 172.21.15.19:0/115333131 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f51b81f1b80 0x7f51b814e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:46.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.462+0000 7f51bc9d5640 1 -- 172.21.15.19:0/115333131 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f51b8155300 con 0x7f51b81f1b80 2024-08-31T12:25:46.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.462+0000 7f51bc9d5640 1 --2- 172.21.15.19:0/115333131 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f51b81f1b80 0x7f51b814e8f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f51ac00e400 tx=0x7f51ac00e8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:46.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.462+0000 7f51b57fa640 1 -- 172.21.15.19:0/115333131 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ac019070 con 0x7f51b81f1b80 2024-08-31T12:25:46.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.462+0000 7f51b57fa640 1 -- 172.21.15.19:0/115333131 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f51b0048460 con 0x7f51b80c9850 2024-08-31T12:25:46.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.463+0000 7f51b57fa640 1 --2- 172.21.15.19:0/115333131 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f519c077950 0x7f519c079e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:46.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.463+0000 7f51b57fa640 1 -- 172.21.15.19:0/115333131 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f51b00ba790 con 0x7f51b80c9850 2024-08-31T12:25:46.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.463+0000 7f51b57fa640 1 -- 172.21.15.19:0/115333131 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f51ac0052e0 con 0x7f51b81f1b80 2024-08-31T12:25:46.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.463+0000 7f51b57fa640 1 -- 172.21.15.19:0/115333131 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f51b81f1b80 msgr2=0x7f51b814e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:46.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.463+0000 7f51b7fff640 1 --2- 172.21.15.19:0/115333131 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f519c077950 0x7f519c079e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:46.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.463+0000 7f51b57fa640 1 --2- 172.21.15.19:0/115333131 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f51b81f1b80 0x7f51b814e8f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f51ac00e400 tx=0x7f51ac00e8c0 comp rx=0 tx=0).stop 2024-08-31T12:25:46.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.463+0000 7f51b57fa640 1 -- 172.21.15.19:0/115333131 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f51b81f1b80 msgr2=0x7f51b814e8f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:25:46.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.463+0000 7f51b57fa640 1 --2- 172.21.15.19:0/115333131 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f51b81f1b80 0x7f51b814e8f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:46.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.464+0000 7f51b7fff640 1 --2- 172.21.15.19:0/115333131 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f519c077950 0x7f519c079e60 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f51a4000c00 tx=0x7f51a4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:46.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.473+0000 7f51bec39640 1 --2- 172.21.15.19:0/115333131 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f51b806d1e0 0x7f51b80c2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:46.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.473+0000 7f51bec39640 1 -- 172.21.15.19:0/115333131 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f51b80ccf20 con 0x7f51b806d1e0 2024-08-31T12:25:46.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.473+0000 7f51bc9d5640 1 --2- 172.21.15.19:0/115333131 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f51b806d1e0 0x7f51b80c2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:46.476 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.474+0000 7f51bc9d5640 1 -- 172.21.15.19:0/115333131 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f51b80ccb30 con 0x7f51b806d1e0 2024-08-31T12:25:46.476 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.474+0000 7f51bc9d5640 1 --2- 172.21.15.19:0/115333131 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f51b806d1e0 0x7f51b80c2410 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f51ac016000 tx=0x7f51ac000fe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:46.476 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.474+0000 7f51b57fa640 1 -- 172.21.15.19:0/115333131 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ac01c2a0 con 0x7f51b806d1e0 2024-08-31T12:25:46.477 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.474+0000 7f51b57fa640 1 -- 172.21.15.19:0/115333131 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5042 (secure 0 0 0) 0x7f51ac01c2a0 con 0x7f51b806d1e0 2024-08-31T12:25:46.477 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.474+0000 7f51b57fa640 1 -- 172.21.15.19:0/115333131 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f51b806d1e0 msgr2=0x7f51b80c2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:46.477 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.474+0000 7f51b57fa640 1 --2- 172.21.15.19:0/115333131 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f51b806d1e0 0x7f51b80c2410 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f51ac016000 tx=0x7f51ac000fe0 comp rx=0 tx=0).stop 2024-08-31T12:25:46.477 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.474+0000 7f51b57fa640 1 -- 172.21.15.19:0/115333131 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f51b806d1e0 msgr2=0x7f51b80c2410 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:25:46.477 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.474+0000 7f51b57fa640 1 --2- 172.21.15.19:0/115333131 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f51b806d1e0 0x7f51b80c2410 secure :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f51ac016000 tx=0x7f51ac000fe0 comp rx=0 tx=0).stop 2024-08-31T12:25:46.477 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:25:46.477 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:25:46.477 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:25:46.477 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:25:46.478 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 80, 2024-08-31T12:25:46.478 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 133833, 2024-08-31T12:25:46.478 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:25:46.478 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:25:46.478 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:25:46.478 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:25:46.478 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:46.478 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 2, 2024-08-31T12:25:46.478 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:25:46.478 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:25:46.478 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:25:46.478 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:25:46.478 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:25:46.478 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:25:46.479 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:25:46.479 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:25:46.479 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:25:46.479 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:25:46.479 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:25:46.479 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:25:46.479 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:25:46.479 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:25:46.479 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:46.479 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:25:46.479 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:25:46.479 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:25:46.479 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:25:46.480 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:25:46.480 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:25:46.480 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:25:46.480 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:25:46.480 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:25:46.480 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:25:46.480 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:25:46.480 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:25:46.480 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:25:46.480 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:46.480 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:46.480 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:25:46.481 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:25:46.481 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:25:46.481 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:25:46.481 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:25:46.481 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:25:46.481 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:25:46.481 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:25:46.481 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:25:46.481 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:25:46.481 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:25:46.481 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:25:46.481 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:25:46.481 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:25:46.482 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:25:46.482 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:25:46.482 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:25:46.482 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:25:46.482 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:25:46.482 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:25:46.482 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:25:46.482 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:25:46.482 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:25:46.482 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:25:46.482 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:25:46.482 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:25:46.482 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:25:46.482 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:25:46.483 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:46.483 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:25:46.483 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:46.483 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:25:46.483 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:46.483 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:25:46.483 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:25:46.483 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:25:46.483 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:25:46.483 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:46.483 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:25:46.483 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:25:46.483 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:46.483 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:46.483 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:46.484 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:25:46.484 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:25:46.484 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:46.484 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:46.484 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:46.484 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:46.484 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:25:46.484 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:25:46.484 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:25:46.484 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:25:46.484 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:25:46.484 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:46.484 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:46.485 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:25:46.485 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:25:46.485 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:25:46.485 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:25:46.485 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:46.485 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:25:46.485 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:25:46.485 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:46.485 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:46.485 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:46.485 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:25:46.485 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:25:46.485 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:46.486 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:46.486 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:46.486 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:46.486 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:25:46.486 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:25:46.486 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:25:46.486 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:25:46.486 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:25:46.486 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:46.486 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:46.486 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:25:46.486 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:25:46.486 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:25:46.486 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:25:46.487 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:46.487 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:25:46.487 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:25:46.487 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:46.487 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:46.487 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:46.487 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:25:46.487 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:25:46.487 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:46.487 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:46.487 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:46.487 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:46.487 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:25:46.488 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:25:46.488 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:25:46.488 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:25:46.488 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:25:46.488 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:46.488 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:46.488 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:46.488 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:25:46.488 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:25:46.488 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:46.488 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:46.488 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:46.488 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:25:46.488 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:46.488 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:46.489 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:25:46.489 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:46.489 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:46.489 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:46.489 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 3 2024-08-31T12:25:46.489 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:46.489 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:46.489 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:25:46.489 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:46.489 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:46.489 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:46.489 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 5 2024-08-31T12:25:46.490 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:46.490 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:46.490 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:25:46.490 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:46.490 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:46.490 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:46.490 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 3 2024-08-31T12:25:46.490 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:46.490 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:46.490 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:25:46.490 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:46.490 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:46.490 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:46.490 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:25:46.490 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:46.491 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:46.491 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:46.491 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:25:46.491 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:25:46.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.475+0000 7f51bec39640 1 -- 172.21.15.19:0/115333131 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f519c077950 msgr2=0x7f519c079e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:46.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.475+0000 7f51bec39640 1 --2- 172.21.15.19:0/115333131 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f519c077950 0x7f519c079e60 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f51a4000c00 tx=0x7f51a4009040 comp rx=0 tx=0).stop 2024-08-31T12:25:46.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.475+0000 7f51bec39640 1 -- 172.21.15.19:0/115333131 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f51b80c9850 msgr2=0x7f51b81f1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:46.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.475+0000 7f51bec39640 1 --2- 172.21.15.19:0/115333131 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f51b80c9850 0x7f51b81f1470 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f51b0005fd0 tx=0x7f51b0004660 comp rx=0 tx=0).stop 2024-08-31T12:25:46.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.475+0000 7f51bd1d6640 1 -- 172.21.15.19:0/115333131 reap_dead start 2024-08-31T12:25:46.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.475+0000 7f51bd1d6640 1 -- 172.21.15.19:0/115333131 reap_dead start 2024-08-31T12:25:46.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.475+0000 7f51bec39640 1 -- 172.21.15.19:0/115333131 shutdown_connections 2024-08-31T12:25:46.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.475+0000 7f51bec39640 1 -- 172.21.15.19:0/115333131 >> 172.21.15.19:0/115333131 conn(0x7f51b80690d0 msgr2=0x7f51b80cb2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:46.493 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.475+0000 7f51bec39640 1 -- 172.21.15.19:0/115333131 shutdown_connections 2024-08-31T12:25:46.493 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.475+0000 7f51bec39640 1 -- 172.21.15.19:0/115333131 wait complete. 2024-08-31T12:25:46.493 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:25:46.493 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-08-31T12:25:46.493 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-08-31T12:25:46.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.615+0000 7f8fbb1c6640 1 -- 172.21.15.19:0/764911583 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8fb406e0e0 msgr2=0x7f8fb4155300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:46.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.615+0000 7f8fbb1c6640 1 --2- 172.21.15.19:0/764911583 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8fb406e0e0 0x7f8fb4155300 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8fb40cb2b0 tx=0x7f8fa002a5d0 comp rx=0 tx=0).stop 2024-08-31T12:25:46.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.615+0000 7f8fbb1c6640 1 -- 172.21.15.19:0/764911583 shutdown_connections 2024-08-31T12:25:46.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.615+0000 7f8fbb1c6640 1 --2- 172.21.15.19:0/764911583 >> v2:172.21.15.43:3300/0 conn(0x7f8fb415b6e0 0x7f8fb415baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:46.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.615+0000 7f8fbb1c6640 1 --2- 172.21.15.19:0/764911583 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8fb406e0e0 0x7f8fb4155300 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:46.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.615+0000 7f8fbb1c6640 1 --2- 172.21.15.19:0/764911583 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8fb40c8a40 0x7f8fb40c9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:46.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.615+0000 7f8fbb1c6640 1 -- 172.21.15.19:0/764911583 >> 172.21.15.19:0/764911583 conn(0x7f8fb40690d0 msgr2=0x7f8fb40d0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:46.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.615+0000 7f8fbb1c6640 1 -- 172.21.15.19:0/764911583 shutdown_connections 2024-08-31T12:25:46.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.615+0000 7f8fbb1c6640 1 -- 172.21.15.19:0/764911583 wait complete. 2024-08-31T12:25:46.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.617+0000 7f8fbb1c6640 1 Processor -- start 2024-08-31T12:25:46.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.617+0000 7f8fbb1c6640 1 -- start start 2024-08-31T12:25:46.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.619+0000 7f8fbb1c6640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8fb406e0e0 0x7f8fb41561b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:46.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.619+0000 7f8fbb1c6640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8fb40c8a40 0x7f8fb40c61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:46.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.619+0000 7f8fbb1c6640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8fb415b6e0 0x7f8fb40c2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:46.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.619+0000 7f8fbb1c6640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb40c8330 con 0x7f8fb415b6e0 2024-08-31T12:25:46.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.619+0000 7f8fbb1c6640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb40c8050 con 0x7f8fb40c8a40 2024-08-31T12:25:46.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.619+0000 7f8fbb1c6640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f8fb40c8610 con 0x7f8fb406e0e0 2024-08-31T12:25:46.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.619+0000 7f8fb9763640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8fb415b6e0 0x7f8fb40c2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:46.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.619+0000 7f8fb8f62640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8fb406e0e0 0x7f8fb41561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:46.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.619+0000 7f8fb3fff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8fb40c8a40 0x7f8fb40c61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:46.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.619+0000 7f8fb8f62640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8fb406e0e0 0x7f8fb41561b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.19:50562/0 (socket says 172.21.15.19:50562) 2024-08-31T12:25:46.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.619+0000 7f8fb9763640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8fb415b6e0 0x7f8fb40c2870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:48728/0 (socket says 172.21.15.19:48728) 2024-08-31T12:25:46.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.619+0000 7f8fb8f62640 1 -- 172.21.15.19:0/645444925 learned_addr learned my addr 172.21.15.19:0/645444925 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:25:46.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.620+0000 7f8fb8f62640 1 -- 172.21.15.19:0/645444925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8fb40c8a40 msgr2=0x7f8fb40c61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:46.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.620+0000 7f8fb8f62640 1 --2- 172.21.15.19:0/645444925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8fb40c8a40 0x7f8fb40c61d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:46.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.620+0000 7f8fb8f62640 1 -- 172.21.15.19:0/645444925 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8fb415b6e0 msgr2=0x7f8fb40c2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:46.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.620+0000 7f8fb8f62640 1 --2- 172.21.15.19:0/645444925 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8fb415b6e0 0x7f8fb40c2870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:46.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.620+0000 7f8fb8f62640 1 -- 172.21.15.19:0/645444925 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fb4156b90 con 0x7f8fb406e0e0 2024-08-31T12:25:46.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.620+0000 7f8fb8f62640 1 --2- 172.21.15.19:0/645444925 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8fb406e0e0 0x7f8fb41561b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8fa800e7f0 tx=0x7f8fa800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:46.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.620+0000 7f8fb17fa640 1 -- 172.21.15.19:0/645444925 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa800eea0 con 0x7f8fb406e0e0 2024-08-31T12:25:46.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.620+0000 7f8fb17fa640 1 -- 172.21.15.19:0/645444925 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8fa800ccc0 con 0x7f8fb406e0e0 2024-08-31T12:25:46.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.620+0000 7f8fb17fa640 1 -- 172.21.15.19:0/645444925 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa8005280 con 0x7f8fb406e0e0 2024-08-31T12:25:46.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.620+0000 7f8fbb1c6640 1 -- 172.21.15.19:0/645444925 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fb40c2db0 con 0x7f8fb406e0e0 2024-08-31T12:25:46.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.620+0000 7f8fbb1c6640 1 -- 172.21.15.19:0/645444925 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fb40c4780 con 0x7f8fb406e0e0 2024-08-31T12:25:46.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.621+0000 7f8fbb1c6640 1 -- 172.21.15.19:0/645444925 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fb40c9850 con 0x7f8fb406e0e0 2024-08-31T12:25:46.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.622+0000 7f8fb17fa640 1 -- 172.21.15.19:0/645444925 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f8fa8005420 con 0x7f8fb406e0e0 2024-08-31T12:25:46.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.623+0000 7f8fb17fa640 1 --2- 172.21.15.19:0/645444925 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8f98077a80 0x7f8f98079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:46.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.623+0000 7f8fb17fa640 1 -- 172.21.15.19:0/645444925 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f8fa8099fe0 con 0x7f8fb406e0e0 2024-08-31T12:25:46.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.623+0000 7f8fb3fff640 1 --2- 172.21.15.19:0/645444925 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8f98077a80 0x7f8f98079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:46.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.624+0000 7f8fb3fff640 1 --2- 172.21.15.19:0/645444925 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8f98077a80 0x7f8f98079f90 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f8fa0033000 tx=0x7f8fa0004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:46.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.628+0000 7f8fb17fa640 1 -- 172.21.15.19:0/645444925 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f8fa8062570 con 0x7f8fb406e0e0 2024-08-31T12:25:46.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.915+0000 7f8fbb1c6640 1 -- 172.21.15.19:0/645444925 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f8fb40ccf20 con 0x7f8fb406e0e0 2024-08-31T12:25:46.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.917+0000 7f8fb17fa640 1 -- 172.21.15.19:0/645444925 <== mon.2 v2:172.21.15.43:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f8fa806b100 con 0x7f8fb406e0e0 2024-08-31T12:25:46.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.919+0000 7f8fbb1c6640 1 -- 172.21.15.19:0/645444925 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8f98077a80 msgr2=0x7f8f98079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:46.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.919+0000 7f8fbb1c6640 1 --2- 172.21.15.19:0/645444925 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8f98077a80 0x7f8f98079f90 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f8fa0033000 tx=0x7f8fa0004730 comp rx=0 tx=0).stop 2024-08-31T12:25:46.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.919+0000 7f8fbb1c6640 1 -- 172.21.15.19:0/645444925 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8fb406e0e0 msgr2=0x7f8fb41561b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:46.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.919+0000 7f8fbb1c6640 1 --2- 172.21.15.19:0/645444925 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8fb406e0e0 0x7f8fb41561b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8fa800e7f0 tx=0x7f8fa800ecb0 comp rx=0 tx=0).stop 2024-08-31T12:25:46.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.919+0000 7f8fbb1c6640 1 -- 172.21.15.19:0/645444925 shutdown_connections 2024-08-31T12:25:46.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.919+0000 7f8fbb1c6640 1 --2- 172.21.15.19:0/645444925 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8f98077a80 0x7f8f98079f90 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:46.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.919+0000 7f8fbb1c6640 1 --2- 172.21.15.19:0/645444925 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8fb415b6e0 0x7f8fb40c2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:46.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.919+0000 7f8fbb1c6640 1 --2- 172.21.15.19:0/645444925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8fb40c8a40 0x7f8fb40c61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:46.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.919+0000 7f8fbb1c6640 1 --2- 172.21.15.19:0/645444925 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8fb406e0e0 0x7f8fb41561b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:46.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.919+0000 7f8fbb1c6640 1 -- 172.21.15.19:0/645444925 >> 172.21.15.19:0/645444925 conn(0x7f8fb40690d0 msgr2=0x7f8fb4159f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:46.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.919+0000 7f8fbb1c6640 1 -- 172.21.15.19:0/645444925 shutdown_connections 2024-08-31T12:25:46.923 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:46.919+0000 7f8fbb1c6640 1 -- 172.21.15.19:0/645444925 wait complete. 2024-08-31T12:25:46.937 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2024-08-31T12:25:56.938 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-08-31T12:25:56.938 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:25:57.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.426+0000 7f3abb8ce640 1 -- 172.21.15.19:0/3729062728 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ab4071b40 msgr2=0x7f3ab40caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:57.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.426+0000 7f3abb8ce640 1 --2- 172.21.15.19:0/3729062728 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ab4071b40 0x7f3ab40caa20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3aa4009830 tx=0x7f3aa402a5d0 comp rx=0 tx=0).stop 2024-08-31T12:25:57.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.426+0000 7f3abb8ce640 1 -- 172.21.15.19:0/3729062728 shutdown_connections 2024-08-31T12:25:57.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.426+0000 7f3abb8ce640 1 --2- 172.21.15.19:0/3729062728 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ab4071b40 0x7f3ab40caa20 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:57.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.426+0000 7f3abb8ce640 1 -- 172.21.15.19:0/3729062728 >> 172.21.15.19:0/3729062728 conn(0x7f3ab4072c80 msgr2=0x7f3ab4073090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:57.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.426+0000 7f3abb8ce640 1 -- 172.21.15.19:0/3729062728 shutdown_connections 2024-08-31T12:25:57.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.426+0000 7f3abb8ce640 1 -- 172.21.15.19:0/3729062728 wait complete. 2024-08-31T12:25:57.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.428+0000 7f3abb8ce640 1 Processor -- start 2024-08-31T12:25:57.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.428+0000 7f3abb8ce640 1 -- start start 2024-08-31T12:25:57.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.429+0000 7f3abb8ce640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ab4071b40 0x7f3ab4149220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:57.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.429+0000 7f3abb8ce640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3ab4153230 0x7f3ab40cdc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:57.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.429+0000 7f3abb8ce640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3ab4159470 0x7f3ab4157c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:57.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.429+0000 7f3abb8ce640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab4072360 con 0x7f3ab4159470 2024-08-31T12:25:57.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.429+0000 7f3abb8ce640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab40721e0 con 0x7f3ab4071b40 2024-08-31T12:25:57.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.429+0000 7f3abb8ce640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f3ab40711e0 con 0x7f3ab4153230 2024-08-31T12:25:57.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.429+0000 7f3ab9e6b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3ab4159470 0x7f3ab4157c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:57.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.429+0000 7f3ab966a640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ab4071b40 0x7f3ab4149220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:57.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.429+0000 7f3ab9e6b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3ab4159470 0x7f3ab4157c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:34288/0 (socket says 172.21.15.19:34288) 2024-08-31T12:25:57.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.429+0000 7f3ab9e6b640 1 -- 172.21.15.19:0/2166093080 learned_addr learned my addr 172.21.15.19:0/2166093080 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:25:57.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.429+0000 7f3ab8e69640 1 --2- 172.21.15.19:0/2166093080 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3ab4153230 0x7f3ab40cdc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:57.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.430+0000 7f3ab9e6b640 1 -- 172.21.15.19:0/2166093080 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3ab4153230 msgr2=0x7f3ab40cdc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:57.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.430+0000 7f3ab9e6b640 1 --2- 172.21.15.19:0/2166093080 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3ab4153230 0x7f3ab40cdc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:57.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.430+0000 7f3ab9e6b640 1 -- 172.21.15.19:0/2166093080 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ab4071b40 msgr2=0x7f3ab4149220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:57.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.430+0000 7f3ab9e6b640 1 --2- 172.21.15.19:0/2166093080 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ab4071b40 0x7f3ab4149220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:57.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.430+0000 7f3ab9e6b640 1 -- 172.21.15.19:0/2166093080 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ab415b050 con 0x7f3ab4159470 2024-08-31T12:25:57.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.430+0000 7f3ab9e6b640 1 --2- 172.21.15.19:0/2166093080 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3ab4159470 0x7f3ab4157c70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3aac00cbf0 tx=0x7f3aac00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:57.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.430+0000 7f3aa9ffb640 1 -- 172.21.15.19:0/2166093080 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aac016020 con 0x7f3ab4159470 2024-08-31T12:25:57.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.430+0000 7f3aa9ffb640 1 -- 172.21.15.19:0/2166093080 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3aac00ae10 con 0x7f3ab4159470 2024-08-31T12:25:57.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.430+0000 7f3abb8ce640 1 -- 172.21.15.19:0/2166093080 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ab415b2f0 con 0x7f3ab4159470 2024-08-31T12:25:57.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.430+0000 7f3aa9ffb640 1 -- 172.21.15.19:0/2166093080 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aac012680 con 0x7f3ab4159470 2024-08-31T12:25:57.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.431+0000 7f3abb8ce640 1 -- 172.21.15.19:0/2166093080 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ab40cc860 con 0x7f3ab4159470 2024-08-31T12:25:57.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.432+0000 7f3abb8ce640 1 -- 172.21.15.19:0/2166093080 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ab40caa20 con 0x7f3ab4159470 2024-08-31T12:25:57.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.433+0000 7f3aa9ffb640 1 -- 172.21.15.19:0/2166093080 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f3aac019020 con 0x7f3ab4159470 2024-08-31T12:25:57.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.433+0000 7f3aa9ffb640 1 --2- 172.21.15.19:0/2166093080 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3a98077a50 0x7f3a98079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:57.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.433+0000 7f3aa9ffb640 1 -- 172.21.15.19:0/2166093080 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f3aac0999c0 con 0x7f3ab4159470 2024-08-31T12:25:57.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.433+0000 7f3ab966a640 1 --2- 172.21.15.19:0/2166093080 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3a98077a50 0x7f3a98079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:57.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.434+0000 7f3ab966a640 1 --2- 172.21.15.19:0/2166093080 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3a98077a50 0x7f3a98079f60 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3ab0000f60 tx=0x7f3ab0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:57.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.438+0000 7f3aa9ffb640 1 -- 172.21.15.19:0/2166093080 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f3aac09be20 con 0x7f3ab4159470 2024-08-31T12:25:57.626 INFO:tasks.workunit.client.0.smithi019.stdout:fs setup took 495 secs 2024-08-31T12:25:57.654 DEBUG:teuthology.orchestra.run.smithi019:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:25:57.657 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:25:57.734 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.733+0000 7f3abb8ce640 1 -- 172.21.15.19:0/2166093080 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3ab40ca110 con 0x7f3ab4159470 2024-08-31T12:25:57.734 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.733+0000 7f3aa9ffb640 1 -- 172.21.15.19:0/2166093080 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f3aac020030 con 0x7f3ab4159470 2024-08-31T12:25:57.734 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:25:57.737 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.735+0000 7f3abb8ce640 1 -- 172.21.15.19:0/2166093080 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3a98077a50 msgr2=0x7f3a98079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:57.737 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.735+0000 7f3abb8ce640 1 --2- 172.21.15.19:0/2166093080 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3a98077a50 0x7f3a98079f60 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3ab0000f60 tx=0x7f3ab0009040 comp rx=0 tx=0).stop 2024-08-31T12:25:57.737 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.735+0000 7f3abb8ce640 1 -- 172.21.15.19:0/2166093080 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3ab4159470 msgr2=0x7f3ab4157c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:57.737 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.735+0000 7f3abb8ce640 1 --2- 172.21.15.19:0/2166093080 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3ab4159470 0x7f3ab4157c70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3aac00cbf0 tx=0x7f3aac00a430 comp rx=0 tx=0).stop 2024-08-31T12:25:57.737 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.735+0000 7f3abb8ce640 1 -- 172.21.15.19:0/2166093080 shutdown_connections 2024-08-31T12:25:57.737 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.735+0000 7f3abb8ce640 1 --2- 172.21.15.19:0/2166093080 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3a98077a50 0x7f3a98079f60 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:57.737 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.735+0000 7f3abb8ce640 1 --2- 172.21.15.19:0/2166093080 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3ab4159470 0x7f3ab4157c70 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:57.737 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.735+0000 7f3abb8ce640 1 --2- 172.21.15.19:0/2166093080 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3ab4153230 0x7f3ab40cdc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:57.737 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.735+0000 7f3abb8ce640 1 --2- 172.21.15.19:0/2166093080 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ab4071b40 0x7f3ab4149220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:57.737 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.735+0000 7f3abb8ce640 1 -- 172.21.15.19:0/2166093080 >> 172.21.15.19:0/2166093080 conn(0x7f3ab4072c80 msgr2=0x7f3ab40c3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:57.737 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.735+0000 7f3abb8ce640 1 -- 172.21.15.19:0/2166093080 shutdown_connections 2024-08-31T12:25:57.737 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.735+0000 7f3abb8ce640 1 -- 172.21.15.19:0/2166093080 wait complete. 2024-08-31T12:25:57.739 INFO:tasks.workunit.client.0.smithi019.stdout:Syncing()...0 sec 2024-08-31T12:25:57.740 INFO:tasks.workunit.client.0.smithi019.stdout:Starting Actual Benchmark At: Sat Aug 31 12:25:57 2024 2024-08-31T12:25:57.740 INFO:tasks.workunit.client.0.smithi019.stdout: 2024-08-31T12:25:57.748 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-08-31T12:25:57.748 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-08-31T12:25:57.749 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-08-31T12:25:57.749 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-31T12:25:57.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.897+0000 7f70a9332640 1 -- 172.21.15.19:0/2996268259 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f70a4071dd0 msgr2=0x7f70a4152dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:57.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.897+0000 7f70a9332640 1 --2- 172.21.15.19:0/2996268259 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f70a4071dd0 0x7f70a4152dd0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7098005fd0 tx=0x7f709802a5d0 comp rx=0 tx=0).stop 2024-08-31T12:25:57.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.897+0000 7f70a9332640 1 -- 172.21.15.19:0/2996268259 shutdown_connections 2024-08-31T12:25:57.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.897+0000 7f70a9332640 1 --2- 172.21.15.19:0/2996268259 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f70a4071dd0 0x7f70a4152dd0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:57.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.897+0000 7f70a9332640 1 --2- 172.21.15.19:0/2996268259 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f70a40c3ff0 0x7f70a40c8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:57.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.897+0000 7f70a9332640 1 -- 172.21.15.19:0/2996268259 >> 172.21.15.19:0/2996268259 conn(0x7f70a40694c0 msgr2=0x7f70a40728b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:57.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.897+0000 7f70a9332640 1 -- 172.21.15.19:0/2996268259 shutdown_connections 2024-08-31T12:25:57.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.898+0000 7f70a9332640 1 -- 172.21.15.19:0/2996268259 wait complete. 2024-08-31T12:25:57.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.900+0000 7f70a9332640 1 Processor -- start 2024-08-31T12:25:57.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.900+0000 7f70a9332640 1 -- start start 2024-08-31T12:25:57.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.901+0000 7f70a9332640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f70a4071dd0 0x7f70a41554e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:57.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.901+0000 7f70a9332640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70a40c3ff0 0x7f70a4149260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:57.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.901+0000 7f70a9332640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f70a40ce550 0x7f70a415b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:57.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.901+0000 7f70a9332640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f70a4153310 con 0x7f70a40ce550 2024-08-31T12:25:57.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.901+0000 7f70a9332640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f70a4159950 con 0x7f70a40c3ff0 2024-08-31T12:25:57.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.901+0000 7f70a9332640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f70a415cc80 con 0x7f70a4071dd0 2024-08-31T12:25:57.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.901+0000 7f70a37fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f70a40ce550 0x7f70a415b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:57.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.901+0000 7f70a27fc640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70a40c3ff0 0x7f70a4149260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:57.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.901+0000 7f70a2ffd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f70a4071dd0 0x7f70a41554e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:57.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.901+0000 7f70a37fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f70a40ce550 0x7f70a415b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:34292/0 (socket says 172.21.15.19:34292) 2024-08-31T12:25:57.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.901+0000 7f70a27fc640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70a40c3ff0 0x7f70a4149260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:42128/0 (socket says 172.21.15.19:42128) 2024-08-31T12:25:57.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.901+0000 7f70a37fe640 1 -- 172.21.15.19:0/1387213792 learned_addr learned my addr 172.21.15.19:0/1387213792 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:25:57.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.902+0000 7f70a37fe640 1 -- 172.21.15.19:0/1387213792 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f70a4071dd0 msgr2=0x7f70a41554e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:57.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.902+0000 7f70a37fe640 1 --2- 172.21.15.19:0/1387213792 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f70a4071dd0 0x7f70a41554e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:57.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.902+0000 7f70a37fe640 1 -- 172.21.15.19:0/1387213792 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70a40c3ff0 msgr2=0x7f70a4149260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:57.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.902+0000 7f70a37fe640 1 --2- 172.21.15.19:0/1387213792 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70a40c3ff0 0x7f70a4149260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:57.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.902+0000 7f70a37fe640 1 -- 172.21.15.19:0/1387213792 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70a41497a0 con 0x7f70a40ce550 2024-08-31T12:25:57.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.902+0000 7f70a37fe640 1 --2- 172.21.15.19:0/1387213792 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f70a40ce550 0x7f70a415b6f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f709c00c700 tx=0x7f709c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:57.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.902+0000 7f708b7fe640 1 -- 172.21.15.19:0/1387213792 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709c003b70 con 0x7f70a40ce550 2024-08-31T12:25:57.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.902+0000 7f70a9332640 1 -- 172.21.15.19:0/1387213792 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70a40b38d0 con 0x7f70a40ce550 2024-08-31T12:25:57.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.902+0000 7f708b7fe640 1 -- 172.21.15.19:0/1387213792 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f709c003d10 con 0x7f70a40ce550 2024-08-31T12:25:57.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.902+0000 7f708b7fe640 1 -- 172.21.15.19:0/1387213792 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709c00aab0 con 0x7f70a40ce550 2024-08-31T12:25:57.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.902+0000 7f70a9332640 1 -- 172.21.15.19:0/1387213792 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70a40b2b10 con 0x7f70a40ce550 2024-08-31T12:25:57.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.903+0000 7f70a9332640 1 --2- 172.21.15.19:0/1387213792 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f70a4155f30 0x7f70a415bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:57.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.903+0000 7f70a9332640 1 -- 172.21.15.19:0/1387213792 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f70a4156450 con 0x7f70a4155f30 2024-08-31T12:25:57.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.904+0000 7f70a2ffd640 1 --2- 172.21.15.19:0/1387213792 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f70a4155f30 0x7f70a415bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:57.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.904+0000 7f70a2ffd640 1 -- 172.21.15.19:0/1387213792 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f70a406fc00 con 0x7f70a4155f30 2024-08-31T12:25:57.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.904+0000 7f708b7fe640 1 -- 172.21.15.19:0/1387213792 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f709c00ac50 con 0x7f70a40ce550 2024-08-31T12:25:57.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.904+0000 7f70a2ffd640 1 --2- 172.21.15.19:0/1387213792 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f70a4155f30 0x7f70a415bc30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f70940109a0 tx=0x7f7094010e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:57.907 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.905+0000 7f708b7fe640 1 --2- 172.21.15.19:0/1387213792 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7090077a80 0x7f7090079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:57.907 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.905+0000 7f708b7fe640 1 -- 172.21.15.19:0/1387213792 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f709c0992e0 con 0x7f70a40ce550 2024-08-31T12:25:57.907 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.905+0000 7f708b7fe640 1 -- 172.21.15.19:0/1387213792 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709400edb0 con 0x7f70a4155f30 2024-08-31T12:25:57.907 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.905+0000 7f708b7fe640 1 -- 172.21.15.19:0/1387213792 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f709401c480 con 0x7f70a4155f30 2024-08-31T12:25:57.907 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.905+0000 7f708b7fe640 1 -- 172.21.15.19:0/1387213792 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f70a4155f30 msgr2=0x7f70a415bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:57.907 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.905+0000 7f708b7fe640 1 --2- 172.21.15.19:0/1387213792 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f70a4155f30 0x7f70a415bc30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f70940109a0 tx=0x7f7094010e60 comp rx=0 tx=0).stop 2024-08-31T12:25:57.907 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.905+0000 7f708b7fe640 1 -- 172.21.15.19:0/1387213792 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f70a4155f30 msgr2=0x7f70a415bc30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:25:57.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.905+0000 7f708b7fe640 1 --2- 172.21.15.19:0/1387213792 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f70a4155f30 0x7f70a415bc30 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:57.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.905+0000 7f70a27fc640 1 --2- 172.21.15.19:0/1387213792 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7090077a80 0x7f7090079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:57.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.906+0000 7f70a27fc640 1 --2- 172.21.15.19:0/1387213792 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7090077a80 0x7f7090079f90 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7098033000 tx=0x7f7098004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:57.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.912+0000 7f70a9332640 1 --2- 172.21.15.19:0/1387213792 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f70a40b2d50 0x7f70a40c9e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:57.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.912+0000 7f70a9332640 1 -- 172.21.15.19:0/1387213792 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f70a4152dd0 con 0x7f70a40b2d50 2024-08-31T12:25:57.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.912+0000 7f70a2ffd640 1 --2- 172.21.15.19:0/1387213792 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f70a40b2d50 0x7f70a40c9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:57.914 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.913+0000 7f70a2ffd640 1 -- 172.21.15.19:0/1387213792 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f70a40c8a60 con 0x7f70a40b2d50 2024-08-31T12:25:57.914 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.913+0000 7f70a2ffd640 1 --2- 172.21.15.19:0/1387213792 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f70a40b2d50 0x7f70a40c9e00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f709401b000 tx=0x7f70940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:57.914 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.913+0000 7f708b7fe640 1 -- 172.21.15.19:0/1387213792 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7094027070 con 0x7f70a40b2d50 2024-08-31T12:25:57.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.913+0000 7f708b7fe640 1 -- 172.21.15.19:0/1387213792 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5044 (secure 0 0 0) 0x7f7094011440 con 0x7f70a40b2d50 2024-08-31T12:25:57.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.914+0000 7f708b7fe640 1 -- 172.21.15.19:0/1387213792 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f70a40b2d50 msgr2=0x7f70a40c9e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:57.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.914+0000 7f708b7fe640 1 --2- 172.21.15.19:0/1387213792 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f70a40b2d50 0x7f70a40c9e00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f709401b000 tx=0x7f70940046c0 comp rx=0 tx=0).stop 2024-08-31T12:25:57.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.914+0000 7f708b7fe640 1 -- 172.21.15.19:0/1387213792 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f70a40b2d50 msgr2=0x7f70a40c9e00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:25:57.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.914+0000 7f708b7fe640 1 --2- 172.21.15.19:0/1387213792 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f70a40b2d50 0x7f70a40c9e00 secure :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f709401b000 tx=0x7f70940046c0 comp rx=0 tx=0).stop 2024-08-31T12:25:57.915 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:25:57.915 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:25:57.915 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:25:57.916 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "leader", 2024-08-31T12:25:57.916 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 80, 2024-08-31T12:25:57.916 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 92468, 2024-08-31T12:25:57.916 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:25:57.916 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:25:57.916 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:25:57.916 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:25:57.916 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:57.916 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 13, 2024-08-31T12:25:57.916 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:25:57.916 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:25:57.916 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:25:57.916 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:25:57.916 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:25:57.917 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:25:57.917 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:25:57.917 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:25:57.917 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:25:57.917 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:25:57.917 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:25:57.917 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:25:57.917 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:25:57.917 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:25:57.917 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:57.917 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:25:57.917 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:25:57.917 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:25:57.917 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:25:57.917 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:25:57.918 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:25:57.918 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:25:57.918 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:25:57.918 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:25:57.918 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:25:57.918 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:25:57.918 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:25:57.918 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:25:57.918 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:57.918 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:57.918 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:25:57.918 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:25:57.918 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:25:57.918 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:25:57.918 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:25:57.919 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:25:57.919 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:25:57.919 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:25:57.919 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:25:57.919 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:25:57.919 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:25:57.919 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:25:57.919 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:25:57.919 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:25:57.919 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:25:57.919 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:25:57.919 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:25:57.919 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:25:57.920 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:25:57.920 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:25:57.920 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:25:57.920 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:25:57.920 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:25:57.920 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:25:57.920 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:25:57.920 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:25:57.920 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:25:57.920 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:25:57.920 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:57.920 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:25:57.920 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:57.920 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:25:57.921 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:57.921 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:25:57.921 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:25:57.921 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:25:57.921 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:25:57.921 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:57.921 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:25:57.921 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:25:57.921 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:57.921 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:57.921 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:57.921 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:25:57.921 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:25:57.921 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:57.922 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:57.922 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:57.922 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:57.922 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:25:57.922 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:25:57.922 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:25:57.922 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:25:57.922 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:25:57.922 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:57.922 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:57.922 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:25:57.922 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:25:57.922 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:25:57.922 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:25:57.923 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:57.923 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:25:57.923 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:25:57.923 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:57.923 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:57.923 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:57.923 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:25:57.923 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:25:57.923 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:57.923 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:57.923 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:57.923 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:57.923 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:25:57.924 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:25:57.924 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:25:57.924 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:25:57.924 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:25:57.924 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:57.924 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:57.924 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:25:57.924 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:25:57.924 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:25:57.924 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:25:57.924 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:57.924 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:25:57.925 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:25:57.925 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:57.925 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:57.925 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:57.925 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:25:57.925 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:25:57.925 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:57.925 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:57.925 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:57.925 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:57.925 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:25:57.925 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:25:57.925 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:25:57.925 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:25:57.925 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:25:57.926 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:57.926 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:57.926 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:57.926 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:25:57.926 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:25:57.926 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:57.926 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:57.926 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:57.926 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:25:57.926 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:57.926 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:57.926 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:25:57.926 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:57.927 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:57.927 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:57.927 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 3 2024-08-31T12:25:57.927 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:57.927 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:57.927 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:25:57.927 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:57.927 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:57.927 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:57.927 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 3 2024-08-31T12:25:57.927 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:57.927 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:57.927 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:25:57.927 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:57.928 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:57.928 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:57.928 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 5 2024-08-31T12:25:57.928 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:57.928 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:57.928 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:25:57.928 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:57.928 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:57.928 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:57.928 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:25:57.928 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:57.928 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:57.928 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:57.929 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:25:57.929 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:25:57.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.914+0000 7f70a9332640 1 -- 172.21.15.19:0/1387213792 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7090077a80 msgr2=0x7f7090079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:57.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.914+0000 7f70a9332640 1 --2- 172.21.15.19:0/1387213792 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7090077a80 0x7f7090079f90 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7098033000 tx=0x7f7098004730 comp rx=0 tx=0).stop 2024-08-31T12:25:57.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.914+0000 7f70a37fe640 1 -- 172.21.15.19:0/1387213792 reap_dead start 2024-08-31T12:25:57.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.914+0000 7f70a9332640 1 -- 172.21.15.19:0/1387213792 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f70a40ce550 msgr2=0x7f70a415b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:57.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.914+0000 7f70a9332640 1 --2- 172.21.15.19:0/1387213792 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f70a40ce550 0x7f70a415b6f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f709c00c700 tx=0x7f709c00cbc0 comp rx=0 tx=0).stop 2024-08-31T12:25:57.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.914+0000 7f70a9332640 1 -- 172.21.15.19:0/1387213792 shutdown_connections 2024-08-31T12:25:57.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.914+0000 7f70a9332640 1 --2- 172.21.15.19:0/1387213792 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f70a40ce550 0x7f70a415b6f0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:57.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.914+0000 7f70a9332640 1 -- 172.21.15.19:0/1387213792 >> 172.21.15.19:0/1387213792 conn(0x7f70a40694c0 msgr2=0x7f70a40cc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:57.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.914+0000 7f70a9332640 1 -- 172.21.15.19:0/1387213792 shutdown_connections 2024-08-31T12:25:57.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:57.914+0000 7f70a9332640 1 -- 172.21.15.19:0/1387213792 wait complete. 2024-08-31T12:25:57.931 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:25:57.931 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-08-31T12:25:57.931 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-31T12:25:58.099 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.097+0000 7f6c20ed5640 1 -- 172.21.15.19:0/3328120199 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6c1c06d4a0 msgr2=0x7f6c1c0c8420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:58.099 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.097+0000 7f6c20ed5640 1 --2- 172.21.15.19:0/3328120199 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6c1c06d4a0 0x7f6c1c0c8420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6c04005fd0 tx=0x7f6c0402a5d0 comp rx=0 tx=0).stop 2024-08-31T12:25:58.099 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.097+0000 7f6c20ed5640 1 -- 172.21.15.19:0/3328120199 shutdown_connections 2024-08-31T12:25:58.099 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.097+0000 7f6c20ed5640 1 --2- 172.21.15.19:0/3328120199 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6c1c06d4a0 0x7f6c1c0c8420 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:58.099 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.097+0000 7f6c20ed5640 1 --2- 172.21.15.19:0/3328120199 >> v2:172.21.15.19:3300/0 conn(0x7f6c1c0ca820 0x7f6c1c14f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:58.099 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.097+0000 7f6c20ed5640 1 -- 172.21.15.19:0/3328120199 >> 172.21.15.19:0/3328120199 conn(0x7f6c1c0cda10 msgr2=0x7f6c1c0c9e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:58.099 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.097+0000 7f6c20ed5640 1 -- 172.21.15.19:0/3328120199 shutdown_connections 2024-08-31T12:25:58.099 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.097+0000 7f6c20ed5640 1 -- 172.21.15.19:0/3328120199 wait complete. 2024-08-31T12:25:58.101 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.100+0000 7f6c20ed5640 1 Processor -- start 2024-08-31T12:25:58.101 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.101+0000 7f6c20ed5640 1 -- start start 2024-08-31T12:25:58.104 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.102+0000 7f6c20ed5640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c1c06d4a0 0x7f6c1c154c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:58.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.102+0000 7f6c20ed5640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6c1c0c8960 0x7f6c1c0c71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:58.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.103+0000 7f6c20ed5640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6c1c0ca820 0x7f6c1c1f47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:58.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.103+0000 7f6c20ed5640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f6c1c0c77e0 con 0x7f6c1c0c8960 2024-08-31T12:25:58.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.103+0000 7f6c20ed5640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6c1c0c7f00 con 0x7f6c1c06d4a0 2024-08-31T12:25:58.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.103+0000 7f6c20ed5640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f6c1c06e9c0 con 0x7f6c1c0ca820 2024-08-31T12:25:58.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.103+0000 7f6c1a59c640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c1c06d4a0 0x7f6c1c154c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:58.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.103+0000 7f6c1a59c640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c1c06d4a0 0x7f6c1c154c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:42136/0 (socket says 172.21.15.19:42136) 2024-08-31T12:25:58.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.103+0000 7f6c1a59c640 1 -- 172.21.15.19:0/2629272743 learned_addr learned my addr 172.21.15.19:0/2629272743 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:25:58.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.103+0000 7f6c1ad9d640 1 --2- 172.21.15.19:0/2629272743 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6c1c0ca820 0x7f6c1c1f47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:58.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.103+0000 7f6c11d9b640 1 --2- 172.21.15.19:0/2629272743 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6c1c0c8960 0x7f6c1c0c71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:58.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.103+0000 7f6c1a59c640 1 -- 172.21.15.19:0/2629272743 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6c1c0ca820 msgr2=0x7f6c1c1f47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:58.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.103+0000 7f6c1a59c640 1 --2- 172.21.15.19:0/2629272743 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6c1c0ca820 0x7f6c1c1f47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:58.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.103+0000 7f6c1a59c640 1 -- 172.21.15.19:0/2629272743 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6c1c0c8960 msgr2=0x7f6c1c0c71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:58.106 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.103+0000 7f6c1a59c640 1 --2- 172.21.15.19:0/2629272743 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6c1c0c8960 0x7f6c1c0c71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:58.106 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.103+0000 7f6c1a59c640 1 -- 172.21.15.19:0/2629272743 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c1c0bf360 con 0x7f6c1c06d4a0 2024-08-31T12:25:58.106 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.103+0000 7f6c11d9b640 1 --2- 172.21.15.19:0/2629272743 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6c1c0c8960 0x7f6c1c0c71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:25:58.106 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.104+0000 7f6c1a59c640 1 --2- 172.21.15.19:0/2629272743 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c1c06d4a0 0x7f6c1c154c00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c00d770 tx=0x7f6c0c00dc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:58.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.104+0000 7f6c137fe640 1 -- 172.21.15.19:0/2629272743 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c0c017070 con 0x7f6c1c06d4a0 2024-08-31T12:25:58.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.104+0000 7f6c137fe640 1 -- 172.21.15.19:0/2629272743 <== mon.1 v2:172.21.15.43:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6c0c009d20 con 0x7f6c1c06d4a0 2024-08-31T12:25:58.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.104+0000 7f6c137fe640 1 -- 172.21.15.19:0/2629272743 <== mon.1 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c0c0106c0 con 0x7f6c1c06d4a0 2024-08-31T12:25:58.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.104+0000 7f6c20ed5640 1 -- 172.21.15.19:0/2629272743 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c1c1f14d0 con 0x7f6c1c06d4a0 2024-08-31T12:25:58.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.104+0000 7f6c20ed5640 1 -- 172.21.15.19:0/2629272743 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c1c1f1a90 con 0x7f6c1c06d4a0 2024-08-31T12:25:58.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.105+0000 7f6c20ed5640 1 --2- 172.21.15.19:0/2629272743 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c1c0c42e0 0x7f6c1c14e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:58.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.105+0000 7f6c20ed5640 1 -- 172.21.15.19:0/2629272743 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6c1c0c9ef0 con 0x7f6c1c0c42e0 2024-08-31T12:25:58.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.106+0000 7f6c11d9b640 1 --2- 172.21.15.19:0/2629272743 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c1c0c42e0 0x7f6c1c14e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:58.108 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.106+0000 7f6c11d9b640 1 -- 172.21.15.19:0/2629272743 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6c1c0c8e00 con 0x7f6c1c0c42e0 2024-08-31T12:25:58.108 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.107+0000 7f6c137fe640 1 -- 172.21.15.19:0/2629272743 <== mon.1 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f6c0c01a070 con 0x7f6c1c06d4a0 2024-08-31T12:25:58.108 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.107+0000 7f6c11d9b640 1 --2- 172.21.15.19:0/2629272743 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c1c0c42e0 0x7f6c1c14e100 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c159f60 tx=0x7f6c04003820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:58.108 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.108+0000 7f6c137fe640 1 --2- 172.21.15.19:0/2629272743 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6bfc077a50 0x7f6bfc079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:58.109 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.108+0000 7f6c1ad9d640 1 --2- 172.21.15.19:0/2629272743 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6bfc077a50 0x7f6bfc079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:58.109 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.108+0000 7f6c137fe640 1 -- 172.21.15.19:0/2629272743 <== mon.1 v2:172.21.15.43:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f6c0c0998d0 con 0x7f6c1c06d4a0 2024-08-31T12:25:58.110 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.108+0000 7f6c137fe640 1 -- 172.21.15.19:0/2629272743 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c04040070 con 0x7f6c1c0c42e0 2024-08-31T12:25:58.110 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.108+0000 7f6c137fe640 1 -- 172.21.15.19:0/2629272743 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6c04003e40 con 0x7f6c1c0c42e0 2024-08-31T12:25:58.110 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.108+0000 7f6c137fe640 1 -- 172.21.15.19:0/2629272743 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c1c0c42e0 msgr2=0x7f6c1c14e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:58.110 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.108+0000 7f6c137fe640 1 --2- 172.21.15.19:0/2629272743 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c1c0c42e0 0x7f6c1c14e100 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c159f60 tx=0x7f6c04003820 comp rx=0 tx=0).stop 2024-08-31T12:25:58.110 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.108+0000 7f6c137fe640 1 -- 172.21.15.19:0/2629272743 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c1c0c42e0 msgr2=0x7f6c1c14e100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:25:58.110 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.108+0000 7f6c137fe640 1 --2- 172.21.15.19:0/2629272743 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c1c0c42e0 0x7f6c1c14e100 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:58.110 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.108+0000 7f6c1ad9d640 1 --2- 172.21.15.19:0/2629272743 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6bfc077a50 0x7f6bfc079f60 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c0c2560 tx=0x7f6c14009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:58.120 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.119+0000 7f6c20ed5640 1 --2- 172.21.15.19:0/2629272743 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c1c1f4d20 0x7f6c1c1520f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:58.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.119+0000 7f6c20ed5640 1 -- 172.21.15.19:0/2629272743 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6c1c0ca1b0 con 0x7f6c1c1f4d20 2024-08-31T12:25:58.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.120+0000 7f6c11d9b640 1 --2- 172.21.15.19:0/2629272743 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c1c1f4d20 0x7f6c1c1520f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:58.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.120+0000 7f6c11d9b640 1 -- 172.21.15.19:0/2629272743 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6c1c0cce00 con 0x7f6c1c1f4d20 2024-08-31T12:25:58.122 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.121+0000 7f6c11d9b640 1 --2- 172.21.15.19:0/2629272743 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c1c1f4d20 0x7f6c1c1520f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6c04009850 tx=0x7f6c04009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:58.122 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.121+0000 7f6c137fe640 1 -- 172.21.15.19:0/2629272743 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c04035040 con 0x7f6c1c1f4d20 2024-08-31T12:25:58.123 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.122+0000 7f6c137fe640 1 -- 172.21.15.19:0/2629272743 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4541 (secure 0 0 0) 0x7f6c04049c70 con 0x7f6c1c1f4d20 2024-08-31T12:25:58.123 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.122+0000 7f6c137fe640 1 -- 172.21.15.19:0/2629272743 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c1c1f4d20 msgr2=0x7f6c1c1520f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:58.123 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.122+0000 7f6c137fe640 1 --2- 172.21.15.19:0/2629272743 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c1c1f4d20 0x7f6c1c1520f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6c04009850 tx=0x7f6c04009260 comp rx=0 tx=0).stop 2024-08-31T12:25:58.123 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.122+0000 7f6c137fe640 1 -- 172.21.15.19:0/2629272743 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c1c1f4d20 msgr2=0x7f6c1c1520f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:25:58.123 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.122+0000 7f6c137fe640 1 --2- 172.21.15.19:0/2629272743 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c1c1f4d20 0x7f6c1c1520f0 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6c04009850 tx=0x7f6c04009260 comp rx=0 tx=0).stop 2024-08-31T12:25:58.123 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:25:58.123 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:25:58.124 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:25:58.124 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:25:58.124 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 80, 2024-08-31T12:25:58.124 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 19408, 2024-08-31T12:25:58.124 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:25:58.124 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:25:58.124 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:25:58.124 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:25:58.124 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:58.124 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 13, 2024-08-31T12:25:58.124 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:25:58.124 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:25:58.124 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:25:58.125 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:25:58.125 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:25:58.125 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:25:58.125 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:25:58.125 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:25:58.125 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:25:58.125 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:25:58.125 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:25:58.125 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:25:58.125 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:25:58.125 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:25:58.125 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:58.125 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:25:58.126 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:25:58.126 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:25:58.126 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:25:58.126 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:25:58.126 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:25:58.126 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:25:58.126 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:25:58.126 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:25:58.126 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:25:58.126 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:25:58.126 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:25:58.126 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:25:58.126 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:58.126 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:58.127 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:25:58.127 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:25:58.127 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:25:58.127 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:25:58.127 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:25:58.127 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:25:58.127 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:25:58.127 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:25:58.127 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:25:58.127 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:25:58.127 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:25:58.127 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:25:58.127 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:25:58.128 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:25:58.128 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:25:58.128 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:25:58.128 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:25:58.128 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:25:58.128 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:25:58.128 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:25:58.128 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:25:58.128 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:25:58.128 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:25:58.128 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:25:58.128 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:25:58.128 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:25:58.128 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:25:58.129 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:25:58.129 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:58.129 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:25:58.129 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:58.129 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:25:58.129 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:58.129 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:25:58.129 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:25:58.129 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:25:58.129 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:25:58.129 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:58.129 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:25:58.129 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:25:58.130 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:58.130 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:58.130 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:58.130 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:25:58.130 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:25:58.130 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:58.130 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:58.130 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:58.130 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:58.130 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:25:58.130 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:25:58.130 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:25:58.130 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:25:58.131 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:25:58.131 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:58.131 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:58.131 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:25:58.131 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:25:58.131 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:25:58.131 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:25:58.131 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:58.131 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:25:58.131 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:25:58.131 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:58.131 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:58.131 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:58.131 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:25:58.132 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:25:58.132 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:58.132 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:58.132 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:58.132 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:58.132 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:25:58.132 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:25:58.132 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:25:58.132 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:25:58.132 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:25:58.132 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:58.132 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:58.132 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:25:58.133 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:25:58.133 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:25:58.133 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:25:58.133 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:58.133 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:25:58.133 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:25:58.133 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:58.133 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:58.133 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:58.133 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:25:58.133 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:25:58.133 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:58.133 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:58.134 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:58.134 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:58.134 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:25:58.134 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:25:58.134 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:25:58.134 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:25:58.134 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:25:58.134 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:58.134 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:58.134 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:58.134 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:25:58.134 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:25:58.134 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:58.134 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:58.135 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:58.135 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:25:58.135 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:58.135 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:58.135 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:25:58.135 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:58.135 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:58.135 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:58.135 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:25:58.135 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:58.135 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:58.135 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:58.135 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:25:58.135 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:25:58.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.123+0000 7f6c20ed5640 1 -- 172.21.15.19:0/2629272743 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6bfc077a50 msgr2=0x7f6bfc079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:58.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.123+0000 7f6c20ed5640 1 --2- 172.21.15.19:0/2629272743 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6bfc077a50 0x7f6bfc079f60 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c0c2560 tx=0x7f6c14009040 comp rx=0 tx=0).stop 2024-08-31T12:25:58.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.123+0000 7f6c20ed5640 1 -- 172.21.15.19:0/2629272743 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c1c06d4a0 msgr2=0x7f6c1c154c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:58.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.123+0000 7f6c20ed5640 1 --2- 172.21.15.19:0/2629272743 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c1c06d4a0 0x7f6c1c154c00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c00d770 tx=0x7f6c0c00dc30 comp rx=0 tx=0).stop 2024-08-31T12:25:58.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.123+0000 7f6c1ad9d640 1 -- 172.21.15.19:0/2629272743 reap_dead start 2024-08-31T12:25:58.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.123+0000 7f6c1ad9d640 1 -- 172.21.15.19:0/2629272743 reap_dead start 2024-08-31T12:25:58.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.123+0000 7f6c20ed5640 1 -- 172.21.15.19:0/2629272743 shutdown_connections 2024-08-31T12:25:58.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.123+0000 7f6c20ed5640 1 -- 172.21.15.19:0/2629272743 >> 172.21.15.19:0/2629272743 conn(0x7f6c1c0cda10 msgr2=0x7f6c1c156ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:58.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.123+0000 7f6c20ed5640 1 -- 172.21.15.19:0/2629272743 shutdown_connections 2024-08-31T12:25:58.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.123+0000 7f6c20ed5640 1 -- 172.21.15.19:0/2629272743 wait complete. 2024-08-31T12:25:58.140 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:25:58.140 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-08-31T12:25:58.140 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-31T12:25:58.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.307+0000 7fcf6e244640 1 -- 172.21.15.19:0/2920849307 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcf6815ed90 msgr2=0x7fcf68165a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:58.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.308+0000 7fcf6e244640 1 --2- 172.21.15.19:0/2920849307 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcf6815ed90 0x7fcf68165a10 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcf60005fd0 tx=0x7fcf6002a820 comp rx=0 tx=0).stop 2024-08-31T12:25:58.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.308+0000 7fcf6e244640 1 -- 172.21.15.19:0/2920849307 shutdown_connections 2024-08-31T12:25:58.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.308+0000 7fcf6e244640 1 --2- 172.21.15.19:0/2920849307 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcf6815ed90 0x7fcf68165a10 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:58.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.308+0000 7fcf6e244640 1 --2- 172.21.15.19:0/2920849307 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcf6806df40 0x7fcf680c5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:58.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.308+0000 7fcf6e244640 1 -- 172.21.15.19:0/2920849307 >> 172.21.15.19:0/2920849307 conn(0x7fcf680cef50 msgr2=0x7fcf680cb340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:58.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.308+0000 7fcf6e244640 1 -- 172.21.15.19:0/2920849307 shutdown_connections 2024-08-31T12:25:58.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.308+0000 7fcf6e244640 1 -- 172.21.15.19:0/2920849307 wait complete. 2024-08-31T12:25:58.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.311+0000 7fcf6e244640 1 Processor -- start 2024-08-31T12:25:58.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.312+0000 7fcf6e244640 1 -- start start 2024-08-31T12:25:58.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.314+0000 7fcf6e244640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcf6806df40 0x7fcf681f1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:58.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.314+0000 7fcf6e244640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcf680cade0 0x7fcf6815a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:58.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.314+0000 7fcf6e244640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcf6815ed90 0x7fcf68152530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:58.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.314+0000 7fcf6e244640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fcf680c9240 con 0x7fcf680cade0 2024-08-31T12:25:58.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.314+0000 7fcf6e244640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fcf680c39a0 con 0x7fcf6815ed90 2024-08-31T12:25:58.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.314+0000 7fcf6e244640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fcf681f17b0 con 0x7fcf6806df40 2024-08-31T12:25:58.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.314+0000 7fcf66ffd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcf680cade0 0x7fcf6815a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:58.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.314+0000 7fcf67fff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcf6815ed90 0x7fcf68152530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:58.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.314+0000 7fcf66ffd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcf680cade0 0x7fcf6815a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:54758/0 (socket says 172.21.15.19:54758) 2024-08-31T12:25:58.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.314+0000 7fcf66ffd640 1 -- 172.21.15.19:0/3075322333 learned_addr learned my addr 172.21.15.19:0/3075322333 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:25:58.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.314+0000 7fcf677fe640 1 --2- 172.21.15.19:0/3075322333 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcf6806df40 0x7fcf681f1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:58.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.314+0000 7fcf67fff640 1 -- 172.21.15.19:0/3075322333 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcf6806df40 msgr2=0x7fcf681f1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:58.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.314+0000 7fcf67fff640 1 --2- 172.21.15.19:0/3075322333 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcf6806df40 0x7fcf681f1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:58.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.314+0000 7fcf67fff640 1 -- 172.21.15.19:0/3075322333 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcf680cade0 msgr2=0x7fcf6815a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:58.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.314+0000 7fcf67fff640 1 --2- 172.21.15.19:0/3075322333 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcf680cade0 0x7fcf6815a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:58.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.314+0000 7fcf67fff640 1 -- 172.21.15.19:0/3075322333 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf680b0870 con 0x7fcf6815ed90 2024-08-31T12:25:58.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.315+0000 7fcf677fe640 1 --2- 172.21.15.19:0/3075322333 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcf6806df40 0x7fcf681f1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:25:58.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.315+0000 7fcf67fff640 1 --2- 172.21.15.19:0/3075322333 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcf6815ed90 0x7fcf68152530 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcf60009000 tx=0x7fcf60032be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:58.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.315+0000 7fcf4ffff640 1 -- 172.21.15.19:0/3075322333 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf60037040 con 0x7fcf6815ed90 2024-08-31T12:25:58.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.315+0000 7fcf4ffff640 1 -- 172.21.15.19:0/3075322333 <== mon.1 v2:172.21.15.43:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcf60042070 con 0x7fcf6815ed90 2024-08-31T12:25:58.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.315+0000 7fcf6e244640 1 -- 172.21.15.19:0/3075322333 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf680b0b50 con 0x7fcf6815ed90 2024-08-31T12:25:58.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.315+0000 7fcf6e244640 1 -- 172.21.15.19:0/3075322333 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf680ae8c0 con 0x7fcf6815ed90 2024-08-31T12:25:58.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.315+0000 7fcf4ffff640 1 -- 172.21.15.19:0/3075322333 <== mon.1 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf60004040 con 0x7fcf6815ed90 2024-08-31T12:25:58.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.317+0000 7fcf6e244640 1 --2- 172.21.15.19:0/3075322333 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcf68158f00 0x7fcf680af570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:58.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.317+0000 7fcf6e244640 1 -- 172.21.15.19:0/3075322333 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fcf680c93c0 con 0x7fcf68158f00 2024-08-31T12:25:58.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.317+0000 7fcf677fe640 1 --2- 172.21.15.19:0/3075322333 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcf68158f00 0x7fcf680af570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:58.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.318+0000 7fcf4ffff640 1 -- 172.21.15.19:0/3075322333 <== mon.1 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fcf60004200 con 0x7fcf6815ed90 2024-08-31T12:25:58.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.318+0000 7fcf677fe640 1 -- 172.21.15.19:0/3075322333 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcf680b0600 con 0x7fcf68158f00 2024-08-31T12:25:58.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.318+0000 7fcf4ffff640 1 --2- 172.21.15.19:0/3075322333 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fcf54077a40 0x7fcf54079f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:58.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.319+0000 7fcf4ffff640 1 -- 172.21.15.19:0/3075322333 <== mon.1 v2:172.21.15.43:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fcf600ba2b0 con 0x7fcf6815ed90 2024-08-31T12:25:58.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.319+0000 7fcf677fe640 1 --2- 172.21.15.19:0/3075322333 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcf68158f00 0x7fcf680af570 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcf5800fae0 tx=0x7fcf58001e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:58.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.319+0000 7fcf66ffd640 1 --2- 172.21.15.19:0/3075322333 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fcf54077a40 0x7fcf54079f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:58.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.319+0000 7fcf4ffff640 1 -- 172.21.15.19:0/3075322333 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf58018030 con 0x7fcf68158f00 2024-08-31T12:25:58.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.319+0000 7fcf4ffff640 1 -- 172.21.15.19:0/3075322333 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcf58005020 con 0x7fcf68158f00 2024-08-31T12:25:58.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.319+0000 7fcf4ffff640 1 -- 172.21.15.19:0/3075322333 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcf68158f00 msgr2=0x7fcf680af570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:58.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.319+0000 7fcf4ffff640 1 --2- 172.21.15.19:0/3075322333 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcf68158f00 0x7fcf680af570 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcf5800fae0 tx=0x7fcf58001e60 comp rx=0 tx=0).stop 2024-08-31T12:25:58.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.319+0000 7fcf4ffff640 1 -- 172.21.15.19:0/3075322333 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcf68158f00 msgr2=0x7fcf680af570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:25:58.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.319+0000 7fcf4ffff640 1 --2- 172.21.15.19:0/3075322333 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcf68158f00 0x7fcf680af570 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:58.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.319+0000 7fcf66ffd640 1 --2- 172.21.15.19:0/3075322333 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fcf54077a40 0x7fcf54079f50 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fcf680cbd80 tx=0x7fcf5c005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:58.331 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.330+0000 7fcf6e244640 1 --2- 172.21.15.19:0/3075322333 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcf681e60b0 0x7fcf680afab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:58.331 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.330+0000 7fcf6e244640 1 -- 172.21.15.19:0/3075322333 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fcf681e5710 con 0x7fcf681e60b0 2024-08-31T12:25:58.331 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.330+0000 7fcf677fe640 1 --2- 172.21.15.19:0/3075322333 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcf681e60b0 0x7fcf680afab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:58.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.330+0000 7fcf677fe640 1 -- 172.21.15.19:0/3075322333 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcf680b0d90 con 0x7fcf681e60b0 2024-08-31T12:25:58.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.331+0000 7fcf677fe640 1 --2- 172.21.15.19:0/3075322333 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcf681e60b0 0x7fcf680afab0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fcf58018000 tx=0x7fcf5800dfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:58.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.331+0000 7fcf4ffff640 1 -- 172.21.15.19:0/3075322333 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf5801ed50 con 0x7fcf681e60b0 2024-08-31T12:25:58.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.331+0000 7fcf4ffff640 1 -- 172.21.15.19:0/3075322333 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5043 (secure 0 0 0) 0x7fcf5801ed50 con 0x7fcf681e60b0 2024-08-31T12:25:58.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.331+0000 7fcf4ffff640 1 -- 172.21.15.19:0/3075322333 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcf681e60b0 msgr2=0x7fcf680afab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:58.333 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:25:58.333 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:25:58.333 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:25:58.333 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:25:58.333 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 80, 2024-08-31T12:25:58.333 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 145688, 2024-08-31T12:25:58.333 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:25:58.333 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:25:58.333 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:25:58.333 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:25:58.334 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:58.334 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 14, 2024-08-31T12:25:58.334 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:25:58.334 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:25:58.334 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:25:58.334 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:25:58.334 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:25:58.334 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:25:58.334 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:25:58.334 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:25:58.334 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:25:58.334 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:25:58.334 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:25:58.335 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:25:58.335 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:25:58.335 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:25:58.335 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:58.335 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:25:58.335 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:25:58.335 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:25:58.335 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:25:58.335 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:25:58.335 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:25:58.335 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:25:58.335 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:25:58.335 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:25:58.336 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:25:58.336 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:25:58.336 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:25:58.336 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:25:58.336 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:58.336 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:58.336 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:25:58.336 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:25:58.336 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:25:58.336 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:25:58.336 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:25:58.336 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:25:58.336 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:25:58.337 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:25:58.337 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:25:58.337 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:25:58.337 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:25:58.337 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:25:58.337 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:25:58.337 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:25:58.337 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:25:58.337 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:25:58.337 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:25:58.337 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:25:58.337 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:25:58.337 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:25:58.337 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:25:58.338 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:25:58.338 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:25:58.338 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:25:58.338 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:25:58.338 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:25:58.338 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:25:58.338 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:25:58.338 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:58.338 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:25:58.338 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:58.338 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:25:58.338 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:58.338 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:25:58.338 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:25:58.339 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:25:58.339 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:25:58.339 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:58.339 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:25:58.339 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:25:58.339 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:58.339 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:58.339 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:58.339 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:25:58.339 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:25:58.339 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:58.339 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:58.339 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:58.340 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:58.340 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:25:58.340 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:25:58.340 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:25:58.340 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:25:58.340 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:25:58.340 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:58.340 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:58.340 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:25:58.340 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:25:58.340 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:25:58.340 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:25:58.340 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:58.341 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:25:58.341 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:25:58.341 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:58.341 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:58.341 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:58.341 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:25:58.341 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:25:58.341 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:58.341 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:58.341 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:58.341 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:58.341 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:25:58.341 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:25:58.342 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:25:58.342 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:25:58.342 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:25:58.342 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:58.342 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:58.342 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:25:58.342 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:25:58.342 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:25:58.342 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:25:58.342 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:58.342 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:25:58.342 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:25:58.342 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:58.342 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:58.343 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:58.343 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:25:58.343 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:25:58.343 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:25:58.343 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:58.343 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:58.343 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:58.343 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:25:58.343 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:25:58.343 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:25:58.343 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:25:58.343 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:25:58.343 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:58.344 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:58.344 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:58.344 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:25:58.344 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:25:58.344 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:58.344 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:58.344 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:58.344 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:25:58.344 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:58.344 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:58.344 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:25:58.344 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:58.344 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:58.344 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:58.345 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 3 2024-08-31T12:25:58.345 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:58.345 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:58.345 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:25:58.345 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:58.345 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:58.345 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:58.345 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 5 2024-08-31T12:25:58.345 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:58.345 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:58.345 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:25:58.345 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:58.345 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:58.346 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:58.346 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 3 2024-08-31T12:25:58.346 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:58.346 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:25:58.346 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:25:58.346 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:25:58.346 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:25:58.346 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:25:58.346 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:25:58.346 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:25:58.346 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:25:58.346 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:25:58.346 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:25:58.346 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:25:58.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.331+0000 7fcf4ffff640 1 --2- 172.21.15.19:0/3075322333 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcf681e60b0 0x7fcf680afab0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fcf58018000 tx=0x7fcf5800dfd0 comp rx=0 tx=0).stop 2024-08-31T12:25:58.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.331+0000 7fcf4ffff640 1 -- 172.21.15.19:0/3075322333 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcf681e60b0 msgr2=0x7fcf680afab0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:25:58.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.331+0000 7fcf4ffff640 1 --2- 172.21.15.19:0/3075322333 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcf681e60b0 0x7fcf680afab0 secure :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fcf58018000 tx=0x7fcf5800dfd0 comp rx=0 tx=0).stop 2024-08-31T12:25:58.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.332+0000 7fcf6e244640 1 -- 172.21.15.19:0/3075322333 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fcf54077a40 msgr2=0x7fcf54079f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:58.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.332+0000 7fcf6e244640 1 --2- 172.21.15.19:0/3075322333 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fcf54077a40 0x7fcf54079f50 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fcf680cbd80 tx=0x7fcf5c005f90 comp rx=0 tx=0).stop 2024-08-31T12:25:58.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.332+0000 7fcf67fff640 1 -- 172.21.15.19:0/3075322333 reap_dead start 2024-08-31T12:25:58.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.332+0000 7fcf6e244640 1 -- 172.21.15.19:0/3075322333 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcf6815ed90 msgr2=0x7fcf68152530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:58.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.332+0000 7fcf6e244640 1 --2- 172.21.15.19:0/3075322333 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcf6815ed90 0x7fcf68152530 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcf60009000 tx=0x7fcf60032be0 comp rx=0 tx=0).stop 2024-08-31T12:25:58.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.332+0000 7fcf6e244640 1 -- 172.21.15.19:0/3075322333 shutdown_connections 2024-08-31T12:25:58.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.332+0000 7fcf6e244640 1 --2- 172.21.15.19:0/3075322333 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcf6815ed90 0x7fcf68152530 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:58.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.332+0000 7fcf6e244640 1 -- 172.21.15.19:0/3075322333 >> 172.21.15.19:0/3075322333 conn(0x7fcf680cef50 msgr2=0x7fcf681566f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:58.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.332+0000 7fcf6e244640 1 -- 172.21.15.19:0/3075322333 shutdown_connections 2024-08-31T12:25:58.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.332+0000 7fcf6e244640 1 -- 172.21.15.19:0/3075322333 wait complete. 2024-08-31T12:25:58.350 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:25:58.350 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-08-31T12:25:58.350 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-08-31T12:25:58.350 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-08-31T12:25:58.351 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-08-31T12:25:58.351 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-08-31T12:25:58.351 INFO:teuthology.orchestra.run:waiting for 300 2024-08-31T12:25:58.365 INFO:tasks.ceph.mon.c:Stopped 2024-08-31T12:25:58.366 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-08-31T12:25:58.366 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:25:58.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.531+0000 7f7e63562640 1 -- 172.21.15.19:0/1117181817 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e5c0c3470 msgr2=0x7f7e5c0c3880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:58.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.531+0000 7f7e63562640 1 --2- 172.21.15.19:0/1117181817 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e5c0c3470 0x7f7e5c0c3880 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c009830 tx=0x7f7e4c02a530 comp rx=0 tx=0).stop 2024-08-31T12:25:58.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.531+0000 7f7e63562640 1 -- 172.21.15.19:0/1117181817 shutdown_connections 2024-08-31T12:25:58.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.531+0000 7f7e63562640 1 --2- 172.21.15.19:0/1117181817 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e5c0c3470 0x7f7e5c0c3880 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c009830 tx=0x7f7e4c02a530 comp rx=0 tx=0).stop 2024-08-31T12:25:58.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.531+0000 7f7e63562640 1 --2- 172.21.15.19:0/1117181817 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7e5c14e970 0x7f7e5c0c8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:58.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.531+0000 7f7e63562640 1 -- 172.21.15.19:0/1117181817 >> 172.21.15.19:0/1117181817 conn(0x7f7e5c0ccda0 msgr2=0x7f7e5c0cd1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:58.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.531+0000 7f7e63562640 1 -- 172.21.15.19:0/1117181817 shutdown_connections 2024-08-31T12:25:58.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.532+0000 7f7e63562640 1 -- 172.21.15.19:0/1117181817 wait complete. 2024-08-31T12:25:58.535 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.534+0000 7f7e63562640 1 Processor -- start 2024-08-31T12:25:58.535 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.534+0000 7f7e63562640 1 -- start start 2024-08-31T12:25:58.537 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.536+0000 7f7e63562640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7e5c14e970 0x7f7e5c1f8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:58.537 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.536+0000 7f7e63562640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e5c15b580 0x7f7e5c145970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:58.538 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.536+0000 7f7e63562640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7e5c15a850 0x7f7e5c159800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:58.538 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.536+0000 7f7e612fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7e5c14e970 0x7f7e5c1f8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:58.538 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.536+0000 7f7e63562640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f7e5c1525a0 con 0x7f7e5c14e970 2024-08-31T12:25:58.539 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.536+0000 7f7e63562640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7e5c0c2fe0 con 0x7f7e5c15b580 2024-08-31T12:25:58.539 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.536+0000 7f7e612fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7e5c14e970 0x7f7e5c1f8fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:54800/0 (socket says 172.21.15.19:54800) 2024-08-31T12:25:58.539 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.536+0000 7f7e612fe640 1 -- 172.21.15.19:0/3259669055 learned_addr learned my addr 172.21.15.19:0/3259669055 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:25:58.539 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.536+0000 7f7e60afd640 1 --2- 172.21.15.19:0/3259669055 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e5c15b580 0x7f7e5c145970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:58.539 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.536+0000 7f7e61aff640 1 -- 172.21.15.19:0/3259669055 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7e5c15a850 msgr2=0x7f7e5c159800 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3301/0 2024-08-31T12:25:58.539 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.536+0000 7f7e61aff640 1 --2- 172.21.15.19:0/3259669055 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7e5c15a850 0x7f7e5c159800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:25:58.539 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.536+0000 7f7e63562640 1 -- 172.21.15.19:0/3259669055 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f7e5c14ed80 con 0x7f7e5c15a850 2024-08-31T12:25:58.539 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.537+0000 7f7e612fe640 1 -- 172.21.15.19:0/3259669055 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7e5c15a850 msgr2=0x7f7e5c159800 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:25:58.539 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.537+0000 7f7e612fe640 1 --2- 172.21.15.19:0/3259669055 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7e5c15a850 0x7f7e5c159800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:58.539 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.537+0000 7f7e612fe640 1 -- 172.21.15.19:0/3259669055 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e5c15b580 msgr2=0x7f7e5c145970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:58.539 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.537+0000 7f7e612fe640 1 --2- 172.21.15.19:0/3259669055 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e5c15b580 0x7f7e5c145970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:58.539 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.537+0000 7f7e612fe640 1 -- 172.21.15.19:0/3259669055 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e5c164240 con 0x7f7e5c14e970 2024-08-31T12:25:58.539 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.537+0000 7f7e60afd640 1 --2- 172.21.15.19:0/3259669055 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e5c15b580 0x7f7e5c145970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:25:58.540 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.537+0000 7f7e612fe640 1 --2- 172.21.15.19:0/3259669055 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7e5c14e970 0x7f7e5c1f8fb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7e5800b550 tx=0x7f7e5800ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:58.540 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.537+0000 7f7e51ffb640 1 -- 172.21.15.19:0/3259669055 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e58014070 con 0x7f7e5c14e970 2024-08-31T12:25:58.540 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.537+0000 7f7e51ffb640 1 -- 172.21.15.19:0/3259669055 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7e580109a0 con 0x7f7e5c14e970 2024-08-31T12:25:58.540 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.537+0000 7f7e51ffb640 1 -- 172.21.15.19:0/3259669055 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e58010df0 con 0x7f7e5c14e970 2024-08-31T12:25:58.540 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.537+0000 7f7e63562640 1 -- 172.21.15.19:0/3259669055 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e5c1f94f0 con 0x7f7e5c14e970 2024-08-31T12:25:58.540 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.537+0000 7f7e63562640 1 -- 172.21.15.19:0/3259669055 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e5c15b330 con 0x7f7e5c14e970 2024-08-31T12:25:58.541 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.540+0000 7f7e51ffb640 1 -- 172.21.15.19:0/3259669055 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f7e58017070 con 0x7f7e5c14e970 2024-08-31T12:25:58.542 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.541+0000 7f7e51ffb640 1 --2- 172.21.15.19:0/3259669055 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7e40077a00 0x7f7e40079f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:25:58.542 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.541+0000 7f7e51ffb640 1 -- 172.21.15.19:0/3259669055 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f7e580995a0 con 0x7f7e5c14e970 2024-08-31T12:25:58.542 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.541+0000 7f7e60afd640 1 --2- 172.21.15.19:0/3259669055 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7e40077a00 0x7f7e40079f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:25:58.542 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.541+0000 7f7e63562640 1 -- 172.21.15.19:0/3259669055 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e5c0c77a0 con 0x7f7e5c14e970 2024-08-31T12:25:58.542 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.541+0000 7f7e60afd640 1 --2- 172.21.15.19:0/3259669055 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7e40077a00 0x7f7e40079f10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c034000 tx=0x7f7e4c004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:25:58.551 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.550+0000 7f7e51ffb640 1 -- 172.21.15.19:0/3259669055 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f7e58061ba0 con 0x7f7e5c14e970 2024-08-31T12:25:58.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.979+0000 7f7e63562640 1 -- 172.21.15.19:0/3259669055 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7e5c0b16d0 con 0x7f7e5c14e970 2024-08-31T12:25:58.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.980+0000 7f7e51ffb640 1 -- 172.21.15.19:0/3259669055 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f7e58066360 con 0x7f7e5c14e970 2024-08-31T12:25:58.981 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:25:58.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.983+0000 7f7e63562640 1 -- 172.21.15.19:0/3259669055 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7e40077a00 msgr2=0x7f7e40079f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:58.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.983+0000 7f7e63562640 1 --2- 172.21.15.19:0/3259669055 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7e40077a00 0x7f7e40079f10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c034000 tx=0x7f7e4c004730 comp rx=0 tx=0).stop 2024-08-31T12:25:58.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.983+0000 7f7e63562640 1 -- 172.21.15.19:0/3259669055 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7e5c14e970 msgr2=0x7f7e5c1f8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:25:58.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.983+0000 7f7e63562640 1 --2- 172.21.15.19:0/3259669055 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7e5c14e970 0x7f7e5c1f8fb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7e5800b550 tx=0x7f7e5800ba10 comp rx=0 tx=0).stop 2024-08-31T12:25:58.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.983+0000 7f7e63562640 1 -- 172.21.15.19:0/3259669055 shutdown_connections 2024-08-31T12:25:58.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.983+0000 7f7e63562640 1 --2- 172.21.15.19:0/3259669055 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7e40077a00 0x7f7e40079f10 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:58.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.983+0000 7f7e63562640 1 --2- 172.21.15.19:0/3259669055 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7e5c15a850 0x7f7e5c159800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:58.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.983+0000 7f7e63562640 1 --2- 172.21.15.19:0/3259669055 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e5c15b580 0x7f7e5c145970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:58.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.983+0000 7f7e63562640 1 --2- 172.21.15.19:0/3259669055 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7e5c14e970 0x7f7e5c1f8fb0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:25:58.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.983+0000 7f7e63562640 1 -- 172.21.15.19:0/3259669055 >> 172.21.15.19:0/3259669055 conn(0x7f7e5c0ccda0 msgr2=0x7f7e5c0c8080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:25:58.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.983+0000 7f7e63562640 1 -- 172.21.15.19:0/3259669055 shutdown_connections 2024-08-31T12:25:58.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:25:58.983+0000 7f7e63562640 1 -- 172.21.15.19:0/3259669055 wait complete. 2024-08-31T12:26:02.001 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:26:02.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.166+0000 7fc571f50640 1 -- 172.21.15.19:0/3307546023 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc56c0ccbb0 msgr2=0x7fc56c0cb280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:02.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.166+0000 7fc571f50640 1 --2- 172.21.15.19:0/3307546023 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc56c0ccbb0 0x7fc56c0cb280 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc56c0ce330 tx=0x7fc56402ab40 comp rx=0 tx=0).stop 2024-08-31T12:26:02.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.166+0000 7fc571f50640 1 -- 172.21.15.19:0/3307546023 shutdown_connections 2024-08-31T12:26:02.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.166+0000 7fc571f50640 1 --2- 172.21.15.19:0/3307546023 >> v2:172.21.15.43:3300/0 conn(0x7fc56c0cc070 0x7fc56c0cc480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:02.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.166+0000 7fc571f50640 1 --2- 172.21.15.19:0/3307546023 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc56c0ccbb0 0x7fc56c0cb280 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:02.168 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.166+0000 7fc571f50640 1 -- 172.21.15.19:0/3307546023 >> 172.21.15.19:0/3307546023 conn(0x7fc56c074810 msgr2=0x7fc56c068aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:26:02.168 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.166+0000 7fc571f50640 1 -- 172.21.15.19:0/3307546023 shutdown_connections 2024-08-31T12:26:02.168 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.166+0000 7fc571f50640 1 -- 172.21.15.19:0/3307546023 wait complete. 2024-08-31T12:26:02.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.169+0000 7fc571f50640 1 Processor -- start 2024-08-31T12:26:02.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.169+0000 7fc571f50640 1 -- start start 2024-08-31T12:26:02.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.171+0000 7fc571f50640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc56c0cc070 0x7fc56c0d11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:02.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.171+0000 7fc571f50640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc56c0ccbb0 0x7fc56c156da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:02.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.171+0000 7fc571f50640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc56c15b120 0x7fc56c1f18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:02.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.171+0000 7fc571f50640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fc56c1557e0 con 0x7fc56c0cc070 2024-08-31T12:26:02.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.171+0000 7fc571f50640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc56c155660 con 0x7fc56c15b120 2024-08-31T12:26:02.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.171+0000 7fc571f50640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fc56c06e1a0 con 0x7fc56c0ccbb0 2024-08-31T12:26:02.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.171+0000 7fc56affd640 1 -- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc56c0ccbb0 msgr2=0x7fc56c156da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3301/0 2024-08-31T12:26:02.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.171+0000 7fc56affd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc56c0ccbb0 0x7fc56c156da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:26:02.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.171+0000 7fc56bfff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc56c15b120 0x7fc56c1f18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:02.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.171+0000 7fc56bfff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc56c15b120 0x7fc56c1f18d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:56406/0 (socket says 172.21.15.19:56406) 2024-08-31T12:26:02.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.171+0000 7fc56bfff640 1 -- 172.21.15.19:0/476274599 learned_addr learned my addr 172.21.15.19:0/476274599 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:26:02.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.171+0000 7fc56b7fe640 1 --2- 172.21.15.19:0/476274599 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc56c0cc070 0x7fc56c0d11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:02.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.172+0000 7fc56bfff640 1 -- 172.21.15.19:0/476274599 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc56c0ccbb0 msgr2=0x7fc56c156da0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:26:02.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.172+0000 7fc56bfff640 1 --2- 172.21.15.19:0/476274599 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc56c0ccbb0 0x7fc56c156da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:02.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.172+0000 7fc56bfff640 1 -- 172.21.15.19:0/476274599 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc56c0cc070 msgr2=0x7fc56c0d11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:02.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.172+0000 7fc56bfff640 1 --2- 172.21.15.19:0/476274599 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc56c0cc070 0x7fc56c0d11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:02.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.172+0000 7fc56bfff640 1 -- 172.21.15.19:0/476274599 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc56c1572e0 con 0x7fc56c15b120 2024-08-31T12:26:02.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.172+0000 7fc56bfff640 1 --2- 172.21.15.19:0/476274599 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc56c15b120 0x7fc56c1f18d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc56000c3c0 tx=0x7fc56000c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:02.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.173+0000 7fc553fff640 1 -- 172.21.15.19:0/476274599 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc560017070 con 0x7fc56c15b120 2024-08-31T12:26:02.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.173+0000 7fc553fff640 1 -- 172.21.15.19:0/476274599 <== mon.1 v2:172.21.15.43:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc5600053c0 con 0x7fc56c15b120 2024-08-31T12:26:02.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.173+0000 7fc571f50640 1 -- 172.21.15.19:0/476274599 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc56c1574f0 con 0x7fc56c15b120 2024-08-31T12:26:02.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.173+0000 7fc553fff640 1 -- 172.21.15.19:0/476274599 <== mon.1 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc560003850 con 0x7fc56c15b120 2024-08-31T12:26:02.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.173+0000 7fc571f50640 1 -- 172.21.15.19:0/476274599 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc56c157b40 con 0x7fc56c15b120 2024-08-31T12:26:02.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.175+0000 7fc571f50640 1 -- 172.21.15.19:0/476274599 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc56c0cb280 con 0x7fc56c15b120 2024-08-31T12:26:02.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.175+0000 7fc553fff640 1 -- 172.21.15.19:0/476274599 <== mon.1 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fc56001a070 con 0x7fc56c15b120 2024-08-31T12:26:02.177 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.176+0000 7fc553fff640 1 --2- 172.21.15.19:0/476274599 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc558077a80 0x7fc558079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:02.177 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.176+0000 7fc553fff640 1 -- 172.21.15.19:0/476274599 <== mon.1 v2:172.21.15.43:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fc5600988c0 con 0x7fc56c15b120 2024-08-31T12:26:02.178 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.177+0000 7fc56b7fe640 1 --2- 172.21.15.19:0/476274599 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc558077a80 0x7fc558079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:02.179 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.178+0000 7fc56b7fe640 1 --2- 172.21.15.19:0/476274599 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc558077a80 0x7fc558079f90 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fc564035000 tx=0x7fc564038040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:02.182 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.181+0000 7fc553fff640 1 -- 172.21.15.19:0/476274599 <== mon.1 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fc560060f40 con 0x7fc56c15b120 2024-08-31T12:26:02.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.623+0000 7fc571f50640 1 -- 172.21.15.19:0/476274599 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc56c158b40 con 0x7fc56c15b120 2024-08-31T12:26:02.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.623+0000 7fc553fff640 1 -- 172.21.15.19:0/476274599 <== mon.1 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fc560065700 con 0x7fc56c15b120 2024-08-31T12:26:02.625 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":18,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:26:02.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.627+0000 7fc571f50640 1 -- 172.21.15.19:0/476274599 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc558077a80 msgr2=0x7fc558079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:02.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.627+0000 7fc571f50640 1 --2- 172.21.15.19:0/476274599 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc558077a80 0x7fc558079f90 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fc564035000 tx=0x7fc564038040 comp rx=0 tx=0).stop 2024-08-31T12:26:02.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.627+0000 7fc571f50640 1 -- 172.21.15.19:0/476274599 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc56c15b120 msgr2=0x7fc56c1f18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:02.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.627+0000 7fc571f50640 1 --2- 172.21.15.19:0/476274599 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc56c15b120 0x7fc56c1f18d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc56000c3c0 tx=0x7fc56000c880 comp rx=0 tx=0).stop 2024-08-31T12:26:02.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.627+0000 7fc571f50640 1 -- 172.21.15.19:0/476274599 shutdown_connections 2024-08-31T12:26:02.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.627+0000 7fc571f50640 1 --2- 172.21.15.19:0/476274599 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc558077a80 0x7fc558079f90 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:02.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.627+0000 7fc571f50640 1 --2- 172.21.15.19:0/476274599 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc56c15b120 0x7fc56c1f18d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:02.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.627+0000 7fc571f50640 1 --2- 172.21.15.19:0/476274599 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc56c0ccbb0 0x7fc56c156da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:02.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.627+0000 7fc571f50640 1 --2- 172.21.15.19:0/476274599 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc56c0cc070 0x7fc56c0d11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:02.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.627+0000 7fc571f50640 1 -- 172.21.15.19:0/476274599 >> 172.21.15.19:0/476274599 conn(0x7fc56c074810 msgr2=0x7fc56c15a770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:26:02.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.627+0000 7fc571f50640 1 -- 172.21.15.19:0/476274599 shutdown_connections 2024-08-31T12:26:02.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:02.627+0000 7fc571f50640 1 -- 172.21.15.19:0/476274599 wait complete. 2024-08-31T12:26:05.645 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:26:05.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:05.816+0000 7f0f1fd39640 1 -- 172.21.15.19:0/605351515 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0f1814e970 msgr2=0x7f0f180c8340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:05.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:05.816+0000 7f0f1fd39640 1 --2- 172.21.15.19:0/605351515 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0f1814e970 0x7f0f180c8340 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0f14009830 tx=0x7f0f1402a530 comp rx=0 tx=0).stop 2024-08-31T12:26:05.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:05.816+0000 7f0f1fd39640 1 -- 172.21.15.19:0/605351515 shutdown_connections 2024-08-31T12:26:05.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:05.816+0000 7f0f1fd39640 1 --2- 172.21.15.19:0/605351515 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0f18160900 0x7f0f18162d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:05.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:05.817+0000 7f0f1fd39640 1 --2- 172.21.15.19:0/605351515 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0f1814e970 0x7f0f180c8340 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:05.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:05.817+0000 7f0f1fd39640 1 -- 172.21.15.19:0/605351515 >> 172.21.15.19:0/605351515 conn(0x7f0f180ccda0 msgr2=0x7f0f180cd1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:26:05.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:05.817+0000 7f0f1fd39640 1 -- 172.21.15.19:0/605351515 shutdown_connections 2024-08-31T12:26:05.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:05.817+0000 7f0f1fd39640 1 -- 172.21.15.19:0/605351515 wait complete. 2024-08-31T12:26:05.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:05.820+0000 7f0f1fd39640 1 Processor -- start 2024-08-31T12:26:05.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:05.820+0000 7f0f1fd39640 1 -- start start 2024-08-31T12:26:05.823 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:05.822+0000 7f0f1fd39640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0f180c3470 0x7f0f181469a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:05.823 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:05.822+0000 7f0f1fd39640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0f1814e970 0x7f0f180cb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:05.823 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:05.822+0000 7f0f1fd39640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0f18160900 0x7f0f18148ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:05.823 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:05.822+0000 7f0f1fd39640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f0f180c2fe0 con 0x7f0f180c3470 2024-08-31T12:26:05.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:05.822+0000 7f0f1fd39640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0f18158660 con 0x7f0f1814e970 2024-08-31T12:26:05.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:05.822+0000 7f0f1fd39640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f0f181525a0 con 0x7f0f18160900 2024-08-31T12:26:05.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:05.822+0000 7f0f1e2d6640 1 -- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0f18160900 msgr2=0x7f0f18148ec0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3301/0 2024-08-31T12:26:05.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:05.822+0000 7f0f1e2d6640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0f18160900 0x7f0f18148ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:26:05.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:05.822+0000 7f0f1dad5640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0f180c3470 0x7f0f181469a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:05.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:05.822+0000 7f0f1dad5640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0f180c3470 0x7f0f181469a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:54844/0 (socket says 172.21.15.19:54844) 2024-08-31T12:26:05.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:05.822+0000 7f0f1dad5640 1 -- 172.21.15.19:0/2049455270 learned_addr learned my addr 172.21.15.19:0/2049455270 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:26:05.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:05.822+0000 7f0f1d2d4640 1 --2- 172.21.15.19:0/2049455270 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0f1814e970 0x7f0f180cb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:05.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:05.822+0000 7f0f1dad5640 1 -- 172.21.15.19:0/2049455270 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0f18160900 msgr2=0x7f0f18148ec0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:26:05.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:05.822+0000 7f0f1dad5640 1 --2- 172.21.15.19:0/2049455270 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0f18160900 0x7f0f18148ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:05.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:05.822+0000 7f0f1dad5640 1 -- 172.21.15.19:0/2049455270 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0f1814e970 msgr2=0x7f0f180cb7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:05.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:05.822+0000 7f0f1dad5640 1 --2- 172.21.15.19:0/2049455270 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0f1814e970 0x7f0f180cb7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:05.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:05.822+0000 7f0f1dad5640 1 -- 172.21.15.19:0/2049455270 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f180cbd20 con 0x7f0f180c3470 2024-08-31T12:26:05.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:05.823+0000 7f0f1dad5640 1 --2- 172.21.15.19:0/2049455270 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0f180c3470 0x7f0f181469a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0f14000c00 tx=0x7f0f14033ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:05.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:05.823+0000 7f0f0e7fc640 1 -- 172.21.15.19:0/2049455270 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f14031d60 con 0x7f0f180c3470 2024-08-31T12:26:05.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:05.823+0000 7f0f0e7fc640 1 -- 172.21.15.19:0/2049455270 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0f140330d0 con 0x7f0f180c3470 2024-08-31T12:26:05.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:05.823+0000 7f0f1fd39640 1 -- 172.21.15.19:0/2049455270 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f180cad40 con 0x7f0f180c3470 2024-08-31T12:26:05.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:05.823+0000 7f0f0e7fc640 1 -- 172.21.15.19:0/2049455270 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f140334e0 con 0x7f0f180c3470 2024-08-31T12:26:05.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:05.823+0000 7f0f1fd39640 1 -- 172.21.15.19:0/2049455270 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f180cafe0 con 0x7f0f180c3470 2024-08-31T12:26:05.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:05.825+0000 7f0efbfff640 1 -- 172.21.15.19:0/2049455270 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f1814ed80 con 0x7f0f180c3470 2024-08-31T12:26:05.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:05.826+0000 7f0f0e7fc640 1 -- 172.21.15.19:0/2049455270 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f0f14033680 con 0x7f0f180c3470 2024-08-31T12:26:05.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:05.826+0000 7f0f0e7fc640 1 --2- 172.21.15.19:0/2049455270 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0efc077a00 0x7f0efc079f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:05.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:05.827+0000 7f0f0e7fc640 1 -- 172.21.15.19:0/2049455270 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f0f140bac20 con 0x7f0f180c3470 2024-08-31T12:26:05.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:05.827+0000 7f0f1d2d4640 1 --2- 172.21.15.19:0/2049455270 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0efc077a00 0x7f0efc079f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:05.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:05.828+0000 7f0f1d2d4640 1 --2- 172.21.15.19:0/2049455270 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0efc077a00 0x7f0efc079f10 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0f080046b0 tx=0x7f0f08007e90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:05.834 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:05.834+0000 7f0f0e7fc640 1 -- 172.21.15.19:0/2049455270 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f0f1403f030 con 0x7f0f180c3470 2024-08-31T12:26:06.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:06.295+0000 7f0efbfff640 1 -- 172.21.15.19:0/2049455270 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0f180732b0 con 0x7f0f180c3470 2024-08-31T12:26:06.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:06.295+0000 7f0f0e7fc640 1 -- 172.21.15.19:0/2049455270 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f0f14083920 con 0x7f0f180c3470 2024-08-31T12:26:06.297 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:26:06.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:06.299+0000 7f0efbfff640 1 -- 172.21.15.19:0/2049455270 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0efc077a00 msgr2=0x7f0efc079f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:06.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:06.299+0000 7f0efbfff640 1 --2- 172.21.15.19:0/2049455270 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0efc077a00 0x7f0efc079f10 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0f080046b0 tx=0x7f0f08007e90 comp rx=0 tx=0).stop 2024-08-31T12:26:06.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:06.299+0000 7f0efbfff640 1 -- 172.21.15.19:0/2049455270 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0f180c3470 msgr2=0x7f0f181469a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:06.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:06.299+0000 7f0efbfff640 1 --2- 172.21.15.19:0/2049455270 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0f180c3470 0x7f0f181469a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0f14000c00 tx=0x7f0f14033ce0 comp rx=0 tx=0).stop 2024-08-31T12:26:06.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:06.299+0000 7f0efbfff640 1 -- 172.21.15.19:0/2049455270 shutdown_connections 2024-08-31T12:26:06.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:06.299+0000 7f0efbfff640 1 --2- 172.21.15.19:0/2049455270 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0efc077a00 0x7f0efc079f10 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:06.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:06.299+0000 7f0efbfff640 1 --2- 172.21.15.19:0/2049455270 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0f18160900 0x7f0f18148ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:06.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:06.299+0000 7f0efbfff640 1 --2- 172.21.15.19:0/2049455270 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0f1814e970 0x7f0f180cb7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:06.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:06.299+0000 7f0efbfff640 1 --2- 172.21.15.19:0/2049455270 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0f180c3470 0x7f0f181469a0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:06.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:06.299+0000 7f0efbfff640 1 -- 172.21.15.19:0/2049455270 >> 172.21.15.19:0/2049455270 conn(0x7f0f180ccda0 msgr2=0x7f0f181427b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:26:06.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:06.299+0000 7f0efbfff640 1 -- 172.21.15.19:0/2049455270 shutdown_connections 2024-08-31T12:26:06.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:06.300+0000 7f0efbfff640 1 -- 172.21.15.19:0/2049455270 wait complete. 2024-08-31T12:26:09.318 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:26:13.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.084+0000 7f65334c5640 1 -- 172.21.15.19:0/3664604641 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f65100036c0 msgr2=0x7f6510005b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:13.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.084+0000 7f65334c5640 1 --2- 172.21.15.19:0/3664604641 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f65100036c0 0x7f6510005b90 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6528005e50 tx=0x7f652802c6d0 comp rx=0 tx=0).stop 2024-08-31T12:26:13.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.084+0000 7f6531a62640 1 -- 172.21.15.19:0/3664604641 reap_dead start 2024-08-31T12:26:13.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.084+0000 7f65334c5640 1 -- 172.21.15.19:0/3664604641 shutdown_connections 2024-08-31T12:26:13.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.084+0000 7f65334c5640 1 --2- 172.21.15.19:0/3664604641 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f65100036c0 0x7f6510005b90 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:13.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.084+0000 7f65334c5640 1 -- 172.21.15.19:0/3664604641 >> 172.21.15.19:0/3664604641 conn(0x7f652c0cc770 msgr2=0x7f652c0ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:26:13.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.084+0000 7f65334c5640 1 -- 172.21.15.19:0/3664604641 shutdown_connections 2024-08-31T12:26:13.087 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.084+0000 7f65334c5640 1 -- 172.21.15.19:0/3664604641 wait complete. 2024-08-31T12:26:13.088 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.087+0000 7f65334c5640 1 Processor -- start 2024-08-31T12:26:13.088 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.087+0000 7f65334c5640 1 -- start start 2024-08-31T12:26:13.090 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.089+0000 7f65334c5640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f65100036c0 0x7f652c1ddc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:13.090 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.089+0000 7f65334c5640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f652c1deb70 0x7f652c153da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:13.090 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.089+0000 7f65334c5640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f652c14eec0 0x7f652c14f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:13.090 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.089+0000 7f65334c5640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f652c070140 con 0x7f652c14eec0 2024-08-31T12:26:13.090 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.089+0000 7f65334c5640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f652c073b50 con 0x7f65100036c0 2024-08-31T12:26:13.091 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.089+0000 7f65334c5640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f652c070420 con 0x7f652c1deb70 2024-08-31T12:26:13.091 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.089+0000 7f6530a60640 1 -- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f652c1deb70 msgr2=0x7f652c153da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3301/0 2024-08-31T12:26:13.091 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.089+0000 7f6530a60640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f652c1deb70 0x7f652c153da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:26:13.091 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.090+0000 7f6531261640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f65100036c0 0x7f652c1ddc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:13.091 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.090+0000 7f6531a62640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f652c14eec0 0x7f652c14f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:13.092 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.090+0000 7f6531261640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f65100036c0 0x7f652c1ddc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:55292/0 (socket says 172.21.15.19:55292) 2024-08-31T12:26:13.092 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.090+0000 7f6531261640 1 -- 172.21.15.19:0/2557613976 learned_addr learned my addr 172.21.15.19:0/2557613976 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:26:13.092 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.090+0000 7f6531261640 1 -- 172.21.15.19:0/2557613976 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f652c1deb70 msgr2=0x7f652c153da0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:26:13.092 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.090+0000 7f6531261640 1 --2- 172.21.15.19:0/2557613976 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f652c1deb70 0x7f652c153da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:13.092 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.090+0000 7f6531261640 1 -- 172.21.15.19:0/2557613976 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f652c14eec0 msgr2=0x7f652c14f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:13.092 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.090+0000 7f6531261640 1 --2- 172.21.15.19:0/2557613976 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f652c14eec0 0x7f652c14f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:13.092 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.090+0000 7f6531261640 1 -- 172.21.15.19:0/2557613976 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f652c1de1d0 con 0x7f65100036c0 2024-08-31T12:26:13.093 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.091+0000 7f6531261640 1 --2- 172.21.15.19:0/2557613976 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f65100036c0 0x7f652c1ddc90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6528032240 tx=0x7f6528032f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:13.093 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.091+0000 7f6521ffb640 1 -- 172.21.15.19:0/2557613976 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f652803a070 con 0x7f65100036c0 2024-08-31T12:26:13.093 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.091+0000 7f6521ffb640 1 -- 172.21.15.19:0/2557613976 <== mon.1 v2:172.21.15.43:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6528004030 con 0x7f65100036c0 2024-08-31T12:26:13.093 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.091+0000 7f6521ffb640 1 -- 172.21.15.19:0/2557613976 <== mon.1 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6528004480 con 0x7f65100036c0 2024-08-31T12:26:13.093 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.091+0000 7f65334c5640 1 -- 172.21.15.19:0/2557613976 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f652c1e3c80 con 0x7f65100036c0 2024-08-31T12:26:13.093 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.092+0000 7f65334c5640 1 -- 172.21.15.19:0/2557613976 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f652c1dd490 con 0x7f65100036c0 2024-08-31T12:26:13.094 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.093+0000 7f65334c5640 1 -- 172.21.15.19:0/2557613976 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f652c1dd280 con 0x7f65100036c0 2024-08-31T12:26:13.095 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.094+0000 7f6521ffb640 1 -- 172.21.15.19:0/2557613976 <== mon.1 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f6528047020 con 0x7f65100036c0 2024-08-31T12:26:13.096 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.094+0000 7f6521ffb640 1 --2- 172.21.15.19:0/2557613976 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6510078a90 0x7f651007afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:13.096 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.095+0000 7f6521ffb640 1 -- 172.21.15.19:0/2557613976 <== mon.1 v2:172.21.15.43:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f65280ba810 con 0x7f65100036c0 2024-08-31T12:26:13.096 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.095+0000 7f6530a60640 1 --2- 172.21.15.19:0/2557613976 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6510078a90 0x7f651007afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:13.096 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.096+0000 7f6530a60640 1 --2- 172.21.15.19:0/2557613976 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6510078a90 0x7f651007afa0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f652c0c32d0 tx=0x7f651c007970 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:13.101 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.100+0000 7f6521ffb640 1 -- 172.21.15.19:0/2557613976 <== mon.1 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f6528083460 con 0x7f65100036c0 2024-08-31T12:26:13.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.562+0000 7f65334c5640 1 -- 172.21.15.19:0/2557613976 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f652c0c9320 con 0x7f65100036c0 2024-08-31T12:26:13.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.563+0000 7f6521ffb640 1 -- 172.21.15.19:0/2557613976 <== mon.1 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7f6528082d60 con 0x7f65100036c0 2024-08-31T12:26:13.565 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":84,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:26:13.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.567+0000 7f65334c5640 1 -- 172.21.15.19:0/2557613976 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6510078a90 msgr2=0x7f651007afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:13.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.567+0000 7f65334c5640 1 --2- 172.21.15.19:0/2557613976 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6510078a90 0x7f651007afa0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f652c0c32d0 tx=0x7f651c007970 comp rx=0 tx=0).stop 2024-08-31T12:26:13.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.567+0000 7f65334c5640 1 -- 172.21.15.19:0/2557613976 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f65100036c0 msgr2=0x7f652c1ddc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:13.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.567+0000 7f65334c5640 1 --2- 172.21.15.19:0/2557613976 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f65100036c0 0x7f652c1ddc90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6528032240 tx=0x7f6528032f40 comp rx=0 tx=0).stop 2024-08-31T12:26:13.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.567+0000 7f65334c5640 1 -- 172.21.15.19:0/2557613976 shutdown_connections 2024-08-31T12:26:13.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.567+0000 7f65334c5640 1 --2- 172.21.15.19:0/2557613976 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6510078a90 0x7f651007afa0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:13.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.567+0000 7f65334c5640 1 --2- 172.21.15.19:0/2557613976 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f652c14eec0 0x7f652c14f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:13.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.567+0000 7f65334c5640 1 --2- 172.21.15.19:0/2557613976 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f652c1deb70 0x7f652c153da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:13.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.567+0000 7f65334c5640 1 --2- 172.21.15.19:0/2557613976 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f65100036c0 0x7f652c1ddc90 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:13.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.567+0000 7f65334c5640 1 -- 172.21.15.19:0/2557613976 >> 172.21.15.19:0/2557613976 conn(0x7f652c0cc770 msgr2=0x7f652c0ca720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:26:13.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.568+0000 7f65334c5640 1 -- 172.21.15.19:0/2557613976 shutdown_connections 2024-08-31T12:26:13.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.568+0000 7f65334c5640 1 -- 172.21.15.19:0/2557613976 wait complete. 2024-08-31T12:26:13.585 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-08-31T12:26:13.585 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-08-31T12:26:13.585 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-31T12:26:13.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.757+0000 7faa4d59a640 1 -- 172.21.15.19:0/1751306419 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa44004a00 con 0x7faa500c8d70 2024-08-31T12:26:13.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.758+0000 7faa54bad640 1 -- 172.21.15.19:0/1751306419 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faa500c8d70 msgr2=0x7faa500c5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:13.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.758+0000 7faa54bad640 1 --2- 172.21.15.19:0/1751306419 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faa500c8d70 0x7faa500c5390 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7faa44005fd0 tx=0x7faa4402c370 comp rx=0 tx=0).stop 2024-08-31T12:26:13.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.758+0000 7faa54bad640 1 -- 172.21.15.19:0/1751306419 shutdown_connections 2024-08-31T12:26:13.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.758+0000 7faa54bad640 1 --2- 172.21.15.19:0/1751306419 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faa5014df80 0x7faa50157680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:13.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.758+0000 7faa54bad640 1 --2- 172.21.15.19:0/1751306419 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faa500c8d70 0x7faa500c5390 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:13.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.758+0000 7faa54bad640 1 -- 172.21.15.19:0/1751306419 >> 172.21.15.19:0/1751306419 conn(0x7faa500716c0 msgr2=0x7faa500ccc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:26:13.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.758+0000 7faa54bad640 1 -- 172.21.15.19:0/1751306419 shutdown_connections 2024-08-31T12:26:13.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.758+0000 7faa54bad640 1 -- 172.21.15.19:0/1751306419 wait complete. 2024-08-31T12:26:13.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.761+0000 7faa54bad640 1 Processor -- start 2024-08-31T12:26:13.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.761+0000 7faa54bad640 1 -- start start 2024-08-31T12:26:13.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.763+0000 7faa54bad640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faa500c4100 0x7faa50159d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:13.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.763+0000 7faa54bad640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa5014df80 0x7faa501f2730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:13.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.763+0000 7faa4e59c640 1 -- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faa500c4100 msgr2=0x7faa50159d30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3301/0 2024-08-31T12:26:13.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.763+0000 7faa4e59c640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faa500c4100 0x7faa50159d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:26:13.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.763+0000 7faa54bad640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faa50154470 0x7faa50158800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:13.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.763+0000 7faa54bad640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7faa50151ba0 con 0x7faa50154470 2024-08-31T12:26:13.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.763+0000 7faa54bad640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7faa50151a20 con 0x7faa5014df80 2024-08-31T12:26:13.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.763+0000 7faa54bad640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7faa500c6210 con 0x7faa500c4100 2024-08-31T12:26:13.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.763+0000 7faa4ed9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faa50154470 0x7faa50158800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:13.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.764+0000 7faa4dd9b640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa5014df80 0x7faa501f2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:13.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.764+0000 7faa4ed9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faa50154470 0x7faa50158800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:41244/0 (socket says 172.21.15.19:41244) 2024-08-31T12:26:13.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.764+0000 7faa4dd9b640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa5014df80 0x7faa501f2730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:55306/0 (socket says 172.21.15.19:55306) 2024-08-31T12:26:13.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.764+0000 7faa4ed9d640 1 -- 172.21.15.19:0/933450241 learned_addr learned my addr 172.21.15.19:0/933450241 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:26:13.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.764+0000 7faa4ed9d640 1 -- 172.21.15.19:0/933450241 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faa500c4100 msgr2=0x7faa50159d30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:26:13.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.764+0000 7faa4ed9d640 1 --2- 172.21.15.19:0/933450241 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faa500c4100 0x7faa50159d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:13.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.764+0000 7faa4ed9d640 1 -- 172.21.15.19:0/933450241 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa5014df80 msgr2=0x7faa501f2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:13.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.764+0000 7faa4ed9d640 1 --2- 172.21.15.19:0/933450241 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa5014df80 0x7faa501f2730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:13.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.764+0000 7faa4ed9d640 1 -- 172.21.15.19:0/933450241 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa5015aae0 con 0x7faa50154470 2024-08-31T12:26:13.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.764+0000 7faa4ed9d640 1 --2- 172.21.15.19:0/933450241 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faa50154470 0x7faa50158800 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7faa4800b9c0 tx=0x7faa4800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:13.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.765+0000 7faa3effd640 1 -- 172.21.15.19:0/933450241 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa48019070 con 0x7faa50154470 2024-08-31T12:26:13.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.765+0000 7faa54bad640 1 -- 172.21.15.19:0/933450241 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa5015adc0 con 0x7faa50154470 2024-08-31T12:26:13.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.765+0000 7faa54bad640 1 -- 172.21.15.19:0/933450241 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa5015b6a0 con 0x7faa50154470 2024-08-31T12:26:13.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.765+0000 7faa3effd640 1 -- 172.21.15.19:0/933450241 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faa4800cc30 con 0x7faa50154470 2024-08-31T12:26:13.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.765+0000 7faa3effd640 1 -- 172.21.15.19:0/933450241 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa4800ce60 con 0x7faa50154470 2024-08-31T12:26:13.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.766+0000 7faa54bad640 1 --2- 172.21.15.19:0/933450241 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faa34002a70 0x7faa34004f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:13.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.767+0000 7faa54bad640 1 -- 172.21.15.19:0/933450241 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7faa34007f10 con 0x7faa34002a70 2024-08-31T12:26:13.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.767+0000 7faa4e59c640 1 --2- 172.21.15.19:0/933450241 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faa34002a70 0x7faa34004f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:13.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.767+0000 7faa4e59c640 1 -- 172.21.15.19:0/933450241 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faa34008110 con 0x7faa34002a70 2024-08-31T12:26:13.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.767+0000 7faa4e59c640 1 --2- 172.21.15.19:0/933450241 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faa34002a70 0x7faa34004f80 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7faa5006dde0 tx=0x7faa44034670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:13.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.767+0000 7faa3effd640 1 -- 172.21.15.19:0/933450241 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa44042070 con 0x7faa34002a70 2024-08-31T12:26:13.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.768+0000 7faa3effd640 1 -- 172.21.15.19:0/933450241 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7faa4400b730 con 0x7faa34002a70 2024-08-31T12:26:13.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.768+0000 7faa3effd640 1 -- 172.21.15.19:0/933450241 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faa34002a70 msgr2=0x7faa34004f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:13.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.768+0000 7faa3effd640 1 --2- 172.21.15.19:0/933450241 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faa34002a70 0x7faa34004f80 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7faa5006dde0 tx=0x7faa44034670 comp rx=0 tx=0).stop 2024-08-31T12:26:13.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.768+0000 7faa3effd640 1 -- 172.21.15.19:0/933450241 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faa34002a70 msgr2=0x7faa34004f80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:26:13.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.768+0000 7faa3effd640 1 --2- 172.21.15.19:0/933450241 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faa34002a70 0x7faa34004f80 secure :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7faa5006dde0 tx=0x7faa44034670 comp rx=0 tx=0).stop 2024-08-31T12:26:13.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.769+0000 7faa3effd640 1 -- 172.21.15.19:0/933450241 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7faa4801c070 con 0x7faa50154470 2024-08-31T12:26:13.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.769+0000 7faa3effd640 1 --2- 172.21.15.19:0/933450241 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7faa3007fd40 0x7faa30082250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:13.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.770+0000 7faa3effd640 1 -- 172.21.15.19:0/933450241 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7faa480992f0 con 0x7faa50154470 2024-08-31T12:26:13.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.770+0000 7faa4e59c640 1 --2- 172.21.15.19:0/933450241 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7faa3007fd40 0x7faa30082250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:13.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.771+0000 7faa4e59c640 1 --2- 172.21.15.19:0/933450241 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7faa3007fd40 0x7faa30082250 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7faa340080e0 tx=0x7faa4403e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:13.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.780+0000 7faa54bad640 1 --2- 172.21.15.19:0/933450241 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faa34006f30 0x7faa340073f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:13.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.780+0000 7faa54bad640 1 -- 172.21.15.19:0/933450241 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7faa3400a4f0 con 0x7faa34006f30 2024-08-31T12:26:13.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.780+0000 7faa4dd9b640 1 --2- 172.21.15.19:0/933450241 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faa34006f30 0x7faa340073f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:13.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.781+0000 7faa4dd9b640 1 -- 172.21.15.19:0/933450241 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7faa34007930 con 0x7faa34006f30 2024-08-31T12:26:13.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.782+0000 7faa4dd9b640 1 --2- 172.21.15.19:0/933450241 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faa34006f30 0x7faa340073f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7faa4000f540 tx=0x7faa4000fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:13.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.782+0000 7faa3effd640 1 -- 172.21.15.19:0/933450241 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa400052e0 con 0x7faa34006f30 2024-08-31T12:26:13.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.783+0000 7faa3effd640 1 -- 172.21.15.19:0/933450241 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5033 (secure 0 0 0) 0x7faa400052e0 con 0x7faa34006f30 2024-08-31T12:26:13.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.783+0000 7faa3effd640 1 -- 172.21.15.19:0/933450241 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faa34006f30 msgr2=0x7faa340073f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:13.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.783+0000 7faa3effd640 1 --2- 172.21.15.19:0/933450241 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faa34006f30 0x7faa340073f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7faa4000f540 tx=0x7faa4000fa00 comp rx=0 tx=0).stop 2024-08-31T12:26:13.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.783+0000 7faa3effd640 1 -- 172.21.15.19:0/933450241 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faa34006f30 msgr2=0x7faa340073f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:26:13.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.783+0000 7faa3effd640 1 --2- 172.21.15.19:0/933450241 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faa34006f30 0x7faa340073f0 secure :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7faa4000f540 tx=0x7faa4000fa00 comp rx=0 tx=0).stop 2024-08-31T12:26:13.784 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:26:13.785 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:26:13.785 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:26:13.785 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "leader", 2024-08-31T12:26:13.785 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 84, 2024-08-31T12:26:13.785 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 108337, 2024-08-31T12:26:13.785 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:26:13.785 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:26:13.785 INFO:teuthology.orchestra.run.smithi019.stdout: 1 2024-08-31T12:26:13.785 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:13.785 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 0, 2024-08-31T12:26:13.785 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:26:13.785 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:26:13.786 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:26:13.786 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:26:13.786 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:26:13.786 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:26:13.786 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:26:13.786 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:26:13.786 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:26:13.786 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:26:13.786 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:26:13.786 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:26:13.786 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:26:13.786 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:26:13.786 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:13.786 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:26:13.787 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:26:13.787 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:26:13.787 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:26:13.787 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:26:13.787 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:26:13.787 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:26:13.787 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:26:13.787 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:26:13.787 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:26:13.787 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:26:13.787 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:26:13.787 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:26:13.787 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:13.787 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:13.787 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:26:13.788 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:26:13.788 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:26:13.788 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:26:13.788 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:26:13.788 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:26:13.788 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:26:13.788 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:26:13.788 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:26:13.788 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:26:13.788 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:26:13.788 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:26:13.788 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:26:13.788 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:26:13.789 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:26:13.789 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:26:13.789 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:26:13.789 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:26:13.789 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:26:13.789 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:26:13.789 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:26:13.789 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:26:13.789 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:26:13.789 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:26:13.789 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:26:13.789 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:26:13.790 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:26:13.790 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:26:13.790 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:13.790 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:26:13.790 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:13.790 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:26:13.790 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:13.790 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:26:13.790 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:26:13.790 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:26:13.790 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:26:13.790 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:13.790 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:26:13.790 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:26:13.790 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:13.791 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:13.791 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:13.791 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:26:13.791 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:26:13.791 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:13.791 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:13.791 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:13.791 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:13.791 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:26:13.791 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:26:13.791 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:26:13.791 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:26:13.791 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:26:13.792 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:13.792 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:13.792 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:26:13.792 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:26:13.792 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:26:13.792 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:26:13.792 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:13.792 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:26:13.792 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:26:13.792 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:13.792 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:13.792 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:13.792 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:26:13.793 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:26:13.793 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:13.793 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:13.793 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:13.793 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:13.793 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:26:13.793 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:26:13.793 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:26:13.793 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:26:13.793 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:26:13.793 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:13.793 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:13.793 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:26:13.793 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:26:13.794 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:26:13.794 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:26:13.794 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:13.794 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:26:13.794 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:26:13.794 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:13.794 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:13.794 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:13.794 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:26:13.794 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:26:13.794 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:13.794 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:13.794 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:13.794 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:13.795 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:26:13.795 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:26:13.795 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:26:13.795 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:26:13.795 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:26:13.795 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:13.795 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:13.795 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:13.795 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:26:13.795 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:26:13.795 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:13.795 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:26:13.795 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:26:13.795 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:26:13.795 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:13.796 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:13.796 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:26:13.796 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:13.796 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:26:13.796 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:26:13.796 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 4 2024-08-31T12:26:13.796 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:13.796 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:13.796 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:26:13.796 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:13.796 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:26:13.796 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:26:13.796 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 6 2024-08-31T12:26:13.796 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:13.797 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:13.797 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:26:13.797 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:13.797 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:26:13.797 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:26:13.797 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 5 2024-08-31T12:26:13.797 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:13.797 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:13.797 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:26:13.797 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:13.797 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:26:13.797 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:26:13.797 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:26:13.797 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:13.797 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:13.798 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:13.798 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:26:13.798 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:26:13.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.783+0000 7faa54bad640 1 -- 172.21.15.19:0/933450241 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7faa3007fd40 msgr2=0x7faa30082250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:13.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.783+0000 7faa54bad640 1 --2- 172.21.15.19:0/933450241 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7faa3007fd40 0x7faa30082250 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7faa340080e0 tx=0x7faa4403e000 comp rx=0 tx=0).stop 2024-08-31T12:26:13.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.783+0000 7faa54bad640 1 -- 172.21.15.19:0/933450241 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faa50154470 msgr2=0x7faa50158800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:13.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.783+0000 7faa4ed9d640 1 -- 172.21.15.19:0/933450241 reap_dead start 2024-08-31T12:26:13.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.783+0000 7faa54bad640 1 --2- 172.21.15.19:0/933450241 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faa50154470 0x7faa50158800 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7faa4800b9c0 tx=0x7faa4800be80 comp rx=0 tx=0).stop 2024-08-31T12:26:13.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.783+0000 7faa54bad640 1 -- 172.21.15.19:0/933450241 shutdown_connections 2024-08-31T12:26:13.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.783+0000 7faa54bad640 1 --2- 172.21.15.19:0/933450241 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faa50154470 0x7faa50158800 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:13.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.784+0000 7faa54bad640 1 -- 172.21.15.19:0/933450241 >> 172.21.15.19:0/933450241 conn(0x7faa500716c0 msgr2=0x7faa500c7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:26:13.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.784+0000 7faa54bad640 1 -- 172.21.15.19:0/933450241 shutdown_connections 2024-08-31T12:26:13.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.784+0000 7faa54bad640 1 -- 172.21.15.19:0/933450241 wait complete. 2024-08-31T12:26:13.800 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-08-31T12:26:13.800 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-08-31T12:26:13.800 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-31T12:26:13.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.967+0000 7fd32ad9d640 1 -- 172.21.15.19:0/2210772243 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3240c4ca0 msgr2=0x7fd324152ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:13.969 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.967+0000 7fd32ad9d640 1 --2- 172.21.15.19:0/2210772243 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3240c4ca0 0x7fd324152ac0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd320009830 tx=0x7fd32002a5c0 comp rx=0 tx=0).stop 2024-08-31T12:26:13.969 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.967+0000 7fd32ad9d640 1 -- 172.21.15.19:0/2210772243 shutdown_connections 2024-08-31T12:26:13.969 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.967+0000 7fd32ad9d640 1 --2- 172.21.15.19:0/2210772243 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd3240ce820 0x7fd3240cec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:13.969 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.967+0000 7fd32ad9d640 1 --2- 172.21.15.19:0/2210772243 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3240c4ca0 0x7fd324152ac0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:13.969 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.967+0000 7fd32ad9d640 1 -- 172.21.15.19:0/2210772243 >> 172.21.15.19:0/2210772243 conn(0x7fd324068ec0 msgr2=0x7fd3240692d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:26:13.969 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.967+0000 7fd32ad9d640 1 -- 172.21.15.19:0/2210772243 shutdown_connections 2024-08-31T12:26:13.969 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.967+0000 7fd32ad9d640 1 -- 172.21.15.19:0/2210772243 wait complete. 2024-08-31T12:26:13.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.970+0000 7fd32ad9d640 1 Processor -- start 2024-08-31T12:26:13.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.970+0000 7fd32ad9d640 1 -- start start 2024-08-31T12:26:13.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.972+0000 7fd32ad9d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd3240c4ca0 0x7fd3241516d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:13.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.972+0000 7fd32ad9d640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3240c8c50 0x7fd324142d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:13.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.972+0000 7fd32ad9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd3240ce820 0x7fd3241432b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:13.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.972+0000 7fd32ad9d640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fd324153000 con 0x7fd3240ce820 2024-08-31T12:26:13.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.972+0000 7fd32ad9d640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd324159490 con 0x7fd3240c8c50 2024-08-31T12:26:13.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.972+0000 7fd32ad9d640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fd3240c9160 con 0x7fd3240c4ca0 2024-08-31T12:26:13.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.972+0000 7fd329d9b640 1 -- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd3240c4ca0 msgr2=0x7fd3241516d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3301/0 2024-08-31T12:26:13.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.972+0000 7fd329d9b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd3240c4ca0 0x7fd3241516d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:26:13.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.972+0000 7fd32959a640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3240c8c50 0x7fd324142d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:13.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.972+0000 7fd32959a640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3240c8c50 0x7fd324142d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:55312/0 (socket says 172.21.15.19:55312) 2024-08-31T12:26:13.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.972+0000 7fd32959a640 1 -- 172.21.15.19:0/1465902485 learned_addr learned my addr 172.21.15.19:0/1465902485 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:26:13.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.972+0000 7fd32a59c640 1 --2- 172.21.15.19:0/1465902485 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd3240ce820 0x7fd3241432b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:13.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.973+0000 7fd32959a640 1 -- 172.21.15.19:0/1465902485 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd3240c4ca0 msgr2=0x7fd3241516d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:26:13.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.973+0000 7fd32959a640 1 --2- 172.21.15.19:0/1465902485 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd3240c4ca0 0x7fd3241516d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:13.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.973+0000 7fd32959a640 1 -- 172.21.15.19:0/1465902485 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd3240ce820 msgr2=0x7fd3241432b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:13.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.973+0000 7fd32959a640 1 --2- 172.21.15.19:0/1465902485 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd3240ce820 0x7fd3241432b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:13.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.973+0000 7fd32959a640 1 -- 172.21.15.19:0/1465902485 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd324063960 con 0x7fd3240c8c50 2024-08-31T12:26:13.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.973+0000 7fd32a59c640 1 --2- 172.21.15.19:0/1465902485 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd3240ce820 0x7fd3241432b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:26:13.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.973+0000 7fd32959a640 1 --2- 172.21.15.19:0/1465902485 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3240c8c50 0x7fd324142d70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd320009950 tx=0x7fd320033ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:13.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.974+0000 7fd31a7fc640 1 -- 172.21.15.19:0/1465902485 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd320031d60 con 0x7fd3240c8c50 2024-08-31T12:26:13.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.974+0000 7fd32ad9d640 1 -- 172.21.15.19:0/1465902485 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd324143af0 con 0x7fd3240c8c50 2024-08-31T12:26:13.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.974+0000 7fd31a7fc640 1 -- 172.21.15.19:0/1465902485 <== mon.1 v2:172.21.15.43:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd320043070 con 0x7fd3240c8c50 2024-08-31T12:26:13.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.974+0000 7fd31a7fc640 1 -- 172.21.15.19:0/1465902485 <== mon.1 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd320033330 con 0x7fd3240c8c50 2024-08-31T12:26:13.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.974+0000 7fd32ad9d640 1 -- 172.21.15.19:0/1465902485 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3241efad0 con 0x7fd3240c8c50 2024-08-31T12:26:13.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.975+0000 7fd32ad9d640 1 --2- 172.21.15.19:0/1465902485 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd32415c720 0x7fd3240ca0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:13.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.975+0000 7fd32ad9d640 1 -- 172.21.15.19:0/1465902485 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd32415aee0 con 0x7fd32415c720 2024-08-31T12:26:13.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.975+0000 7fd329d9b640 1 --2- 172.21.15.19:0/1465902485 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd32415c720 0x7fd3240ca0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:13.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.976+0000 7fd329d9b640 1 -- 172.21.15.19:0/1465902485 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd3241ef5e0 con 0x7fd32415c720 2024-08-31T12:26:13.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.977+0000 7fd329d9b640 1 --2- 172.21.15.19:0/1465902485 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd32415c720 0x7fd3240ca0d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd314003610 tx=0x7fd31400c7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:13.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.977+0000 7fd31a7fc640 1 -- 172.21.15.19:0/1465902485 <== mon.1 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fd3200334d0 con 0x7fd3240c8c50 2024-08-31T12:26:13.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.977+0000 7fd31a7fc640 1 --2- 172.21.15.19:0/1465902485 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd308077a30 0x7fd308079f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:13.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.978+0000 7fd31a7fc640 1 -- 172.21.15.19:0/1465902485 <== mon.1 v2:172.21.15.43:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fd3200ba700 con 0x7fd3240c8c50 2024-08-31T12:26:13.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.978+0000 7fd32a59c640 1 --2- 172.21.15.19:0/1465902485 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd308077a30 0x7fd308079f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:13.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.978+0000 7fd31a7fc640 1 -- 172.21.15.19:0/1465902485 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd314017070 con 0x7fd32415c720 2024-08-31T12:26:13.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.978+0000 7fd31a7fc640 1 -- 172.21.15.19:0/1465902485 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd314005c90 con 0x7fd32415c720 2024-08-31T12:26:13.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.978+0000 7fd31a7fc640 1 -- 172.21.15.19:0/1465902485 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd32415c720 msgr2=0x7fd3240ca0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:13.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.978+0000 7fd31a7fc640 1 --2- 172.21.15.19:0/1465902485 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd32415c720 0x7fd3240ca0d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd314003610 tx=0x7fd31400c7f0 comp rx=0 tx=0).stop 2024-08-31T12:26:13.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.978+0000 7fd31a7fc640 1 -- 172.21.15.19:0/1465902485 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd32415c720 msgr2=0x7fd3240ca0d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:26:13.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.978+0000 7fd31a7fc640 1 --2- 172.21.15.19:0/1465902485 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd32415c720 0x7fd3240ca0d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:13.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.978+0000 7fd32a59c640 1 --2- 172.21.15.19:0/1465902485 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd308077a30 0x7fd308079f40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd324147c40 tx=0x7fd31c00d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:13.991 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.990+0000 7fd32ad9d640 1 --2- 172.21.15.19:0/1465902485 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3241f02d0 0x7fd3240ca610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:13.991 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.990+0000 7fd32ad9d640 1 -- 172.21.15.19:0/1465902485 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd3240c3c20 con 0x7fd3241f02d0 2024-08-31T12:26:13.991 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.990+0000 7fd329d9b640 1 --2- 172.21.15.19:0/1465902485 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3241f02d0 0x7fd3240ca610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:13.992 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.991+0000 7fd329d9b640 1 -- 172.21.15.19:0/1465902485 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd324156c20 con 0x7fd3241f02d0 2024-08-31T12:26:13.992 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.991+0000 7fd329d9b640 1 --2- 172.21.15.19:0/1465902485 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3241f02d0 0x7fd3240ca610 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd314003610 tx=0x7fd314001080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:13.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.994+0000 7fd31a7fc640 1 -- 172.21.15.19:0/1465902485 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd314017040 con 0x7fd3241f02d0 2024-08-31T12:26:13.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.994+0000 7fd31a7fc640 1 -- 172.21.15.19:0/1465902485 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4863 (secure 0 0 0) 0x7fd314012400 con 0x7fd3241f02d0 2024-08-31T12:26:13.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.994+0000 7fd31a7fc640 1 -- 172.21.15.19:0/1465902485 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3241f02d0 msgr2=0x7fd3240ca610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:13.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.994+0000 7fd31a7fc640 1 --2- 172.21.15.19:0/1465902485 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3241f02d0 0x7fd3240ca610 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd314003610 tx=0x7fd314001080 comp rx=0 tx=0).stop 2024-08-31T12:26:13.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.994+0000 7fd31a7fc640 1 -- 172.21.15.19:0/1465902485 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3241f02d0 msgr2=0x7fd3240ca610 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:26:13.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.994+0000 7fd31a7fc640 1 --2- 172.21.15.19:0/1465902485 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3241f02d0 0x7fd3240ca610 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:13.996 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:26:13.996 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:26:13.996 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:26:13.996 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:26:13.996 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 84, 2024-08-31T12:26:13.997 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 35280, 2024-08-31T12:26:13.997 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:26:13.997 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:26:13.997 INFO:teuthology.orchestra.run.smithi019.stdout: 1 2024-08-31T12:26:13.997 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:13.997 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 1, 2024-08-31T12:26:13.997 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:26:13.997 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:26:13.997 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:26:13.997 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:26:13.997 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:26:13.997 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:26:13.997 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:26:13.998 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:26:13.998 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:26:13.998 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:26:13.998 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:26:13.998 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:26:13.998 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:26:13.998 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:26:13.998 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:13.998 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:26:13.998 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:26:13.998 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:26:13.998 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:26:13.998 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:26:13.999 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:26:13.999 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:26:13.999 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:26:13.999 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:26:13.999 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:26:13.999 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:26:13.999 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:26:13.999 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:26:13.999 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:13.999 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:13.999 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:26:13.999 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:26:13.999 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:26:13.999 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:26:14.000 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:26:14.000 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:26:14.000 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:26:14.000 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:26:14.000 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:26:14.000 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:26:14.000 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:26:14.000 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:26:14.000 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:26:14.000 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:26:14.000 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:26:14.000 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:26:14.001 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:26:14.001 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:26:14.001 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:26:14.001 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:26:14.001 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:26:14.001 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:26:14.001 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:26:14.001 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:26:14.001 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:26:14.001 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:26:14.001 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:26:14.001 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:26:14.001 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:14.001 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:26:14.002 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:14.002 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:26:14.002 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:14.002 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:26:14.002 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:26:14.002 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:26:14.002 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:26:14.002 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:14.002 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:26:14.002 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:26:14.002 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:14.002 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:14.002 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:14.002 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:26:14.003 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:26:14.003 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:14.003 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:14.003 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:14.003 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:14.003 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:26:14.003 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:26:14.003 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:26:14.003 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:26:14.003 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:26:14.003 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:14.003 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:14.003 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:26:14.004 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:26:14.004 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:26:14.004 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:26:14.004 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:14.004 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:26:14.004 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:26:14.004 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:14.004 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:14.004 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:14.004 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:26:14.004 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:26:14.004 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:14.004 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:14.005 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:14.005 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:14.005 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:26:14.005 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:26:14.005 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:26:14.005 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:26:14.005 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:26:14.005 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:14.005 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:14.005 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:26:14.005 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:26:14.005 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:26:14.005 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:26:14.005 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:14.006 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:26:14.006 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:26:14.006 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:14.006 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:14.006 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:14.006 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:26:14.006 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:26:14.006 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:14.006 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:14.006 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:14.006 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:14.006 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:26:14.006 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:26:14.006 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:26:14.007 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:26:14.007 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:26:14.007 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:14.007 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:14.007 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:14.007 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:26:14.007 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:26:14.007 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:14.007 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:26:14.007 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:26:14.007 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:26:14.007 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:14.007 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:14.008 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:26:14.008 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:14.008 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:26:14.008 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:26:14.008 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:26:14.008 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:14.008 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:14.008 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:26:14.008 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:14.008 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:26:14.008 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:26:14.008 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:26:14.008 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:14.008 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:14.009 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:26:14.009 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:14.009 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:26:14.009 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:26:14.009 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 4 2024-08-31T12:26:14.009 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:14.009 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:14.009 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:14.009 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:26:14.009 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:26:14.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.995+0000 7fd32ad9d640 1 -- 172.21.15.19:0/1465902485 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd308077a30 msgr2=0x7fd308079f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:14.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.995+0000 7fd32ad9d640 1 --2- 172.21.15.19:0/1465902485 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd308077a30 0x7fd308079f40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd324147c40 tx=0x7fd31c00d040 comp rx=0 tx=0).stop 2024-08-31T12:26:14.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.995+0000 7fd32ad9d640 1 -- 172.21.15.19:0/1465902485 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3240c8c50 msgr2=0x7fd324142d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:14.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.995+0000 7fd32ad9d640 1 --2- 172.21.15.19:0/1465902485 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3240c8c50 0x7fd324142d70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd320009950 tx=0x7fd320033ce0 comp rx=0 tx=0).stop 2024-08-31T12:26:14.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.995+0000 7fd32a59c640 1 -- 172.21.15.19:0/1465902485 reap_dead start 2024-08-31T12:26:14.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.995+0000 7fd32a59c640 1 -- 172.21.15.19:0/1465902485 reap_dead start 2024-08-31T12:26:14.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.995+0000 7fd32ad9d640 1 -- 172.21.15.19:0/1465902485 shutdown_connections 2024-08-31T12:26:14.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.995+0000 7fd32ad9d640 1 -- 172.21.15.19:0/1465902485 >> 172.21.15.19:0/1465902485 conn(0x7fd324068ec0 msgr2=0x7fd3240cd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:26:14.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.995+0000 7fd32ad9d640 1 -- 172.21.15.19:0/1465902485 shutdown_connections 2024-08-31T12:26:14.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:13.995+0000 7fd32ad9d640 1 -- 172.21.15.19:0/1465902485 wait complete. 2024-08-31T12:26:14.013 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-08-31T12:26:14.013 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-08-31T12:26:27.832 DEBUG:teuthology.orchestra.run.smithi019:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:26:27.834 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:26:34.014 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-08-31T12:26:34.014 INFO:tasks.ceph.mon.c:Restarting daemon 2024-08-31T12:26:34.015 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-08-31T12:26:34.017 INFO:tasks.ceph.mon.c:Started 2024-08-31T12:26:34.017 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-08-31T12:26:34.018 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:26:34.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.187+0000 7f2815d39640 1 -- 172.21.15.19:0/3208845672 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2810071960 msgr2=0x7f2810071d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:34.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.187+0000 7f2815d39640 1 --2- 172.21.15.19:0/3208845672 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2810071960 0x7f2810071d70 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f28100c79d0 tx=0x7f280802a5c0 comp rx=0 tx=0).stop 2024-08-31T12:26:34.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.187+0000 7f2815d39640 1 -- 172.21.15.19:0/3208845672 shutdown_connections 2024-08-31T12:26:34.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.187+0000 7f2815d39640 1 --2- 172.21.15.19:0/3208845672 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2810071960 0x7f2810071d70 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:34.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.187+0000 7f2815d39640 1 -- 172.21.15.19:0/3208845672 >> 172.21.15.19:0/3208845672 conn(0x7f28100cd7d0 msgr2=0x7f28100cdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:26:34.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.187+0000 7f2815d39640 1 -- 172.21.15.19:0/3208845672 shutdown_connections 2024-08-31T12:26:34.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.187+0000 7f2815d39640 1 -- 172.21.15.19:0/3208845672 wait complete. 2024-08-31T12:26:34.191 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.190+0000 7f2815d39640 1 Processor -- start 2024-08-31T12:26:34.192 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.190+0000 7f2815d39640 1 -- start start 2024-08-31T12:26:34.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.192+0000 7f2815d39640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2810071960 0x7f28101f91f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:34.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.192+0000 7f2815d39640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f28100c72a0 0x7f28100cbaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:34.194 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.192+0000 7f2815d39640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f28100c7d60 0x7f28100bb380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:34.194 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.192+0000 7f2815d39640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f281006d7b0 con 0x7f28100c72a0 2024-08-31T12:26:34.194 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.192+0000 7f2815d39640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f281006cb80 con 0x7f2810071960 2024-08-31T12:26:34.194 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.192+0000 7f2815d39640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f2810158f00 con 0x7f28100c7d60 2024-08-31T12:26:34.194 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.192+0000 7f280f7fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2810071960 0x7f28101f91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:34.194 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.192+0000 7f280f7fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2810071960 0x7f28101f91f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:43794/0 (socket says 172.21.15.19:43794) 2024-08-31T12:26:34.194 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.192+0000 7f280f7fe640 1 -- 172.21.15.19:0/4213206420 learned_addr learned my addr 172.21.15.19:0/4213206420 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:26:34.194 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.192+0000 7f280effd640 1 --2- 172.21.15.19:0/4213206420 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f28100c72a0 0x7f28100cbaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:34.194 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.192+0000 7f280ffff640 1 --2- 172.21.15.19:0/4213206420 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f28100c7d60 0x7f28100bb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:34.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.193+0000 7f280ffff640 1 -- 172.21.15.19:0/4213206420 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f28100c7d60 msgr2=0x7f28100bb380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-08-31T12:26:34.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.193+0000 7f280ffff640 1 -- 172.21.15.19:0/4213206420 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f28100c7d60 msgr2=0x7f28100bb380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-31T12:26:34.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.193+0000 7f280ffff640 1 --2- 172.21.15.19:0/4213206420 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f28100c7d60 0x7f28100bb380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-31T12:26:34.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.193+0000 7f280ffff640 1 --2- 172.21.15.19:0/4213206420 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f28100c7d60 0x7f28100bb380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:26:34.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.193+0000 7f280f7fe640 1 -- 172.21.15.19:0/4213206420 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f28100c7d60 msgr2=0x7f28100bb380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:26:34.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.193+0000 7f280f7fe640 1 --2- 172.21.15.19:0/4213206420 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f28100c7d60 0x7f28100bb380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:34.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.193+0000 7f280f7fe640 1 -- 172.21.15.19:0/4213206420 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f28100c72a0 msgr2=0x7f28100cbaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:34.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.193+0000 7f280f7fe640 1 --2- 172.21.15.19:0/4213206420 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f28100c72a0 0x7f28100cbaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:34.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.193+0000 7f280f7fe640 1 -- 172.21.15.19:0/4213206420 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2810155480 con 0x7f2810071960 2024-08-31T12:26:34.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.193+0000 7f280f7fe640 1 --2- 172.21.15.19:0/4213206420 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2810071960 0x7f28101f91f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f280000c420 tx=0x7f280000c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:34.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.193+0000 7f2807fff640 1 -- 172.21.15.19:0/4213206420 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28000052d0 con 0x7f2810071960 2024-08-31T12:26:34.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.194+0000 7f2807fff640 1 -- 172.21.15.19:0/4213206420 <== mon.1 v2:172.21.15.43:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2800007020 con 0x7f2810071960 2024-08-31T12:26:34.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.194+0000 7f2807fff640 1 -- 172.21.15.19:0/4213206420 <== mon.1 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2800007470 con 0x7f2810071960 2024-08-31T12:26:34.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.194+0000 7f2815d39640 1 -- 172.21.15.19:0/4213206420 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28101f9730 con 0x7f2810071960 2024-08-31T12:26:34.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.194+0000 7f2815d39640 1 -- 172.21.15.19:0/4213206420 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28101551a0 con 0x7f2810071960 2024-08-31T12:26:34.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.195+0000 7f2815d39640 1 -- 172.21.15.19:0/4213206420 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28101f8e30 con 0x7f2810071960 2024-08-31T12:26:34.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.196+0000 7f2807fff640 1 -- 172.21.15.19:0/4213206420 <== mon.1 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f28000067d0 con 0x7f2810071960 2024-08-31T12:26:34.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.197+0000 7f2807fff640 1 --2- 172.21.15.19:0/4213206420 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f27f8077a50 0x7f27f8079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:34.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.197+0000 7f2807fff640 1 -- 172.21.15.19:0/4213206420 <== mon.1 v2:172.21.15.43:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f2800098b40 con 0x7f2810071960 2024-08-31T12:26:34.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.197+0000 7f280effd640 1 --2- 172.21.15.19:0/4213206420 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f27f8077a50 0x7f27f8079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:34.200 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.199+0000 7f280effd640 1 --2- 172.21.15.19:0/4213206420 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f27f8077a50 0x7f27f8079f60 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f27fc0017a0 tx=0x7f27fc001730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:34.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.204+0000 7f2807fff640 1 -- 172.21.15.19:0/4213206420 <== mon.1 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f28000611e0 con 0x7f2810071960 2024-08-31T12:26:34.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.651+0000 7f2815d39640 1 -- 172.21.15.19:0/4213206420 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f28100c4350 con 0x7f2810071960 2024-08-31T12:26:34.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.651+0000 7f2807fff640 1 -- 172.21.15.19:0/4213206420 <== mon.1 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f28000659a0 con 0x7f2810071960 2024-08-31T12:26:34.653 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":84,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:26:34.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.654+0000 7f2815d39640 1 -- 172.21.15.19:0/4213206420 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f27f8077a50 msgr2=0x7f27f8079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:34.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.655+0000 7f2815d39640 1 --2- 172.21.15.19:0/4213206420 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f27f8077a50 0x7f27f8079f60 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f27fc0017a0 tx=0x7f27fc001730 comp rx=0 tx=0).stop 2024-08-31T12:26:34.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.655+0000 7f2815d39640 1 -- 172.21.15.19:0/4213206420 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2810071960 msgr2=0x7f28101f91f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:34.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.655+0000 7f2815d39640 1 --2- 172.21.15.19:0/4213206420 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2810071960 0x7f28101f91f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f280000c420 tx=0x7f280000c8e0 comp rx=0 tx=0).stop 2024-08-31T12:26:34.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.655+0000 7f2815d39640 1 -- 172.21.15.19:0/4213206420 shutdown_connections 2024-08-31T12:26:34.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.655+0000 7f2815d39640 1 --2- 172.21.15.19:0/4213206420 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f27f8077a50 0x7f27f8079f60 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:34.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.655+0000 7f2815d39640 1 --2- 172.21.15.19:0/4213206420 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f28100c7d60 0x7f28100bb380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:34.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.655+0000 7f2815d39640 1 --2- 172.21.15.19:0/4213206420 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f28100c72a0 0x7f28100cbaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:34.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.655+0000 7f2815d39640 1 --2- 172.21.15.19:0/4213206420 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2810071960 0x7f28101f91f0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:34.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.655+0000 7f2815d39640 1 -- 172.21.15.19:0/4213206420 >> 172.21.15.19:0/4213206420 conn(0x7f28100cd7d0 msgr2=0x7f28101586a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:26:34.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.655+0000 7f2815d39640 1 -- 172.21.15.19:0/4213206420 shutdown_connections 2024-08-31T12:26:34.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:34.655+0000 7f2815d39640 1 -- 172.21.15.19:0/4213206420 wait complete. 2024-08-31T12:26:37.672 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:26:37.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:37.844+0000 7ff2f5a0e640 1 -- 172.21.15.19:0/205108925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff2f00caa20 msgr2=0x7ff2f0070ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:37.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:37.844+0000 7ff2f5a0e640 1 --2- 172.21.15.19:0/205108925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff2f00caa20 0x7ff2f0070ad0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff2f0074b80 tx=0x7ff2e402a4d0 comp rx=0 tx=0).stop 2024-08-31T12:26:37.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:37.844+0000 7ff2f5a0e640 1 -- 172.21.15.19:0/205108925 shutdown_connections 2024-08-31T12:26:37.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:37.844+0000 7ff2f5a0e640 1 --2- 172.21.15.19:0/205108925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff2f00caa20 0x7ff2f0070ad0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:37.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:37.844+0000 7ff2f5a0e640 1 --2- 172.21.15.19:0/205108925 >> v2:172.21.15.19:3300/0 conn(0x7ff2f0153230 0x7ff2f0071b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:37.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:37.844+0000 7ff2f5a0e640 1 -- 172.21.15.19:0/205108925 >> 172.21.15.19:0/205108925 conn(0x7ff2f0072c80 msgr2=0x7ff2f0073090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:26:37.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:37.844+0000 7ff2f5a0e640 1 -- 172.21.15.19:0/205108925 shutdown_connections 2024-08-31T12:26:37.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:37.844+0000 7ff2f5a0e640 1 -- 172.21.15.19:0/205108925 wait complete. 2024-08-31T12:26:37.847 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:37.846+0000 7ff2f5a0e640 1 Processor -- start 2024-08-31T12:26:37.847 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:37.847+0000 7ff2f5a0e640 1 -- start start 2024-08-31T12:26:37.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:37.848+0000 7ff2f5a0e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff2f0071010 0x7ff2f015bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:37.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:37.848+0000 7ff2f5a0e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff2f00caa20 0x7ff2f01f20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:37.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:37.848+0000 7ff2f5a0e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff2f0153230 0x7ff2f00b10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:37.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:37.848+0000 7ff2f5a0e640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7ff2f00721e0 con 0x7ff2f00caa20 2024-08-31T12:26:37.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:37.848+0000 7ff2f5a0e640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff2f00cae30 con 0x7ff2f0153230 2024-08-31T12:26:37.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:37.848+0000 7ff2f5a0e640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7ff2f0072360 con 0x7ff2f0071010 2024-08-31T12:26:37.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:37.848+0000 7ff2ee7fc640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff2f00caa20 0x7ff2f01f20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:37.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:37.848+0000 7ff2eeffd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff2f0071010 0x7ff2f015bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:37.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:37.848+0000 7ff2eeffd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff2f0071010 0x7ff2f015bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.19:55062/0 (socket says 172.21.15.19:55062) 2024-08-31T12:26:37.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:37.848+0000 7ff2eeffd640 1 -- 172.21.15.19:0/1677673058 learned_addr learned my addr 172.21.15.19:0/1677673058 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:26:37.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:37.848+0000 7ff2ef7fe640 1 --2- 172.21.15.19:0/1677673058 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff2f0153230 0x7ff2f00b10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:37.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:37.849+0000 7ff2eeffd640 1 -- 172.21.15.19:0/1677673058 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff2f0071010 msgr2=0x7ff2f015bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-08-31T12:26:37.852 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:37.849+0000 7ff2eeffd640 1 -- 172.21.15.19:0/1677673058 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff2f0071010 msgr2=0x7ff2f015bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-31T12:26:37.852 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:37.849+0000 7ff2eeffd640 1 --2- 172.21.15.19:0/1677673058 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff2f0071010 0x7ff2f015bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-31T12:26:37.852 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:37.849+0000 7ff2eeffd640 1 --2- 172.21.15.19:0/1677673058 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff2f0071010 0x7ff2f015bab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:26:37.852 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:37.849+0000 7ff2ee7fc640 1 -- 172.21.15.19:0/1677673058 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff2f0071010 msgr2=0x7ff2f015bab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:26:37.852 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:37.849+0000 7ff2ee7fc640 1 --2- 172.21.15.19:0/1677673058 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff2f0071010 0x7ff2f015bab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:37.852 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:37.849+0000 7ff2ee7fc640 1 -- 172.21.15.19:0/1677673058 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff2f0153230 msgr2=0x7ff2f00b10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:37.852 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:37.849+0000 7ff2ee7fc640 1 --2- 172.21.15.19:0/1677673058 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff2f0153230 0x7ff2f00b10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:37.852 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:37.849+0000 7ff2ee7fc640 1 -- 172.21.15.19:0/1677673058 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2f01f25f0 con 0x7ff2f00caa20 2024-08-31T12:26:37.852 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:37.849+0000 7ff2ee7fc640 1 --2- 172.21.15.19:0/1677673058 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff2f00caa20 0x7ff2f01f20b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff2e4031000 tx=0x7ff2e4033f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:37.853 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:37.849+0000 7ff2d7fff640 1 -- 172.21.15.19:0/1677673058 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2e4031d60 con 0x7ff2f00caa20 2024-08-31T12:26:37.853 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:37.850+0000 7ff2d7fff640 1 -- 172.21.15.19:0/1677673058 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff2e4005020 con 0x7ff2f00caa20 2024-08-31T12:26:37.853 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:37.850+0000 7ff2d7fff640 1 -- 172.21.15.19:0/1677673058 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2e4005470 con 0x7ff2f00caa20 2024-08-31T12:26:37.853 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:37.850+0000 7ff2f5a0e640 1 -- 172.21.15.19:0/1677673058 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2f006dbf0 con 0x7ff2f00caa20 2024-08-31T12:26:37.853 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:37.850+0000 7ff2f5a0e640 1 -- 172.21.15.19:0/1677673058 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2f006de90 con 0x7ff2f00caa20 2024-08-31T12:26:37.853 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:37.852+0000 7ff2f5a0e640 1 -- 172.21.15.19:0/1677673058 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2f00ca110 con 0x7ff2f00caa20 2024-08-31T12:26:37.854 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:37.853+0000 7ff2d7fff640 1 -- 172.21.15.19:0/1677673058 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7ff2e40047a0 con 0x7ff2f00caa20 2024-08-31T12:26:37.855 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:37.854+0000 7ff2d7fff640 1 --2- 172.21.15.19:0/1677673058 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff2dc077ad0 0x7ff2dc079fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:37.856 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:37.854+0000 7ff2d7fff640 1 -- 172.21.15.19:0/1677673058 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7ff2e40ba700 con 0x7ff2f00caa20 2024-08-31T12:26:37.856 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:37.855+0000 7ff2eeffd640 1 --2- 172.21.15.19:0/1677673058 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff2dc077ad0 0x7ff2dc079fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:37.856 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:37.855+0000 7ff2eeffd640 1 --2- 172.21.15.19:0/1677673058 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff2dc077ad0 0x7ff2dc079fe0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff2e0006fd0 tx=0x7ff2e0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:37.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:37.862+0000 7ff2d7fff640 1 -- 172.21.15.19:0/1677673058 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7ff2e403f030 con 0x7ff2f00caa20 2024-08-31T12:26:38.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:38.317+0000 7ff2f5a0e640 1 -- 172.21.15.19:0/1677673058 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff2f0158030 con 0x7ff2f00caa20 2024-08-31T12:26:38.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:38.318+0000 7ff2d7fff640 1 -- 172.21.15.19:0/1677673058 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7ff2e4031770 con 0x7ff2f00caa20 2024-08-31T12:26:38.319 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":84,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":25,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:26:38.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:38.321+0000 7ff2f5a0e640 1 -- 172.21.15.19:0/1677673058 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff2dc077ad0 msgr2=0x7ff2dc079fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:38.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:38.321+0000 7ff2f5a0e640 1 --2- 172.21.15.19:0/1677673058 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff2dc077ad0 0x7ff2dc079fe0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff2e0006fd0 tx=0x7ff2e0008040 comp rx=0 tx=0).stop 2024-08-31T12:26:38.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:38.321+0000 7ff2f5a0e640 1 -- 172.21.15.19:0/1677673058 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff2f00caa20 msgr2=0x7ff2f01f20b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:38.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:38.321+0000 7ff2f5a0e640 1 --2- 172.21.15.19:0/1677673058 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff2f00caa20 0x7ff2f01f20b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff2e4031000 tx=0x7ff2e4033f90 comp rx=0 tx=0).stop 2024-08-31T12:26:38.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:38.321+0000 7ff2f5a0e640 1 -- 172.21.15.19:0/1677673058 shutdown_connections 2024-08-31T12:26:38.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:38.321+0000 7ff2f5a0e640 1 --2- 172.21.15.19:0/1677673058 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff2dc077ad0 0x7ff2dc079fe0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:38.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:38.321+0000 7ff2f5a0e640 1 --2- 172.21.15.19:0/1677673058 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff2f0153230 0x7ff2f00b10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:38.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:38.321+0000 7ff2f5a0e640 1 --2- 172.21.15.19:0/1677673058 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff2f00caa20 0x7ff2f01f20b0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:38.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:38.321+0000 7ff2f5a0e640 1 --2- 172.21.15.19:0/1677673058 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff2f0071010 0x7ff2f015bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:38.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:38.321+0000 7ff2f5a0e640 1 -- 172.21.15.19:0/1677673058 >> 172.21.15.19:0/1677673058 conn(0x7ff2f0072c80 msgr2=0x7ff2f00c3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:26:38.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:38.322+0000 7ff2f5a0e640 1 -- 172.21.15.19:0/1677673058 shutdown_connections 2024-08-31T12:26:38.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:38.322+0000 7ff2f5a0e640 1 -- 172.21.15.19:0/1677673058 wait complete. 2024-08-31T12:26:41.341 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:26:41.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.517+0000 7f2c92ed3640 1 -- 172.21.15.19:0/3020332548 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2c8c0cc260 msgr2=0x7f2c8c0cc670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:41.519 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.517+0000 7f2c92ed3640 1 --2- 172.21.15.19:0/3020332548 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2c8c0cc260 0x7f2c8c0cc670 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c071d30 tx=0x7f2c8402aa30 comp rx=0 tx=0).stop 2024-08-31T12:26:41.519 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.517+0000 7f2c92ed3640 1 -- 172.21.15.19:0/3020332548 shutdown_connections 2024-08-31T12:26:41.519 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.517+0000 7f2c92ed3640 1 --2- 172.21.15.19:0/3020332548 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2c8c0cc260 0x7f2c8c0cc670 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:41.519 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.517+0000 7f2c92ed3640 1 --2- 172.21.15.19:0/3020332548 >> v2:172.21.15.19:3300/0 conn(0x7f2c8c06f080 0x7f2c8c15b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:41.519 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.517+0000 7f2c92ed3640 1 -- 172.21.15.19:0/3020332548 >> 172.21.15.19:0/3020332548 conn(0x7f2c8c06cae0 msgr2=0x7f2c8c06cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:26:41.519 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.517+0000 7f2c92ed3640 1 -- 172.21.15.19:0/3020332548 shutdown_connections 2024-08-31T12:26:41.519 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.517+0000 7f2c92ed3640 1 -- 172.21.15.19:0/3020332548 wait complete. 2024-08-31T12:26:41.521 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.520+0000 7f2c92ed3640 1 Processor -- start 2024-08-31T12:26:41.521 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.520+0000 7f2c92ed3640 1 -- start start 2024-08-31T12:26:41.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.522+0000 7f2c92ed3640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2c8c06f080 0x7f2c8c14f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:41.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.522+0000 7f2c92ed3640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2c8c070140 0x7f2c8c1e6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:41.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.522+0000 7f2c92ed3640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2c8c0cc260 0x7f2c8c14c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:41.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.522+0000 7f2c92ed3640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f2c8c0cb3e0 con 0x7f2c8c070140 2024-08-31T12:26:41.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.522+0000 7f2c92ed3640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2c8c0cb260 con 0x7f2c8c06f080 2024-08-31T12:26:41.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.522+0000 7f2c92ed3640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f2c8c0cb560 con 0x7f2c8c0cc260 2024-08-31T12:26:41.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.522+0000 7f2c8bfff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2c8c070140 0x7f2c8c1e6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:41.524 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.522+0000 7f2c91470640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2c8c0cc260 0x7f2c8c14c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:41.524 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.522+0000 7f2c8bfff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2c8c070140 0x7f2c8c1e6640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:47960/0 (socket says 172.21.15.19:47960) 2024-08-31T12:26:41.524 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.522+0000 7f2c8bfff640 1 -- 172.21.15.19:0/3873155084 learned_addr learned my addr 172.21.15.19:0/3873155084 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:26:41.524 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.522+0000 7f2c90c6f640 1 --2- 172.21.15.19:0/3873155084 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2c8c06f080 0x7f2c8c14f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:41.525 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.522+0000 7f2c91470640 1 -- 172.21.15.19:0/3873155084 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2c8c06f080 msgr2=0x7f2c8c14f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:41.525 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.522+0000 7f2c91470640 1 --2- 172.21.15.19:0/3873155084 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2c8c06f080 0x7f2c8c14f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:41.525 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.522+0000 7f2c91470640 1 -- 172.21.15.19:0/3873155084 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2c8c070140 msgr2=0x7f2c8c1e6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:41.525 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.522+0000 7f2c91470640 1 --2- 172.21.15.19:0/3873155084 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2c8c070140 0x7f2c8c1e6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:41.525 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.522+0000 7f2c91470640 1 -- 172.21.15.19:0/3873155084 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c8c0c7240 con 0x7f2c8c0cc260 2024-08-31T12:26:41.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.523+0000 7f2c91470640 1 --2- 172.21.15.19:0/3873155084 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2c8c0cc260 0x7f2c8c14c800 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2c84032000 tx=0x7f2c84003f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:41.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.523+0000 7f2c897fa640 1 -- 172.21.15.19:0/3873155084 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c84033050 con 0x7f2c8c0cc260 2024-08-31T12:26:41.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.523+0000 7f2c92ed3640 1 -- 172.21.15.19:0/3873155084 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c8c0c6f60 con 0x7f2c8c0cc260 2024-08-31T12:26:41.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.523+0000 7f2c897fa640 1 -- 172.21.15.19:0/3873155084 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2c84004450 con 0x7f2c8c0cc260 2024-08-31T12:26:41.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.523+0000 7f2c897fa640 1 -- 172.21.15.19:0/3873155084 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c84004800 con 0x7f2c8c0cc260 2024-08-31T12:26:41.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.524+0000 7f2c92ed3640 1 -- 172.21.15.19:0/3873155084 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c8c1543b0 con 0x7f2c8c0cc260 2024-08-31T12:26:41.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.525+0000 7f2c92ed3640 1 -- 172.21.15.19:0/3873155084 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c8c0cb980 con 0x7f2c8c0cc260 2024-08-31T12:26:41.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.526+0000 7f2c897fa640 1 -- 172.21.15.19:0/3873155084 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f2c84036070 con 0x7f2c8c0cc260 2024-08-31T12:26:41.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.526+0000 7f2c897fa640 1 --2- 172.21.15.19:0/3873155084 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2c78077a50 0x7f2c78079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:41.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.527+0000 7f2c897fa640 1 -- 172.21.15.19:0/3873155084 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f2c840bab30 con 0x7f2c8c0cc260 2024-08-31T12:26:41.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.527+0000 7f2c90c6f640 1 --2- 172.21.15.19:0/3873155084 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2c78077a50 0x7f2c78079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:41.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.527+0000 7f2c90c6f640 1 --2- 172.21.15.19:0/3873155084 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2c78077a50 0x7f2c78079f60 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c009b30 tx=0x7f2c7c005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:41.536 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.535+0000 7f2c897fa640 1 -- 172.21.15.19:0/3873155084 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f2c840831e0 con 0x7f2c8c0cc260 2024-08-31T12:26:41.992 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.991+0000 7f2c92ed3640 1 -- 172.21.15.19:0/3873155084 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2c8c0c6cc0 con 0x7f2c8c0cc260 2024-08-31T12:26:41.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.992+0000 7f2c897fa640 1 -- 172.21.15.19:0/3873155084 <== mon.2 v2:172.21.15.43:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f2c840879a0 con 0x7f2c8c0cc260 2024-08-31T12:26:41.993 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":86,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:26:41.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.995+0000 7f2c92ed3640 1 -- 172.21.15.19:0/3873155084 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2c78077a50 msgr2=0x7f2c78079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:41.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.995+0000 7f2c92ed3640 1 --2- 172.21.15.19:0/3873155084 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2c78077a50 0x7f2c78079f60 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c009b30 tx=0x7f2c7c005f90 comp rx=0 tx=0).stop 2024-08-31T12:26:41.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.995+0000 7f2c92ed3640 1 -- 172.21.15.19:0/3873155084 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2c8c0cc260 msgr2=0x7f2c8c14c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:41.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.995+0000 7f2c92ed3640 1 --2- 172.21.15.19:0/3873155084 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2c8c0cc260 0x7f2c8c14c800 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2c84032000 tx=0x7f2c84003f10 comp rx=0 tx=0).stop 2024-08-31T12:26:41.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.995+0000 7f2c92ed3640 1 -- 172.21.15.19:0/3873155084 shutdown_connections 2024-08-31T12:26:41.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.995+0000 7f2c92ed3640 1 --2- 172.21.15.19:0/3873155084 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2c78077a50 0x7f2c78079f60 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:41.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.995+0000 7f2c92ed3640 1 --2- 172.21.15.19:0/3873155084 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2c8c0cc260 0x7f2c8c14c800 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:41.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.995+0000 7f2c92ed3640 1 --2- 172.21.15.19:0/3873155084 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2c8c070140 0x7f2c8c1e6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:41.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.995+0000 7f2c92ed3640 1 --2- 172.21.15.19:0/3873155084 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2c8c06f080 0x7f2c8c14f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:41.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.995+0000 7f2c92ed3640 1 -- 172.21.15.19:0/3873155084 >> 172.21.15.19:0/3873155084 conn(0x7f2c8c06cae0 msgr2=0x7f2c8c06cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:26:41.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.995+0000 7f2c92ed3640 1 -- 172.21.15.19:0/3873155084 shutdown_connections 2024-08-31T12:26:41.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:41.995+0000 7f2c92ed3640 1 -- 172.21.15.19:0/3873155084 wait complete. 2024-08-31T12:26:42.012 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-08-31T12:26:42.012 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-08-31T12:26:42.012 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-31T12:26:42.186 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.184+0000 7fc0f7e15640 1 -- 172.21.15.19:0/1127127986 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0f0072f30 msgr2=0x7fc0f0151190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:42.186 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.184+0000 7fc0f7e15640 1 --2- 172.21.15.19:0/1127127986 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0f0072f30 0x7fc0f0151190 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc0f0072ec0 tx=0x7fc0e002a460 comp rx=0 tx=0).stop 2024-08-31T12:26:42.186 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.184+0000 7fc0f7e15640 1 -- 172.21.15.19:0/1127127986 shutdown_connections 2024-08-31T12:26:42.186 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.184+0000 7fc0f7e15640 1 --2- 172.21.15.19:0/1127127986 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc0f00c93e0 0x7fc0f00c97f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:42.186 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.184+0000 7fc0f7e15640 1 --2- 172.21.15.19:0/1127127986 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0f0072f30 0x7fc0f0151190 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:42.186 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.184+0000 7fc0f7e15640 1 --2- 172.21.15.19:0/1127127986 >> v2:172.21.15.19:3300/0 conn(0x7fc0f006ad70 0x7fc0f00c4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:42.186 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.184+0000 7fc0f7e15640 1 -- 172.21.15.19:0/1127127986 >> 172.21.15.19:0/1127127986 conn(0x7fc0f0071100 msgr2=0x7fc0f006cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:26:42.186 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.185+0000 7fc0f7e15640 1 -- 172.21.15.19:0/1127127986 shutdown_connections 2024-08-31T12:26:42.187 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.185+0000 7fc0f7e15640 1 -- 172.21.15.19:0/1127127986 wait complete. 2024-08-31T12:26:42.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.187+0000 7fc0f7e15640 1 Processor -- start 2024-08-31T12:26:42.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.187+0000 7fc0f7e15640 1 -- start start 2024-08-31T12:26:42.191 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.189+0000 7fc0f7e15640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc0f006ad70 0x7fc0f014ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:42.191 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.189+0000 7fc0f7e15640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc0f0072f30 0x7fc0f00cc900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:42.191 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.189+0000 7fc0f7e15640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0f00c93e0 0x7fc0f00c5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:42.191 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.189+0000 7fc0f7e15640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f0074180 con 0x7fc0f0072f30 2024-08-31T12:26:42.191 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.189+0000 7fc0f7e15640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f0073ea0 con 0x7fc0f00c93e0 2024-08-31T12:26:42.191 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.189+0000 7fc0f7e15640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fc0f01516d0 con 0x7fc0f006ad70 2024-08-31T12:26:42.191 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.189+0000 7fc0f5bb1640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc0f006ad70 0x7fc0f014ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:42.191 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.190+0000 7fc0f5bb1640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc0f006ad70 0x7fc0f014ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.19:35726/0 (socket says 172.21.15.19:35726) 2024-08-31T12:26:42.191 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.190+0000 7fc0f5bb1640 1 -- 172.21.15.19:0/1465634122 learned_addr learned my addr 172.21.15.19:0/1465634122 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:26:42.191 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.190+0000 7fc0f53b0640 1 --2- 172.21.15.19:0/1465634122 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc0f0072f30 0x7fc0f00cc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:42.191 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.190+0000 7fc0f63b2640 1 --2- 172.21.15.19:0/1465634122 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0f00c93e0 0x7fc0f00c5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:42.192 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.190+0000 7fc0f5bb1640 1 -- 172.21.15.19:0/1465634122 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0f00c93e0 msgr2=0x7fc0f00c5cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:42.192 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.190+0000 7fc0f5bb1640 1 --2- 172.21.15.19:0/1465634122 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0f00c93e0 0x7fc0f00c5cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:42.192 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.190+0000 7fc0f5bb1640 1 -- 172.21.15.19:0/1465634122 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc0f0072f30 msgr2=0x7fc0f00cc900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:42.192 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.190+0000 7fc0f5bb1640 1 --2- 172.21.15.19:0/1465634122 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc0f0072f30 0x7fc0f00cc900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:42.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.190+0000 7fc0f5bb1640 1 -- 172.21.15.19:0/1465634122 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0f014b240 con 0x7fc0f006ad70 2024-08-31T12:26:42.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.190+0000 7fc0f53b0640 1 --2- 172.21.15.19:0/1465634122 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc0f0072f30 0x7fc0f00cc900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:26:42.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.190+0000 7fc0f5bb1640 1 --2- 172.21.15.19:0/1465634122 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc0f006ad70 0x7fc0f014ad00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec00b410 tx=0x7fc0ec00b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:42.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.191+0000 7fc0e67fc640 1 -- 172.21.15.19:0/1465634122 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0ec015020 con 0x7fc0f006ad70 2024-08-31T12:26:42.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.191+0000 7fc0f7e15640 1 -- 172.21.15.19:0/1465634122 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0f00cce40 con 0x7fc0f006ad70 2024-08-31T12:26:42.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.191+0000 7fc0e67fc640 1 -- 172.21.15.19:0/1465634122 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc0ec011920 con 0x7fc0f006ad70 2024-08-31T12:26:42.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.191+0000 7fc0f7e15640 1 -- 172.21.15.19:0/1465634122 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0f0149170 con 0x7fc0f006ad70 2024-08-31T12:26:42.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.191+0000 7fc0e67fc640 1 -- 172.21.15.19:0/1465634122 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0ec015020 con 0x7fc0f006ad70 2024-08-31T12:26:42.194 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.192+0000 7fc0f7e15640 1 --2- 172.21.15.19:0/1465634122 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc0f0148c10 0x7fc0f0152900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:42.194 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.192+0000 7fc0f7e15640 1 -- 172.21.15.19:0/1465634122 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f00c39e0 con 0x7fc0f0148c10 2024-08-31T12:26:42.194 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.192+0000 7fc0f53b0640 1 --2- 172.21.15.19:0/1465634122 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc0f0148c10 0x7fc0f0152900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:42.194 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.193+0000 7fc0f53b0640 1 -- 172.21.15.19:0/1465634122 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc0f00c4940 con 0x7fc0f0148c10 2024-08-31T12:26:42.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.193+0000 7fc0f53b0640 1 --2- 172.21.15.19:0/1465634122 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc0f0148c10 0x7fc0f0152900 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc0f0072820 tx=0x7fc0e0003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:42.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.194+0000 7fc0e67fc640 1 -- 172.21.15.19:0/1465634122 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e0040070 con 0x7fc0f0148c10 2024-08-31T12:26:42.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.194+0000 7fc0e67fc640 1 -- 172.21.15.19:0/1465634122 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fc0ec018020 con 0x7fc0f006ad70 2024-08-31T12:26:42.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.195+0000 7fc0e67fc640 1 --2- 172.21.15.19:0/1465634122 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc0d4077cf0 0x7fc0d407a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:42.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.195+0000 7fc0e67fc640 1 -- 172.21.15.19:0/1465634122 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fc0ec011630 con 0x7fc0f006ad70 2024-08-31T12:26:42.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.196+0000 7fc0f63b2640 1 --2- 172.21.15.19:0/1465634122 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc0d4077cf0 0x7fc0d407a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:42.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.196+0000 7fc0e67fc640 1 -- 172.21.15.19:0/1465634122 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc0e00049a0 con 0x7fc0f0148c10 2024-08-31T12:26:42.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.196+0000 7fc0e67fc640 1 -- 172.21.15.19:0/1465634122 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc0f0148c10 msgr2=0x7fc0f0152900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:42.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.196+0000 7fc0e67fc640 1 --2- 172.21.15.19:0/1465634122 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc0f0148c10 0x7fc0f0152900 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc0f0072820 tx=0x7fc0e0003b90 comp rx=0 tx=0).stop 2024-08-31T12:26:42.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.196+0000 7fc0e67fc640 1 -- 172.21.15.19:0/1465634122 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc0f0148c10 msgr2=0x7fc0f0152900 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:26:42.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.196+0000 7fc0e67fc640 1 --2- 172.21.15.19:0/1465634122 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc0f0148c10 0x7fc0f0152900 secure :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc0f0072820 tx=0x7fc0e0003b90 comp rx=0 tx=0).stop 2024-08-31T12:26:42.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.196+0000 7fc0f63b2640 1 --2- 172.21.15.19:0/1465634122 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc0d4077cf0 0x7fc0d407a200 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc0e80092f0 tx=0x7fc0e80095b0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:42.209 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.208+0000 7fc0f7e15640 1 --2- 172.21.15.19:0/1465634122 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc0f0152e40 0x7fc0f0153250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:42.209 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.208+0000 7fc0f7e15640 1 -- 172.21.15.19:0/1465634122 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f0153790 con 0x7fc0f0152e40 2024-08-31T12:26:42.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.209+0000 7fc0f53b0640 1 --2- 172.21.15.19:0/1465634122 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc0f0152e40 0x7fc0f0153250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:42.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.209+0000 7fc0f53b0640 1 -- 172.21.15.19:0/1465634122 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc0f014a030 con 0x7fc0f0152e40 2024-08-31T12:26:42.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.209+0000 7fc0f53b0640 1 --2- 172.21.15.19:0/1465634122 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc0f0152e40 0x7fc0f0153250 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc0e0034040 tx=0x7fc0e0009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:42.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.210+0000 7fc0e67fc640 1 -- 172.21.15.19:0/1465634122 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e0035040 con 0x7fc0f0152e40 2024-08-31T12:26:42.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.210+0000 7fc0e67fc640 1 -- 172.21.15.19:0/1465634122 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5044 (secure 0 0 0) 0x7fc0e003c440 con 0x7fc0f0152e40 2024-08-31T12:26:42.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.210+0000 7fc0e67fc640 1 -- 172.21.15.19:0/1465634122 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc0f0152e40 msgr2=0x7fc0f0153250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:42.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.210+0000 7fc0e67fc640 1 --2- 172.21.15.19:0/1465634122 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc0f0152e40 0x7fc0f0153250 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc0e0034040 tx=0x7fc0e0009260 comp rx=0 tx=0).stop 2024-08-31T12:26:42.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.210+0000 7fc0e67fc640 1 -- 172.21.15.19:0/1465634122 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc0f0152e40 msgr2=0x7fc0f0153250 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:26:42.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.210+0000 7fc0e67fc640 1 --2- 172.21.15.19:0/1465634122 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc0f0152e40 0x7fc0f0153250 secure :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc0e0034040 tx=0x7fc0e0009260 comp rx=0 tx=0).stop 2024-08-31T12:26:42.212 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:26:42.212 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:26:42.212 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:26:42.212 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "leader", 2024-08-31T12:26:42.212 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 86, 2024-08-31T12:26:42.212 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 136764, 2024-08-31T12:26:42.213 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:26:42.213 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:26:42.213 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:26:42.213 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:26:42.213 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:42.213 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 3, 2024-08-31T12:26:42.213 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:26:42.213 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:26:42.213 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:26:42.213 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:26:42.213 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:26:42.213 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:26:42.213 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:26:42.214 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:26:42.214 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:26:42.214 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:26:42.214 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:26:42.214 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:26:42.214 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:26:42.214 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:26:42.214 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:42.214 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:26:42.214 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:26:42.214 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:26:42.214 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:26:42.214 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:26:42.214 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:26:42.214 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:26:42.215 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:26:42.215 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:26:42.215 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:26:42.215 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:26:42.215 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:26:42.215 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:26:42.215 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:42.215 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:42.215 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:26:42.215 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:26:42.215 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:26:42.215 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:26:42.215 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:26:42.215 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:26:42.216 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:26:42.216 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:26:42.216 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:26:42.216 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:26:42.216 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:26:42.216 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:26:42.216 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:26:42.216 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:26:42.216 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:26:42.216 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:26:42.216 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:26:42.216 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:26:42.216 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:26:42.216 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:26:42.217 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:26:42.217 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:26:42.217 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:26:42.217 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:26:42.217 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:26:42.217 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:26:42.217 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:26:42.217 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:26:42.217 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:42.217 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:26:42.217 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:42.217 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:26:42.217 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:42.217 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:26:42.218 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:26:42.218 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:26:42.218 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:26:42.218 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:42.218 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:26:42.218 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:26:42.218 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:42.218 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:42.218 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:42.218 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:26:42.218 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:26:42.218 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:42.218 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:42.218 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:42.219 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:42.219 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:26:42.219 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:26:42.219 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:26:42.219 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:26:42.219 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:26:42.219 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:42.219 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:42.219 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:26:42.219 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:26:42.219 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:26:42.219 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:26:42.219 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:42.219 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:26:42.219 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:26:42.220 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:42.220 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:42.220 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:42.220 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:26:42.220 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:26:42.220 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:42.220 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:42.220 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:42.220 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:42.220 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:26:42.220 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:26:42.220 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:26:42.220 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:26:42.221 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:26:42.221 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:42.221 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:42.221 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:26:42.221 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:26:42.221 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:26:42.221 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:26:42.221 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:42.221 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:26:42.221 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:26:42.221 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:42.221 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:42.221 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:42.221 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:26:42.222 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:26:42.222 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:42.222 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:42.222 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:42.222 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:42.222 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:26:42.222 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:26:42.222 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:26:42.222 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:26:42.222 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:26:42.222 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:42.222 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:42.222 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:42.222 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:26:42.222 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:26:42.222 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:42.223 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:26:42.223 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:26:42.223 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:26:42.223 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:42.223 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:42.223 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:26:42.223 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:42.223 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:26:42.223 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:26:42.223 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 4 2024-08-31T12:26:42.223 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:42.223 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:42.223 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:26:42.223 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:42.224 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:26:42.224 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:26:42.224 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 6 2024-08-31T12:26:42.224 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:42.224 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:42.224 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:26:42.224 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:42.224 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:26:42.224 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:26:42.224 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 4 2024-08-31T12:26:42.224 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:42.224 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:42.224 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:26:42.225 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:42.225 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:26:42.225 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:26:42.225 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:26:42.225 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:42.225 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:42.225 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:42.225 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:26:42.225 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:26:42.226 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.211+0000 7fc0f7e15640 1 -- 172.21.15.19:0/1465634122 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc0d4077cf0 msgr2=0x7fc0d407a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:42.226 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.211+0000 7fc0f7e15640 1 --2- 172.21.15.19:0/1465634122 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc0d4077cf0 0x7fc0d407a200 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc0e80092f0 tx=0x7fc0e80095b0 comp rx=0 tx=0).stop 2024-08-31T12:26:42.226 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.211+0000 7fc0f7e15640 1 -- 172.21.15.19:0/1465634122 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc0f006ad70 msgr2=0x7fc0f014ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:42.226 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.211+0000 7fc0f63b2640 1 -- 172.21.15.19:0/1465634122 reap_dead start 2024-08-31T12:26:42.226 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.211+0000 7fc0f7e15640 1 --2- 172.21.15.19:0/1465634122 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc0f006ad70 0x7fc0f014ad00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec00b410 tx=0x7fc0ec00b8d0 comp rx=0 tx=0).stop 2024-08-31T12:26:42.226 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.211+0000 7fc0f7e15640 1 -- 172.21.15.19:0/1465634122 shutdown_connections 2024-08-31T12:26:42.226 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.211+0000 7fc0f7e15640 1 --2- 172.21.15.19:0/1465634122 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc0f006ad70 0x7fc0f014ad00 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:42.226 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.211+0000 7fc0f7e15640 1 -- 172.21.15.19:0/1465634122 >> 172.21.15.19:0/1465634122 conn(0x7fc0f0071100 msgr2=0x7fc0f0157440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:26:42.227 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.211+0000 7fc0f7e15640 1 -- 172.21.15.19:0/1465634122 shutdown_connections 2024-08-31T12:26:42.227 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.211+0000 7fc0f7e15640 1 -- 172.21.15.19:0/1465634122 wait complete. 2024-08-31T12:26:42.228 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:26:42.228 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-08-31T12:26:42.228 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-31T12:26:42.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.400+0000 7fe967dda640 1 -- 172.21.15.19:0/661912121 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe960074730 msgr2=0x7fe960074b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:42.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.400+0000 7fe967dda640 1 --2- 172.21.15.19:0/661912121 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe960074730 0x7fe960074b40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe958005fd0 tx=0x7fe95802bbb0 comp rx=0 tx=0).stop 2024-08-31T12:26:42.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.401+0000 7fe967dda640 1 -- 172.21.15.19:0/661912121 shutdown_connections 2024-08-31T12:26:42.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.401+0000 7fe967dda640 1 --2- 172.21.15.19:0/661912121 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe960074730 0x7fe960074b40 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:42.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.401+0000 7fe967dda640 1 -- 172.21.15.19:0/661912121 >> 172.21.15.19:0/661912121 conn(0x7fe960071100 msgr2=0x7fe96006cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:26:42.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.401+0000 7fe967dda640 1 -- 172.21.15.19:0/661912121 shutdown_connections 2024-08-31T12:26:42.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.401+0000 7fe967dda640 1 -- 172.21.15.19:0/661912121 wait complete. 2024-08-31T12:26:42.404 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.403+0000 7fe967dda640 1 Processor -- start 2024-08-31T12:26:42.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.404+0000 7fe967dda640 1 -- start start 2024-08-31T12:26:42.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.405+0000 7fe967dda640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe960074730 0x7fe960153a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:42.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.405+0000 7fe967dda640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe9600c3e50 0x7fe960156710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:42.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.405+0000 7fe967dda640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe9600c8610 0x7fe9601529c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:42.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.405+0000 7fe967dda640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fe9600c57e0 con 0x7fe9600c3e50 2024-08-31T12:26:42.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.405+0000 7fe967dda640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe960073b00 con 0x7fe960074730 2024-08-31T12:26:42.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.405+0000 7fe967dda640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fe960075080 con 0x7fe9600c8610 2024-08-31T12:26:42.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.406+0000 7fe965375640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe9600c3e50 0x7fe960156710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:42.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.406+0000 7fe966377640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe9600c8610 0x7fe9601529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:42.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.406+0000 7fe966377640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe9600c8610 0x7fe9601529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.19:35728/0 (socket says 172.21.15.19:35728) 2024-08-31T12:26:42.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.406+0000 7fe966377640 1 -- 172.21.15.19:0/4073668250 learned_addr learned my addr 172.21.15.19:0/4073668250 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:26:42.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.406+0000 7fe965375640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe9600c3e50 0x7fe960156710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:47970/0 (socket says 172.21.15.19:47970) 2024-08-31T12:26:42.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.406+0000 7fe965b76640 1 --2- 172.21.15.19:0/4073668250 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe960074730 0x7fe960153a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:42.408 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.406+0000 7fe966377640 1 -- 172.21.15.19:0/4073668250 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe960074730 msgr2=0x7fe960153a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:42.408 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.406+0000 7fe966377640 1 --2- 172.21.15.19:0/4073668250 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe960074730 0x7fe960153a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:42.408 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.406+0000 7fe966377640 1 -- 172.21.15.19:0/4073668250 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe9600c3e50 msgr2=0x7fe960156710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:42.408 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.406+0000 7fe966377640 1 --2- 172.21.15.19:0/4073668250 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe9600c3e50 0x7fe960156710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:42.408 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.406+0000 7fe966377640 1 -- 172.21.15.19:0/4073668250 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe960144df0 con 0x7fe9600c8610 2024-08-31T12:26:42.408 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.406+0000 7fe966377640 1 --2- 172.21.15.19:0/4073668250 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe9600c8610 0x7fe9601529c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe95800ade0 tx=0x7fe958003f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:42.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.407+0000 7fe9567fc640 1 -- 172.21.15.19:0/4073668250 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe95802de80 con 0x7fe9600c8610 2024-08-31T12:26:42.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.407+0000 7fe9567fc640 1 -- 172.21.15.19:0/4073668250 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe958004150 con 0x7fe9600c8610 2024-08-31T12:26:42.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.408+0000 7fe9567fc640 1 -- 172.21.15.19:0/4073668250 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9580045a0 con 0x7fe9600c8610 2024-08-31T12:26:42.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.408+0000 7fe967dda640 1 -- 172.21.15.19:0/4073668250 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe960153fc0 con 0x7fe9600c8610 2024-08-31T12:26:42.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.409+0000 7fe967dda640 1 -- 172.21.15.19:0/4073668250 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe960152140 con 0x7fe9600c8610 2024-08-31T12:26:42.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.410+0000 7fe967dda640 1 --2- 172.21.15.19:0/4073668250 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe960154190 0x7fe960141400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:42.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.411+0000 7fe967dda640 1 -- 172.21.15.19:0/4073668250 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe960151fa0 con 0x7fe960154190 2024-08-31T12:26:42.412 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.411+0000 7fe965b76640 1 --2- 172.21.15.19:0/4073668250 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe960154190 0x7fe960141400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:42.412 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.411+0000 7fe9567fc640 1 -- 172.21.15.19:0/4073668250 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fe958004740 con 0x7fe9600c8610 2024-08-31T12:26:42.412 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.411+0000 7fe965b76640 1 -- 172.21.15.19:0/4073668250 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe9600c9190 con 0x7fe960154190 2024-08-31T12:26:42.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.412+0000 7fe965b76640 1 --2- 172.21.15.19:0/4073668250 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe960154190 0x7fe960141400 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe95c00f8b0 tx=0x7fe95c00fd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:42.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.412+0000 7fe9567fc640 1 --2- 172.21.15.19:0/4073668250 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe944077a50 0x7fe944079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:42.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.413+0000 7fe9567fc640 1 -- 172.21.15.19:0/4073668250 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fe9580ba370 con 0x7fe9600c8610 2024-08-31T12:26:42.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.413+0000 7fe965375640 1 --2- 172.21.15.19:0/4073668250 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe944077a50 0x7fe944079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:42.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.413+0000 7fe9567fc640 1 -- 172.21.15.19:0/4073668250 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe95c019070 con 0x7fe960154190 2024-08-31T12:26:42.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.413+0000 7fe9567fc640 1 -- 172.21.15.19:0/4073668250 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe95c0053e0 con 0x7fe960154190 2024-08-31T12:26:42.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.413+0000 7fe9567fc640 1 -- 172.21.15.19:0/4073668250 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe960154190 msgr2=0x7fe960141400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:42.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.413+0000 7fe9567fc640 1 --2- 172.21.15.19:0/4073668250 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe960154190 0x7fe960141400 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe95c00f8b0 tx=0x7fe95c00fd70 comp rx=0 tx=0).stop 2024-08-31T12:26:42.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.413+0000 7fe9567fc640 1 -- 172.21.15.19:0/4073668250 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe960154190 msgr2=0x7fe960141400 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:26:42.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.413+0000 7fe9567fc640 1 --2- 172.21.15.19:0/4073668250 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe960154190 0x7fe960141400 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:42.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.413+0000 7fe965375640 1 --2- 172.21.15.19:0/4073668250 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe944077a50 0x7fe944079f60 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fe950007fd0 tx=0x7fe950009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:42.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.426+0000 7fe967dda640 1 --2- 172.21.15.19:0/4073668250 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe960152f00 0x7fe960141940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:42.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.426+0000 7fe967dda640 1 -- 172.21.15.19:0/4073668250 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe960141e80 con 0x7fe960152f00 2024-08-31T12:26:42.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.427+0000 7fe965b76640 1 --2- 172.21.15.19:0/4073668250 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe960152f00 0x7fe960141940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:42.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.427+0000 7fe965b76640 1 -- 172.21.15.19:0/4073668250 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe960154fd0 con 0x7fe960152f00 2024-08-31T12:26:42.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.427+0000 7fe965b76640 1 --2- 172.21.15.19:0/4073668250 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe960152f00 0x7fe960141940 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe95c016000 tx=0x7fe95c000fe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:42.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.428+0000 7fe9567fc640 1 -- 172.21.15.19:0/4073668250 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe95c019070 con 0x7fe960152f00 2024-08-31T12:26:42.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.428+0000 7fe9567fc640 1 -- 172.21.15.19:0/4073668250 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4874 (secure 0 0 0) 0x7fe95c01a040 con 0x7fe960152f00 2024-08-31T12:26:42.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.428+0000 7fe9567fc640 1 -- 172.21.15.19:0/4073668250 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe960152f00 msgr2=0x7fe960141940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:42.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.428+0000 7fe9567fc640 1 --2- 172.21.15.19:0/4073668250 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe960152f00 0x7fe960141940 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe95c016000 tx=0x7fe95c000fe0 comp rx=0 tx=0).stop 2024-08-31T12:26:42.430 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:26:42.430 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:26:42.430 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:26:42.430 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:26:42.430 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 86, 2024-08-31T12:26:42.430 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 63714, 2024-08-31T12:26:42.430 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:26:42.430 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:26:42.430 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:26:42.430 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:26:42.430 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:42.430 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 3, 2024-08-31T12:26:42.431 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:26:42.431 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:26:42.431 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:26:42.431 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:26:42.431 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:26:42.431 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:26:42.431 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:26:42.431 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:26:42.431 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:26:42.431 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:26:42.431 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:26:42.431 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:26:42.431 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:26:42.432 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:26:42.432 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:42.432 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:26:42.432 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:26:42.432 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:26:42.432 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:26:42.432 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:26:42.432 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:26:42.432 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:26:42.432 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:26:42.432 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:26:42.432 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:26:42.433 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:26:42.433 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:26:42.433 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:26:42.433 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:42.433 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:42.433 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:26:42.433 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:26:42.433 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:26:42.433 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:26:42.433 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:26:42.433 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:26:42.433 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:26:42.433 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:26:42.434 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:26:42.434 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:26:42.434 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:26:42.434 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:26:42.434 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:26:42.434 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:26:42.434 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:26:42.434 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:26:42.434 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:26:42.434 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:26:42.434 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:26:42.434 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:26:42.434 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:26:42.435 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:26:42.435 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:26:42.435 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:26:42.435 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:26:42.435 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:26:42.435 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:26:42.435 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:26:42.435 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:42.435 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:26:42.435 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:42.435 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:26:42.435 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:42.435 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:26:42.435 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:26:42.436 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:26:42.436 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:26:42.436 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:42.436 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:26:42.436 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:26:42.436 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:42.436 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:42.436 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:42.436 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:26:42.436 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:26:42.436 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:42.436 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:42.436 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:42.437 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:42.437 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:26:42.437 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:26:42.437 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:26:42.437 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:26:42.437 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:26:42.437 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:42.437 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:42.437 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:26:42.437 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:26:42.437 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:26:42.437 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:26:42.437 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:42.438 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:26:42.438 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:26:42.438 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:42.438 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:42.438 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:42.438 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:26:42.438 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:26:42.438 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:42.438 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:42.438 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:42.438 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:42.438 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:26:42.438 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:26:42.438 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:26:42.439 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:26:42.439 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:26:42.439 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:42.439 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:42.439 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:26:42.439 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:26:42.439 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:26:42.439 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:26:42.439 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:42.439 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:26:42.439 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:26:42.439 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:42.439 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:42.440 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:42.440 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:26:42.440 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:26:42.440 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:42.440 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:42.440 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:42.440 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:42.440 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:26:42.440 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:26:42.440 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:26:42.440 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:26:42.440 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:26:42.440 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:42.440 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:42.440 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:42.441 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:26:42.441 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:26:42.441 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:42.441 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:26:42.441 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:26:42.441 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:26:42.441 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:42.441 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:42.441 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:26:42.441 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:42.441 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:26:42.441 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:26:42.441 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:26:42.441 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:42.441 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:42.442 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:26:42.442 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:42.442 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:26:42.442 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:26:42.442 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:26:42.442 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:42.442 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:42.442 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:26:42.442 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:42.442 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:26:42.442 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:26:42.442 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 3 2024-08-31T12:26:42.442 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:42.442 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:42.442 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:42.443 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:26:42.443 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:26:42.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.428+0000 7fe9567fc640 1 -- 172.21.15.19:0/4073668250 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe960152f00 msgr2=0x7fe960141940 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:26:42.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.428+0000 7fe9567fc640 1 --2- 172.21.15.19:0/4073668250 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe960152f00 0x7fe960141940 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:42.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.428+0000 7fe967dda640 1 -- 172.21.15.19:0/4073668250 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe944077a50 msgr2=0x7fe944079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:42.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.428+0000 7fe967dda640 1 --2- 172.21.15.19:0/4073668250 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe944077a50 0x7fe944079f60 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fe950007fd0 tx=0x7fe950009040 comp rx=0 tx=0).stop 2024-08-31T12:26:42.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.428+0000 7fe967dda640 1 -- 172.21.15.19:0/4073668250 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe9600c8610 msgr2=0x7fe9601529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:42.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.428+0000 7fe966377640 1 -- 172.21.15.19:0/4073668250 reap_dead start 2024-08-31T12:26:42.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.428+0000 7fe967dda640 1 --2- 172.21.15.19:0/4073668250 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe9600c8610 0x7fe9601529c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe95800ade0 tx=0x7fe958003f60 comp rx=0 tx=0).stop 2024-08-31T12:26:42.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.429+0000 7fe967dda640 1 -- 172.21.15.19:0/4073668250 shutdown_connections 2024-08-31T12:26:42.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.429+0000 7fe967dda640 1 --2- 172.21.15.19:0/4073668250 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe9600c8610 0x7fe9601529c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:42.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.429+0000 7fe967dda640 1 -- 172.21.15.19:0/4073668250 >> 172.21.15.19:0/4073668250 conn(0x7fe960071100 msgr2=0x7fe9600c36f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:26:42.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.429+0000 7fe967dda640 1 -- 172.21.15.19:0/4073668250 shutdown_connections 2024-08-31T12:26:42.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.429+0000 7fe967dda640 1 -- 172.21.15.19:0/4073668250 wait complete. 2024-08-31T12:26:42.446 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:26:42.446 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-08-31T12:26:42.446 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-31T12:26:42.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.619+0000 7fb12f543640 1 -- 172.21.15.19:0/1963120275 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1280cb680 msgr2=0x7fb12806dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:42.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.619+0000 7fb12f543640 1 --2- 172.21.15.19:0/1963120275 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1280cb680 0x7fb12806dbf0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb120005fd0 tx=0x7fb12002aa90 comp rx=0 tx=0).stop 2024-08-31T12:26:42.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.619+0000 7fb12f543640 1 -- 172.21.15.19:0/1963120275 shutdown_connections 2024-08-31T12:26:42.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.619+0000 7fb12f543640 1 --2- 172.21.15.19:0/1963120275 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1280cb680 0x7fb12806dbf0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:42.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.619+0000 7fb12f543640 1 -- 172.21.15.19:0/1963120275 >> 172.21.15.19:0/1963120275 conn(0x7fb1280690d0 msgr2=0x7fb128072800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:26:42.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.619+0000 7fb12f543640 1 -- 172.21.15.19:0/1963120275 shutdown_connections 2024-08-31T12:26:42.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.619+0000 7fb12f543640 1 -- 172.21.15.19:0/1963120275 wait complete. 2024-08-31T12:26:42.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.623+0000 7fb12f543640 1 Processor -- start 2024-08-31T12:26:42.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.623+0000 7fb12f543640 1 -- start start 2024-08-31T12:26:42.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.625+0000 7fb12f543640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb128073380 0x7fb12814b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:42.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.625+0000 7fb12f543640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb1280ca870 0x7fb128156de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:42.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.625+0000 7fb12f543640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb1280cb680 0x7fb1280ccea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:42.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.625+0000 7fb12f543640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fb1280c8f40 con 0x7fb1280ca870 2024-08-31T12:26:42.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.625+0000 7fb12f543640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb12806ea30 con 0x7fb128073380 2024-08-31T12:26:42.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.625+0000 7fb12f543640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fb12806e130 con 0x7fb1280cb680 2024-08-31T12:26:42.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.626+0000 7fb12d2df640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb128073380 0x7fb12814b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:42.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.626+0000 7fb12cade640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb1280ca870 0x7fb128156de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:42.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.626+0000 7fb12d2df640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb128073380 0x7fb12814b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:34408/0 (socket says 172.21.15.19:34408) 2024-08-31T12:26:42.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.626+0000 7fb12cade640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb1280ca870 0x7fb128156de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:47972/0 (socket says 172.21.15.19:47972) 2024-08-31T12:26:42.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.626+0000 7fb12cade640 1 -- 172.21.15.19:0/1189777784 learned_addr learned my addr 172.21.15.19:0/1189777784 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:26:42.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.626+0000 7fb12dae0640 1 --2- 172.21.15.19:0/1189777784 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb1280cb680 0x7fb1280ccea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:42.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.626+0000 7fb12d2df640 1 -- 172.21.15.19:0/1189777784 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb1280cb680 msgr2=0x7fb1280ccea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:42.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.626+0000 7fb12d2df640 1 --2- 172.21.15.19:0/1189777784 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb1280cb680 0x7fb1280ccea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:42.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.626+0000 7fb12d2df640 1 -- 172.21.15.19:0/1189777784 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb1280ca870 msgr2=0x7fb128156de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:42.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.626+0000 7fb12d2df640 1 --2- 172.21.15.19:0/1189777784 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb1280ca870 0x7fb128156de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:42.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.626+0000 7fb12d2df640 1 -- 172.21.15.19:0/1189777784 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb128157320 con 0x7fb128073380 2024-08-31T12:26:42.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.626+0000 7fb12d2df640 1 --2- 172.21.15.19:0/1189777784 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb128073380 0x7fb12814b580 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb12400c470 tx=0x7fb12400c930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:42.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.627+0000 7fb11dffb640 1 -- 172.21.15.19:0/1189777784 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb124017070 con 0x7fb128073380 2024-08-31T12:26:42.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.627+0000 7fb11dffb640 1 -- 172.21.15.19:0/1189777784 <== mon.1 v2:172.21.15.43:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb1240035f0 con 0x7fb128073380 2024-08-31T12:26:42.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.627+0000 7fb11dffb640 1 -- 172.21.15.19:0/1189777784 <== mon.1 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb124003a40 con 0x7fb128073380 2024-08-31T12:26:42.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.627+0000 7fb12f543640 1 -- 172.21.15.19:0/1189777784 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1281560a0 con 0x7fb128073380 2024-08-31T12:26:42.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.627+0000 7fb12f543640 1 -- 172.21.15.19:0/1189777784 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1280cc3a0 con 0x7fb128073380 2024-08-31T12:26:42.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.629+0000 7fb12f543640 1 --2- 172.21.15.19:0/1189777784 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb1281574f0 0x7fb1281580c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:42.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.629+0000 7fb12f543640 1 -- 172.21.15.19:0/1189777784 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fb12806e610 con 0x7fb1281574f0 2024-08-31T12:26:42.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.629+0000 7fb12cade640 1 --2- 172.21.15.19:0/1189777784 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb1281574f0 0x7fb1281580c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:42.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.629+0000 7fb12cade640 1 -- 172.21.15.19:0/1189777784 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb1281537e0 con 0x7fb1281574f0 2024-08-31T12:26:42.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.630+0000 7fb12cade640 1 --2- 172.21.15.19:0/1189777784 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb1281574f0 0x7fb1281580c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb1280cebc0 tx=0x7fb11800ec00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:42.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.630+0000 7fb11dffb640 1 -- 172.21.15.19:0/1189777784 <== mon.1 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fb124003be0 con 0x7fb128073380 2024-08-31T12:26:42.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.630+0000 7fb11dffb640 1 --2- 172.21.15.19:0/1189777784 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb10c077ab0 0x7fb10c079fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:42.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.631+0000 7fb11dffb640 1 -- 172.21.15.19:0/1189777784 <== mon.1 v2:172.21.15.43:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fb1240986e0 con 0x7fb128073380 2024-08-31T12:26:42.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.631+0000 7fb12dae0640 1 --2- 172.21.15.19:0/1189777784 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb10c077ab0 0x7fb10c079fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:42.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.631+0000 7fb11dffb640 1 -- 172.21.15.19:0/1189777784 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb118017070 con 0x7fb1281574f0 2024-08-31T12:26:42.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.631+0000 7fb11dffb640 1 -- 172.21.15.19:0/1189777784 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb11800c7a0 con 0x7fb1281574f0 2024-08-31T12:26:42.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.631+0000 7fb11dffb640 1 -- 172.21.15.19:0/1189777784 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb1281574f0 msgr2=0x7fb1281580c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:42.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.631+0000 7fb11dffb640 1 --2- 172.21.15.19:0/1189777784 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb1281574f0 0x7fb1281580c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb1280cebc0 tx=0x7fb11800ec00 comp rx=0 tx=0).stop 2024-08-31T12:26:42.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.631+0000 7fb11dffb640 1 -- 172.21.15.19:0/1189777784 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb1281574f0 msgr2=0x7fb1281580c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:26:42.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.631+0000 7fb11dffb640 1 --2- 172.21.15.19:0/1189777784 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb1281574f0 0x7fb1281580c0 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb1280cebc0 tx=0x7fb11800ec00 comp rx=0 tx=0).stop 2024-08-31T12:26:42.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.631+0000 7fb12dae0640 1 --2- 172.21.15.19:0/1189777784 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb10c077ab0 0x7fb10c079fc0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb120035000 tx=0x7fb120038040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:42.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.643+0000 7fb12f543640 1 --2- 172.21.15.19:0/1189777784 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb1280ce6c0 0x7fb128158600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:42.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.643+0000 7fb12f543640 1 -- 172.21.15.19:0/1189777784 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fb1280641e0 con 0x7fb1280ce6c0 2024-08-31T12:26:42.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.644+0000 7fb12cade640 1 --2- 172.21.15.19:0/1189777784 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb1280ce6c0 0x7fb128158600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:42.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.645+0000 7fb12cade640 1 -- 172.21.15.19:0/1189777784 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb128156a00 con 0x7fb1280ce6c0 2024-08-31T12:26:42.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.645+0000 7fb12cade640 1 --2- 172.21.15.19:0/1189777784 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb1280ce6c0 0x7fb128158600 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb1280cebc0 tx=0x7fb1180046f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:42.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.645+0000 7fb11dffb640 1 -- 172.21.15.19:0/1189777784 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb118017050 con 0x7fb1280ce6c0 2024-08-31T12:26:42.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.646+0000 7fb11dffb640 1 -- 172.21.15.19:0/1189777784 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4539 (secure 0 0 0) 0x7fb118006310 con 0x7fb1280ce6c0 2024-08-31T12:26:42.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.646+0000 7fb11dffb640 1 -- 172.21.15.19:0/1189777784 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb1280ce6c0 msgr2=0x7fb128158600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:42.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.646+0000 7fb11dffb640 1 --2- 172.21.15.19:0/1189777784 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb1280ce6c0 0x7fb128158600 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb1280cebc0 tx=0x7fb1180046f0 comp rx=0 tx=0).stop 2024-08-31T12:26:42.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.646+0000 7fb11dffb640 1 -- 172.21.15.19:0/1189777784 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb1280ce6c0 msgr2=0x7fb128158600 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:26:42.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.646+0000 7fb11dffb640 1 --2- 172.21.15.19:0/1189777784 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb1280ce6c0 0x7fb128158600 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb1280cebc0 tx=0x7fb1180046f0 comp rx=0 tx=0).stop 2024-08-31T12:26:42.648 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:26:42.648 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:26:42.648 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:26:42.648 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:26:42.648 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 86, 2024-08-31T12:26:42.649 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 8514, 2024-08-31T12:26:42.649 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:26:42.649 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:26:42.649 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:26:42.649 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:26:42.649 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:42.649 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 3, 2024-08-31T12:26:42.649 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:26:42.649 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:26:42.650 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:26:42.650 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:26:42.650 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:26:42.650 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:26:42.650 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:26:42.650 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:26:42.650 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:26:42.650 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:26:42.650 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:26:42.651 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:26:42.651 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:26:42.651 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:26:42.651 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:42.651 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:26:42.651 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:26:42.651 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:26:42.651 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:26:42.651 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:26:42.651 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:26:42.652 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:26:42.652 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:26:42.652 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:26:42.652 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:26:42.652 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:26:42.652 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:26:42.652 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:26:42.652 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:42.652 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:42.653 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:26:42.653 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:26:42.653 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:26:42.653 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:26:42.653 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:26:42.653 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:26:42.653 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:26:42.653 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:26:42.653 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:26:42.653 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:26:42.653 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:26:42.653 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:26:42.654 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:26:42.654 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:26:42.654 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:26:42.654 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:26:42.654 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:26:42.654 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:26:42.654 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:26:42.654 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:26:42.654 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:26:42.654 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:26:42.654 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:26:42.654 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:26:42.655 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:26:42.655 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:26:42.655 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:26:42.655 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:26:42.655 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:42.655 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:26:42.655 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:42.655 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:26:42.655 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:42.655 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:26:42.655 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:26:42.655 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:26:42.655 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:26:42.656 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:42.656 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:26:42.656 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:26:42.656 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:42.656 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:42.656 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:42.656 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:26:42.656 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:26:42.656 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:42.656 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:42.656 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:42.656 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:42.656 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:26:42.656 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:26:42.657 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:26:42.657 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:26:42.657 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:26:42.657 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:42.657 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:42.657 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:26:42.657 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:26:42.657 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:26:42.657 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:26:42.657 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:42.657 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:26:42.657 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:26:42.657 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:42.657 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:42.658 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:42.658 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:26:42.658 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:26:42.658 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:42.658 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:42.658 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:42.658 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:42.658 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:26:42.658 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:26:42.658 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:26:42.658 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:26:42.658 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:26:42.658 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:42.658 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:42.659 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:26:42.659 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:26:42.659 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:26:42.659 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:26:42.659 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:42.659 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:26:42.659 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:26:42.659 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:42.659 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:42.659 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:42.659 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:26:42.659 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:26:42.659 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:42.659 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:42.660 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:42.660 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:42.660 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:26:42.660 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:26:42.660 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:26:42.660 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:26:42.660 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:26:42.660 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:42.660 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:42.660 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:42.660 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:26:42.660 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:26:42.660 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:42.660 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:26:42.661 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:26:42.661 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:26:42.661 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:42.661 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:42.661 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:26:42.661 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:42.661 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:26:42.661 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:26:42.661 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:26:42.661 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:42.661 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:42.661 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:42.661 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:26:42.661 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:26:42.663 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.647+0000 7fb12f543640 1 -- 172.21.15.19:0/1189777784 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb10c077ab0 msgr2=0x7fb10c079fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:42.663 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.647+0000 7fb12f543640 1 --2- 172.21.15.19:0/1189777784 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb10c077ab0 0x7fb10c079fc0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb120035000 tx=0x7fb120038040 comp rx=0 tx=0).stop 2024-08-31T12:26:42.663 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.647+0000 7fb12f543640 1 -- 172.21.15.19:0/1189777784 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb128073380 msgr2=0x7fb12814b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:42.663 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.647+0000 7fb12f543640 1 --2- 172.21.15.19:0/1189777784 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb128073380 0x7fb12814b580 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb12400c470 tx=0x7fb12400c930 comp rx=0 tx=0).stop 2024-08-31T12:26:42.663 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.647+0000 7fb12dae0640 1 -- 172.21.15.19:0/1189777784 reap_dead start 2024-08-31T12:26:42.663 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.647+0000 7fb12dae0640 1 -- 172.21.15.19:0/1189777784 reap_dead start 2024-08-31T12:26:42.663 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.647+0000 7fb12f543640 1 -- 172.21.15.19:0/1189777784 shutdown_connections 2024-08-31T12:26:42.663 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.647+0000 7fb12f543640 1 -- 172.21.15.19:0/1189777784 >> 172.21.15.19:0/1189777784 conn(0x7fb1280690d0 msgr2=0x7fb128153d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:26:42.663 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.647+0000 7fb12f543640 1 -- 172.21.15.19:0/1189777784 shutdown_connections 2024-08-31T12:26:42.664 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.647+0000 7fb12f543640 1 -- 172.21.15.19:0/1189777784 wait complete. 2024-08-31T12:26:42.664 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:26:42.664 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-08-31T12:26:42.664 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-08-31T12:26:42.834 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.832+0000 7fbf290ec640 1 -- 172.21.15.19:0/2339777356 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbf24155520 msgr2=0x7fbf2415b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:42.834 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.832+0000 7fbf290ec640 1 --2- 172.21.15.19:0/2339777356 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbf24155520 0x7fbf2415b020 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fbf240ce650 tx=0x7fbf1402a5f0 comp rx=0 tx=0).stop 2024-08-31T12:26:42.834 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.833+0000 7fbf290ec640 1 -- 172.21.15.19:0/2339777356 shutdown_connections 2024-08-31T12:26:42.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.833+0000 7fbf290ec640 1 --2- 172.21.15.19:0/2339777356 >> v2:172.21.15.43:3300/0 conn(0x7fbf2415b560 0x7fbf240cbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:42.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.833+0000 7fbf290ec640 1 --2- 172.21.15.19:0/2339777356 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbf24155520 0x7fbf2415b020 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:42.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.833+0000 7fbf290ec640 1 -- 172.21.15.19:0/2339777356 >> 172.21.15.19:0/2339777356 conn(0x7fbf24074a90 msgr2=0x7fbf24068aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:26:42.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.833+0000 7fbf290ec640 1 -- 172.21.15.19:0/2339777356 shutdown_connections 2024-08-31T12:26:42.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.833+0000 7fbf290ec640 1 -- 172.21.15.19:0/2339777356 wait complete. 2024-08-31T12:26:42.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.836+0000 7fbf290ec640 1 Processor -- start 2024-08-31T12:26:42.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.836+0000 7fbf290ec640 1 -- start start 2024-08-31T12:26:42.839 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.838+0000 7fbf290ec640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbf240c5c20 0x7fbf241514b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:42.839 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.838+0000 7fbf290ec640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbf24155520 0x7fbf240c9200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:42.839 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.838+0000 7fbf290ec640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf2415b560 0x7fbf2415a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:42.839 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.838+0000 7fbf290ec640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fbf240d1140 con 0x7fbf24155520 2024-08-31T12:26:42.839 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.838+0000 7fbf290ec640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fbf240c6750 con 0x7fbf2415b560 2024-08-31T12:26:42.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.838+0000 7fbf290ec640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fbf240cb800 con 0x7fbf240c5c20 2024-08-31T12:26:42.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.838+0000 7fbf2259c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbf24155520 0x7fbf240c9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:42.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.838+0000 7fbf22d9d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbf240c5c20 0x7fbf241514b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:42.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.838+0000 7fbf2359e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf2415b560 0x7fbf2415a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:42.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.838+0000 7fbf22d9d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbf240c5c20 0x7fbf241514b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.19:35762/0 (socket says 172.21.15.19:35762) 2024-08-31T12:26:42.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.838+0000 7fbf22d9d640 1 -- 172.21.15.19:0/1360139733 learned_addr learned my addr 172.21.15.19:0/1360139733 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:26:42.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.838+0000 7fbf2259c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbf24155520 0x7fbf240c9200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:47996/0 (socket says 172.21.15.19:47996) 2024-08-31T12:26:42.841 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.838+0000 7fbf2359e640 1 -- 172.21.15.19:0/1360139733 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbf240c5c20 msgr2=0x7fbf241514b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:42.841 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.838+0000 7fbf2359e640 1 --2- 172.21.15.19:0/1360139733 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbf240c5c20 0x7fbf241514b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:42.841 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.839+0000 7fbf2359e640 1 -- 172.21.15.19:0/1360139733 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbf24155520 msgr2=0x7fbf240c9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:42.841 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.839+0000 7fbf2359e640 1 --2- 172.21.15.19:0/1360139733 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbf24155520 0x7fbf240c9200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:42.841 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.839+0000 7fbf2359e640 1 -- 172.21.15.19:0/1360139733 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf241519f0 con 0x7fbf2415b560 2024-08-31T12:26:42.841 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.839+0000 7fbf2359e640 1 --2- 172.21.15.19:0/1360139733 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf2415b560 0x7fbf2415a6a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c00c700 tx=0x7fbf1c00cbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:42.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.839+0000 7fbf1b7fe640 1 -- 172.21.15.19:0/1360139733 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf1c003b70 con 0x7fbf2415b560 2024-08-31T12:26:42.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.839+0000 7fbf1b7fe640 1 -- 172.21.15.19:0/1360139733 <== mon.1 v2:172.21.15.43:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbf1c003d10 con 0x7fbf2415b560 2024-08-31T12:26:42.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.839+0000 7fbf1b7fe640 1 -- 172.21.15.19:0/1360139733 <== mon.1 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf1c005330 con 0x7fbf2415b560 2024-08-31T12:26:42.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.839+0000 7fbf290ec640 1 -- 172.21.15.19:0/1360139733 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf240c9740 con 0x7fbf2415b560 2024-08-31T12:26:42.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.839+0000 7fbf290ec640 1 -- 172.21.15.19:0/1360139733 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf241565e0 con 0x7fbf2415b560 2024-08-31T12:26:42.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.841+0000 7fbf290ec640 1 -- 172.21.15.19:0/1360139733 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf240cb2f0 con 0x7fbf2415b560 2024-08-31T12:26:42.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.841+0000 7fbf1b7fe640 1 -- 172.21.15.19:0/1360139733 <== mon.1 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fbf1c0054d0 con 0x7fbf2415b560 2024-08-31T12:26:42.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.842+0000 7fbf1b7fe640 1 --2- 172.21.15.19:0/1360139733 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fbf04077a50 0x7fbf04079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:42.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.842+0000 7fbf1b7fe640 1 -- 172.21.15.19:0/1360139733 <== mon.1 v2:172.21.15.43:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fbf1c099270 con 0x7fbf2415b560 2024-08-31T12:26:42.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.842+0000 7fbf22d9d640 1 --2- 172.21.15.19:0/1360139733 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fbf04077a50 0x7fbf04079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:42.844 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.843+0000 7fbf22d9d640 1 --2- 172.21.15.19:0/1360139733 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fbf04077a50 0x7fbf04079f60 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fbf140096c0 tx=0x7fbf14004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:42.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:42.848+0000 7fbf1b7fe640 1 -- 172.21.15.19:0/1360139733 <== mon.1 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fbf241519f0 con 0x7fbf2415b560 2024-08-31T12:26:43.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:43.314+0000 7fbf290ec640 1 -- 172.21.15.19:0/1360139733 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fbf240b3bb0 con 0x7fbf2415b560 2024-08-31T12:26:43.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:43.317+0000 7fbf1b7fe640 1 -- 172.21.15.19:0/1360139733 <== mon.1 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fbf1c062080 con 0x7fbf2415b560 2024-08-31T12:26:43.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:43.321+0000 7fbf290ec640 1 -- 172.21.15.19:0/1360139733 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fbf04077a50 msgr2=0x7fbf04079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:43.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:43.321+0000 7fbf290ec640 1 --2- 172.21.15.19:0/1360139733 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fbf04077a50 0x7fbf04079f60 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fbf140096c0 tx=0x7fbf14004730 comp rx=0 tx=0).stop 2024-08-31T12:26:43.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:43.321+0000 7fbf290ec640 1 -- 172.21.15.19:0/1360139733 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf2415b560 msgr2=0x7fbf2415a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:43.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:43.321+0000 7fbf290ec640 1 --2- 172.21.15.19:0/1360139733 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf2415b560 0x7fbf2415a6a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c00c700 tx=0x7fbf1c00cbc0 comp rx=0 tx=0).stop 2024-08-31T12:26:43.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:43.321+0000 7fbf290ec640 1 -- 172.21.15.19:0/1360139733 shutdown_connections 2024-08-31T12:26:43.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:43.321+0000 7fbf290ec640 1 --2- 172.21.15.19:0/1360139733 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fbf04077a50 0x7fbf04079f60 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:43.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:43.321+0000 7fbf290ec640 1 --2- 172.21.15.19:0/1360139733 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf2415b560 0x7fbf2415a6a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:43.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:43.321+0000 7fbf290ec640 1 --2- 172.21.15.19:0/1360139733 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbf24155520 0x7fbf240c9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:43.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:43.321+0000 7fbf290ec640 1 --2- 172.21.15.19:0/1360139733 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbf240c5c20 0x7fbf241514b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:43.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:43.322+0000 7fbf290ec640 1 -- 172.21.15.19:0/1360139733 >> 172.21.15.19:0/1360139733 conn(0x7fbf24074a90 msgr2=0x7fbf240cbf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:26:43.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:43.322+0000 7fbf290ec640 1 -- 172.21.15.19:0/1360139733 shutdown_connections 2024-08-31T12:26:43.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:43.322+0000 7fbf290ec640 1 -- 172.21.15.19:0/1360139733 wait complete. 2024-08-31T12:26:43.341 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2024-08-31T12:26:53.342 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-08-31T12:26:53.342 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:26:53.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:53.517+0000 7faf4c207640 1 -- 172.21.15.19:0/3481286321 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faf441581e0 msgr2=0x7faf440c3e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:53.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:53.517+0000 7faf4c207640 1 --2- 172.21.15.19:0/3481286321 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faf441581e0 0x7faf440c3e50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7faf4406db10 tx=0x7faf3c02a5a0 comp rx=0 tx=0).stop 2024-08-31T12:26:53.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:53.517+0000 7faf4c207640 1 -- 172.21.15.19:0/3481286321 shutdown_connections 2024-08-31T12:26:53.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:53.517+0000 7faf4c207640 1 --2- 172.21.15.19:0/3481286321 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faf441581e0 0x7faf440c3e50 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:53.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:53.517+0000 7faf4c207640 1 --2- 172.21.15.19:0/3481286321 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faf44151ea0 0x7faf44157ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:53.521 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:53.517+0000 7faf4c207640 1 --2- 172.21.15.19:0/3481286321 >> v2:172.21.15.43:3300/0 conn(0x7faf440c3720 0x7faf440c5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:53.521 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:53.517+0000 7faf4c207640 1 -- 172.21.15.19:0/3481286321 >> 172.21.15.19:0/3481286321 conn(0x7faf4406d110 msgr2=0x7faf440cc950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:26:53.521 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:53.518+0000 7faf4c207640 1 -- 172.21.15.19:0/3481286321 shutdown_connections 2024-08-31T12:26:53.521 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:53.518+0000 7faf4c207640 1 -- 172.21.15.19:0/3481286321 wait complete. 2024-08-31T12:26:53.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:53.521+0000 7faf4c207640 1 Processor -- start 2024-08-31T12:26:53.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:53.521+0000 7faf4c207640 1 -- start start 2024-08-31T12:26:53.525 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:53.523+0000 7faf4c207640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faf440c3720 0x7faf441423c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:53.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:53.523+0000 7faf49fa3640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faf440c3720 0x7faf441423c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:53.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:53.523+0000 7faf49fa3640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faf440c3720 0x7faf441423c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:56390/0 (socket says 172.21.15.19:56390) 2024-08-31T12:26:53.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:53.523+0000 7faf49fa3640 1 -- 172.21.15.19:0/1452372497 learned_addr learned my addr 172.21.15.19:0/1452372497 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:26:53.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:53.524+0000 7faf4c207640 1 --2- 172.21.15.19:0/1452372497 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faf44151ea0 0x7faf4414c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:53.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:53.524+0000 7faf4c207640 1 --2- 172.21.15.19:0/1452372497 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf441581e0 0x7faf440ca750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:53.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:53.524+0000 7faf4c207640 1 -- 172.21.15.19:0/1452372497 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7faf441522b0 con 0x7faf440c3720 2024-08-31T12:26:53.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:53.524+0000 7faf4c207640 1 -- 172.21.15.19:0/1452372497 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7faf440c6d70 con 0x7faf441581e0 2024-08-31T12:26:53.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:53.524+0000 7faf4c207640 1 -- 172.21.15.19:0/1452372497 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7faf440cd580 con 0x7faf44151ea0 2024-08-31T12:26:53.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:53.524+0000 7faf4a7a4640 1 --2- 172.21.15.19:0/1452372497 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf441581e0 0x7faf440ca750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:53.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:53.524+0000 7faf497a2640 1 --2- 172.21.15.19:0/1452372497 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faf44151ea0 0x7faf4414c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:53.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:53.524+0000 7faf49fa3640 1 -- 172.21.15.19:0/1452372497 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faf44151ea0 msgr2=0x7faf4414c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:53.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:53.524+0000 7faf49fa3640 1 --2- 172.21.15.19:0/1452372497 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faf44151ea0 0x7faf4414c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:53.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:53.524+0000 7faf49fa3640 1 -- 172.21.15.19:0/1452372497 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf441581e0 msgr2=0x7faf440ca750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:53.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:53.524+0000 7faf49fa3640 1 --2- 172.21.15.19:0/1452372497 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf441581e0 0x7faf440ca750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:53.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:53.524+0000 7faf49fa3640 1 -- 172.21.15.19:0/1452372497 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf441463b0 con 0x7faf440c3720 2024-08-31T12:26:53.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:53.525+0000 7faf497a2640 1 --2- 172.21.15.19:0/1452372497 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faf44151ea0 0x7faf4414c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:26:53.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:53.525+0000 7faf49fa3640 1 --2- 172.21.15.19:0/1452372497 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faf440c3720 0x7faf441423c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7faf400058e0 tx=0x7faf4000cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:53.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:53.525+0000 7faf3a7fc640 1 -- 172.21.15.19:0/1452372497 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf40007bb0 con 0x7faf440c3720 2024-08-31T12:26:53.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:53.525+0000 7faf3a7fc640 1 -- 172.21.15.19:0/1452372497 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faf40007e40 con 0x7faf440c3720 2024-08-31T12:26:53.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:53.526+0000 7faf3a7fc640 1 -- 172.21.15.19:0/1452372497 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf40005270 con 0x7faf440c3720 2024-08-31T12:26:53.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:53.526+0000 7faf4c207640 1 -- 172.21.15.19:0/1452372497 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf4414c9c0 con 0x7faf440c3720 2024-08-31T12:26:53.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:53.527+0000 7faf4c207640 1 -- 172.21.15.19:0/1452372497 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf441468f0 con 0x7faf440c3720 2024-08-31T12:26:53.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:53.528+0000 7faf4c207640 1 -- 172.21.15.19:0/1452372497 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf440c8ff0 con 0x7faf440c3720 2024-08-31T12:26:53.531 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:53.529+0000 7faf3a7fc640 1 -- 172.21.15.19:0/1452372497 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7faf40005410 con 0x7faf440c3720 2024-08-31T12:26:53.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:53.530+0000 7faf3a7fc640 1 --2- 172.21.15.19:0/1452372497 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7faf28077a50 0x7faf28079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:53.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:53.530+0000 7faf3a7fc640 1 -- 172.21.15.19:0/1452372497 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7faf40099b60 con 0x7faf440c3720 2024-08-31T12:26:53.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:53.530+0000 7faf497a2640 1 --2- 172.21.15.19:0/1452372497 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7faf28077a50 0x7faf28079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:53.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:53.531+0000 7faf497a2640 1 --2- 172.21.15.19:0/1452372497 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7faf28077a50 0x7faf28079f60 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7faf441efaa0 tx=0x7faf3400f040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:53.541 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:53.539+0000 7faf3a7fc640 1 -- 172.21.15.19:0/1452372497 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7faf440c8ff0 con 0x7faf440c3720 2024-08-31T12:26:53.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:53.997+0000 7faf4c207640 1 -- 172.21.15.19:0/1452372497 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7faf44157ca0 con 0x7faf440c3720 2024-08-31T12:26:53.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:53.997+0000 7faf3a7fc640 1 -- 172.21.15.19:0/1452372497 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7faf441463b0 con 0x7faf440c3720 2024-08-31T12:26:54.000 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":86,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:26:54.003 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.000+0000 7faf4c207640 1 -- 172.21.15.19:0/1452372497 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7faf28077a50 msgr2=0x7faf28079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:54.003 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.001+0000 7faf4c207640 1 --2- 172.21.15.19:0/1452372497 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7faf28077a50 0x7faf28079f60 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7faf441efaa0 tx=0x7faf3400f040 comp rx=0 tx=0).stop 2024-08-31T12:26:54.003 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.001+0000 7faf4c207640 1 -- 172.21.15.19:0/1452372497 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faf440c3720 msgr2=0x7faf441423c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:54.003 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.001+0000 7faf4c207640 1 --2- 172.21.15.19:0/1452372497 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faf440c3720 0x7faf441423c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7faf400058e0 tx=0x7faf4000cdd0 comp rx=0 tx=0).stop 2024-08-31T12:26:54.003 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.001+0000 7faf4c207640 1 -- 172.21.15.19:0/1452372497 shutdown_connections 2024-08-31T12:26:54.003 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.001+0000 7faf4c207640 1 --2- 172.21.15.19:0/1452372497 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7faf28077a50 0x7faf28079f60 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:54.003 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.001+0000 7faf4c207640 1 --2- 172.21.15.19:0/1452372497 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf441581e0 0x7faf440ca750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:54.003 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.001+0000 7faf4c207640 1 --2- 172.21.15.19:0/1452372497 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faf44151ea0 0x7faf4414c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:54.003 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.001+0000 7faf4c207640 1 --2- 172.21.15.19:0/1452372497 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faf440c3720 0x7faf441423c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:54.004 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.001+0000 7faf4c207640 1 -- 172.21.15.19:0/1452372497 >> 172.21.15.19:0/1452372497 conn(0x7faf4406d110 msgr2=0x7faf44072ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:26:54.004 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.001+0000 7faf4c207640 1 -- 172.21.15.19:0/1452372497 shutdown_connections 2024-08-31T12:26:54.004 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.001+0000 7faf4c207640 1 -- 172.21.15.19:0/1452372497 wait complete. 2024-08-31T12:26:54.019 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-08-31T12:26:54.019 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-08-31T12:26:54.020 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-08-31T12:26:54.020 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-31T12:26:54.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.187+0000 7f6146941640 1 -- 172.21.15.19:0/2680319060 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f61400ca7f0 msgr2=0x7f6140071bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:54.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.187+0000 7f6146941640 1 --2- 172.21.15.19:0/2680319060 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f61400ca7f0 0x7f6140071bd0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6140072450 tx=0x7f613002a560 comp rx=0 tx=0).stop 2024-08-31T12:26:54.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.187+0000 7f6146941640 1 -- 172.21.15.19:0/2680319060 shutdown_connections 2024-08-31T12:26:54.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.187+0000 7f6146941640 1 --2- 172.21.15.19:0/2680319060 >> v2:172.21.15.19:3300/0 conn(0x7f61400c9880 0x7f614015c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:54.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.187+0000 7f6146941640 1 --2- 172.21.15.19:0/2680319060 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f61401532f0 0x7f614015c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:54.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.187+0000 7f6146941640 1 --2- 172.21.15.19:0/2680319060 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f61400ca7f0 0x7f6140071bd0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:54.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.187+0000 7f6146941640 1 -- 172.21.15.19:0/2680319060 >> 172.21.15.19:0/2680319060 conn(0x7f61400cefb0 msgr2=0x7f61400cb340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:26:54.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.187+0000 7f6146941640 1 -- 172.21.15.19:0/2680319060 shutdown_connections 2024-08-31T12:26:54.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.187+0000 7f6146941640 1 -- 172.21.15.19:0/2680319060 wait complete. 2024-08-31T12:26:54.194 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.192+0000 7f6146941640 1 Processor -- start 2024-08-31T12:26:54.194 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.192+0000 7f6146941640 1 -- start start 2024-08-31T12:26:54.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.194+0000 7f6146941640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f61400c9880 0x7f61400c6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:54.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.194+0000 7f6146941640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f61400ca7f0 0x7f614006e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:54.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.194+0000 7f6146941640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f61401532f0 0x7f6140147340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:54.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.195+0000 7f6146941640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f614015d330 con 0x7f61401532f0 2024-08-31T12:26:54.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.195+0000 7f6146941640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6140072110 con 0x7f61400c9880 2024-08-31T12:26:54.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.195+0000 7f6146941640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f61400c3860 con 0x7f61400ca7f0 2024-08-31T12:26:54.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.195+0000 7f613ffff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f61400c9880 0x7f61400c6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:54.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.195+0000 7f613f7fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f61400ca7f0 0x7f614006e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:54.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.195+0000 7f613ffff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f61400c9880 0x7f61400c6120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:46268/0 (socket says 172.21.15.19:46268) 2024-08-31T12:26:54.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.195+0000 7f613ffff640 1 -- 172.21.15.19:0/4268926228 learned_addr learned my addr 172.21.15.19:0/4268926228 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:26:54.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.195+0000 7f6144ede640 1 --2- 172.21.15.19:0/4268926228 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f61401532f0 0x7f6140147340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:54.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.195+0000 7f613ffff640 1 -- 172.21.15.19:0/4268926228 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f61400ca7f0 msgr2=0x7f614006e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:54.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.195+0000 7f613ffff640 1 --2- 172.21.15.19:0/4268926228 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f61400ca7f0 0x7f614006e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:54.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.195+0000 7f613ffff640 1 -- 172.21.15.19:0/4268926228 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f61401532f0 msgr2=0x7f6140147340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:54.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.195+0000 7f613ffff640 1 --2- 172.21.15.19:0/4268926228 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f61401532f0 0x7f6140147340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:54.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.195+0000 7f613ffff640 1 -- 172.21.15.19:0/4268926228 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61401f5e80 con 0x7f61400c9880 2024-08-31T12:26:54.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.195+0000 7f613ffff640 1 --2- 172.21.15.19:0/4268926228 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f61400c9880 0x7f61400c6120 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6130000c00 tx=0x7f61300038a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:54.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.196+0000 7f613cff9640 1 -- 172.21.15.19:0/4268926228 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f613003e030 con 0x7f61400c9880 2024-08-31T12:26:54.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.196+0000 7f6146941640 1 -- 172.21.15.19:0/4268926228 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f614006ea40 con 0x7f61400c9880 2024-08-31T12:26:54.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.196+0000 7f613cff9640 1 -- 172.21.15.19:0/4268926228 <== mon.1 v2:172.21.15.43:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6130003ca0 con 0x7f61400c9880 2024-08-31T12:26:54.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.196+0000 7f613cff9640 1 -- 172.21.15.19:0/4268926228 <== mon.1 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6130035040 con 0x7f61400c9880 2024-08-31T12:26:54.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.196+0000 7f6146941640 1 -- 172.21.15.19:0/4268926228 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61401f5be0 con 0x7f61400c9880 2024-08-31T12:26:54.200 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.198+0000 7f6146941640 1 --2- 172.21.15.19:0/4268926228 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6140147880 0x7f61400b34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:54.200 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.198+0000 7f6146941640 1 -- 172.21.15.19:0/4268926228 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f614006f8e0 con 0x7f6140147880 2024-08-31T12:26:54.200 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.198+0000 7f613f7fe640 1 --2- 172.21.15.19:0/4268926228 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6140147880 0x7f61400b34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:54.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.198+0000 7f613f7fe640 1 -- 172.21.15.19:0/4268926228 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f614006e1f0 con 0x7f6140147880 2024-08-31T12:26:54.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.199+0000 7f613f7fe640 1 --2- 172.21.15.19:0/4268926228 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6140147880 0x7f61400b34c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6134011a10 tx=0x7f6134011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:54.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.199+0000 7f613cff9640 1 -- 172.21.15.19:0/4268926228 <== mon.1 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f6130041030 con 0x7f61400c9880 2024-08-31T12:26:54.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.199+0000 7f613cff9640 1 --2- 172.21.15.19:0/4268926228 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f612c077a80 0x7f612c079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:54.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.199+0000 7f613cff9640 1 -- 172.21.15.19:0/4268926228 <== mon.1 v2:172.21.15.43:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f61300b9ef0 con 0x7f61400c9880 2024-08-31T12:26:54.202 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.200+0000 7f613cff9640 1 -- 172.21.15.19:0/4268926228 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f613401e070 con 0x7f6140147880 2024-08-31T12:26:54.202 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.200+0000 7f6144ede640 1 --2- 172.21.15.19:0/4268926228 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f612c077a80 0x7f612c079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:54.202 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.200+0000 7f613cff9640 1 -- 172.21.15.19:0/4268926228 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f613400fc20 con 0x7f6140147880 2024-08-31T12:26:54.202 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.200+0000 7f613cff9640 1 -- 172.21.15.19:0/4268926228 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6140147880 msgr2=0x7f61400b34c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:54.202 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.200+0000 7f613cff9640 1 --2- 172.21.15.19:0/4268926228 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6140147880 0x7f61400b34c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6134011a10 tx=0x7f6134011ed0 comp rx=0 tx=0).stop 2024-08-31T12:26:54.202 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.200+0000 7f613cff9640 1 -- 172.21.15.19:0/4268926228 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6140147880 msgr2=0x7f61400b34c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:26:54.202 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.200+0000 7f613cff9640 1 --2- 172.21.15.19:0/4268926228 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6140147880 0x7f61400b34c0 secure :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6134011a10 tx=0x7f6134011ed0 comp rx=0 tx=0).stop 2024-08-31T12:26:54.202 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.201+0000 7f6144ede640 1 --2- 172.21.15.19:0/4268926228 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f612c077a80 0x7f612c079f90 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f614014d3b0 tx=0x7f6138009500 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:54.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.213+0000 7f6146941640 1 --2- 172.21.15.19:0/4268926228 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f61400b3a00 0x7f61401f9040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:54.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.213+0000 7f6146941640 1 -- 172.21.15.19:0/4268926228 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f61400b3e10 con 0x7f61400b3a00 2024-08-31T12:26:54.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.214+0000 7f613f7fe640 1 --2- 172.21.15.19:0/4268926228 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f61400b3a00 0x7f61401f9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:54.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.214+0000 7f613f7fe640 1 -- 172.21.15.19:0/4268926228 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f614006fb30 con 0x7f61400b3a00 2024-08-31T12:26:54.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.214+0000 7f613f7fe640 1 --2- 172.21.15.19:0/4268926228 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f61400b3a00 0x7f61401f9040 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f613401b000 tx=0x7f61340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:54.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.214+0000 7f613cff9640 1 -- 172.21.15.19:0/4268926228 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f613401e040 con 0x7f61400b3a00 2024-08-31T12:26:54.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.215+0000 7f613cff9640 1 -- 172.21.15.19:0/4268926228 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5045 (secure 0 0 0) 0x7f613400fe90 con 0x7f61400b3a00 2024-08-31T12:26:54.217 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:26:54.217 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:26:54.217 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:26:54.217 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "leader", 2024-08-31T12:26:54.217 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 86, 2024-08-31T12:26:54.217 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 148769, 2024-08-31T12:26:54.217 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:26:54.218 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:26:54.218 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:26:54.218 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:26:54.218 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:54.218 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 15, 2024-08-31T12:26:54.218 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:26:54.218 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:26:54.218 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:26:54.218 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:26:54.218 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:26:54.218 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:26:54.218 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:26:54.218 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:26:54.219 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:26:54.219 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:26:54.219 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:26:54.219 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:26:54.219 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:26:54.219 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:26:54.219 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:54.219 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:26:54.219 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:26:54.219 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:26:54.219 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:26:54.219 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:26:54.219 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:26:54.219 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:26:54.220 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:26:54.220 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:26:54.220 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:26:54.220 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:26:54.220 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:26:54.220 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:26:54.220 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:54.220 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:54.220 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:26:54.220 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:26:54.220 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:26:54.220 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:26:54.220 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:26:54.220 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:26:54.221 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:26:54.221 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:26:54.221 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:26:54.221 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:26:54.221 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:26:54.221 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:26:54.221 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:26:54.221 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:26:54.221 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:26:54.221 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:26:54.221 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:26:54.221 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:26:54.221 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:26:54.221 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:26:54.222 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:26:54.222 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:26:54.222 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:26:54.222 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:26:54.222 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:26:54.222 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:26:54.222 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:26:54.222 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:26:54.222 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:54.222 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:26:54.222 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:54.222 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:26:54.222 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:54.222 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:26:54.222 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:26:54.223 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:26:54.223 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:26:54.223 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:54.223 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:26:54.223 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:26:54.223 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:54.223 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:54.223 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:54.223 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:26:54.223 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:26:54.223 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:54.223 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:54.223 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:54.223 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:54.224 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:26:54.224 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:26:54.224 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:26:54.224 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:26:54.224 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:26:54.224 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:54.224 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:54.224 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:26:54.224 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:26:54.224 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:26:54.224 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:26:54.224 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:54.224 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:26:54.224 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:26:54.225 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:54.225 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:54.225 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:54.225 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:26:54.225 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:26:54.225 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:54.225 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:54.225 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:54.225 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:54.225 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:26:54.225 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:26:54.225 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:26:54.226 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:26:54.226 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:26:54.226 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:54.226 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:54.226 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:26:54.226 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:26:54.226 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:26:54.226 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:26:54.226 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:54.226 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:26:54.226 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:26:54.226 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:54.226 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:54.226 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:54.226 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:26:54.227 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:26:54.227 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:54.227 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:54.227 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:54.227 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:54.227 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:26:54.227 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:26:54.227 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:26:54.227 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:26:54.227 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:26:54.227 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:54.227 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:54.227 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:54.227 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:26:54.228 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:26:54.228 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:54.228 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:26:54.228 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:26:54.228 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:26:54.228 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:54.228 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:54.228 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:26:54.228 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:54.228 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:26:54.228 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:26:54.228 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 4 2024-08-31T12:26:54.228 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:54.228 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:54.228 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:26:54.229 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:54.229 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:26:54.229 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:26:54.229 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 6 2024-08-31T12:26:54.229 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:54.229 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:54.229 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:26:54.229 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:54.229 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:26:54.229 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:26:54.229 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 4 2024-08-31T12:26:54.229 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:54.229 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:54.229 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:26:54.229 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:54.230 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:26:54.230 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:26:54.230 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:26:54.230 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:54.230 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:54.230 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:54.230 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:26:54.230 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:26:54.231 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.215+0000 7f613cff9640 1 -- 172.21.15.19:0/4268926228 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f61400b3a00 msgr2=0x7f61401f9040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:54.231 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.215+0000 7f613cff9640 1 --2- 172.21.15.19:0/4268926228 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f61400b3a00 0x7f61401f9040 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f613401b000 tx=0x7f61340046c0 comp rx=0 tx=0).stop 2024-08-31T12:26:54.231 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.215+0000 7f613cff9640 1 -- 172.21.15.19:0/4268926228 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f61400b3a00 msgr2=0x7f61401f9040 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:26:54.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.215+0000 7f613cff9640 1 --2- 172.21.15.19:0/4268926228 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f61400b3a00 0x7f61401f9040 secure :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f613401b000 tx=0x7f61340046c0 comp rx=0 tx=0).stop 2024-08-31T12:26:54.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.215+0000 7f6146941640 1 -- 172.21.15.19:0/4268926228 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f612c077a80 msgr2=0x7f612c079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:54.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.215+0000 7f6146941640 1 --2- 172.21.15.19:0/4268926228 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f612c077a80 0x7f612c079f90 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f614014d3b0 tx=0x7f6138009500 comp rx=0 tx=0).stop 2024-08-31T12:26:54.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.215+0000 7f6144ede640 1 -- 172.21.15.19:0/4268926228 reap_dead start 2024-08-31T12:26:54.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.215+0000 7f6146941640 1 -- 172.21.15.19:0/4268926228 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f61400c9880 msgr2=0x7f61400c6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:54.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.215+0000 7f6146941640 1 --2- 172.21.15.19:0/4268926228 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f61400c9880 0x7f61400c6120 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6130000c00 tx=0x7f61300038a0 comp rx=0 tx=0).stop 2024-08-31T12:26:54.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.215+0000 7f6146941640 1 -- 172.21.15.19:0/4268926228 shutdown_connections 2024-08-31T12:26:54.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.215+0000 7f6146941640 1 --2- 172.21.15.19:0/4268926228 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f61400c9880 0x7f61400c6120 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:54.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.215+0000 7f6146941640 1 -- 172.21.15.19:0/4268926228 >> 172.21.15.19:0/4268926228 conn(0x7f61400cefb0 msgr2=0x7f614014e0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:26:54.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.216+0000 7f6146941640 1 -- 172.21.15.19:0/4268926228 shutdown_connections 2024-08-31T12:26:54.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.216+0000 7f6146941640 1 -- 172.21.15.19:0/4268926228 wait complete. 2024-08-31T12:26:54.233 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:26:54.234 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-08-31T12:26:54.234 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-31T12:26:54.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.414+0000 7f82625f1640 1 -- 172.21.15.19:0/3901139017 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8248003a30 msgr2=0x7f8248003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:54.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.414+0000 7f82625f1640 1 --2- 172.21.15.19:0/3901139017 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8248003a30 0x7f8248003ea0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f825c0713a0 tx=0x7f8248005830 comp rx=0 tx=0).stop 2024-08-31T12:26:54.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.414+0000 7f8261df0640 1 -- 172.21.15.19:0/3901139017 reap_dead start 2024-08-31T12:26:54.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.415+0000 7f82625f1640 1 -- 172.21.15.19:0/3901139017 shutdown_connections 2024-08-31T12:26:54.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.415+0000 7f82625f1640 1 -- 172.21.15.19:0/3901139017 >> 172.21.15.19:0/3901139017 conn(0x7f825c068ec0 msgr2=0x7f825c0692d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:26:54.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.417+0000 7f82625f1640 1 -- 172.21.15.19:0/3901139017 shutdown_connections 2024-08-31T12:26:54.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.417+0000 7f82625f1640 1 -- 172.21.15.19:0/3901139017 wait complete. 2024-08-31T12:26:54.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.420+0000 7f82625f1640 1 Processor -- start 2024-08-31T12:26:54.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.420+0000 7f82625f1640 1 -- start start 2024-08-31T12:26:54.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.421+0000 7f82625f1640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8248003a30 0x7f825c14a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:54.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.421+0000 7f82625f1640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f825c1f9160 0x7f825c06f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:54.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.421+0000 7f82625f1640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f825c148550 0x7f825c148960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:54.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.421+0000 7f82625f1640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f825c0713d0 con 0x7f825c1f9160 2024-08-31T12:26:54.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.421+0000 7f82625f1640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f825c070d70 con 0x7f8248003a30 2024-08-31T12:26:54.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.421+0000 7f82615ef640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8248003a30 0x7f825c14a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:54.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.422+0000 7f82615ef640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8248003a30 0x7f825c14a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:46276/0 (socket says 172.21.15.19:46276) 2024-08-31T12:26:54.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.422+0000 7f82615ef640 1 -- 172.21.15.19:0/3438302229 learned_addr learned my addr 172.21.15.19:0/3438302229 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:26:54.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.422+0000 7f82625f1640 1 -- 172.21.15.19:0/3438302229 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f825c152f80 con 0x7f825c148550 2024-08-31T12:26:54.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.422+0000 7f8261df0640 1 --2- 172.21.15.19:0/3438302229 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f825c148550 0x7f825c148960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:54.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.422+0000 7f8260dee640 1 --2- 172.21.15.19:0/3438302229 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f825c1f9160 0x7f825c06f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:54.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.422+0000 7f82615ef640 1 -- 172.21.15.19:0/3438302229 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f825c148550 msgr2=0x7f825c148960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:54.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.422+0000 7f82615ef640 1 --2- 172.21.15.19:0/3438302229 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f825c148550 0x7f825c148960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:54.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.422+0000 7f82615ef640 1 -- 172.21.15.19:0/3438302229 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f825c1f9160 msgr2=0x7f825c06f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:54.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.422+0000 7f82615ef640 1 --2- 172.21.15.19:0/3438302229 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f825c1f9160 0x7f825c06f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:54.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.422+0000 7f82615ef640 1 -- 172.21.15.19:0/3438302229 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f825c148ea0 con 0x7f8248003a30 2024-08-31T12:26:54.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.422+0000 7f8260dee640 1 --2- 172.21.15.19:0/3438302229 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f825c1f9160 0x7f825c06f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:26:54.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.423+0000 7f82615ef640 1 --2- 172.21.15.19:0/3438302229 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8248003a30 0x7f825c14a510 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f825802b310 tx=0x7f8258004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:54.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.423+0000 7f8251ffb640 1 -- 172.21.15.19:0/3438302229 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8258038070 con 0x7f8248003a30 2024-08-31T12:26:54.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.423+0000 7f82625f1640 1 -- 172.21.15.19:0/3438302229 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f825c1587a0 con 0x7f8248003a30 2024-08-31T12:26:54.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.423+0000 7f82625f1640 1 -- 172.21.15.19:0/3438302229 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f825c158a40 con 0x7f8248003a30 2024-08-31T12:26:54.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.423+0000 7f8251ffb640 1 -- 172.21.15.19:0/3438302229 <== mon.1 v2:172.21.15.43:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8258039040 con 0x7f8248003a30 2024-08-31T12:26:54.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.423+0000 7f8251ffb640 1 -- 172.21.15.19:0/3438302229 <== mon.1 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8258003830 con 0x7f8248003a30 2024-08-31T12:26:54.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.425+0000 7f82625f1640 1 --2- 172.21.15.19:0/3438302229 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f823c002a70 0x7f823c004f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:54.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.425+0000 7f82625f1640 1 -- 172.21.15.19:0/3438302229 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f823c007f10 con 0x7f823c002a70 2024-08-31T12:26:54.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.425+0000 7f8260dee640 1 --2- 172.21.15.19:0/3438302229 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f823c002a70 0x7f823c004f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:54.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.426+0000 7f8260dee640 1 -- 172.21.15.19:0/3438302229 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f823c008110 con 0x7f823c002a70 2024-08-31T12:26:54.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.426+0000 7f8251ffb640 1 -- 172.21.15.19:0/3438302229 <== mon.1 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f82580039d0 con 0x7f8248003a30 2024-08-31T12:26:54.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.426+0000 7f8260dee640 1 --2- 172.21.15.19:0/3438302229 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f823c002a70 0x7f823c004f80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f825c14eb10 tx=0x7f824c003e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:54.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.427+0000 7f8251ffb640 1 --2- 172.21.15.19:0/3438302229 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8240077a80 0x7f8240079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:54.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.427+0000 7f8251ffb640 1 -- 172.21.15.19:0/3438302229 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f824c00f3e0 con 0x7f823c002a70 2024-08-31T12:26:54.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.427+0000 7f8251ffb640 1 -- 172.21.15.19:0/3438302229 <== mon.1 v2:172.21.15.43:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f82580b9750 con 0x7f8248003a30 2024-08-31T12:26:54.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.427+0000 7f8251ffb640 1 -- 172.21.15.19:0/3438302229 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f824c00f580 con 0x7f823c002a70 2024-08-31T12:26:54.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.427+0000 7f8251ffb640 1 -- 172.21.15.19:0/3438302229 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f823c002a70 msgr2=0x7f823c004f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:54.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.427+0000 7f8251ffb640 1 --2- 172.21.15.19:0/3438302229 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f823c002a70 0x7f823c004f80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f825c14eb10 tx=0x7f824c003e10 comp rx=0 tx=0).stop 2024-08-31T12:26:54.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.427+0000 7f8251ffb640 1 -- 172.21.15.19:0/3438302229 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f823c002a70 msgr2=0x7f823c004f80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:26:54.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.427+0000 7f8251ffb640 1 --2- 172.21.15.19:0/3438302229 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f823c002a70 0x7f823c004f80 secure :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f825c14eb10 tx=0x7f824c003e10 comp rx=0 tx=0).stop 2024-08-31T12:26:54.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.428+0000 7f8261df0640 1 --2- 172.21.15.19:0/3438302229 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8240077a80 0x7f8240079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:54.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.429+0000 7f8261df0640 1 --2- 172.21.15.19:0/3438302229 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8240077a80 0x7f8240079f90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f825c0c88d0 tx=0x7f8254006ef0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:54.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.439+0000 7f82625f1640 1 --2- 172.21.15.19:0/3438302229 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f823c006fa0 0x7f823c007480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:54.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.439+0000 7f82625f1640 1 -- 172.21.15.19:0/3438302229 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f823c00a4f0 con 0x7f823c006fa0 2024-08-31T12:26:54.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.440+0000 7f8260dee640 1 --2- 172.21.15.19:0/3438302229 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f823c006fa0 0x7f823c007480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:54.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.440+0000 7f8260dee640 1 -- 172.21.15.19:0/3438302229 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f823c0079c0 con 0x7f823c006fa0 2024-08-31T12:26:54.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.441+0000 7f8260dee640 1 --2- 172.21.15.19:0/3438302229 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f823c006fa0 0x7f823c007480 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f823c0080e0 tx=0x7f824c001890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:54.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.441+0000 7f8251ffb640 1 -- 172.21.15.19:0/3438302229 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f824c00f3e0 con 0x7f823c006fa0 2024-08-31T12:26:54.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.441+0000 7f8251ffb640 1 -- 172.21.15.19:0/3438302229 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4875 (secure 0 0 0) 0x7f824c01f110 con 0x7f823c006fa0 2024-08-31T12:26:54.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.441+0000 7f8251ffb640 1 -- 172.21.15.19:0/3438302229 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f823c006fa0 msgr2=0x7f823c007480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:54.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.441+0000 7f8251ffb640 1 --2- 172.21.15.19:0/3438302229 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f823c006fa0 0x7f823c007480 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f823c0080e0 tx=0x7f824c001890 comp rx=0 tx=0).stop 2024-08-31T12:26:54.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.441+0000 7f8251ffb640 1 -- 172.21.15.19:0/3438302229 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f823c006fa0 msgr2=0x7f823c007480 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:26:54.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.441+0000 7f8251ffb640 1 --2- 172.21.15.19:0/3438302229 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f823c006fa0 0x7f823c007480 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f823c0080e0 tx=0x7f824c001890 comp rx=0 tx=0).stop 2024-08-31T12:26:54.444 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:26:54.444 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:26:54.444 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:26:54.444 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:26:54.444 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 86, 2024-08-31T12:26:54.444 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 75729, 2024-08-31T12:26:54.444 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:26:54.444 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:26:54.444 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:26:54.445 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:26:54.445 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:54.445 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 15, 2024-08-31T12:26:54.445 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:26:54.445 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:26:54.445 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:26:54.445 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:26:54.445 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:26:54.445 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:26:54.445 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:26:54.445 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:26:54.445 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:26:54.446 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:26:54.446 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:26:54.446 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:26:54.446 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:26:54.446 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:26:54.446 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:54.446 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:26:54.446 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:26:54.446 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:26:54.446 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:26:54.446 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:26:54.446 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:26:54.447 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:26:54.447 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:26:54.447 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:26:54.447 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:26:54.447 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:26:54.447 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:26:54.447 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:26:54.447 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:54.447 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:54.447 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:26:54.447 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:26:54.447 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:26:54.447 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:26:54.447 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:26:54.448 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:26:54.448 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:26:54.448 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:26:54.448 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:26:54.448 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:26:54.448 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:26:54.448 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:26:54.448 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:26:54.448 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:26:54.448 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:26:54.448 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:26:54.448 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:26:54.448 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:26:54.449 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:26:54.449 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:26:54.449 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:26:54.449 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:26:54.449 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:26:54.449 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:26:54.449 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:26:54.449 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:26:54.449 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:26:54.449 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:26:54.449 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:54.449 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:26:54.449 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:54.449 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:26:54.450 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:54.450 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:26:54.450 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:26:54.450 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:26:54.450 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:26:54.450 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:54.450 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:26:54.450 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:26:54.450 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:54.450 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:54.450 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:54.450 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:26:54.450 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:26:54.450 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:54.451 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:54.451 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:54.451 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:54.451 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:26:54.451 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:26:54.451 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:26:54.451 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:26:54.451 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:26:54.451 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:54.451 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:54.451 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:26:54.451 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:26:54.451 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:26:54.452 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:26:54.452 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:54.452 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:26:54.452 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:26:54.452 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:54.452 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:54.452 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:54.452 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:26:54.452 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:26:54.452 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:54.452 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:54.452 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:54.452 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:54.452 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:26:54.453 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:26:54.453 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:26:54.453 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:26:54.453 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:26:54.453 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:54.453 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:54.453 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:26:54.453 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:26:54.453 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:26:54.453 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:26:54.453 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:54.453 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:26:54.453 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:26:54.454 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:54.454 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:54.454 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:54.454 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:26:54.454 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:26:54.454 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:54.454 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:54.454 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:54.454 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:54.454 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:26:54.454 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:26:54.454 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:26:54.454 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:26:54.454 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:26:54.455 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:54.455 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:54.455 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:54.455 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:26:54.455 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:26:54.455 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:54.455 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:26:54.455 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:26:54.455 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:26:54.455 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:54.455 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:54.455 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:26:54.455 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:54.455 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:26:54.456 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:26:54.456 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:26:54.456 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:54.456 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:54.456 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:26:54.456 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:54.456 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:26:54.456 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:26:54.456 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:26:54.456 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:54.456 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:54.456 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:26:54.456 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:54.456 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:26:54.456 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:26:54.457 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 4 2024-08-31T12:26:54.457 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:54.457 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:54.457 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:54.457 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:26:54.457 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:26:54.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.442+0000 7f82625f1640 1 -- 172.21.15.19:0/3438302229 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8240077a80 msgr2=0x7f8240079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:54.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.442+0000 7f82625f1640 1 --2- 172.21.15.19:0/3438302229 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8240077a80 0x7f8240079f90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f825c0c88d0 tx=0x7f8254006ef0 comp rx=0 tx=0).stop 2024-08-31T12:26:54.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.442+0000 7f82625f1640 1 -- 172.21.15.19:0/3438302229 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8248003a30 msgr2=0x7f825c14a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:54.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.442+0000 7f82625f1640 1 --2- 172.21.15.19:0/3438302229 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8248003a30 0x7f825c14a510 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f825802b310 tx=0x7f8258004270 comp rx=0 tx=0).stop 2024-08-31T12:26:54.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.442+0000 7f8261df0640 1 -- 172.21.15.19:0/3438302229 reap_dead start 2024-08-31T12:26:54.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.442+0000 7f8261df0640 1 -- 172.21.15.19:0/3438302229 reap_dead start 2024-08-31T12:26:54.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.442+0000 7f82625f1640 1 -- 172.21.15.19:0/3438302229 shutdown_connections 2024-08-31T12:26:54.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.442+0000 7f82625f1640 1 -- 172.21.15.19:0/3438302229 >> 172.21.15.19:0/3438302229 conn(0x7f825c068ec0 msgr2=0x7f825c0725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:26:54.459 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.442+0000 7f82625f1640 1 -- 172.21.15.19:0/3438302229 shutdown_connections 2024-08-31T12:26:54.459 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.442+0000 7f82625f1640 1 -- 172.21.15.19:0/3438302229 wait complete. 2024-08-31T12:26:54.460 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:26:54.460 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-08-31T12:26:54.460 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-31T12:26:54.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.639+0000 7fba450ab640 1 -- 172.21.15.19:0/660255769 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba28003a80 msgr2=0x7fba28003ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:54.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.639+0000 7fba450ab640 1 --2- 172.21.15.19:0/660255769 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba28003a80 0x7fba28003ef0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fba34006f80 tx=0x7fba28005830 comp rx=0 tx=0).stop 2024-08-31T12:26:54.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.639+0000 7fba3f59e640 1 -- 172.21.15.19:0/660255769 reap_dead start 2024-08-31T12:26:54.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.639+0000 7fba450ab640 1 -- 172.21.15.19:0/660255769 shutdown_connections 2024-08-31T12:26:54.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.639+0000 7fba450ab640 1 -- 172.21.15.19:0/660255769 >> 172.21.15.19:0/660255769 conn(0x7fba400cc950 msgr2=0x7fba400ccd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:26:54.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.640+0000 7fba450ab640 1 -- 172.21.15.19:0/660255769 shutdown_connections 2024-08-31T12:26:54.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.640+0000 7fba450ab640 1 -- 172.21.15.19:0/660255769 wait complete. 2024-08-31T12:26:54.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.642+0000 7fba450ab640 1 Processor -- start 2024-08-31T12:26:54.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.642+0000 7fba450ab640 1 -- start start 2024-08-31T12:26:54.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.644+0000 7fba450ab640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba28003a80 0x7fba4013ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:54.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.644+0000 7fba450ab640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fba40158f40 0x7fba40159a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:54.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.644+0000 7fba450ab640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fba40159fb0 0x7fba401f07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:54.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.644+0000 7fba450ab640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fba400c6240 con 0x7fba40159fb0 2024-08-31T12:26:54.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.644+0000 7fba450ab640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fba400c5f60 con 0x7fba28003a80 2024-08-31T12:26:54.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.644+0000 7fba450ab640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fba400c6520 con 0x7fba40158f40 2024-08-31T12:26:54.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.644+0000 7fba3ed9d640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba28003a80 0x7fba4013ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:54.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.644+0000 7fba3ed9d640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba28003a80 0x7fba4013ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:46318/0 (socket says 172.21.15.19:46318) 2024-08-31T12:26:54.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.644+0000 7fba3ed9d640 1 -- 172.21.15.19:0/522923537 learned_addr learned my addr 172.21.15.19:0/522923537 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:26:54.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.644+0000 7fba3f59e640 1 --2- 172.21.15.19:0/522923537 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fba40159fb0 0x7fba401f07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:54.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.644+0000 7fba3e59c640 1 --2- 172.21.15.19:0/522923537 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fba40158f40 0x7fba40159a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:54.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.644+0000 7fba3ed9d640 1 -- 172.21.15.19:0/522923537 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fba40158f40 msgr2=0x7fba40159a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:54.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.644+0000 7fba3ed9d640 1 --2- 172.21.15.19:0/522923537 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fba40158f40 0x7fba40159a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:54.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.644+0000 7fba3ed9d640 1 -- 172.21.15.19:0/522923537 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fba40159fb0 msgr2=0x7fba401f07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:54.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.645+0000 7fba3ed9d640 1 --2- 172.21.15.19:0/522923537 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fba40159fb0 0x7fba401f07a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:54.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.645+0000 7fba3ed9d640 1 -- 172.21.15.19:0/522923537 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba40155d50 con 0x7fba28003a80 2024-08-31T12:26:54.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.645+0000 7fba3f59e640 1 --2- 172.21.15.19:0/522923537 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fba40159fb0 0x7fba401f07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:26:54.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.645+0000 7fba3ed9d640 1 --2- 172.21.15.19:0/522923537 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba28003a80 0x7fba4013ff00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fba34000c00 tx=0x7fba340064a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:54.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.645+0000 7fba277fe640 1 -- 172.21.15.19:0/522923537 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba34003a20 con 0x7fba28003a80 2024-08-31T12:26:54.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.645+0000 7fba277fe640 1 -- 172.21.15.19:0/522923537 <== mon.1 v2:172.21.15.43:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fba34003cb0 con 0x7fba28003a80 2024-08-31T12:26:54.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.645+0000 7fba277fe640 1 -- 172.21.15.19:0/522923537 <== mon.1 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba34003e60 con 0x7fba28003a80 2024-08-31T12:26:54.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.645+0000 7fba450ab640 1 -- 172.21.15.19:0/522923537 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba401ed310 con 0x7fba28003a80 2024-08-31T12:26:54.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.645+0000 7fba450ab640 1 -- 172.21.15.19:0/522923537 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba401ed8d0 con 0x7fba28003a80 2024-08-31T12:26:54.650 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.647+0000 7fba450ab640 1 --2- 172.21.15.19:0/522923537 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fba401593a0 0x7fba401f0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:54.650 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.647+0000 7fba450ab640 1 -- 172.21.15.19:0/522923537 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fba40155b00 con 0x7fba401593a0 2024-08-31T12:26:54.650 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.647+0000 7fba3e59c640 1 --2- 172.21.15.19:0/522923537 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fba401593a0 0x7fba401f0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:54.650 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.647+0000 7fba3e59c640 1 -- 172.21.15.19:0/522923537 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fba400c5c60 con 0x7fba401593a0 2024-08-31T12:26:54.650 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.648+0000 7fba3e59c640 1 --2- 172.21.15.19:0/522923537 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fba401593a0 0x7fba401f0ce0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fba400c80b0 tx=0x7fba30015600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:54.650 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.648+0000 7fba277fe640 1 -- 172.21.15.19:0/522923537 <== mon.1 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fba34035020 con 0x7fba28003a80 2024-08-31T12:26:54.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.648+0000 7fba277fe640 1 --2- 172.21.15.19:0/522923537 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fba2c077a80 0x7fba2c079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:54.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.649+0000 7fba277fe640 1 -- 172.21.15.19:0/522923537 <== mon.1 v2:172.21.15.43:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fba340b9ae0 con 0x7fba28003a80 2024-08-31T12:26:54.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.649+0000 7fba3f59e640 1 --2- 172.21.15.19:0/522923537 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fba2c077a80 0x7fba2c079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:54.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.649+0000 7fba277fe640 1 -- 172.21.15.19:0/522923537 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba3001f030 con 0x7fba401593a0 2024-08-31T12:26:54.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.649+0000 7fba277fe640 1 -- 172.21.15.19:0/522923537 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fba30005020 con 0x7fba401593a0 2024-08-31T12:26:54.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.649+0000 7fba277fe640 1 -- 172.21.15.19:0/522923537 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fba401593a0 msgr2=0x7fba401f0ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:54.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.649+0000 7fba3f59e640 1 --2- 172.21.15.19:0/522923537 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fba2c077a80 0x7fba2c079f90 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fba40071730 tx=0x7fba38009840 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:54.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.650+0000 7fba277fe640 1 --2- 172.21.15.19:0/522923537 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fba401593a0 0x7fba401f0ce0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fba400c80b0 tx=0x7fba30015600 comp rx=0 tx=0).stop 2024-08-31T12:26:54.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.650+0000 7fba277fe640 1 -- 172.21.15.19:0/522923537 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fba401593a0 msgr2=0x7fba401f0ce0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:26:54.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.650+0000 7fba277fe640 1 --2- 172.21.15.19:0/522923537 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fba401593a0 0x7fba401f0ce0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:54.664 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.661+0000 7fba450ab640 1 --2- 172.21.15.19:0/522923537 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fba401f1220 0x7fba400c1450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:54.664 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.661+0000 7fba450ab640 1 -- 172.21.15.19:0/522923537 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fba401f1630 con 0x7fba401f1220 2024-08-31T12:26:54.664 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.662+0000 7fba3e59c640 1 --2- 172.21.15.19:0/522923537 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fba401f1220 0x7fba400c1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:54.664 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.662+0000 7fba3e59c640 1 -- 172.21.15.19:0/522923537 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fba400c7ad0 con 0x7fba401f1220 2024-08-31T12:26:54.665 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.663+0000 7fba3e59c640 1 --2- 172.21.15.19:0/522923537 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fba401f1220 0x7fba400c1450 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fba400c80b0 tx=0x7fba30004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:54.665 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.663+0000 7fba277fe640 1 -- 172.21.15.19:0/522923537 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba30004e30 con 0x7fba401f1220 2024-08-31T12:26:54.666 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:26:54.666 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:26:54.666 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:26:54.666 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:26:54.666 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 86, 2024-08-31T12:26:54.666 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 20532, 2024-08-31T12:26:54.666 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:26:54.666 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:26:54.666 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:26:54.666 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:26:54.666 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:54.666 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 15, 2024-08-31T12:26:54.667 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:26:54.667 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:26:54.667 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:26:54.667 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:26:54.667 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:26:54.667 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:26:54.667 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:26:54.667 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:26:54.667 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:26:54.667 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:26:54.667 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:26:54.667 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:26:54.667 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:26:54.667 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:26:54.667 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:54.668 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:26:54.668 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:26:54.668 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:26:54.668 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:26:54.668 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:26:54.668 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:26:54.668 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:26:54.668 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:26:54.668 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:26:54.668 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:26:54.668 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:26:54.668 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:26:54.668 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:26:54.668 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:54.669 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:54.669 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:26:54.669 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:26:54.669 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:26:54.669 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:26:54.669 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:26:54.669 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:26:54.669 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:26:54.670 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:26:54.670 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:26:54.670 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:26:54.670 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:26:54.670 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:26:54.670 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:26:54.670 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:26:54.670 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:26:54.670 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:26:54.670 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:26:54.670 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:26:54.671 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:26:54.671 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:26:54.671 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:26:54.671 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:26:54.671 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:26:54.671 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:26:54.671 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:26:54.671 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:26:54.671 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:26:54.671 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:26:54.672 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:54.672 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:26:54.672 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:54.672 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:26:54.672 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:54.673 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:26:54.674 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:26:54.674 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:26:54.675 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:26:54.675 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:54.675 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:26:54.675 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:26:54.675 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:54.675 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:54.676 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:54.676 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:26:54.676 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:26:54.676 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:54.676 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:54.676 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:54.676 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:54.677 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:26:54.677 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:26:54.677 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:26:54.677 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:26:54.677 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:26:54.677 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:54.677 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:54.678 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:26:54.678 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:26:54.678 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:26:54.678 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:26:54.678 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:54.678 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:26:54.679 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:26:54.679 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:54.679 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:54.679 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:54.679 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:26:54.679 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:26:54.679 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:54.679 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:54.679 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:54.679 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:54.679 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:26:54.680 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:26:54.680 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:26:54.680 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:26:54.680 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:26:54.680 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:54.680 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:54.680 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:26:54.680 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:26:54.680 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:26:54.681 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:26:54.681 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:54.681 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:26:54.681 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:26:54.681 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:54.681 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:54.682 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:54.682 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:26:54.682 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:26:54.682 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:26:54.682 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:54.682 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:54.683 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:54.683 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:26:54.683 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:26:54.683 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:26:54.683 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:26:54.683 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:26:54.683 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:54.683 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:54.683 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:54.683 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:26:54.683 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:26:54.684 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:54.684 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:26:54.684 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:26:54.684 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:26:54.684 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:54.684 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:26:54.684 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:26:54.684 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:26:54.684 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:26:54.685 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:26:54.685 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:26:54.685 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:26:54.685 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:26:54.685 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:26:54.685 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:26:54.685 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:26:54.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.663+0000 7fba277fe640 1 -- 172.21.15.19:0/522923537 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4541 (secure 0 0 0) 0x7fba30004e30 con 0x7fba401f1220 2024-08-31T12:26:54.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.663+0000 7fba277fe640 1 -- 172.21.15.19:0/522923537 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fba401f1220 msgr2=0x7fba400c1450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:54.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.663+0000 7fba277fe640 1 --2- 172.21.15.19:0/522923537 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fba401f1220 0x7fba400c1450 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fba400c80b0 tx=0x7fba30004730 comp rx=0 tx=0).stop 2024-08-31T12:26:54.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.663+0000 7fba277fe640 1 -- 172.21.15.19:0/522923537 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fba401f1220 msgr2=0x7fba400c1450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:26:54.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.663+0000 7fba277fe640 1 --2- 172.21.15.19:0/522923537 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fba401f1220 0x7fba400c1450 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:54.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.663+0000 7fba450ab640 1 -- 172.21.15.19:0/522923537 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fba2c077a80 msgr2=0x7fba2c079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:54.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.663+0000 7fba450ab640 1 --2- 172.21.15.19:0/522923537 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fba2c077a80 0x7fba2c079f90 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fba40071730 tx=0x7fba38009840 comp rx=0 tx=0).stop 2024-08-31T12:26:54.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.663+0000 7fba450ab640 1 -- 172.21.15.19:0/522923537 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba28003a80 msgr2=0x7fba4013ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:54.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.663+0000 7fba450ab640 1 --2- 172.21.15.19:0/522923537 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba28003a80 0x7fba4013ff00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fba34000c00 tx=0x7fba340064a0 comp rx=0 tx=0).stop 2024-08-31T12:26:54.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.663+0000 7fba3f59e640 1 -- 172.21.15.19:0/522923537 reap_dead start 2024-08-31T12:26:54.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.664+0000 7fba3f59e640 1 -- 172.21.15.19:0/522923537 reap_dead start 2024-08-31T12:26:54.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.664+0000 7fba450ab640 1 -- 172.21.15.19:0/522923537 shutdown_connections 2024-08-31T12:26:54.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.664+0000 7fba450ab640 1 -- 172.21.15.19:0/522923537 >> 172.21.15.19:0/522923537 conn(0x7fba400cc950 msgr2=0x7fba400c6c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:26:54.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.664+0000 7fba450ab640 1 -- 172.21.15.19:0/522923537 shutdown_connections 2024-08-31T12:26:54.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.664+0000 7fba450ab640 1 -- 172.21.15.19:0/522923537 wait complete. 2024-08-31T12:26:54.689 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:26:54.689 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-08-31T12:26:54.690 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-08-31T12:26:54.690 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-08-31T12:26:54.690 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-08-31T12:26:54.690 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-08-31T12:26:54.690 INFO:teuthology.orchestra.run:waiting for 300 2024-08-31T12:26:54.706 INFO:tasks.ceph.mon.b:Stopped 2024-08-31T12:26:54.706 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-08-31T12:26:54.706 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:26:54.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.846+0000 7fb7ba312640 1 -- 172.21.15.19:0/930481235 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb7b40cc070 msgr2=0x7fb7b40cc480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:54.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.846+0000 7fb7ba312640 1 --2- 172.21.15.19:0/930481235 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb7b40cc070 0x7fb7b40cc480 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb7a8005fd0 tx=0x7fb7a802a580 comp rx=0 tx=0).stop 2024-08-31T12:26:54.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.846+0000 7fb7ba312640 1 -- 172.21.15.19:0/930481235 shutdown_connections 2024-08-31T12:26:54.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.846+0000 7fb7ba312640 1 --2- 172.21.15.19:0/930481235 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb7b40cc070 0x7fb7b40cc480 secure :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb7a8005fd0 tx=0x7fb7a802a580 comp rx=0 tx=0).stop 2024-08-31T12:26:54.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.846+0000 7fb7ba312640 1 --2- 172.21.15.19:0/930481235 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb7b40ccbb0 0x7fb7b40cb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:54.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.846+0000 7fb7ba312640 1 -- 172.21.15.19:0/930481235 >> 172.21.15.19:0/930481235 conn(0x7fb7b4074810 msgr2=0x7fb7b4068aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:26:54.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.847+0000 7fb7ba312640 1 -- 172.21.15.19:0/930481235 shutdown_connections 2024-08-31T12:26:54.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.847+0000 7fb7ba312640 1 -- 172.21.15.19:0/930481235 wait complete. 2024-08-31T12:26:54.852 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.850+0000 7fb7ba312640 1 Processor -- start 2024-08-31T12:26:54.852 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.850+0000 7fb7ba312640 1 -- start start 2024-08-31T12:26:54.854 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.853+0000 7fb7ba312640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb7b40ccbb0 0x7fb7b414eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:54.855 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.853+0000 7fb7ba312640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb7b415b120 0x7fb7b40c7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:54.856 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.853+0000 7fb7b3fff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb7b40ccbb0 0x7fb7b414eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:54.856 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.853+0000 7fb7b3fff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb7b40ccbb0 0x7fb7b414eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.19:53290/0 (socket says 172.21.15.19:53290) 2024-08-31T12:26:54.856 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.853+0000 7fb7b37fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb7b415b120 0x7fb7b40c7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:54.856 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.853+0000 7fb7b37fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb7b415b120 0x7fb7b40c7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:56444/0 (socket says 172.21.15.19:56444) 2024-08-31T12:26:54.856 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.853+0000 7fb7ba312640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb7b40d11a0 0x7fb7b415d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:54.856 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.853+0000 7fb7ba312640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b41557e0 con 0x7fb7b415b120 2024-08-31T12:26:54.856 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.853+0000 7fb7ba312640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b4155660 con 0x7fb7b40d11a0 2024-08-31T12:26:54.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.853+0000 7fb7ba312640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fb7b406e1a0 con 0x7fb7b40ccbb0 2024-08-31T12:26:54.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.853+0000 7fb7b3fff640 1 -- 172.21.15.19:0/3844281173 learned_addr learned my addr 172.21.15.19:0/3844281173 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:26:54.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.853+0000 7fb7b88af640 1 -- 172.21.15.19:0/3844281173 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb7b40d11a0 msgr2=0x7fb7b415d590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3300/0 2024-08-31T12:26:54.858 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.853+0000 7fb7b88af640 1 --2- 172.21.15.19:0/3844281173 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb7b40d11a0 0x7fb7b415d590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:26:54.858 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.854+0000 7fb7b37fe640 1 -- 172.21.15.19:0/3844281173 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb7b40ccbb0 msgr2=0x7fb7b414eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:54.858 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.854+0000 7fb7b37fe640 1 --2- 172.21.15.19:0/3844281173 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb7b40ccbb0 0x7fb7b414eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:54.858 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.854+0000 7fb7b37fe640 1 -- 172.21.15.19:0/3844281173 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb7b40d11a0 msgr2=0x7fb7b415d590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:26:54.858 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.854+0000 7fb7b37fe640 1 --2- 172.21.15.19:0/3844281173 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb7b40d11a0 0x7fb7b415d590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:54.858 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.854+0000 7fb7b37fe640 1 -- 172.21.15.19:0/3844281173 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7b40c8200 con 0x7fb7b415b120 2024-08-31T12:26:54.858 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.854+0000 7fb7b37fe640 1 --2- 172.21.15.19:0/3844281173 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb7b415b120 0x7fb7b40c7cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb7b406e560 tx=0x7fb7a8003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:54.858 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.855+0000 7fb7b0ff9640 1 -- 172.21.15.19:0/3844281173 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a8035030 con 0x7fb7b415b120 2024-08-31T12:26:54.858 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.855+0000 7fb7b0ff9640 1 -- 172.21.15.19:0/3844281173 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb7a8042070 con 0x7fb7b415b120 2024-08-31T12:26:54.858 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.855+0000 7fb7b0ff9640 1 -- 172.21.15.19:0/3844281173 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a8004a00 con 0x7fb7b415b120 2024-08-31T12:26:54.858 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.855+0000 7fb7ba312640 1 -- 172.21.15.19:0/3844281173 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7b41f6520 con 0x7fb7b415b120 2024-08-31T12:26:54.858 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.855+0000 7fb7ba312640 1 -- 172.21.15.19:0/3844281173 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7b41f6ae0 con 0x7fb7b415b120 2024-08-31T12:26:54.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.858+0000 7fb7ba312640 1 -- 172.21.15.19:0/3844281173 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7b4063cf0 con 0x7fb7b415b120 2024-08-31T12:26:54.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.859+0000 7fb7b0ff9640 1 -- 172.21.15.19:0/3844281173 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fb7a8004ba0 con 0x7fb7b415b120 2024-08-31T12:26:54.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.860+0000 7fb7b0ff9640 1 --2- 172.21.15.19:0/3844281173 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb7a0077a30 0x7fb7a0079f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:54.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.860+0000 7fb7b3fff640 1 --2- 172.21.15.19:0/3844281173 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb7a0077a30 0x7fb7a0079f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:54.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.860+0000 7fb7b0ff9640 1 -- 172.21.15.19:0/3844281173 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fb7a80ba4a0 con 0x7fb7b415b120 2024-08-31T12:26:54.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.861+0000 7fb7b3fff640 1 --2- 172.21.15.19:0/3844281173 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb7a0077a30 0x7fb7a0079f40 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac009f80 tx=0x7fb7ac009600 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:54.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:54.868+0000 7fb7b0ff9640 1 -- 172.21.15.19:0/3844281173 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fb7b4063cf0 con 0x7fb7b415b120 2024-08-31T12:26:55.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:55.322+0000 7fb7ba312640 1 -- 172.21.15.19:0/3844281173 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb7b40cb280 con 0x7fb7b415b120 2024-08-31T12:26:55.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:55.323+0000 7fb7b0ff9640 1 -- 172.21.15.19:0/3844281173 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fb7a8083190 con 0x7fb7b415b120 2024-08-31T12:26:55.325 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":86,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:26:55.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:55.327+0000 7fb7ba312640 1 -- 172.21.15.19:0/3844281173 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb7a0077a30 msgr2=0x7fb7a0079f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:55.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:55.327+0000 7fb7ba312640 1 --2- 172.21.15.19:0/3844281173 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb7a0077a30 0x7fb7a0079f40 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac009f80 tx=0x7fb7ac009600 comp rx=0 tx=0).stop 2024-08-31T12:26:55.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:55.328+0000 7fb7ba312640 1 -- 172.21.15.19:0/3844281173 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb7b415b120 msgr2=0x7fb7b40c7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:55.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:55.328+0000 7fb7ba312640 1 --2- 172.21.15.19:0/3844281173 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb7b415b120 0x7fb7b40c7cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb7b406e560 tx=0x7fb7a8003b90 comp rx=0 tx=0).stop 2024-08-31T12:26:55.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:55.328+0000 7fb7ba312640 1 -- 172.21.15.19:0/3844281173 shutdown_connections 2024-08-31T12:26:55.331 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:55.328+0000 7fb7ba312640 1 --2- 172.21.15.19:0/3844281173 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fb7a0077a30 0x7fb7a0079f40 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:55.331 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:55.328+0000 7fb7ba312640 1 --2- 172.21.15.19:0/3844281173 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb7b40d11a0 0x7fb7b415d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:55.331 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:55.328+0000 7fb7ba312640 1 --2- 172.21.15.19:0/3844281173 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb7b415b120 0x7fb7b40c7cc0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:55.331 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:55.328+0000 7fb7ba312640 1 --2- 172.21.15.19:0/3844281173 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb7b40ccbb0 0x7fb7b414eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:55.331 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:55.328+0000 7fb7ba312640 1 -- 172.21.15.19:0/3844281173 >> 172.21.15.19:0/3844281173 conn(0x7fb7b4074810 msgr2=0x7fb7b4068aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:26:55.331 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:55.329+0000 7fb7ba312640 1 -- 172.21.15.19:0/3844281173 shutdown_connections 2024-08-31T12:26:55.331 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:55.329+0000 7fb7ba312640 1 -- 172.21.15.19:0/3844281173 wait complete. 2024-08-31T12:26:57.899 DEBUG:teuthology.orchestra.run.smithi019:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:26:57.901 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:26:58.349 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:26:58.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:58.517+0000 7f1f59e86640 1 -- 172.21.15.19:0/2344544247 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1f540c6dd0 msgr2=0x7f1f540c71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:58.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:58.517+0000 7f1f59e86640 1 --2- 172.21.15.19:0/2344544247 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1f540c6dd0 0x7f1f540c71e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c005fd0 tx=0x7f1f4c02a820 comp rx=0 tx=0).stop 2024-08-31T12:26:58.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:58.517+0000 7f1f59e86640 1 -- 172.21.15.19:0/2344544247 shutdown_connections 2024-08-31T12:26:58.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:58.517+0000 7f1f59e86640 1 --2- 172.21.15.19:0/2344544247 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1f540c6dd0 0x7f1f540c71e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:58.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:58.517+0000 7f1f59e86640 1 -- 172.21.15.19:0/2344544247 >> 172.21.15.19:0/2344544247 conn(0x7f1f540ca630 msgr2=0x7f1f540cd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:26:58.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:58.518+0000 7f1f59e86640 1 -- 172.21.15.19:0/2344544247 shutdown_connections 2024-08-31T12:26:58.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:58.518+0000 7f1f59e86640 1 -- 172.21.15.19:0/2344544247 wait complete. 2024-08-31T12:26:58.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:58.521+0000 7f1f59e86640 1 Processor -- start 2024-08-31T12:26:58.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:58.521+0000 7f1f59e86640 1 -- start start 2024-08-31T12:26:58.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:58.523+0000 7f1f59e86640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f540c53c0 0x7f1f54149dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:58.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:58.523+0000 7f1f59e86640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1f540c6dd0 0x7f1f54073dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:58.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:58.523+0000 7f1f59e86640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1f540c7f00 0x7f1f54145c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:58.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:58.523+0000 7f1f537fe640 1 -- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f540c53c0 msgr2=0x7f1f54149dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3300/0 2024-08-31T12:26:58.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:58.523+0000 7f1f537fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f540c53c0 0x7f1f54149dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:26:58.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:58.524+0000 7f1f52ffd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1f540c6dd0 0x7f1f54073dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:58.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:58.524+0000 7f1f52ffd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1f540c6dd0 0x7f1f54073dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.19:50880/0 (socket says 172.21.15.19:50880) 2024-08-31T12:26:58.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:58.524+0000 7f1f53fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1f540c7f00 0x7f1f54145c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:58.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:58.524+0000 7f1f53fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1f540c7f00 0x7f1f54145c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:54184/0 (socket says 172.21.15.19:54184) 2024-08-31T12:26:58.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:58.524+0000 7f1f52ffd640 1 -- 172.21.15.19:0/1237964322 learned_addr learned my addr 172.21.15.19:0/1237964322 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:26:58.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:58.524+0000 7f1f59e86640 1 -- 172.21.15.19:0/1237964322 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f1f540ce240 con 0x7f1f540c7f00 2024-08-31T12:26:58.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:58.524+0000 7f1f59e86640 1 -- 172.21.15.19:0/1237964322 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1f540cded0 con 0x7f1f540c53c0 2024-08-31T12:26:58.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:58.524+0000 7f1f59e86640 1 -- 172.21.15.19:0/1237964322 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f1f5406e300 con 0x7f1f540c6dd0 2024-08-31T12:26:58.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:58.525+0000 7f1f53fff640 1 -- 172.21.15.19:0/1237964322 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1f540c6dd0 msgr2=0x7f1f54073dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:58.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:58.525+0000 7f1f53fff640 1 --2- 172.21.15.19:0/1237964322 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1f540c6dd0 0x7f1f54073dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:58.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:58.525+0000 7f1f53fff640 1 -- 172.21.15.19:0/1237964322 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f540c53c0 msgr2=0x7f1f54149dc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:26:58.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:58.525+0000 7f1f53fff640 1 --2- 172.21.15.19:0/1237964322 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f540c53c0 0x7f1f54149dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:58.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:58.525+0000 7f1f53fff640 1 -- 172.21.15.19:0/1237964322 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f541492d0 con 0x7f1f540c7f00 2024-08-31T12:26:58.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:58.525+0000 7f1f53fff640 1 --2- 172.21.15.19:0/1237964322 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1f540c7f00 0x7f1f54145c70 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1f5406e7a0 tx=0x7f1f4c0092e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:58.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:58.526+0000 7f1f3bfff640 1 -- 172.21.15.19:0/1237964322 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f4c037040 con 0x7f1f540c7f00 2024-08-31T12:26:58.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:58.526+0000 7f1f3bfff640 1 -- 172.21.15.19:0/1237964322 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1f4c042070 con 0x7f1f540c7f00 2024-08-31T12:26:58.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:58.526+0000 7f1f59e86640 1 -- 172.21.15.19:0/1237964322 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f54152ce0 con 0x7f1f540c7f00 2024-08-31T12:26:58.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:58.526+0000 7f1f3bfff640 1 -- 172.21.15.19:0/1237964322 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f4c0043d0 con 0x7f1f540c7f00 2024-08-31T12:26:58.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:58.526+0000 7f1f59e86640 1 -- 172.21.15.19:0/1237964322 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f541eee40 con 0x7f1f540c7f00 2024-08-31T12:26:58.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:58.528+0000 7f1f59e86640 1 -- 172.21.15.19:0/1237964322 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f34005290 con 0x7f1f540c7f00 2024-08-31T12:26:58.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:58.528+0000 7f1f3bfff640 1 -- 172.21.15.19:0/1237964322 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f1f4c004570 con 0x7f1f540c7f00 2024-08-31T12:26:58.531 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:58.529+0000 7f1f3bfff640 1 --2- 172.21.15.19:0/1237964322 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1f40079c40 0x7f1f4007c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:26:58.531 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:58.529+0000 7f1f3bfff640 1 -- 172.21.15.19:0/1237964322 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f1f4c0ba280 con 0x7f1f540c7f00 2024-08-31T12:26:58.531 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:58.529+0000 7f1f537fe640 1 --2- 172.21.15.19:0/1237964322 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1f40079c40 0x7f1f4007c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:26:58.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:58.530+0000 7f1f537fe640 1 --2- 172.21.15.19:0/1237964322 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1f40079c40 0x7f1f4007c150 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1f440010a0 tx=0x7f1f44006d10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:26:58.539 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:58.537+0000 7f1f3bfff640 1 -- 172.21.15.19:0/1237964322 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f1f541492d0 con 0x7f1f540c7f00 2024-08-31T12:26:59.002 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:59.000+0000 7f1f59e86640 1 -- 172.21.15.19:0/1237964322 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1f34005850 con 0x7f1f540c7f00 2024-08-31T12:26:59.003 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:59.001+0000 7f1f3bfff640 1 -- 172.21.15.19:0/1237964322 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f1f4c082f80 con 0x7f1f540c7f00 2024-08-31T12:26:59.003 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":86,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":19,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:26:59.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:59.004+0000 7f1f59e86640 1 -- 172.21.15.19:0/1237964322 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1f40079c40 msgr2=0x7f1f4007c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:59.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:59.004+0000 7f1f59e86640 1 --2- 172.21.15.19:0/1237964322 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1f40079c40 0x7f1f4007c150 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1f440010a0 tx=0x7f1f44006d10 comp rx=0 tx=0).stop 2024-08-31T12:26:59.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:59.005+0000 7f1f59e86640 1 -- 172.21.15.19:0/1237964322 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1f540c7f00 msgr2=0x7f1f54145c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:26:59.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:59.005+0000 7f1f59e86640 1 --2- 172.21.15.19:0/1237964322 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1f540c7f00 0x7f1f54145c70 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1f5406e7a0 tx=0x7f1f4c0092e0 comp rx=0 tx=0).stop 2024-08-31T12:26:59.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:59.005+0000 7f1f59e86640 1 -- 172.21.15.19:0/1237964322 shutdown_connections 2024-08-31T12:26:59.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:59.005+0000 7f1f59e86640 1 --2- 172.21.15.19:0/1237964322 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1f40079c40 0x7f1f4007c150 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:59.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:59.005+0000 7f1f59e86640 1 --2- 172.21.15.19:0/1237964322 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1f540c7f00 0x7f1f54145c70 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:59.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:59.005+0000 7f1f59e86640 1 --2- 172.21.15.19:0/1237964322 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1f540c6dd0 0x7f1f54073dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:59.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:59.005+0000 7f1f59e86640 1 --2- 172.21.15.19:0/1237964322 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f540c53c0 0x7f1f54149dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:26:59.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:59.005+0000 7f1f59e86640 1 -- 172.21.15.19:0/1237964322 >> 172.21.15.19:0/1237964322 conn(0x7f1f540ca630 msgr2=0x7f1f54151ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:26:59.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:59.005+0000 7f1f59e86640 1 -- 172.21.15.19:0/1237964322 shutdown_connections 2024-08-31T12:26:59.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:26:59.005+0000 7f1f59e86640 1 -- 172.21.15.19:0/1237964322 wait complete. 2024-08-31T12:27:02.024 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:27:02.203 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.200+0000 7f80fa9c3640 1 -- 172.21.15.19:0/2038206324 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f80f40c6e60 msgr2=0x7f80f40c4470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:02.203 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.200+0000 7f80fa9c3640 1 --2- 172.21.15.19:0/2038206324 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f80f40c6e60 0x7f80f40c4470 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f80e8009830 tx=0x7f80e802a510 comp rx=0 tx=0).stop 2024-08-31T12:27:02.203 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.200+0000 7f80fa9c3640 1 -- 172.21.15.19:0/2038206324 shutdown_connections 2024-08-31T12:27:02.203 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.200+0000 7f80fa9c3640 1 --2- 172.21.15.19:0/2038206324 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f80f4151ee0 0x7f80f4157e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:02.203 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.200+0000 7f80fa9c3640 1 --2- 172.21.15.19:0/2038206324 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f80f40c6e60 0x7f80f40c4470 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:02.203 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.201+0000 7f80fa9c3640 1 -- 172.21.15.19:0/2038206324 >> 172.21.15.19:0/2038206324 conn(0x7f80f40cc770 msgr2=0x7f80f40ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:27:02.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.201+0000 7f80fa9c3640 1 -- 172.21.15.19:0/2038206324 shutdown_connections 2024-08-31T12:27:02.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.201+0000 7f80fa9c3640 1 -- 172.21.15.19:0/2038206324 wait complete. 2024-08-31T12:27:02.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.203+0000 7f80fa9c3640 1 Processor -- start 2024-08-31T12:27:02.206 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.204+0000 7f80fa9c3640 1 -- start start 2024-08-31T12:27:02.208 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.206+0000 7f80fa9c3640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f80f40c56c0 0x7f80f41f21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:02.208 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.206+0000 7f80fa9c3640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80f4151ee0 0x7f80f414daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:02.208 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.206+0000 7f80fa9c3640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f80f41e3630 0x7f80f40cb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:02.209 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.206+0000 7f80fa9c3640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f80f406d550 con 0x7f80f41e3630 2024-08-31T12:27:02.209 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.206+0000 7f80fa9c3640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f80f40c6230 con 0x7f80f4151ee0 2024-08-31T12:27:02.209 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.206+0000 7f80fa9c3640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f80f406d6d0 con 0x7f80f40c56c0 2024-08-31T12:27:02.209 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.206+0000 7f80f37fe640 1 -- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80f4151ee0 msgr2=0x7f80f414daa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3300/0 2024-08-31T12:27:02.209 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.206+0000 7f80f37fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80f4151ee0 0x7f80f414daa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:27:02.209 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.206+0000 7f80f3fff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f80f40c56c0 0x7f80f41f21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:02.209 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.206+0000 7f80f3fff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f80f40c56c0 0x7f80f41f21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.19:50902/0 (socket says 172.21.15.19:50902) 2024-08-31T12:27:02.209 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.206+0000 7f80f3fff640 1 -- 172.21.15.19:0/2265061346 learned_addr learned my addr 172.21.15.19:0/2265061346 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:27:02.209 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.206+0000 7f80f8f60640 1 --2- 172.21.15.19:0/2265061346 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f80f41e3630 0x7f80f40cb6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:02.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.207+0000 7f80f3fff640 1 -- 172.21.15.19:0/2265061346 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80f4151ee0 msgr2=0x7f80f414daa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:27:02.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.207+0000 7f80f3fff640 1 --2- 172.21.15.19:0/2265061346 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80f4151ee0 0x7f80f414daa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:02.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.207+0000 7f80f3fff640 1 -- 172.21.15.19:0/2265061346 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f80f41e3630 msgr2=0x7f80f40cb6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:02.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.207+0000 7f80f3fff640 1 --2- 172.21.15.19:0/2265061346 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f80f41e3630 0x7f80f40cb6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:02.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.207+0000 7f80f3fff640 1 -- 172.21.15.19:0/2265061346 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80f41e4160 con 0x7f80f40c56c0 2024-08-31T12:27:02.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.207+0000 7f80f3fff640 1 --2- 172.21.15.19:0/2265061346 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f80f40c56c0 0x7f80f41f21d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f80e400c3a0 tx=0x7f80e400c860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:02.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.207+0000 7f80f0ff9640 1 -- 172.21.15.19:0/2265061346 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e4011790 con 0x7f80f40c56c0 2024-08-31T12:27:02.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.207+0000 7f80fa9c3640 1 -- 172.21.15.19:0/2265061346 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80f41e2310 con 0x7f80f40c56c0 2024-08-31T12:27:02.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.207+0000 7f80f0ff9640 1 -- 172.21.15.19:0/2265061346 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f80e4017070 con 0x7f80f40c56c0 2024-08-31T12:27:02.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.208+0000 7f80f0ff9640 1 -- 172.21.15.19:0/2265061346 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e4011cc0 con 0x7f80f40c56c0 2024-08-31T12:27:02.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.208+0000 7f80fa9c3640 1 -- 172.21.15.19:0/2265061346 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f80f41e28d0 con 0x7f80f40c56c0 2024-08-31T12:27:02.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.209+0000 7f80fa9c3640 1 -- 172.21.15.19:0/2265061346 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80f40c4470 con 0x7f80f40c56c0 2024-08-31T12:27:02.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.210+0000 7f80f0ff9640 1 -- 172.21.15.19:0/2265061346 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f80e401c460 con 0x7f80f40c56c0 2024-08-31T12:27:02.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.210+0000 7f80f0ff9640 1 --2- 172.21.15.19:0/2265061346 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f80e0077a30 0x7f80e0079f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:02.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.210+0000 7f80f0ff9640 1 -- 172.21.15.19:0/2265061346 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f80e4099480 con 0x7f80f40c56c0 2024-08-31T12:27:02.214 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.212+0000 7f80f37fe640 1 --2- 172.21.15.19:0/2265061346 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f80e0077a30 0x7f80e0079f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:02.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.213+0000 7f80f37fe640 1 --2- 172.21.15.19:0/2265061346 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f80e0077a30 0x7f80e0079f40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f80e8034000 tx=0x7f80e8004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:02.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.216+0000 7f80f0ff9640 1 -- 172.21.15.19:0/2265061346 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f80f41522f0 con 0x7f80f40c56c0 2024-08-31T12:27:02.682 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.680+0000 7f80fa9c3640 1 -- 172.21.15.19:0/2265061346 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f80f40c7b40 con 0x7f80f40c56c0 2024-08-31T12:27:02.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.681+0000 7f80f0ff9640 1 -- 172.21.15.19:0/2265061346 <== mon.2 v2:172.21.15.43:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f80e4062170 con 0x7f80f40c56c0 2024-08-31T12:27:02.683 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":86,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":23,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:27:02.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.684+0000 7f80fa9c3640 1 -- 172.21.15.19:0/2265061346 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f80e0077a30 msgr2=0x7f80e0079f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:02.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.684+0000 7f80fa9c3640 1 --2- 172.21.15.19:0/2265061346 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f80e0077a30 0x7f80e0079f40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f80e8034000 tx=0x7f80e8004730 comp rx=0 tx=0).stop 2024-08-31T12:27:02.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.684+0000 7f80fa9c3640 1 -- 172.21.15.19:0/2265061346 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f80f40c56c0 msgr2=0x7f80f41f21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:02.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.684+0000 7f80fa9c3640 1 --2- 172.21.15.19:0/2265061346 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f80f40c56c0 0x7f80f41f21d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f80e400c3a0 tx=0x7f80e400c860 comp rx=0 tx=0).stop 2024-08-31T12:27:02.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.685+0000 7f80fa9c3640 1 -- 172.21.15.19:0/2265061346 shutdown_connections 2024-08-31T12:27:02.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.685+0000 7f80fa9c3640 1 --2- 172.21.15.19:0/2265061346 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f80e0077a30 0x7f80e0079f40 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:02.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.685+0000 7f80fa9c3640 1 --2- 172.21.15.19:0/2265061346 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f80f41e3630 0x7f80f40cb6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:02.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.685+0000 7f80fa9c3640 1 --2- 172.21.15.19:0/2265061346 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80f4151ee0 0x7f80f414daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:02.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.685+0000 7f80fa9c3640 1 --2- 172.21.15.19:0/2265061346 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f80f40c56c0 0x7f80f41f21d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:02.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.685+0000 7f80fa9c3640 1 -- 172.21.15.19:0/2265061346 >> 172.21.15.19:0/2265061346 conn(0x7f80f40cc770 msgr2=0x7f80f40c86e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:27:02.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.685+0000 7f80fa9c3640 1 -- 172.21.15.19:0/2265061346 shutdown_connections 2024-08-31T12:27:02.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:02.685+0000 7f80fa9c3640 1 -- 172.21.15.19:0/2265061346 wait complete. 2024-08-31T12:27:05.704 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:27:09.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.116+0000 7f18c224d640 1 -- 172.21.15.19:0/1713246209 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f18bc157f60 msgr2=0x7f18a40039f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:09.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.116+0000 7f18c224d640 1 --2- 172.21.15.19:0/1713246209 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f18bc157f60 0x7f18a40039f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f18ac009f60 tx=0x7f18a4005830 comp rx=0 tx=0).stop 2024-08-31T12:27:09.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.116+0000 7f18c224d640 1 -- 172.21.15.19:0/1713246209 shutdown_connections 2024-08-31T12:27:09.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.116+0000 7f18c224d640 1 --2- 172.21.15.19:0/1713246209 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f18a4004e10 0x7f18a40052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:09.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.116+0000 7f18c224d640 1 --2- 172.21.15.19:0/1713246209 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f18a4004100 0x7f18a4004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:09.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.116+0000 7f18c224d640 1 --2- 172.21.15.19:0/1713246209 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f18bc157f60 0x7f18a40039f0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:09.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.116+0000 7f18c224d640 1 -- 172.21.15.19:0/1713246209 >> 172.21.15.19:0/1713246209 conn(0x7f18bc06cae0 msgr2=0x7f18bc06cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:27:09.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.116+0000 7f18c224d640 1 -- 172.21.15.19:0/1713246209 shutdown_connections 2024-08-31T12:27:09.120 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.116+0000 7f18c224d640 1 -- 172.21.15.19:0/1713246209 wait complete. 2024-08-31T12:27:09.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.119+0000 7f18c224d640 1 Processor -- start 2024-08-31T12:27:09.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.119+0000 7f18c224d640 1 -- start start 2024-08-31T12:27:09.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.121+0000 7f18c224d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f18a4004100 0x7f18bc1edb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:09.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.121+0000 7f18bb7fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f18a4004100 0x7f18bc1edb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:09.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.122+0000 7f18bb7fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f18a4004100 0x7f18bc1edb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:49444/0 (socket says 172.21.15.19:49444) 2024-08-31T12:27:09.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.122+0000 7f18bb7fe640 1 -- 172.21.15.19:0/1992854194 learned_addr learned my addr 172.21.15.19:0/1992854194 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:27:09.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.122+0000 7f18c224d640 1 --2- 172.21.15.19:0/1992854194 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f18a4004e10 0x7f18bc0cbf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:09.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.122+0000 7f18c224d640 1 --2- 172.21.15.19:0/1992854194 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f18bc157f60 0x7f18bc154610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:09.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.122+0000 7f18c224d640 1 -- 172.21.15.19:0/1992854194 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f18bc0c8c90 con 0x7f18a4004100 2024-08-31T12:27:09.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.122+0000 7f18bbfff640 1 -- 172.21.15.19:0/1992854194 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f18bc157f60 msgr2=0x7f18bc154610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3300/0 2024-08-31T12:27:09.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.122+0000 7f18bbfff640 1 --2- 172.21.15.19:0/1992854194 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f18bc157f60 0x7f18bc154610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:27:09.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.122+0000 7f18c224d640 1 -- 172.21.15.19:0/1992854194 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f18bc157760 con 0x7f18bc157f60 2024-08-31T12:27:09.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.122+0000 7f18c224d640 1 -- 172.21.15.19:0/1992854194 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f18bc157410 con 0x7f18a4004e10 2024-08-31T12:27:09.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.123+0000 7f18baffd640 1 --2- 172.21.15.19:0/1992854194 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f18a4004e10 0x7f18bc0cbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:09.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.123+0000 7f18bb7fe640 1 -- 172.21.15.19:0/1992854194 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f18a4004e10 msgr2=0x7f18bc0cbf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:09.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.123+0000 7f18bb7fe640 1 --2- 172.21.15.19:0/1992854194 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f18a4004e10 0x7f18bc0cbf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:09.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.123+0000 7f18bb7fe640 1 -- 172.21.15.19:0/1992854194 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f18bc157f60 msgr2=0x7f18bc154610 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:27:09.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.123+0000 7f18bb7fe640 1 --2- 172.21.15.19:0/1992854194 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f18bc157f60 0x7f18bc154610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:09.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.123+0000 7f18bb7fe640 1 -- 172.21.15.19:0/1992854194 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18bc145f60 con 0x7f18a4004100 2024-08-31T12:27:09.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.123+0000 7f18baffd640 1 --2- 172.21.15.19:0/1992854194 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f18a4004e10 0x7f18bc0cbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-31T12:27:09.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.124+0000 7f18bb7fe640 1 --2- 172.21.15.19:0/1992854194 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f18a4004100 0x7f18bc1edb10 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f18ac00b140 tx=0x7f18ac031e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:09.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.124+0000 7f18a3fff640 1 -- 172.21.15.19:0/1992854194 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18ac040070 con 0x7f18a4004100 2024-08-31T12:27:09.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.124+0000 7f18c224d640 1 -- 172.21.15.19:0/1992854194 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18bc146200 con 0x7f18a4004100 2024-08-31T12:27:09.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.124+0000 7f18c224d640 1 -- 172.21.15.19:0/1992854194 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18bc14d580 con 0x7f18a4004100 2024-08-31T12:27:09.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.124+0000 7f18a3fff640 1 -- 172.21.15.19:0/1992854194 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f18ac03c620 con 0x7f18a4004100 2024-08-31T12:27:09.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.124+0000 7f18a3fff640 1 -- 172.21.15.19:0/1992854194 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18ac009480 con 0x7f18a4004100 2024-08-31T12:27:09.129 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.126+0000 7f18c224d640 1 -- 172.21.15.19:0/1992854194 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f189c005290 con 0x7f18a4004100 2024-08-31T12:27:09.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.128+0000 7f18a3fff640 1 -- 172.21.15.19:0/1992854194 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f18ac039040 con 0x7f18a4004100 2024-08-31T12:27:09.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.129+0000 7f18a3fff640 1 --2- 172.21.15.19:0/1992854194 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f18a8078550 0x7f18a807aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:09.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.129+0000 7f18a3fff640 1 -- 172.21.15.19:0/1992854194 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f18ac0ba150 con 0x7f18a4004100 2024-08-31T12:27:09.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.129+0000 7f18baffd640 1 --2- 172.21.15.19:0/1992854194 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f18a8078550 0x7f18a807aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:09.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.130+0000 7f18baffd640 1 --2- 172.21.15.19:0/1992854194 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f18a8078550 0x7f18a807aa60 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f18b0005940 tx=0x7f18b00058d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:09.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.137+0000 7f18a3fff640 1 -- 172.21.15.19:0/1992854194 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f18ac082e30 con 0x7f18a4004100 2024-08-31T12:27:09.594 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.592+0000 7f18c224d640 1 -- 172.21.15.19:0/1992854194 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f189c005580 con 0x7f18a4004100 2024-08-31T12:27:09.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.593+0000 7f18a3fff640 1 -- 172.21.15.19:0/1992854194 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7f18ac082730 con 0x7f18a4004100 2024-08-31T12:27:09.595 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":90,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:27:09.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.596+0000 7f18c224d640 1 -- 172.21.15.19:0/1992854194 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f18a8078550 msgr2=0x7f18a807aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:09.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.596+0000 7f18c224d640 1 --2- 172.21.15.19:0/1992854194 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f18a8078550 0x7f18a807aa60 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f18b0005940 tx=0x7f18b00058d0 comp rx=0 tx=0).stop 2024-08-31T12:27:09.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.597+0000 7f18c224d640 1 -- 172.21.15.19:0/1992854194 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f18a4004100 msgr2=0x7f18bc1edb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:09.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.597+0000 7f18c224d640 1 --2- 172.21.15.19:0/1992854194 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f18a4004100 0x7f18bc1edb10 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f18ac00b140 tx=0x7f18ac031e20 comp rx=0 tx=0).stop 2024-08-31T12:27:09.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.597+0000 7f18c224d640 1 -- 172.21.15.19:0/1992854194 shutdown_connections 2024-08-31T12:27:09.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.597+0000 7f18c224d640 1 --2- 172.21.15.19:0/1992854194 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f18a8078550 0x7f18a807aa60 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:09.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.597+0000 7f18c224d640 1 --2- 172.21.15.19:0/1992854194 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f18bc157f60 0x7f18bc154610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:09.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.597+0000 7f18c224d640 1 --2- 172.21.15.19:0/1992854194 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f18a4004e10 0x7f18bc0cbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:09.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.597+0000 7f18c224d640 1 --2- 172.21.15.19:0/1992854194 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f18a4004100 0x7f18bc1edb10 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:09.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.597+0000 7f18c224d640 1 -- 172.21.15.19:0/1992854194 >> 172.21.15.19:0/1992854194 conn(0x7f18bc06cae0 msgr2=0x7f18bc150f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:27:09.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.597+0000 7f18c224d640 1 -- 172.21.15.19:0/1992854194 shutdown_connections 2024-08-31T12:27:09.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.597+0000 7f18c224d640 1 -- 172.21.15.19:0/1992854194 wait complete. 2024-08-31T12:27:09.616 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-08-31T12:27:09.616 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-08-31T12:27:09.616 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-31T12:27:09.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.791+0000 7f950a50e640 1 -- 172.21.15.19:0/224540599 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f950415bcf0 msgr2=0x7f95040ccf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:09.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.791+0000 7f950a50e640 1 --2- 172.21.15.19:0/224540599 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f950415bcf0 0x7f95040ccf90 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9500005fd0 tx=0x7f950002a540 comp rx=0 tx=0).stop 2024-08-31T12:27:09.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.792+0000 7f950a50e640 1 -- 172.21.15.19:0/224540599 shutdown_connections 2024-08-31T12:27:09.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.792+0000 7f950a50e640 1 --2- 172.21.15.19:0/224540599 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f950415e6f0 0x7f9504154d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:09.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.792+0000 7f950a50e640 1 --2- 172.21.15.19:0/224540599 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f950415bcf0 0x7f95040ccf90 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:09.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.792+0000 7f950a50e640 1 -- 172.21.15.19:0/224540599 >> 172.21.15.19:0/224540599 conn(0x7f9504068e90 msgr2=0x7f95040692a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:27:09.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.792+0000 7f950a50e640 1 -- 172.21.15.19:0/224540599 shutdown_connections 2024-08-31T12:27:09.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.792+0000 7f950a50e640 1 -- 172.21.15.19:0/224540599 wait complete. 2024-08-31T12:27:09.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.794+0000 7f950a50e640 1 Processor -- start 2024-08-31T12:27:09.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.794+0000 7f950a50e640 1 -- start start 2024-08-31T12:27:09.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.796+0000 7f950a50e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f95040c6120 0x7f9504148270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:09.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.796+0000 7f950a50e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f950415bcf0 0x7f95040cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:09.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.796+0000 7f950a50e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f950415e6f0 0x7f95040d0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:09.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.796+0000 7f950a50e640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f95040b43a0 con 0x7f950415bcf0 2024-08-31T12:27:09.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.796+0000 7f950a50e640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f950415b4f0 con 0x7f950415e6f0 2024-08-31T12:27:09.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.796+0000 7f950a50e640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f95040b4520 con 0x7f95040c6120 2024-08-31T12:27:09.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.796+0000 7f950950c640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f95040c6120 0x7f9504148270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:09.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.796+0000 7f9508d0b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f950415bcf0 0x7f95040cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:09.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.796+0000 7f950950c640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f95040c6120 0x7f9504148270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.19:58370/0 (socket says 172.21.15.19:58370) 2024-08-31T12:27:09.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.796+0000 7f950950c640 1 -- 172.21.15.19:0/1831401673 learned_addr learned my addr 172.21.15.19:0/1831401673 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:27:09.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.796+0000 7f9509d0d640 1 -- 172.21.15.19:0/1831401673 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f950415e6f0 msgr2=0x7f95040d0020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3300/0 2024-08-31T12:27:09.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.796+0000 7f9509d0d640 1 --2- 172.21.15.19:0/1831401673 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f950415e6f0 0x7f95040d0020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:27:09.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.797+0000 7f950950c640 1 -- 172.21.15.19:0/1831401673 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f950415e6f0 msgr2=0x7f95040d0020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:27:09.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.797+0000 7f950950c640 1 --2- 172.21.15.19:0/1831401673 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f950415e6f0 0x7f95040d0020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:09.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.797+0000 7f950950c640 1 -- 172.21.15.19:0/1831401673 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f950415bcf0 msgr2=0x7f95040cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:09.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.797+0000 7f950950c640 1 --2- 172.21.15.19:0/1831401673 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f950415bcf0 0x7f95040cfae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:09.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.797+0000 7f950950c640 1 -- 172.21.15.19:0/1831401673 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9504149f20 con 0x7f95040c6120 2024-08-31T12:27:09.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.797+0000 7f950950c640 1 --2- 172.21.15.19:0/1831401673 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f95040c6120 0x7f9504148270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f94f400b580 tx=0x7f94f400ba40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:09.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.798+0000 7f94f9ffb640 1 -- 172.21.15.19:0/1831401673 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f4014070 con 0x7f95040c6120 2024-08-31T12:27:09.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.798+0000 7f94f9ffb640 1 -- 172.21.15.19:0/1831401673 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f94f40109a0 con 0x7f95040c6120 2024-08-31T12:27:09.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.798+0000 7f94f9ffb640 1 -- 172.21.15.19:0/1831401673 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f4010df0 con 0x7f95040c6120 2024-08-31T12:27:09.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.798+0000 7f950a50e640 1 -- 172.21.15.19:0/1831401673 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f950414a430 con 0x7f95040c6120 2024-08-31T12:27:09.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.798+0000 7f950a50e640 1 -- 172.21.15.19:0/1831401673 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f950415df70 con 0x7f95040c6120 2024-08-31T12:27:09.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.800+0000 7f950a50e640 1 --2- 172.21.15.19:0/1831401673 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9504149a70 0x7f95040cbfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:09.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.800+0000 7f950a50e640 1 -- 172.21.15.19:0/1831401673 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f95040cf150 con 0x7f9504149a70 2024-08-31T12:27:09.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.800+0000 7f9508d0b640 1 --2- 172.21.15.19:0/1831401673 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9504149a70 0x7f95040cbfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:09.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.801+0000 7f9508d0b640 1 -- 172.21.15.19:0/1831401673 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f950415dad0 con 0x7f9504149a70 2024-08-31T12:27:09.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.801+0000 7f94f9ffb640 1 -- 172.21.15.19:0/1831401673 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f94f4017070 con 0x7f95040c6120 2024-08-31T12:27:09.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.801+0000 7f9508d0b640 1 --2- 172.21.15.19:0/1831401673 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9504149a70 0x7f95040cbfb0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f95040699b0 tx=0x7f95000045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:09.805 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.802+0000 7f94f9ffb640 1 --2- 172.21.15.19:0/1831401673 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f94e8077a80 0x7f94e8079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:09.805 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.802+0000 7f94f9ffb640 1 -- 172.21.15.19:0/1831401673 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f94f4099480 con 0x7f95040c6120 2024-08-31T12:27:09.805 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.802+0000 7f94f9ffb640 1 -- 172.21.15.19:0/1831401673 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9500040070 con 0x7f9504149a70 2024-08-31T12:27:09.805 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.802+0000 7f94f9ffb640 1 -- 172.21.15.19:0/1831401673 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9500003bc0 con 0x7f9504149a70 2024-08-31T12:27:09.805 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.802+0000 7f94f9ffb640 1 -- 172.21.15.19:0/1831401673 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9504149a70 msgr2=0x7f95040cbfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:09.805 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.802+0000 7f9509d0d640 1 --2- 172.21.15.19:0/1831401673 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f94e8077a80 0x7f94e8079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:09.805 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.802+0000 7f94f9ffb640 1 --2- 172.21.15.19:0/1831401673 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9504149a70 0x7f95040cbfb0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f95040699b0 tx=0x7f95000045e0 comp rx=0 tx=0).stop 2024-08-31T12:27:09.805 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.802+0000 7f94f9ffb640 1 -- 172.21.15.19:0/1831401673 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9504149a70 msgr2=0x7f95040cbfb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:27:09.805 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.802+0000 7f94f9ffb640 1 --2- 172.21.15.19:0/1831401673 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9504149a70 0x7f95040cbfb0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:09.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.804+0000 7f9509d0d640 1 --2- 172.21.15.19:0/1831401673 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f94e8077a80 0x7f94e8079f90 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f94fc005fd0 tx=0x7f94fc009500 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:09.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.816+0000 7f950a50e640 1 --2- 172.21.15.19:0/1831401673 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f95040cc4f0 0x7f95041f8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:09.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.816+0000 7f950a50e640 1 -- 172.21.15.19:0/1831401673 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f950415eb00 con 0x7f95040cc4f0 2024-08-31T12:27:09.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.816+0000 7f9508d0b640 1 --2- 172.21.15.19:0/1831401673 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f95040cc4f0 0x7f95041f8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:09.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.817+0000 7f9508d0b640 1 -- 172.21.15.19:0/1831401673 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f950414a600 con 0x7f95040cc4f0 2024-08-31T12:27:09.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.817+0000 7f9508d0b640 1 --2- 172.21.15.19:0/1831401673 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f95040cc4f0 0x7f95041f8dd0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9500009850 tx=0x7f950002a540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:09.820 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.818+0000 7f94f9ffb640 1 -- 172.21.15.19:0/1831401673 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9500035040 con 0x7f95040cc4f0 2024-08-31T12:27:09.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.818+0000 7f94f9ffb640 1 -- 172.21.15.19:0/1831401673 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5033 (secure 0 0 0) 0x7f9500049570 con 0x7f95040cc4f0 2024-08-31T12:27:09.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.818+0000 7f94f9ffb640 1 -- 172.21.15.19:0/1831401673 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f95040cc4f0 msgr2=0x7f95041f8dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:09.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.818+0000 7f94f9ffb640 1 --2- 172.21.15.19:0/1831401673 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f95040cc4f0 0x7f95041f8dd0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9500009850 tx=0x7f950002a540 comp rx=0 tx=0).stop 2024-08-31T12:27:09.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.818+0000 7f94f9ffb640 1 -- 172.21.15.19:0/1831401673 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f95040cc4f0 msgr2=0x7f95041f8dd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:27:09.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.818+0000 7f94f9ffb640 1 --2- 172.21.15.19:0/1831401673 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f95040cc4f0 0x7f95041f8dd0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:09.822 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:27:09.822 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:27:09.822 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:27:09.822 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "leader", 2024-08-31T12:27:09.822 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 90, 2024-08-31T12:27:09.822 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 164372, 2024-08-31T12:27:09.822 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:27:09.822 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:27:09.822 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:27:09.822 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:09.822 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 0, 2024-08-31T12:27:09.823 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:27:09.823 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:27:09.823 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:27:09.823 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:27:09.823 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:27:09.823 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:27:09.823 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:27:09.823 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:27:09.823 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:27:09.823 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:27:09.823 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:27:09.823 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:27:09.823 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:27:09.824 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:27:09.824 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:09.824 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:27:09.824 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:27:09.824 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:27:09.824 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:27:09.824 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:27:09.824 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:27:09.824 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:27:09.824 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:27:09.824 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:27:09.825 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:27:09.825 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:27:09.825 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:27:09.825 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:27:09.825 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:09.825 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:09.825 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:27:09.825 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:27:09.825 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:27:09.825 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:27:09.825 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:27:09.825 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:27:09.826 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:27:09.826 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:27:09.826 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:27:09.826 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:27:09.826 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:27:09.826 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:27:09.826 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:27:09.826 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:27:09.826 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:27:09.826 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:27:09.826 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:27:09.826 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:27:09.826 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:27:09.826 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:27:09.827 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:27:09.827 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:27:09.827 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:27:09.827 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:27:09.827 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:27:09.827 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:27:09.827 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:27:09.827 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:27:09.827 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:09.827 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:27:09.827 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:09.827 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:27:09.827 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:09.827 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:27:09.827 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:27:09.828 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:27:09.828 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:27:09.828 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:09.828 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:27:09.828 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:27:09.828 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:09.828 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:09.828 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:09.828 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:27:09.828 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:27:09.828 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:09.828 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:09.828 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:09.828 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:09.829 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:27:09.829 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:27:09.829 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:27:09.829 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:27:09.829 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:27:09.829 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:09.829 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:09.829 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:27:09.829 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:27:09.829 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:27:09.829 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:27:09.829 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:09.829 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:27:09.830 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:27:09.830 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:09.830 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:09.830 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:09.830 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:27:09.830 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:27:09.830 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:09.830 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:09.830 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:09.830 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:09.830 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:27:09.830 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:27:09.830 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:27:09.830 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:27:09.831 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:27:09.831 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:09.831 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:09.831 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:27:09.831 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:27:09.831 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:27:09.831 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:27:09.831 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:09.831 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:27:09.831 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:27:09.831 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:09.831 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:09.831 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:09.831 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:27:09.831 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:27:09.832 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:09.832 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:09.832 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:09.832 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:09.832 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:27:09.832 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:27:09.832 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:27:09.832 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:27:09.832 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:27:09.832 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:09.832 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:09.832 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:09.832 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:27:09.832 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:27:09.833 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:09.833 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:27:09.833 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:27:09.833 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:27:09.833 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:09.833 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:09.833 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:27:09.833 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:09.833 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:27:09.833 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:27:09.833 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 5 2024-08-31T12:27:09.833 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:09.833 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:09.833 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:27:09.833 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:09.834 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:27:09.834 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:27:09.834 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 7 2024-08-31T12:27:09.834 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:09.834 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:09.834 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:27:09.834 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:09.834 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:27:09.834 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:27:09.834 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 4 2024-08-31T12:27:09.834 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:09.834 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:09.834 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:27:09.835 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:09.835 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:27:09.835 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:27:09.835 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:27:09.835 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:09.835 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:09.835 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:09.835 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:27:09.835 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:27:09.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.819+0000 7f950a50e640 1 -- 172.21.15.19:0/1831401673 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f94e8077a80 msgr2=0x7f94e8079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:09.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.819+0000 7f950a50e640 1 --2- 172.21.15.19:0/1831401673 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f94e8077a80 0x7f94e8079f90 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f94fc005fd0 tx=0x7f94fc009500 comp rx=0 tx=0).stop 2024-08-31T12:27:09.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.819+0000 7f950a50e640 1 -- 172.21.15.19:0/1831401673 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f95040c6120 msgr2=0x7f9504148270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:09.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.819+0000 7f950a50e640 1 --2- 172.21.15.19:0/1831401673 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f95040c6120 0x7f9504148270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f94f400b580 tx=0x7f94f400ba40 comp rx=0 tx=0).stop 2024-08-31T12:27:09.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.819+0000 7f9509d0d640 1 -- 172.21.15.19:0/1831401673 reap_dead start 2024-08-31T12:27:09.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.820+0000 7f950a50e640 1 -- 172.21.15.19:0/1831401673 shutdown_connections 2024-08-31T12:27:09.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.820+0000 7f950a50e640 1 --2- 172.21.15.19:0/1831401673 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f95040c6120 0x7f9504148270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:09.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.820+0000 7f950a50e640 1 -- 172.21.15.19:0/1831401673 >> 172.21.15.19:0/1831401673 conn(0x7f9504068e90 msgr2=0x7f950414d560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:27:09.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.820+0000 7f950a50e640 1 -- 172.21.15.19:0/1831401673 shutdown_connections 2024-08-31T12:27:09.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:09.821+0000 7f950a50e640 1 -- 172.21.15.19:0/1831401673 wait complete. 2024-08-31T12:27:09.838 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-08-31T12:27:09.839 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-08-31T12:27:09.839 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-31T12:27:10.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.013+0000 7f83ec85c640 1 -- 172.21.15.19:0/2024887430 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f83e80c7110 msgr2=0x7f83e815b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:10.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.013+0000 7f83ec85c640 1 --2- 172.21.15.19:0/2024887430 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f83e80c7110 0x7f83e815b080 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f83d0009be0 tx=0x7f83d002c8f0 comp rx=0 tx=0).stop 2024-08-31T12:27:10.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.014+0000 7f83ec85c640 1 -- 172.21.15.19:0/2024887430 shutdown_connections 2024-08-31T12:27:10.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.014+0000 7f83ec85c640 1 --2- 172.21.15.19:0/2024887430 >> v2:172.21.15.19:3300/0 conn(0x7f83e80738a0 0x7f83e8161930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:10.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.014+0000 7f83ec85c640 1 --2- 172.21.15.19:0/2024887430 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f83e80c7110 0x7f83e815b080 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:10.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.014+0000 7f83ec85c640 1 -- 172.21.15.19:0/2024887430 >> 172.21.15.19:0/2024887430 conn(0x7f83e806cae0 msgr2=0x7f83e806cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:27:10.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.014+0000 7f83ec85c640 1 -- 172.21.15.19:0/2024887430 shutdown_connections 2024-08-31T12:27:10.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.014+0000 7f83ec85c640 1 -- 172.21.15.19:0/2024887430 wait complete. 2024-08-31T12:27:10.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.017+0000 7f83ec85c640 1 Processor -- start 2024-08-31T12:27:10.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.017+0000 7f83ec85c640 1 -- start start 2024-08-31T12:27:10.029 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.018+0000 7f83ec85c640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f83e80738a0 0x7f83e81f06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:10.029 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.019+0000 7f83ec85c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83e80c7110 0x7f83e8148830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:10.030 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.019+0000 7f83ec85c640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83e8151040 0x7f83e8072790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:10.030 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.019+0000 7f83ec85c640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f83e815a5a0 con 0x7f83e80c7110 2024-08-31T12:27:10.031 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.019+0000 7f83ec85c640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f83e806f150 con 0x7f83e8151040 2024-08-31T12:27:10.031 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.019+0000 7f83ec85c640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f83e815a880 con 0x7f83e80738a0 2024-08-31T12:27:10.031 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.019+0000 7f83e6d9d640 1 -- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83e8151040 msgr2=0x7f83e8072790 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3300/0 2024-08-31T12:27:10.032 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.019+0000 7f83e6d9d640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83e8151040 0x7f83e8072790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:27:10.032 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.019+0000 7f83e5d9b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83e80c7110 0x7f83e8148830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:10.032 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.019+0000 7f83e5d9b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83e80c7110 0x7f83e8148830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:49482/0 (socket says 172.21.15.19:49482) 2024-08-31T12:27:10.032 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.019+0000 7f83e5d9b640 1 -- 172.21.15.19:0/1427186570 learned_addr learned my addr 172.21.15.19:0/1427186570 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:27:10.032 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.019+0000 7f83e659c640 1 --2- 172.21.15.19:0/1427186570 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f83e80738a0 0x7f83e81f06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:10.033 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.019+0000 7f83e5d9b640 1 -- 172.21.15.19:0/1427186570 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f83e80738a0 msgr2=0x7f83e81f06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:10.033 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.019+0000 7f83e5d9b640 1 --2- 172.21.15.19:0/1427186570 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f83e80738a0 0x7f83e81f06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:10.033 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.019+0000 7f83e5d9b640 1 -- 172.21.15.19:0/1427186570 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83e8151040 msgr2=0x7f83e8072790 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:27:10.033 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.019+0000 7f83e5d9b640 1 --2- 172.21.15.19:0/1427186570 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83e8151040 0x7f83e8072790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:10.033 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.019+0000 7f83e5d9b640 1 -- 172.21.15.19:0/1427186570 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83e8145b10 con 0x7f83e80c7110 2024-08-31T12:27:10.033 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.020+0000 7f83e5d9b640 1 --2- 172.21.15.19:0/1427186570 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83e80c7110 0x7f83e8148830 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f83d800c400 tx=0x7f83d800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:10.034 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.020+0000 7f83deffd640 1 -- 172.21.15.19:0/1427186570 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d8011790 con 0x7f83e80c7110 2024-08-31T12:27:10.034 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.020+0000 7f83ec85c640 1 -- 172.21.15.19:0/1427186570 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83e81f0c20 con 0x7f83e80c7110 2024-08-31T12:27:10.034 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.020+0000 7f83deffd640 1 -- 172.21.15.19:0/1427186570 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f83d8017070 con 0x7f83e80c7110 2024-08-31T12:27:10.034 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.020+0000 7f83ec85c640 1 -- 172.21.15.19:0/1427186570 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83e8146170 con 0x7f83e80c7110 2024-08-31T12:27:10.034 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.021+0000 7f83dcff9640 1 --2- 172.21.15.19:0/1427186570 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f83c0002a70 0x7f83c0004f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:10.035 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.021+0000 7f83dcff9640 1 -- 172.21.15.19:0/1427186570 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f83c0007f10 con 0x7f83c0002a70 2024-08-31T12:27:10.035 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.022+0000 7f83e659c640 1 --2- 172.21.15.19:0/1427186570 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f83c0002a70 0x7f83c0004f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:10.035 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.022+0000 7f83deffd640 1 -- 172.21.15.19:0/1427186570 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d8011ce0 con 0x7f83e80c7110 2024-08-31T12:27:10.035 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.023+0000 7f83e659c640 1 -- 172.21.15.19:0/1427186570 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f83c0008110 con 0x7f83c0002a70 2024-08-31T12:27:10.035 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.024+0000 7f83deffd640 1 -- 172.21.15.19:0/1427186570 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f83d801c460 con 0x7f83e80c7110 2024-08-31T12:27:10.036 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.024+0000 7f83e659c640 1 --2- 172.21.15.19:0/1427186570 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f83c0002a70 0x7f83c0004f80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f83e80ccd10 tx=0x7f83d0004250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:10.036 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.025+0000 7f83deffd640 1 --2- 172.21.15.19:0/1427186570 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f83cc07be40 0x7f83cc07e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:10.037 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.025+0000 7f83deffd640 1 -- 172.21.15.19:0/1427186570 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f83d809d170 con 0x7f83e80c7110 2024-08-31T12:27:10.038 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.025+0000 7f83e6d9d640 1 --2- 172.21.15.19:0/1427186570 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f83cc07be40 0x7f83cc07e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:10.038 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.025+0000 7f83deffd640 1 -- 172.21.15.19:0/1427186570 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d00047a0 con 0x7f83c0002a70 2024-08-31T12:27:10.038 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.025+0000 7f83deffd640 1 -- 172.21.15.19:0/1427186570 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f83d0041070 con 0x7f83c0002a70 2024-08-31T12:27:10.038 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.025+0000 7f83deffd640 1 -- 172.21.15.19:0/1427186570 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f83c0002a70 msgr2=0x7f83c0004f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:10.038 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.025+0000 7f83deffd640 1 --2- 172.21.15.19:0/1427186570 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f83c0002a70 0x7f83c0004f80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f83e80ccd10 tx=0x7f83d0004250 comp rx=0 tx=0).stop 2024-08-31T12:27:10.038 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.025+0000 7f83deffd640 1 -- 172.21.15.19:0/1427186570 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f83c0002a70 msgr2=0x7f83c0004f80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:27:10.038 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.026+0000 7f83deffd640 1 --2- 172.21.15.19:0/1427186570 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f83c0002a70 0x7f83c0004f80 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:10.038 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.026+0000 7f83e6d9d640 1 --2- 172.21.15.19:0/1427186570 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f83cc07be40 0x7f83cc07e350 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f83e80cd200 tx=0x7f83e0005ee0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:10.039 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.035+0000 7f83dcff9640 1 --2- 172.21.15.19:0/1427186570 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f83c0006f40 0x7f83c0007420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:10.039 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.035+0000 7f83dcff9640 1 -- 172.21.15.19:0/1427186570 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f83c000a4f0 con 0x7f83c0006f40 2024-08-31T12:27:10.040 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.035+0000 7f83e659c640 1 --2- 172.21.15.19:0/1427186570 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f83c0006f40 0x7f83c0007420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:10.040 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.036+0000 7f83e659c640 1 -- 172.21.15.19:0/1427186570 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f83c0007960 con 0x7f83c0006f40 2024-08-31T12:27:10.040 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.036+0000 7f83e659c640 1 --2- 172.21.15.19:0/1427186570 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f83c0006f40 0x7f83c0007420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f83d0009b30 tx=0x7f83d0009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:10.040 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.036+0000 7f83deffd640 1 -- 172.21.15.19:0/1427186570 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d0045820 con 0x7f83c0006f40 2024-08-31T12:27:10.040 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:27:10.040 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:27:10.040 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:27:10.040 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:27:10.040 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 90, 2024-08-31T12:27:10.041 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 35906, 2024-08-31T12:27:10.041 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:27:10.041 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:27:10.041 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:27:10.041 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:10.041 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 0, 2024-08-31T12:27:10.041 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:27:10.041 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:27:10.041 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:27:10.041 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:27:10.041 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:27:10.041 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:27:10.041 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:27:10.042 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:27:10.042 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:27:10.042 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:27:10.042 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:27:10.042 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:27:10.042 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:27:10.042 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:27:10.042 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:10.042 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:27:10.042 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:27:10.042 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:27:10.042 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:27:10.043 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:27:10.043 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:27:10.043 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:27:10.043 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:27:10.043 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:27:10.043 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:27:10.043 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:27:10.043 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:27:10.043 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:27:10.043 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:10.043 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:10.043 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:27:10.043 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:27:10.044 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:27:10.044 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:27:10.044 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:27:10.044 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:27:10.044 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:27:10.044 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:27:10.044 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:27:10.044 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:27:10.044 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:27:10.044 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:27:10.044 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:27:10.044 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:27:10.044 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:27:10.045 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:27:10.045 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:27:10.045 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:27:10.045 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:27:10.045 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:27:10.045 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:27:10.045 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:27:10.045 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:27:10.045 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:27:10.045 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:27:10.045 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:27:10.045 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:27:10.046 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:27:10.046 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:10.046 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:27:10.046 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:10.046 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:27:10.046 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:10.046 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:27:10.046 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:27:10.046 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:27:10.046 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:27:10.046 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:10.046 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:27:10.046 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:27:10.047 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:10.047 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:10.047 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:10.047 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:27:10.047 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:27:10.047 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:10.047 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:10.047 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:10.047 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:10.047 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:27:10.047 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:27:10.047 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:27:10.047 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:27:10.047 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:27:10.048 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:10.048 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:10.048 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:27:10.048 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:27:10.048 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:27:10.048 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:27:10.048 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:10.048 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:27:10.048 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:27:10.048 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:10.048 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:10.048 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:10.048 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:27:10.049 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:27:10.049 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:10.049 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:10.049 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:10.049 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:10.049 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:27:10.049 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:27:10.049 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:27:10.049 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:27:10.049 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:27:10.049 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:10.049 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:10.049 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:27:10.050 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:27:10.050 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:27:10.050 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:27:10.050 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:10.050 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:27:10.050 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:27:10.050 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:10.050 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:10.050 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:10.050 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:27:10.050 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:27:10.050 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:10.050 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:10.051 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:10.051 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:10.051 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:27:10.051 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:27:10.051 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:27:10.051 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:27:10.051 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:27:10.051 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:10.051 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:10.051 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:10.051 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:27:10.051 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:27:10.051 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:10.052 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:27:10.052 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:27:10.052 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:27:10.052 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:10.052 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:10.052 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:27:10.052 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:10.052 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:27:10.052 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:27:10.052 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:27:10.052 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:10.052 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:10.052 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:27:10.053 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:10.053 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:27:10.053 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:27:10.053 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:27:10.053 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:10.053 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:10.053 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:27:10.053 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:10.053 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:27:10.053 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:27:10.053 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 3 2024-08-31T12:27:10.053 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:10.053 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:10.054 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:10.054 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:27:10.054 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:27:10.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.036+0000 7f83deffd640 1 -- 172.21.15.19:0/1427186570 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4863 (secure 0 0 0) 0x7f83d003d410 con 0x7f83c0006f40 2024-08-31T12:27:10.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.036+0000 7f83deffd640 1 -- 172.21.15.19:0/1427186570 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f83c0006f40 msgr2=0x7f83c0007420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:10.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.036+0000 7f83deffd640 1 --2- 172.21.15.19:0/1427186570 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f83c0006f40 0x7f83c0007420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f83d0009b30 tx=0x7f83d0009500 comp rx=0 tx=0).stop 2024-08-31T12:27:10.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.036+0000 7f83deffd640 1 -- 172.21.15.19:0/1427186570 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f83c0006f40 msgr2=0x7f83c0007420 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:27:10.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.036+0000 7f83deffd640 1 --2- 172.21.15.19:0/1427186570 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f83c0006f40 0x7f83c0007420 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f83d0009b30 tx=0x7f83d0009500 comp rx=0 tx=0).stop 2024-08-31T12:27:10.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.037+0000 7f83dcff9640 1 -- 172.21.15.19:0/1427186570 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f83cc07be40 msgr2=0x7f83cc07e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:10.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.037+0000 7f83dcff9640 1 --2- 172.21.15.19:0/1427186570 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f83cc07be40 0x7f83cc07e350 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f83e80cd200 tx=0x7f83e0005ee0 comp rx=0 tx=0).stop 2024-08-31T12:27:10.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.037+0000 7f83e6d9d640 1 -- 172.21.15.19:0/1427186570 reap_dead start 2024-08-31T12:27:10.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.037+0000 7f83dcff9640 1 -- 172.21.15.19:0/1427186570 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83e80c7110 msgr2=0x7f83e8148830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:10.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.037+0000 7f83dcff9640 1 --2- 172.21.15.19:0/1427186570 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83e80c7110 0x7f83e8148830 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f83d800c400 tx=0x7f83d800c8c0 comp rx=0 tx=0).stop 2024-08-31T12:27:10.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.037+0000 7f83dcff9640 1 -- 172.21.15.19:0/1427186570 shutdown_connections 2024-08-31T12:27:10.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.037+0000 7f83dcff9640 1 --2- 172.21.15.19:0/1427186570 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83e80c7110 0x7f83e8148830 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:10.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.037+0000 7f83dcff9640 1 -- 172.21.15.19:0/1427186570 >> 172.21.15.19:0/1427186570 conn(0x7f83e806cae0 msgr2=0x7f83e80c6860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:27:10.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.037+0000 7f83dcff9640 1 -- 172.21.15.19:0/1427186570 shutdown_connections 2024-08-31T12:27:10.057 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:10.037+0000 7f83dcff9640 1 -- 172.21.15.19:0/1427186570 wait complete. 2024-08-31T12:27:10.057 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-08-31T12:27:10.057 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-08-31T12:27:28.165 DEBUG:teuthology.orchestra.run.smithi019:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:27:28.168 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:27:30.058 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-08-31T12:27:30.058 INFO:tasks.ceph.mon.b:Restarting daemon 2024-08-31T12:27:30.058 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-08-31T12:27:30.061 INFO:tasks.ceph.mon.b:Started 2024-08-31T12:27:30.061 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-08-31T12:27:30.062 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:27:30.231 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.229+0000 7f1b3effd640 1 -- 172.21.15.19:0/1277290432 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b38006bb0 con 0x7f1b2c003a00 2024-08-31T12:27:30.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.230+0000 7f1b465a3640 1 -- 172.21.15.19:0/1277290432 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1b2c003a00 msgr2=0x7f1b2c003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:30.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.230+0000 7f1b465a3640 1 --2- 172.21.15.19:0/1277290432 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1b2c003a00 0x7f1b2c003e70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1b38006f90 tx=0x7f1b2c005830 comp rx=0 tx=0).stop 2024-08-31T12:27:30.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.230+0000 7f1b44b40640 1 -- 172.21.15.19:0/1277290432 reap_dead start 2024-08-31T12:27:30.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.230+0000 7f1b465a3640 1 -- 172.21.15.19:0/1277290432 shutdown_connections 2024-08-31T12:27:30.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.230+0000 7f1b465a3640 1 -- 172.21.15.19:0/1277290432 >> 172.21.15.19:0/1277290432 conn(0x7f1b4006d110 msgr2=0x7f1b400cc950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:27:30.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.230+0000 7f1b465a3640 1 -- 172.21.15.19:0/1277290432 shutdown_connections 2024-08-31T12:27:30.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.230+0000 7f1b465a3640 1 -- 172.21.15.19:0/1277290432 wait complete. 2024-08-31T12:27:30.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.234+0000 7f1b465a3640 1 Processor -- start 2024-08-31T12:27:30.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.234+0000 7f1b465a3640 1 -- start start 2024-08-31T12:27:30.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.236+0000 7f1b465a3640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1b2c003a00 0x7f1b401ed870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:30.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.236+0000 7f1b465a3640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b40147bc0 0x7f1b40142560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:30.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.236+0000 7f1b465a3640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1b400cc0b0 0x7f1b40142aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:30.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.236+0000 7f1b3ffff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1b2c003a00 0x7f1b401ed870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:30.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.236+0000 7f1b3ffff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1b2c003a00 0x7f1b401ed870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.19:47940/0 (socket says 172.21.15.19:47940) 2024-08-31T12:27:30.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.236+0000 7f1b465a3640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f1b400c68f0 con 0x7f1b400cc0b0 2024-08-31T12:27:30.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.236+0000 7f1b3ffff640 1 -- 172.21.15.19:0/3404027851 learned_addr learned my addr 172.21.15.19:0/3404027851 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:27:30.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.236+0000 7f1b465a3640 1 -- 172.21.15.19:0/3404027851 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1b400c6610 con 0x7f1b40147bc0 2024-08-31T12:27:30.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.236+0000 7f1b465a3640 1 -- 172.21.15.19:0/3404027851 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f1b40074220 con 0x7f1b2c003a00 2024-08-31T12:27:30.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.237+0000 7f1b3f7fe640 1 --2- 172.21.15.19:0/3404027851 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b40147bc0 0x7f1b40142560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:30.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.237+0000 7f1b44b40640 1 --2- 172.21.15.19:0/3404027851 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1b400cc0b0 0x7f1b40142aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:30.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.237+0000 7f1b3f7fe640 1 -- 172.21.15.19:0/3404027851 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b40147bc0 msgr2=0x7f1b40142560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-08-31T12:27:30.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.237+0000 7f1b3f7fe640 1 -- 172.21.15.19:0/3404027851 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b40147bc0 msgr2=0x7f1b40142560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-31T12:27:30.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.237+0000 7f1b3f7fe640 1 --2- 172.21.15.19:0/3404027851 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b40147bc0 0x7f1b40142560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-31T12:27:30.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.237+0000 7f1b3f7fe640 1 --2- 172.21.15.19:0/3404027851 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b40147bc0 0x7f1b40142560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:27:30.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.237+0000 7f1b3ffff640 1 -- 172.21.15.19:0/3404027851 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b40147bc0 msgr2=0x7f1b40142560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:27:30.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.237+0000 7f1b3ffff640 1 --2- 172.21.15.19:0/3404027851 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b40147bc0 0x7f1b40142560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:30.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.237+0000 7f1b3ffff640 1 -- 172.21.15.19:0/3404027851 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1b400cc0b0 msgr2=0x7f1b40142aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:30.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.237+0000 7f1b3ffff640 1 --2- 172.21.15.19:0/3404027851 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1b400cc0b0 0x7f1b40142aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:30.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.237+0000 7f1b3ffff640 1 -- 172.21.15.19:0/3404027851 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b40148670 con 0x7f1b2c003a00 2024-08-31T12:27:30.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.237+0000 7f1b3ffff640 1 --2- 172.21.15.19:0/3404027851 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1b2c003a00 0x7f1b401ed870 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1b3802b0f0 tx=0x7f1b38006650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:30.242 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.237+0000 7f1b3cff9640 1 -- 172.21.15.19:0/3404027851 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b38031040 con 0x7f1b2c003a00 2024-08-31T12:27:30.242 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.238+0000 7f1b465a3640 1 -- 172.21.15.19:0/3404027851 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b400cc5d0 con 0x7f1b2c003a00 2024-08-31T12:27:30.242 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.238+0000 7f1b3cff9640 1 -- 172.21.15.19:0/3404027851 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1b38003800 con 0x7f1b2c003a00 2024-08-31T12:27:30.242 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.238+0000 7f1b3cff9640 1 -- 172.21.15.19:0/3404027851 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b38003c10 con 0x7f1b2c003a00 2024-08-31T12:27:30.242 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.238+0000 7f1b465a3640 1 -- 172.21.15.19:0/3404027851 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b40156840 con 0x7f1b2c003a00 2024-08-31T12:27:30.242 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.239+0000 7f1b465a3640 1 -- 172.21.15.19:0/3404027851 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b400c7630 con 0x7f1b2c003a00 2024-08-31T12:27:30.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.240+0000 7f1b3cff9640 1 -- 172.21.15.19:0/3404027851 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f1b38034070 con 0x7f1b2c003a00 2024-08-31T12:27:30.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.241+0000 7f1b3cff9640 1 --2- 172.21.15.19:0/3404027851 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1b20077a50 0x7f1b20079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:30.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.241+0000 7f1b3cff9640 1 -- 172.21.15.19:0/3404027851 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f1b380b9830 con 0x7f1b2c003a00 2024-08-31T12:27:30.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.242+0000 7f1b3f7fe640 1 --2- 172.21.15.19:0/3404027851 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1b20077a50 0x7f1b20079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:30.244 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.242+0000 7f1b3f7fe640 1 --2- 172.21.15.19:0/3404027851 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1b20077a50 0x7f1b20079f60 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1b4014be00 tx=0x7f1b30007040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:30.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.248+0000 7f1b3cff9640 1 -- 172.21.15.19:0/3404027851 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f1b400c7630 con 0x7f1b2c003a00 2024-08-31T12:27:30.735 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.732+0000 7f1b465a3640 1 -- 172.21.15.19:0/3404027851 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1b400c3380 con 0x7f1b2c003a00 2024-08-31T12:27:30.735 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.733+0000 7f1b3cff9640 1 -- 172.21.15.19:0/3404027851 <== mon.2 v2:172.21.15.43:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f1b400c7630 con 0x7f1b2c003a00 2024-08-31T12:27:30.735 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":90,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:27:30.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.737+0000 7f1b465a3640 1 -- 172.21.15.19:0/3404027851 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1b20077a50 msgr2=0x7f1b20079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:30.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.737+0000 7f1b465a3640 1 --2- 172.21.15.19:0/3404027851 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1b20077a50 0x7f1b20079f60 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1b4014be00 tx=0x7f1b30007040 comp rx=0 tx=0).stop 2024-08-31T12:27:30.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.737+0000 7f1b465a3640 1 -- 172.21.15.19:0/3404027851 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1b2c003a00 msgr2=0x7f1b401ed870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:30.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.737+0000 7f1b465a3640 1 --2- 172.21.15.19:0/3404027851 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1b2c003a00 0x7f1b401ed870 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1b3802b0f0 tx=0x7f1b38006650 comp rx=0 tx=0).stop 2024-08-31T12:27:30.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.737+0000 7f1b465a3640 1 -- 172.21.15.19:0/3404027851 shutdown_connections 2024-08-31T12:27:30.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.737+0000 7f1b465a3640 1 --2- 172.21.15.19:0/3404027851 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1b20077a50 0x7f1b20079f60 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:30.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.737+0000 7f1b465a3640 1 --2- 172.21.15.19:0/3404027851 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1b400cc0b0 0x7f1b40142aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:30.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.737+0000 7f1b465a3640 1 --2- 172.21.15.19:0/3404027851 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b40147bc0 0x7f1b40142560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:30.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.737+0000 7f1b465a3640 1 --2- 172.21.15.19:0/3404027851 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1b2c003a00 0x7f1b401ed870 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:30.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.737+0000 7f1b465a3640 1 -- 172.21.15.19:0/3404027851 >> 172.21.15.19:0/3404027851 conn(0x7f1b4006d110 msgr2=0x7f1b40072ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:27:30.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.737+0000 7f1b465a3640 1 -- 172.21.15.19:0/3404027851 shutdown_connections 2024-08-31T12:27:30.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:30.737+0000 7f1b465a3640 1 -- 172.21.15.19:0/3404027851 wait complete. 2024-08-31T12:27:33.759 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:27:33.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:33.932+0000 7fdbcb59e640 1 -- 172.21.15.19:0/357331481 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdbb4004160 msgr2=0x7fdbb4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:33.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:33.932+0000 7fdbcb59e640 1 --2- 172.21.15.19:0/357331481 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdbb4004160 0x7fdbb4004800 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc0c3990 tx=0x7fdbb4005830 comp rx=0 tx=0).stop 2024-08-31T12:27:33.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:33.933+0000 7fdbcad9d640 1 -- 172.21.15.19:0/357331481 reap_dead start 2024-08-31T12:27:33.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:33.933+0000 7fdbcb59e640 1 -- 172.21.15.19:0/357331481 shutdown_connections 2024-08-31T12:27:33.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:33.933+0000 7fdbcb59e640 1 -- 172.21.15.19:0/357331481 >> 172.21.15.19:0/357331481 conn(0x7fdbcc071290 msgr2=0x7fdbcc0cca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:27:33.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:33.933+0000 7fdbcb59e640 1 -- 172.21.15.19:0/357331481 shutdown_connections 2024-08-31T12:27:33.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:33.933+0000 7fdbcb59e640 1 -- 172.21.15.19:0/357331481 wait complete. 2024-08-31T12:27:33.938 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:33.936+0000 7fdbcb59e640 1 Processor -- start 2024-08-31T12:27:33.939 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:33.937+0000 7fdbcb59e640 1 -- start start 2024-08-31T12:27:33.943 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:33.939+0000 7fdbcb59e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdbb4004160 0x7fdbcc1e21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:33.943 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:33.939+0000 7fdbcb59e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdbcc1e39b0 0x7fdbcc0cba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:33.943 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:33.939+0000 7fdbcb59e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdbcc1e51d0 0x7fdbcc14f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:33.943 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:33.939+0000 7fdbcb59e640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fdbcc0c83b0 con 0x7fdbcc1e51d0 2024-08-31T12:27:33.943 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:33.939+0000 7fdbcb59e640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fdbcc0c77a0 con 0x7fdbb4004160 2024-08-31T12:27:33.943 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:33.939+0000 7fdbcb59e640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fdbcc0c8530 con 0x7fdbcc1e39b0 2024-08-31T12:27:33.943 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:33.940+0000 7fdbca59c640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdbb4004160 0x7fdbcc1e21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:33.943 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:33.940+0000 7fdbc9d9b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdbcc1e39b0 0x7fdbcc0cba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:33.944 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:33.940+0000 7fdbcad9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdbcc1e51d0 0x7fdbcc14f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:33.944 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:33.940+0000 7fdbc9d9b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdbcc1e39b0 0x7fdbcc0cba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.19:47972/0 (socket says 172.21.15.19:47972) 2024-08-31T12:27:33.944 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:33.940+0000 7fdbc9d9b640 1 -- 172.21.15.19:0/543225075 learned_addr learned my addr 172.21.15.19:0/543225075 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:27:33.944 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:33.940+0000 7fdbca59c640 1 -- 172.21.15.19:0/543225075 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdbb4004160 msgr2=0x7fdbcc1e21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-08-31T12:27:33.944 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:33.940+0000 7fdbca59c640 1 -- 172.21.15.19:0/543225075 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdbb4004160 msgr2=0x7fdbcc1e21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-31T12:27:33.944 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:33.940+0000 7fdbca59c640 1 --2- 172.21.15.19:0/543225075 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdbb4004160 0x7fdbcc1e21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-31T12:27:33.944 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:33.940+0000 7fdbca59c640 1 --2- 172.21.15.19:0/543225075 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdbb4004160 0x7fdbcc1e21d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:27:33.944 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:33.940+0000 7fdbc9d9b640 1 -- 172.21.15.19:0/543225075 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdbb4004160 msgr2=0x7fdbcc1e21d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:27:33.944 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:33.940+0000 7fdbc9d9b640 1 --2- 172.21.15.19:0/543225075 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdbb4004160 0x7fdbcc1e21d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:33.944 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:33.940+0000 7fdbc9d9b640 1 -- 172.21.15.19:0/543225075 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdbcc1e51d0 msgr2=0x7fdbcc14f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:33.944 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:33.940+0000 7fdbc9d9b640 1 --2- 172.21.15.19:0/543225075 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdbcc1e51d0 0x7fdbcc14f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:33.945 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:33.940+0000 7fdbc9d9b640 1 -- 172.21.15.19:0/543225075 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbcc150f40 con 0x7fdbcc1e39b0 2024-08-31T12:27:33.945 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:33.941+0000 7fdbc9d9b640 1 --2- 172.21.15.19:0/543225075 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdbcc1e39b0 0x7fdbcc0cba20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdbc402ab00 tx=0x7fdbc4032db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:33.945 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:33.941+0000 7fdbbaffd640 1 -- 172.21.15.19:0/543225075 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc4040070 con 0x7fdbcc1e39b0 2024-08-31T12:27:33.945 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:33.941+0000 7fdbbaffd640 1 -- 172.21.15.19:0/543225075 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdbc4004250 con 0x7fdbcc1e39b0 2024-08-31T12:27:33.945 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:33.941+0000 7fdbbaffd640 1 -- 172.21.15.19:0/543225075 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc4009be0 con 0x7fdbcc1e39b0 2024-08-31T12:27:33.945 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:33.941+0000 7fdbcb59e640 1 -- 172.21.15.19:0/543225075 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbcc151220 con 0x7fdbcc1e39b0 2024-08-31T12:27:33.945 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:33.941+0000 7fdbcb59e640 1 -- 172.21.15.19:0/543225075 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbcc1e3e20 con 0x7fdbcc1e39b0 2024-08-31T12:27:33.946 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:33.942+0000 7fdbcb59e640 1 -- 172.21.15.19:0/543225075 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbcc155260 con 0x7fdbcc1e39b0 2024-08-31T12:27:33.946 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:33.943+0000 7fdbbaffd640 1 -- 172.21.15.19:0/543225075 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fdbc4039040 con 0x7fdbcc1e39b0 2024-08-31T12:27:33.946 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:33.944+0000 7fdbbaffd640 1 --2- 172.21.15.19:0/543225075 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fdbac077ab0 0x7fdbac079fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:33.946 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:33.944+0000 7fdbbaffd640 1 -- 172.21.15.19:0/543225075 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fdbc40ba8f0 con 0x7fdbcc1e39b0 2024-08-31T12:27:33.946 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:33.944+0000 7fdbca59c640 1 --2- 172.21.15.19:0/543225075 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fdbac077ab0 0x7fdbac079fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:33.947 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:33.945+0000 7fdbca59c640 1 --2- 172.21.15.19:0/543225075 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fdbac077ab0 0x7fdbac079fc0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fdbc0001480 tx=0x7fdbc0001cf0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:33.953 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:33.952+0000 7fdbbaffd640 1 -- 172.21.15.19:0/543225075 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fdbc4083540 con 0x7fdbcc1e39b0 2024-08-31T12:27:34.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:34.411+0000 7fdbcb59e640 1 -- 172.21.15.19:0/543225075 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fdbcc0c3a70 con 0x7fdbcc1e39b0 2024-08-31T12:27:34.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:34.411+0000 7fdbbaffd640 1 -- 172.21.15.19:0/543225075 <== mon.2 v2:172.21.15.43:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7fdbc4082e40 con 0x7fdbcc1e39b0 2024-08-31T12:27:34.414 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":90,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":25,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:27:34.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:34.415+0000 7fdbcb59e640 1 -- 172.21.15.19:0/543225075 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fdbac077ab0 msgr2=0x7fdbac079fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:34.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:34.415+0000 7fdbcb59e640 1 --2- 172.21.15.19:0/543225075 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fdbac077ab0 0x7fdbac079fc0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fdbc0001480 tx=0x7fdbc0001cf0 comp rx=0 tx=0).stop 2024-08-31T12:27:34.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:34.415+0000 7fdbcb59e640 1 -- 172.21.15.19:0/543225075 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdbcc1e39b0 msgr2=0x7fdbcc0cba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:34.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:34.415+0000 7fdbcb59e640 1 --2- 172.21.15.19:0/543225075 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdbcc1e39b0 0x7fdbcc0cba20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdbc402ab00 tx=0x7fdbc4032db0 comp rx=0 tx=0).stop 2024-08-31T12:27:34.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:34.415+0000 7fdbcb59e640 1 -- 172.21.15.19:0/543225075 shutdown_connections 2024-08-31T12:27:34.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:34.415+0000 7fdbcb59e640 1 --2- 172.21.15.19:0/543225075 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fdbac077ab0 0x7fdbac079fc0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:34.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:34.415+0000 7fdbcb59e640 1 --2- 172.21.15.19:0/543225075 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdbcc1e51d0 0x7fdbcc14f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:34.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:34.415+0000 7fdbcb59e640 1 --2- 172.21.15.19:0/543225075 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdbcc1e39b0 0x7fdbcc0cba20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:34.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:34.415+0000 7fdbcb59e640 1 --2- 172.21.15.19:0/543225075 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdbb4004160 0x7fdbcc1e21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:34.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:34.415+0000 7fdbcb59e640 1 -- 172.21.15.19:0/543225075 >> 172.21.15.19:0/543225075 conn(0x7fdbcc071290 msgr2=0x7fdbcc153440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:27:34.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:34.415+0000 7fdbcb59e640 1 -- 172.21.15.19:0/543225075 shutdown_connections 2024-08-31T12:27:34.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:34.415+0000 7fdbcb59e640 1 -- 172.21.15.19:0/543225075 wait complete. 2024-08-31T12:27:37.435 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:27:37.606 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:37.603+0000 7f3ae659f640 1 -- 172.21.15.19:0/2089913677 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3ae00cde10 msgr2=0x7f3ae014e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:37.606 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:37.603+0000 7f3ae659f640 1 --2- 172.21.15.19:0/2089913677 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3ae00cde10 0x7f3ae014e550 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3ad0005fd0 tx=0x7f3ad002a940 comp rx=0 tx=0).stop 2024-08-31T12:27:37.606 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:37.603+0000 7f3ae659f640 1 -- 172.21.15.19:0/2089913677 shutdown_connections 2024-08-31T12:27:37.606 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:37.603+0000 7f3ae659f640 1 --2- 172.21.15.19:0/2089913677 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3ae00cde10 0x7f3ae014e550 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:37.607 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:37.603+0000 7f3ae659f640 1 --2- 172.21.15.19:0/2089913677 >> v2:172.21.15.43:3300/0 conn(0x7f3ae00c5480 0x7f3ae00c7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:37.607 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:37.603+0000 7f3ae659f640 1 -- 172.21.15.19:0/2089913677 >> 172.21.15.19:0/2089913677 conn(0x7f3ae006fe90 msgr2=0x7f3ae00cd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:27:37.607 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:37.604+0000 7f3ae659f640 1 -- 172.21.15.19:0/2089913677 shutdown_connections 2024-08-31T12:27:37.607 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:37.604+0000 7f3ae659f640 1 -- 172.21.15.19:0/2089913677 wait complete. 2024-08-31T12:27:37.608 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:37.606+0000 7f3ae659f640 1 Processor -- start 2024-08-31T12:27:37.610 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:37.608+0000 7f3ae659f640 1 -- start start 2024-08-31T12:27:37.612 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:37.609+0000 7f3ae659f640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3ae006dbf0 0x7f3ae01f1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:37.612 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:37.610+0000 7f3ae659f640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ae00c5480 0x7f3ae0149420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:37.612 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:37.610+0000 7f3ae659f640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3ae00cde10 0x7f3ae01446b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:37.612 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:37.610+0000 7f3ae659f640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae0152290 con 0x7f3ae00cde10 2024-08-31T12:27:37.612 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:37.610+0000 7f3ae659f640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae014ea90 con 0x7f3ae00c5480 2024-08-31T12:27:37.612 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:37.610+0000 7f3ae659f640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f3ae0152410 con 0x7f3ae006dbf0 2024-08-31T12:27:37.613 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:37.610+0000 7f3adffff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3ae006dbf0 0x7f3ae01f1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:37.613 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:37.610+0000 7f3adffff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3ae006dbf0 0x7f3ae01f1c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.19:47990/0 (socket says 172.21.15.19:47990) 2024-08-31T12:27:37.613 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:37.610+0000 7f3adffff640 1 -- 172.21.15.19:0/1628181220 learned_addr learned my addr 172.21.15.19:0/1628181220 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:27:37.613 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:37.610+0000 7f3ae4b3c640 1 --2- 172.21.15.19:0/1628181220 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3ae00cde10 0x7f3ae01446b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:37.613 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:37.610+0000 7f3adf7fe640 1 --2- 172.21.15.19:0/1628181220 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ae00c5480 0x7f3ae0149420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:37.614 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:37.611+0000 7f3adffff640 1 -- 172.21.15.19:0/1628181220 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ae00c5480 msgr2=0x7f3ae0149420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:37.614 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:37.611+0000 7f3adffff640 1 --2- 172.21.15.19:0/1628181220 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ae00c5480 0x7f3ae0149420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:37.614 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:37.611+0000 7f3adffff640 1 -- 172.21.15.19:0/1628181220 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3ae00cde10 msgr2=0x7f3ae01446b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:37.614 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:37.611+0000 7f3adffff640 1 --2- 172.21.15.19:0/1628181220 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3ae00cde10 0x7f3ae01446b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:37.614 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:37.611+0000 7f3adffff640 1 -- 172.21.15.19:0/1628181220 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ae01f2160 con 0x7f3ae006dbf0 2024-08-31T12:27:37.614 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:37.611+0000 7f3adffff640 1 --2- 172.21.15.19:0/1628181220 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3ae006dbf0 0x7f3ae01f1c20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3ad400d980 tx=0x7f3ad400de40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:37.614 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:37.611+0000 7f3adcff9640 1 -- 172.21.15.19:0/1628181220 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad4017070 con 0x7f3ae006dbf0 2024-08-31T12:27:37.614 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:37.611+0000 7f3adcff9640 1 -- 172.21.15.19:0/1628181220 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3ad4009d40 con 0x7f3ae006dbf0 2024-08-31T12:27:37.614 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:37.611+0000 7f3adcff9640 1 -- 172.21.15.19:0/1628181220 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad4005280 con 0x7f3ae006dbf0 2024-08-31T12:27:37.614 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:37.611+0000 7f3ae659f640 1 -- 172.21.15.19:0/1628181220 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ae014d0b0 con 0x7f3ae006dbf0 2024-08-31T12:27:37.614 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:37.611+0000 7f3ae659f640 1 -- 172.21.15.19:0/1628181220 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ae014d390 con 0x7f3ae006dbf0 2024-08-31T12:27:37.615 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:37.613+0000 7f3ae659f640 1 -- 172.21.15.19:0/1628181220 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ae01586c0 con 0x7f3ae006dbf0 2024-08-31T12:27:37.617 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:37.614+0000 7f3adcff9640 1 -- 172.21.15.19:0/1628181220 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f3ad4005420 con 0x7f3ae006dbf0 2024-08-31T12:27:37.617 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:37.614+0000 7f3adcff9640 1 --2- 172.21.15.19:0/1628181220 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3acc077a50 0x7f3acc079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:37.617 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:37.615+0000 7f3adf7fe640 1 --2- 172.21.15.19:0/1628181220 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3acc077a50 0x7f3acc079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:37.617 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:37.615+0000 7f3adcff9640 1 -- 172.21.15.19:0/1628181220 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f3ad4099930 con 0x7f3ae006dbf0 2024-08-31T12:27:37.617 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:37.615+0000 7f3adf7fe640 1 --2- 172.21.15.19:0/1628181220 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3acc077a50 0x7f3acc079f60 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f3ad0034000 tx=0x7f3ad0004ab0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:37.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:37.621+0000 7f3adcff9640 1 -- 172.21.15.19:0/1628181220 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f3ad4061ea0 con 0x7f3ae006dbf0 2024-08-31T12:27:38.080 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.077+0000 7f3ae659f640 1 -- 172.21.15.19:0/1628181220 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3ae00c8c60 con 0x7f3ae006dbf0 2024-08-31T12:27:38.080 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.078+0000 7f3adcff9640 1 -- 172.21.15.19:0/1628181220 <== mon.2 v2:172.21.15.43:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f3ad4066660 con 0x7f3ae006dbf0 2024-08-31T12:27:38.080 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":92,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:27:38.083 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.081+0000 7f3ae659f640 1 -- 172.21.15.19:0/1628181220 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3acc077a50 msgr2=0x7f3acc079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:38.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.081+0000 7f3ae659f640 1 --2- 172.21.15.19:0/1628181220 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3acc077a50 0x7f3acc079f60 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f3ad0034000 tx=0x7f3ad0004ab0 comp rx=0 tx=0).stop 2024-08-31T12:27:38.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.081+0000 7f3ae659f640 1 -- 172.21.15.19:0/1628181220 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3ae006dbf0 msgr2=0x7f3ae01f1c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:38.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.081+0000 7f3ae659f640 1 --2- 172.21.15.19:0/1628181220 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3ae006dbf0 0x7f3ae01f1c20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3ad400d980 tx=0x7f3ad400de40 comp rx=0 tx=0).stop 2024-08-31T12:27:38.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.081+0000 7f3ae659f640 1 -- 172.21.15.19:0/1628181220 shutdown_connections 2024-08-31T12:27:38.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.081+0000 7f3ae659f640 1 --2- 172.21.15.19:0/1628181220 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3acc077a50 0x7f3acc079f60 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:38.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.081+0000 7f3ae659f640 1 --2- 172.21.15.19:0/1628181220 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3ae00cde10 0x7f3ae01446b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:38.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.081+0000 7f3ae659f640 1 --2- 172.21.15.19:0/1628181220 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ae00c5480 0x7f3ae0149420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:38.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.081+0000 7f3ae659f640 1 --2- 172.21.15.19:0/1628181220 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3ae006dbf0 0x7f3ae01f1c20 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:38.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.081+0000 7f3ae659f640 1 -- 172.21.15.19:0/1628181220 >> 172.21.15.19:0/1628181220 conn(0x7f3ae006fe90 msgr2=0x7f3ae006cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:27:38.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.082+0000 7f3ae659f640 1 -- 172.21.15.19:0/1628181220 shutdown_connections 2024-08-31T12:27:38.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.082+0000 7f3ae659f640 1 -- 172.21.15.19:0/1628181220 wait complete. 2024-08-31T12:27:38.100 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-08-31T12:27:38.100 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-08-31T12:27:38.101 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-31T12:27:38.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.315+0000 7f78d49ba640 1 -- 172.21.15.19:0/885601221 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78d0167ec0 msgr2=0x7f78d006cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:38.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.315+0000 7f78d49ba640 1 --2- 172.21.15.19:0/885601221 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78d0167ec0 0x7f78d006cfc0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f78d0072b10 tx=0x7f78c002b670 comp rx=0 tx=0).stop 2024-08-31T12:27:38.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.316+0000 7f78d49ba640 1 -- 172.21.15.19:0/885601221 shutdown_connections 2024-08-31T12:27:38.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.316+0000 7f78d49ba640 1 --2- 172.21.15.19:0/885601221 >> v2:172.21.15.43:3300/0 conn(0x7f78d00c5fa0 0x7f78d00c63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:38.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.316+0000 7f78d49ba640 1 --2- 172.21.15.19:0/885601221 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78d0167ec0 0x7f78d006cfc0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:38.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.316+0000 7f78d49ba640 1 -- 172.21.15.19:0/885601221 >> 172.21.15.19:0/885601221 conn(0x7f78d00cc950 msgr2=0x7f78d00ccd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:27:38.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.316+0000 7f78d49ba640 1 -- 172.21.15.19:0/885601221 shutdown_connections 2024-08-31T12:27:38.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.316+0000 7f78d49ba640 1 -- 172.21.15.19:0/885601221 wait complete. 2024-08-31T12:27:38.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.319+0000 7f78d49ba640 1 Processor -- start 2024-08-31T12:27:38.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.319+0000 7f78d49ba640 1 -- start start 2024-08-31T12:27:38.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.321+0000 7f78d49ba640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f78d008c7b0 0x7f78d0204040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:38.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.321+0000 7f78d49ba640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78d00c5fa0 0x7f78d00cb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:38.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.321+0000 7f78d49ba640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78d0167ec0 0x7f78d015ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:38.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.321+0000 7f78d49ba640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f78d00c6e60 con 0x7f78d00c5fa0 2024-08-31T12:27:38.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.321+0000 7f78d49ba640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f78d00c7da0 con 0x7f78d0167ec0 2024-08-31T12:27:38.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.321+0000 7f78d49ba640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f78d006d500 con 0x7f78d008c7b0 2024-08-31T12:27:38.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.321+0000 7f78c7fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78d00c5fa0 0x7f78d00cb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:38.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.321+0000 7f78c7fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78d00c5fa0 0x7f78d00cb7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:55642/0 (socket says 172.21.15.19:55642) 2024-08-31T12:27:38.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.321+0000 7f78c7fff640 1 -- 172.21.15.19:0/2504199601 learned_addr learned my addr 172.21.15.19:0/2504199601 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:27:38.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.321+0000 7f78ce59c640 1 --2- 172.21.15.19:0/2504199601 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f78d008c7b0 0x7f78d0204040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:38.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.321+0000 7f78ced9d640 1 --2- 172.21.15.19:0/2504199601 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78d0167ec0 0x7f78d015ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:38.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.322+0000 7f78ced9d640 1 -- 172.21.15.19:0/2504199601 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78d0167ec0 msgr2=0x7f78d015ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-08-31T12:27:38.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.322+0000 7f78ced9d640 1 -- 172.21.15.19:0/2504199601 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78d0167ec0 msgr2=0x7f78d015ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-31T12:27:38.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.322+0000 7f78ced9d640 1 --2- 172.21.15.19:0/2504199601 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78d0167ec0 0x7f78d015ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-31T12:27:38.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.322+0000 7f78ced9d640 1 --2- 172.21.15.19:0/2504199601 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78d0167ec0 0x7f78d015ab70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:27:38.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.322+0000 7f78c7fff640 1 -- 172.21.15.19:0/2504199601 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f78d008c7b0 msgr2=0x7f78d0204040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:38.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.322+0000 7f78c7fff640 1 --2- 172.21.15.19:0/2504199601 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f78d008c7b0 0x7f78d0204040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:38.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.322+0000 7f78c7fff640 1 -- 172.21.15.19:0/2504199601 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78d0167ec0 msgr2=0x7f78d015ab70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:27:38.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.322+0000 7f78c7fff640 1 --2- 172.21.15.19:0/2504199601 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78d0167ec0 0x7f78d015ab70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:38.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.322+0000 7f78c7fff640 1 -- 172.21.15.19:0/2504199601 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78d0204580 con 0x7f78d00c5fa0 2024-08-31T12:27:38.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.322+0000 7f78c7fff640 1 --2- 172.21.15.19:0/2504199601 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78d00c5fa0 0x7f78d00cb7c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f78b800c3f0 tx=0x7f78b800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:38.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.323+0000 7f78c6ffd640 1 -- 172.21.15.19:0/2504199601 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b8017070 con 0x7f78d00c5fa0 2024-08-31T12:27:38.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.323+0000 7f78d49ba640 1 -- 172.21.15.19:0/2504199601 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78d015f7a0 con 0x7f78d00c5fa0 2024-08-31T12:27:38.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.323+0000 7f78c6ffd640 1 -- 172.21.15.19:0/2504199601 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f78b80035f0 con 0x7f78d00c5fa0 2024-08-31T12:27:38.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.323+0000 7f78d49ba640 1 -- 172.21.15.19:0/2504199601 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78d0161160 con 0x7f78d00c5fa0 2024-08-31T12:27:38.327 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.323+0000 7f78c6ffd640 1 -- 172.21.15.19:0/2504199601 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b8003a40 con 0x7f78d00c5fa0 2024-08-31T12:27:38.327 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.324+0000 7f78d49ba640 1 --2- 172.21.15.19:0/2504199601 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78d015fa20 0x7f78d015b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:38.327 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.324+0000 7f78d49ba640 1 -- 172.21.15.19:0/2504199601 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f78d0162d50 con 0x7f78d015fa20 2024-08-31T12:27:38.327 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.324+0000 7f78ce59c640 1 --2- 172.21.15.19:0/2504199601 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78d015fa20 0x7f78d015b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:38.327 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.325+0000 7f78ce59c640 1 -- 172.21.15.19:0/2504199601 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f78d00c3ac0 con 0x7f78d015fa20 2024-08-31T12:27:38.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.326+0000 7f78ce59c640 1 --2- 172.21.15.19:0/2504199601 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78d015fa20 0x7f78d015b0b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f78d0072350 tx=0x7f78c0003940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:38.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.326+0000 7f78c6ffd640 1 -- 172.21.15.19:0/2504199601 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c0005020 con 0x7f78d015fa20 2024-08-31T12:27:38.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.326+0000 7f78c6ffd640 1 -- 172.21.15.19:0/2504199601 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f78c00051c0 con 0x7f78d015fa20 2024-08-31T12:27:38.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.326+0000 7f78c6ffd640 1 -- 172.21.15.19:0/2504199601 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78d015fa20 msgr2=0x7f78d015b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:38.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.326+0000 7f78c6ffd640 1 --2- 172.21.15.19:0/2504199601 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78d015fa20 0x7f78d015b0b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f78d0072350 tx=0x7f78c0003940 comp rx=0 tx=0).stop 2024-08-31T12:27:38.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.326+0000 7f78c6ffd640 1 -- 172.21.15.19:0/2504199601 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78d015fa20 msgr2=0x7f78d015b0b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:27:38.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.326+0000 7f78c6ffd640 1 --2- 172.21.15.19:0/2504199601 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78d015fa20 0x7f78d015b0b0 secure :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f78d0072350 tx=0x7f78c0003940 comp rx=0 tx=0).stop 2024-08-31T12:27:38.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.327+0000 7f78c6ffd640 1 -- 172.21.15.19:0/2504199601 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f78b801a070 con 0x7f78d00c5fa0 2024-08-31T12:27:38.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.328+0000 7f78c6ffd640 1 --2- 172.21.15.19:0/2504199601 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f78b0077560 0x7f78b0079a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:38.331 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.329+0000 7f78c6ffd640 1 -- 172.21.15.19:0/2504199601 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f78b8098890 con 0x7f78d00c5fa0 2024-08-31T12:27:38.331 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.329+0000 7f78ce59c640 1 --2- 172.21.15.19:0/2504199601 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f78b0077560 0x7f78b0079a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:38.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.330+0000 7f78ce59c640 1 --2- 172.21.15.19:0/2504199601 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f78b0077560 0x7f78b0079a70 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f78d0077070 tx=0x7f78c003f000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:38.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.339+0000 7f78d49ba640 1 --2- 172.21.15.19:0/2504199601 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78d0200340 0x7f78d0200750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:38.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.339+0000 7f78d49ba640 1 -- 172.21.15.19:0/2504199601 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f78d01eff50 con 0x7f78d0200340 2024-08-31T12:27:38.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.339+0000 7f78ced9d640 1 --2- 172.21.15.19:0/2504199601 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78d0200340 0x7f78d0200750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:38.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.340+0000 7f78ced9d640 1 -- 172.21.15.19:0/2504199601 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f78d0074610 con 0x7f78d0200340 2024-08-31T12:27:38.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.340+0000 7f78ced9d640 1 --2- 172.21.15.19:0/2504199601 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78d0200340 0x7f78d0200750 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f78d00cc530 tx=0x7f78c8007590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:38.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.340+0000 7f78c6ffd640 1 -- 172.21.15.19:0/2504199601 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c801a430 con 0x7f78d0200340 2024-08-31T12:27:38.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.340+0000 7f78c6ffd640 1 -- 172.21.15.19:0/2504199601 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5044 (secure 0 0 0) 0x7f78c801a430 con 0x7f78d0200340 2024-08-31T12:27:38.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.341+0000 7f78c6ffd640 1 -- 172.21.15.19:0/2504199601 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78d0200340 msgr2=0x7f78d0200750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:38.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.341+0000 7f78c6ffd640 1 --2- 172.21.15.19:0/2504199601 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78d0200340 0x7f78d0200750 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f78d00cc530 tx=0x7f78c8007590 comp rx=0 tx=0).stop 2024-08-31T12:27:38.343 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:27:38.343 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:27:38.343 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:27:38.343 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "leader", 2024-08-31T12:27:38.343 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 92, 2024-08-31T12:27:38.343 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 192895, 2024-08-31T12:27:38.344 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:27:38.344 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:27:38.344 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:27:38.344 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:27:38.344 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:38.344 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 2, 2024-08-31T12:27:38.344 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:27:38.344 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:27:38.344 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:27:38.344 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:27:38.344 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:27:38.344 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:27:38.344 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:27:38.344 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:27:38.345 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:27:38.345 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:27:38.345 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:27:38.345 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:27:38.345 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:27:38.345 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:27:38.345 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:38.345 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:27:38.345 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:27:38.345 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:27:38.345 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:27:38.345 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:27:38.345 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:27:38.345 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:27:38.345 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:27:38.346 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:27:38.346 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:27:38.346 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:27:38.346 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:27:38.346 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:27:38.346 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:38.346 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:38.346 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:27:38.346 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:27:38.346 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:27:38.346 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:27:38.346 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:27:38.346 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:27:38.347 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:27:38.347 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:27:38.347 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:27:38.347 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:27:38.347 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:27:38.347 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:27:38.347 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:27:38.347 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:27:38.347 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:27:38.347 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:27:38.347 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:27:38.347 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:27:38.347 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:27:38.347 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:27:38.347 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:27:38.348 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:27:38.348 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:27:38.348 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:27:38.348 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:27:38.348 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:27:38.348 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:27:38.348 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:27:38.348 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:38.348 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:27:38.348 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:38.348 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:27:38.348 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:38.348 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:27:38.348 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:27:38.349 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:27:38.349 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:27:38.349 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:38.349 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:27:38.349 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:27:38.349 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:38.349 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:38.349 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:38.349 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:27:38.349 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:27:38.349 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:38.349 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:38.349 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:38.349 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:38.350 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:27:38.350 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:27:38.350 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:27:38.350 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:27:38.350 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:27:38.350 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:38.350 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:38.350 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:27:38.350 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:27:38.350 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:27:38.351 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:27:38.351 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:38.351 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:27:38.351 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:27:38.351 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:38.351 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:38.351 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:38.351 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:27:38.351 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:27:38.351 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:38.351 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:38.351 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:38.351 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:38.352 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:27:38.352 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:27:38.352 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:27:38.352 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:27:38.352 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:27:38.352 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:38.352 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:38.352 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:27:38.352 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:27:38.352 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:27:38.352 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:27:38.352 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:38.353 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:27:38.353 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:27:38.353 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:38.353 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:38.353 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:38.353 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:27:38.353 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:27:38.353 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:38.353 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:38.353 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:38.353 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:38.353 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:27:38.353 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:27:38.354 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:27:38.354 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:27:38.354 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:27:38.354 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:38.354 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:38.354 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:38.354 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:27:38.354 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:27:38.354 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:38.354 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:27:38.354 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:27:38.354 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:27:38.354 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:38.355 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:38.355 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:27:38.355 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:38.355 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:27:38.355 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:27:38.355 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 5 2024-08-31T12:27:38.355 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:38.355 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:38.355 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:27:38.355 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:38.355 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:27:38.356 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:27:38.356 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 7 2024-08-31T12:27:38.356 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:38.356 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:38.356 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:27:38.356 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:38.356 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:27:38.356 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:27:38.356 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 5 2024-08-31T12:27:38.356 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:38.356 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:38.356 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:27:38.356 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:38.356 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:27:38.356 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:27:38.357 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:27:38.357 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:38.357 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:38.357 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:38.357 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:27:38.357 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:27:38.358 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.341+0000 7f78c6ffd640 1 -- 172.21.15.19:0/2504199601 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78d0200340 msgr2=0x7f78d0200750 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:27:38.358 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.341+0000 7f78c6ffd640 1 --2- 172.21.15.19:0/2504199601 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78d0200340 0x7f78d0200750 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:38.358 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.341+0000 7f78d49ba640 1 -- 172.21.15.19:0/2504199601 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f78b0077560 msgr2=0x7f78b0079a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:38.358 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.341+0000 7f78d49ba640 1 --2- 172.21.15.19:0/2504199601 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f78b0077560 0x7f78b0079a70 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f78d0077070 tx=0x7f78c003f000 comp rx=0 tx=0).stop 2024-08-31T12:27:38.358 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.341+0000 7f78ced9d640 1 -- 172.21.15.19:0/2504199601 reap_dead start 2024-08-31T12:27:38.358 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.341+0000 7f78d49ba640 1 -- 172.21.15.19:0/2504199601 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78d00c5fa0 msgr2=0x7f78d00cb7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:38.358 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.341+0000 7f78d49ba640 1 --2- 172.21.15.19:0/2504199601 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78d00c5fa0 0x7f78d00cb7c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f78b800c3f0 tx=0x7f78b800c8b0 comp rx=0 tx=0).stop 2024-08-31T12:27:38.358 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.341+0000 7f78d49ba640 1 -- 172.21.15.19:0/2504199601 shutdown_connections 2024-08-31T12:27:38.359 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.341+0000 7f78d49ba640 1 --2- 172.21.15.19:0/2504199601 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78d00c5fa0 0x7f78d00cb7c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:38.359 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.341+0000 7f78d49ba640 1 -- 172.21.15.19:0/2504199601 >> 172.21.15.19:0/2504199601 conn(0x7f78d00cc950 msgr2=0x7f78d0167370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:27:38.359 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.341+0000 7f78d49ba640 1 -- 172.21.15.19:0/2504199601 shutdown_connections 2024-08-31T12:27:38.359 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.341+0000 7f78d49ba640 1 -- 172.21.15.19:0/2504199601 wait complete. 2024-08-31T12:27:38.360 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:27:38.360 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-08-31T12:27:38.360 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-31T12:27:38.539 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.536+0000 7f9711513640 1 -- 172.21.15.19:0/4076023188 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f970c0cc400 msgr2=0x7f970c0cc810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:38.539 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.536+0000 7f9711513640 1 --2- 172.21.15.19:0/4076023188 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f970c0cc400 0x7f970c0cc810 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9704005fd0 tx=0x7f970402a940 comp rx=0 tx=0).stop 2024-08-31T12:27:38.539 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.537+0000 7f9711513640 1 -- 172.21.15.19:0/4076023188 shutdown_connections 2024-08-31T12:27:38.539 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.537+0000 7f9711513640 1 --2- 172.21.15.19:0/4076023188 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f970c0cc400 0x7f970c0cc810 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:38.539 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.537+0000 7f9711513640 1 --2- 172.21.15.19:0/4076023188 >> v2:172.21.15.43:3300/0 conn(0x7f970c0c5370 0x7f970c06ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:38.539 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.537+0000 7f9711513640 1 -- 172.21.15.19:0/4076023188 >> 172.21.15.19:0/4076023188 conn(0x7f970c0ca670 msgr2=0x7f970c0cd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:27:38.539 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.537+0000 7f9711513640 1 -- 172.21.15.19:0/4076023188 shutdown_connections 2024-08-31T12:27:38.540 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.538+0000 7f9711513640 1 -- 172.21.15.19:0/4076023188 wait complete. 2024-08-31T12:27:38.542 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.541+0000 7f9711513640 1 Processor -- start 2024-08-31T12:27:38.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.541+0000 7f9711513640 1 -- start start 2024-08-31T12:27:38.546 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.543+0000 7f9711513640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f970c0c5370 0x7f970c1623d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:38.546 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.543+0000 7f9711513640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f970c0cb2b0 0x7f970c1604c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:38.546 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.543+0000 7f9711513640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f970c0cc400 0x7f970c1f8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:38.546 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.543+0000 7f970affd640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f970c0c5370 0x7f970c1623d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:38.546 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.543+0000 7f970affd640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f970c0c5370 0x7f970c1623d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:34998/0 (socket says 172.21.15.19:34998) 2024-08-31T12:27:38.546 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.543+0000 7f970a7fc640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f970c0cb2b0 0x7f970c1604c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:38.546 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.543+0000 7f970a7fc640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f970c0cb2b0 0x7f970c1604c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.19:57960/0 (socket says 172.21.15.19:57960) 2024-08-31T12:27:38.546 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.543+0000 7f970b7fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f970c0cc400 0x7f970c1f8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:38.546 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.543+0000 7f970b7fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f970c0cc400 0x7f970c1f8ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:55670/0 (socket says 172.21.15.19:55670) 2024-08-31T12:27:38.547 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.544+0000 7f970affd640 1 -- 172.21.15.19:0/2096696802 learned_addr learned my addr 172.21.15.19:0/2096696802 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:27:38.547 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.544+0000 7f9711513640 1 -- 172.21.15.19:0/2096696802 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f970c0c72e0 con 0x7f970c0cc400 2024-08-31T12:27:38.547 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.544+0000 7f9711513640 1 -- 172.21.15.19:0/2096696802 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f970c0ccd50 con 0x7f970c0c5370 2024-08-31T12:27:38.547 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.544+0000 7f9711513640 1 -- 172.21.15.19:0/2096696802 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f970c0cb8c0 con 0x7f970c0cb2b0 2024-08-31T12:27:38.548 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.544+0000 7f970b7fe640 1 -- 172.21.15.19:0/2096696802 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f970c0cb2b0 msgr2=0x7f970c1604c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:38.548 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.544+0000 7f970b7fe640 1 --2- 172.21.15.19:0/2096696802 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f970c0cb2b0 0x7f970c1604c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:38.548 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.544+0000 7f970b7fe640 1 -- 172.21.15.19:0/2096696802 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f970c0c5370 msgr2=0x7f970c1623d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:38.548 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.544+0000 7f970b7fe640 1 --2- 172.21.15.19:0/2096696802 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f970c0c5370 0x7f970c1623d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:38.548 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.544+0000 7f970b7fe640 1 -- 172.21.15.19:0/2096696802 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f970c0c8fc0 con 0x7f970c0cc400 2024-08-31T12:27:38.548 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.545+0000 7f970b7fe640 1 --2- 172.21.15.19:0/2096696802 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f970c0cc400 0x7f970c1f8ff0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f970c0ce2c0 tx=0x7f9704031e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:38.548 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.545+0000 7f96f37fe640 1 -- 172.21.15.19:0/2096696802 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f970403e070 con 0x7f970c0cc400 2024-08-31T12:27:38.548 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.545+0000 7f96f37fe640 1 -- 172.21.15.19:0/2096696802 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9704004560 con 0x7f970c0cc400 2024-08-31T12:27:38.548 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.545+0000 7f96f37fe640 1 -- 172.21.15.19:0/2096696802 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9704043460 con 0x7f970c0cc400 2024-08-31T12:27:38.549 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.545+0000 7f9711513640 1 -- 172.21.15.19:0/2096696802 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f970c0c9280 con 0x7f970c0cc400 2024-08-31T12:27:38.549 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.545+0000 7f9711513640 1 -- 172.21.15.19:0/2096696802 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f970c0c8910 con 0x7f970c0cc400 2024-08-31T12:27:38.549 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.547+0000 7f9711513640 1 --2- 172.21.15.19:0/2096696802 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f970c0c99e0 0x7f970c143dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:38.550 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.547+0000 7f9711513640 1 -- 172.21.15.19:0/2096696802 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f970c144ae0 con 0x7f970c0c99e0 2024-08-31T12:27:38.550 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.548+0000 7f970affd640 1 --2- 172.21.15.19:0/2096696802 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f970c0c99e0 0x7f970c143dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:38.550 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.548+0000 7f970affd640 1 -- 172.21.15.19:0/2096696802 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f970c06e760 con 0x7f970c0c99e0 2024-08-31T12:27:38.550 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.548+0000 7f96f37fe640 1 -- 172.21.15.19:0/2096696802 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f970403a440 con 0x7f970c0cc400 2024-08-31T12:27:38.550 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.548+0000 7f970affd640 1 --2- 172.21.15.19:0/2096696802 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f970c0c99e0 0x7f970c143dc0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f96fc0109a0 tx=0x7f96fc010e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:38.551 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.549+0000 7f96f37fe640 1 --2- 172.21.15.19:0/2096696802 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f96f8077a80 0x7f96f8079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:38.552 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.549+0000 7f96f37fe640 1 -- 172.21.15.19:0/2096696802 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f97040bb0e0 con 0x7f970c0cc400 2024-08-31T12:27:38.552 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.549+0000 7f96f37fe640 1 -- 172.21.15.19:0/2096696802 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96fc01c480 con 0x7f970c0c99e0 2024-08-31T12:27:38.552 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.550+0000 7f96f37fe640 1 -- 172.21.15.19:0/2096696802 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f96fc01c620 con 0x7f970c0c99e0 2024-08-31T12:27:38.552 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.550+0000 7f96f37fe640 1 -- 172.21.15.19:0/2096696802 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f970c0c99e0 msgr2=0x7f970c143dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:38.552 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.550+0000 7f96f37fe640 1 --2- 172.21.15.19:0/2096696802 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f970c0c99e0 0x7f970c143dc0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f96fc0109a0 tx=0x7f96fc010e60 comp rx=0 tx=0).stop 2024-08-31T12:27:38.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.550+0000 7f970a7fc640 1 --2- 172.21.15.19:0/2096696802 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f96f8077a80 0x7f96f8079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:38.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.550+0000 7f96f37fe640 1 -- 172.21.15.19:0/2096696802 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f970c0c99e0 msgr2=0x7f970c143dc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:27:38.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.550+0000 7f96f37fe640 1 --2- 172.21.15.19:0/2096696802 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f970c0c99e0 0x7f970c143dc0 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f96fc0109a0 tx=0x7f96fc010e60 comp rx=0 tx=0).stop 2024-08-31T12:27:38.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.550+0000 7f970a7fc640 1 --2- 172.21.15.19:0/2096696802 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f96f8077a80 0x7f96f8079f90 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f970c0c9750 tx=0x7f9700009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:38.564 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.562+0000 7f9711513640 1 --2- 172.21.15.19:0/2096696802 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f970c1f9530 0x7f970c1f9940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:38.564 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.562+0000 7f9711513640 1 -- 172.21.15.19:0/2096696802 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f970c0cbfd0 con 0x7f970c1f9530 2024-08-31T12:27:38.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.562+0000 7f970affd640 1 --2- 172.21.15.19:0/2096696802 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f970c1f9530 0x7f970c1f9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:38.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.562+0000 7f970affd640 1 -- 172.21.15.19:0/2096696802 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f970c06cf40 con 0x7f970c1f9530 2024-08-31T12:27:38.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.563+0000 7f970affd640 1 --2- 172.21.15.19:0/2096696802 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f970c1f9530 0x7f970c1f9940 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f96fc0109a0 tx=0x7f96fc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:38.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.563+0000 7f96f37fe640 1 -- 172.21.15.19:0/2096696802 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96fc027070 con 0x7f970c1f9530 2024-08-31T12:27:38.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.564+0000 7f96f37fe640 1 -- 172.21.15.19:0/2096696802 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4539 (secure 0 0 0) 0x7f96fc021770 con 0x7f970c1f9530 2024-08-31T12:27:38.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.564+0000 7f96f37fe640 1 -- 172.21.15.19:0/2096696802 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f970c1f9530 msgr2=0x7f970c1f9940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:38.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.564+0000 7f96f37fe640 1 --2- 172.21.15.19:0/2096696802 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f970c1f9530 0x7f970c1f9940 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f96fc0109a0 tx=0x7f96fc0046c0 comp rx=0 tx=0).stop 2024-08-31T12:27:38.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.564+0000 7f96f37fe640 1 -- 172.21.15.19:0/2096696802 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f970c1f9530 msgr2=0x7f970c1f9940 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:27:38.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.564+0000 7f96f37fe640 1 --2- 172.21.15.19:0/2096696802 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f970c1f9530 0x7f970c1f9940 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:38.566 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:27:38.566 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:27:38.566 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:27:38.566 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:27:38.567 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 92, 2024-08-31T12:27:38.567 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 8386, 2024-08-31T12:27:38.567 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:27:38.567 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:27:38.567 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:27:38.567 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:27:38.567 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:38.567 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 3, 2024-08-31T12:27:38.567 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:27:38.567 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:27:38.567 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:27:38.567 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:27:38.567 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:27:38.567 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:27:38.567 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:27:38.568 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:27:38.568 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:27:38.568 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:27:38.568 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:27:38.568 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:27:38.568 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:27:38.568 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:27:38.568 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:38.568 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:27:38.568 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:27:38.568 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:27:38.568 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:27:38.568 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:27:38.569 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:27:38.569 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:27:38.569 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:27:38.569 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:27:38.569 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:27:38.569 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:27:38.569 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:27:38.569 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:27:38.569 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:38.569 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:38.569 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:27:38.569 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:27:38.569 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:27:38.570 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:27:38.570 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:27:38.570 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:27:38.570 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:27:38.570 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:27:38.570 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:27:38.570 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:27:38.570 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:27:38.570 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:27:38.570 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:27:38.570 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:27:38.570 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:27:38.570 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:27:38.570 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:27:38.571 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:27:38.571 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:27:38.571 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:27:38.571 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:27:38.571 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:27:38.571 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:27:38.571 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:27:38.571 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:27:38.571 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:27:38.571 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:27:38.571 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:27:38.571 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:38.571 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:27:38.572 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:38.572 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:27:38.572 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:38.572 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:27:38.572 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:27:38.572 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:27:38.572 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:27:38.572 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:38.572 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:27:38.572 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:27:38.572 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:38.572 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:38.572 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:38.573 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:27:38.573 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:27:38.573 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:38.573 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:38.573 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:38.573 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:38.573 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:27:38.573 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:27:38.573 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:27:38.573 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:27:38.573 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:27:38.573 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:38.573 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:38.573 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:27:38.574 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:27:38.574 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:27:38.574 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:27:38.574 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:38.574 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:27:38.574 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:27:38.574 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:38.574 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:38.574 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:38.574 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:27:38.574 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:27:38.574 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:38.574 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:38.575 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:38.575 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:38.575 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:27:38.575 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:27:38.575 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:27:38.575 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:27:38.575 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:27:38.575 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:38.575 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:38.575 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:27:38.575 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:27:38.575 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:27:38.575 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:27:38.575 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:38.576 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:27:38.576 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:27:38.576 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:38.576 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:38.576 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:38.576 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:27:38.576 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:27:38.576 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:38.576 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:38.576 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:38.576 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:38.576 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:27:38.576 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:27:38.577 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:27:38.577 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:27:38.577 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:27:38.577 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:38.577 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:38.577 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:38.577 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:27:38.577 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:27:38.577 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:38.577 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:27:38.577 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:27:38.577 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:27:38.577 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:38.577 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:38.578 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:27:38.578 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:38.578 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:27:38.578 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:27:38.578 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:27:38.578 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:38.578 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:38.578 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:38.578 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:27:38.578 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:27:38.579 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.565+0000 7f9711513640 1 -- 172.21.15.19:0/2096696802 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f96f8077a80 msgr2=0x7f96f8079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:38.579 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.565+0000 7f9711513640 1 --2- 172.21.15.19:0/2096696802 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f96f8077a80 0x7f96f8079f90 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f970c0c9750 tx=0x7f9700009040 comp rx=0 tx=0).stop 2024-08-31T12:27:38.579 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.565+0000 7f970b7fe640 1 -- 172.21.15.19:0/2096696802 reap_dead start 2024-08-31T12:27:38.579 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.565+0000 7f9711513640 1 -- 172.21.15.19:0/2096696802 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f970c0cc400 msgr2=0x7f970c1f8ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:38.579 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.565+0000 7f9711513640 1 --2- 172.21.15.19:0/2096696802 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f970c0cc400 0x7f970c1f8ff0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f970c0ce2c0 tx=0x7f9704031e20 comp rx=0 tx=0).stop 2024-08-31T12:27:38.579 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.565+0000 7f9711513640 1 -- 172.21.15.19:0/2096696802 shutdown_connections 2024-08-31T12:27:38.579 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.565+0000 7f9711513640 1 --2- 172.21.15.19:0/2096696802 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f970c0cc400 0x7f970c1f8ff0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:38.579 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.565+0000 7f9711513640 1 -- 172.21.15.19:0/2096696802 >> 172.21.15.19:0/2096696802 conn(0x7f970c0ca670 msgr2=0x7f970c06d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:27:38.579 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.565+0000 7f9711513640 1 -- 172.21.15.19:0/2096696802 shutdown_connections 2024-08-31T12:27:38.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.565+0000 7f9711513640 1 -- 172.21.15.19:0/2096696802 wait complete. 2024-08-31T12:27:38.584 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:27:38.584 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-08-31T12:27:38.584 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-31T12:27:38.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.751+0000 7f04667fc640 1 -- 172.21.15.19:0/1469309984 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04580062d0 con 0x7f0454004ce0 2024-08-31T12:27:38.754 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.751+0000 7f0467fff640 1 -- 172.21.15.19:0/1469309984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0454004ce0 msgr2=0x7f0454005130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:38.754 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.751+0000 7f0467fff640 1 --2- 172.21.15.19:0/1469309984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0454004ce0 0x7f0454005130 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0458001760 tx=0x7f0454005830 comp rx=0 tx=0).stop 2024-08-31T12:27:38.754 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.751+0000 7f0467fff640 1 -- 172.21.15.19:0/1469309984 shutdown_connections 2024-08-31T12:27:38.754 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.751+0000 7f0467fff640 1 --2- 172.21.15.19:0/1469309984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0454004ce0 0x7f0454005130 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:38.754 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.752+0000 7f0467fff640 1 -- 172.21.15.19:0/1469309984 >> 172.21.15.19:0/1469309984 conn(0x7f04680c99b0 msgr2=0x7f046806ef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:27:38.754 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.752+0000 7f0467fff640 1 -- 172.21.15.19:0/1469309984 shutdown_connections 2024-08-31T12:27:38.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.752+0000 7f0467fff640 1 -- 172.21.15.19:0/1469309984 wait complete. 2024-08-31T12:27:38.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.755+0000 7f0467fff640 1 Processor -- start 2024-08-31T12:27:38.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.755+0000 7f0467fff640 1 -- start start 2024-08-31T12:27:38.759 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.757+0000 7f0467fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0454004ce0 0x7f046815b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:38.759 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.757+0000 7f0467fff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0468074bf0 0x7f0468201a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:38.759 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.757+0000 7f0467fff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0468151220 0x7f0468159c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:38.759 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.757+0000 7f0467fff640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f04680c5170 con 0x7f0454004ce0 2024-08-31T12:27:38.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.757+0000 7f0467fff640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f04680c7db0 con 0x7f0468074bf0 2024-08-31T12:27:38.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.757+0000 7f0467fff640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f046806fd80 con 0x7f0468151220 2024-08-31T12:27:38.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.757+0000 7f045ffff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0468074bf0 0x7f0468201a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:38.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.757+0000 7f0466ffd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0454004ce0 0x7f046815b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:38.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.757+0000 7f0466ffd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0454004ce0 0x7f046815b1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:55700/0 (socket says 172.21.15.19:55700) 2024-08-31T12:27:38.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.757+0000 7f045ffff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0468074bf0 0x7f0468201a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:35034/0 (socket says 172.21.15.19:35034) 2024-08-31T12:27:38.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.757+0000 7f0466ffd640 1 -- 172.21.15.19:0/2810295672 learned_addr learned my addr 172.21.15.19:0/2810295672 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:27:38.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.757+0000 7f04677fe640 1 --2- 172.21.15.19:0/2810295672 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0468151220 0x7f0468159c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:38.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.757+0000 7f0466ffd640 1 -- 172.21.15.19:0/2810295672 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0468151220 msgr2=0x7f0468159c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:38.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.757+0000 7f0466ffd640 1 --2- 172.21.15.19:0/2810295672 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0468151220 0x7f0468159c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:38.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.757+0000 7f0466ffd640 1 -- 172.21.15.19:0/2810295672 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0468074bf0 msgr2=0x7f0468201a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:38.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.757+0000 7f0466ffd640 1 --2- 172.21.15.19:0/2810295672 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0468074bf0 0x7f0468201a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:38.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.757+0000 7f0466ffd640 1 -- 172.21.15.19:0/2810295672 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0468201fc0 con 0x7f0454004ce0 2024-08-31T12:27:38.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.757+0000 7f045ffff640 1 --2- 172.21.15.19:0/2810295672 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0468074bf0 0x7f0468201a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:27:38.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.758+0000 7f0466ffd640 1 --2- 172.21.15.19:0/2810295672 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0454004ce0 0x7f046815b1a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0458001880 tx=0x7f0458006360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:38.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.758+0000 7f045f7fe640 1 -- 172.21.15.19:0/2810295672 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0458003a00 con 0x7f0454004ce0 2024-08-31T12:27:38.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.758+0000 7f045f7fe640 1 -- 172.21.15.19:0/2810295672 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0458039070 con 0x7f0454004ce0 2024-08-31T12:27:38.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.758+0000 7f045f7fe640 1 -- 172.21.15.19:0/2810295672 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0458033420 con 0x7f0454004ce0 2024-08-31T12:27:38.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.759+0000 7f04677fe640 1 --2- 172.21.15.19:0/2810295672 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0468151220 0x7f0468159c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:27:38.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.760+0000 7f0467fff640 1 -- 172.21.15.19:0/2810295672 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0468156710 con 0x7f0454004ce0 2024-08-31T12:27:38.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.760+0000 7f0467fff640 1 -- 172.21.15.19:0/2810295672 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0468156430 con 0x7f0454004ce0 2024-08-31T12:27:38.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.762+0000 7f0467fff640 1 --2- 172.21.15.19:0/2810295672 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0468151630 0x7f046815a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:38.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.762+0000 7f0467fff640 1 -- 172.21.15.19:0/2810295672 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f04681feb80 con 0x7f0468151630 2024-08-31T12:27:38.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.762+0000 7f045ffff640 1 --2- 172.21.15.19:0/2810295672 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0468151630 0x7f046815a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:38.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.762+0000 7f045ffff640 1 -- 172.21.15.19:0/2810295672 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f04681fedd0 con 0x7f0468151630 2024-08-31T12:27:38.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.763+0000 7f045ffff640 1 --2- 172.21.15.19:0/2810295672 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0468151630 0x7f046815a1b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f046806b8f0 tx=0x7f045000d6b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:38.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.763+0000 7f045f7fe640 1 -- 172.21.15.19:0/2810295672 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f0458044020 con 0x7f0454004ce0 2024-08-31T12:27:38.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.764+0000 7f045f7fe640 1 --2- 172.21.15.19:0/2810295672 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0448077a50 0x7f0448079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:38.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.764+0000 7f045f7fe640 1 -- 172.21.15.19:0/2810295672 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f04580b8ec0 con 0x7f0454004ce0 2024-08-31T12:27:38.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.764+0000 7f045f7fe640 1 -- 172.21.15.19:0/2810295672 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0450005020 con 0x7f0468151630 2024-08-31T12:27:38.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.764+0000 7f04677fe640 1 --2- 172.21.15.19:0/2810295672 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0448077a50 0x7f0448079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:38.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.764+0000 7f045f7fe640 1 -- 172.21.15.19:0/2810295672 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f04500051c0 con 0x7f0468151630 2024-08-31T12:27:38.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.764+0000 7f045f7fe640 1 -- 172.21.15.19:0/2810295672 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0468151630 msgr2=0x7f046815a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:38.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.764+0000 7f045f7fe640 1 --2- 172.21.15.19:0/2810295672 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0468151630 0x7f046815a1b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f046806b8f0 tx=0x7f045000d6b0 comp rx=0 tx=0).stop 2024-08-31T12:27:38.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.764+0000 7f045f7fe640 1 -- 172.21.15.19:0/2810295672 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0468151630 msgr2=0x7f046815a1b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:27:38.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.764+0000 7f045f7fe640 1 --2- 172.21.15.19:0/2810295672 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0468151630 0x7f046815a1b0 secure :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f046806b8f0 tx=0x7f045000d6b0 comp rx=0 tx=0).stop 2024-08-31T12:27:38.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.765+0000 7f04677fe640 1 --2- 172.21.15.19:0/2810295672 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0448077a50 0x7f0448079f60 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f046806fd10 tx=0x7f0460006ee0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:38.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.777+0000 7f0467fff640 1 --2- 172.21.15.19:0/2810295672 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f04680cc750 0x7f04680ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:38.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.777+0000 7f0467fff640 1 -- 172.21.15.19:0/2810295672 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f046815b6e0 con 0x7f04680cc750 2024-08-31T12:27:38.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.778+0000 7f045ffff640 1 --2- 172.21.15.19:0/2810295672 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f04680cc750 0x7f04680ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:38.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.779+0000 7f045ffff640 1 -- 172.21.15.19:0/2810295672 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f04680c82d0 con 0x7f04680cc750 2024-08-31T12:27:38.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.779+0000 7f045ffff640 1 --2- 172.21.15.19:0/2810295672 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f04680cc750 0x7f04680ccb60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f045001b000 tx=0x7f04500046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:38.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.779+0000 7f045f7fe640 1 -- 172.21.15.19:0/2810295672 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0450021770 con 0x7f04680cc750 2024-08-31T12:27:38.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.779+0000 7f045f7fe640 1 -- 172.21.15.19:0/2810295672 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4874 (secure 0 0 0) 0x7f0450021770 con 0x7f04680cc750 2024-08-31T12:27:38.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.779+0000 7f045f7fe640 1 -- 172.21.15.19:0/2810295672 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f04680cc750 msgr2=0x7f04680ccb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:38.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.779+0000 7f045f7fe640 1 --2- 172.21.15.19:0/2810295672 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f04680cc750 0x7f04680ccb60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f045001b000 tx=0x7f04500046c0 comp rx=0 tx=0).stop 2024-08-31T12:27:38.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.779+0000 7f045f7fe640 1 -- 172.21.15.19:0/2810295672 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f04680cc750 msgr2=0x7f04680ccb60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:27:38.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.779+0000 7f045f7fe640 1 --2- 172.21.15.19:0/2810295672 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f04680cc750 0x7f04680ccb60 secure :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f045001b000 tx=0x7f04500046c0 comp rx=0 tx=0).stop 2024-08-31T12:27:38.782 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:27:38.782 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:27:38.782 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:27:38.783 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:27:38.783 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 92, 2024-08-31T12:27:38.783 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 64649, 2024-08-31T12:27:38.783 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:27:38.783 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:27:38.783 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:27:38.783 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:27:38.783 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:38.783 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 3, 2024-08-31T12:27:38.783 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:27:38.783 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:27:38.783 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:27:38.783 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:27:38.783 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:27:38.784 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:27:38.784 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:27:38.784 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:27:38.784 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:27:38.784 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:27:38.784 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:27:38.784 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:27:38.784 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:27:38.784 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:27:38.784 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:38.784 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:27:38.784 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:27:38.784 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:27:38.785 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:27:38.785 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:27:38.785 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:27:38.785 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:27:38.785 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:27:38.785 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:27:38.785 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:27:38.785 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:27:38.785 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:27:38.785 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:27:38.785 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:38.785 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:38.785 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:27:38.785 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:27:38.786 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:27:38.786 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:27:38.786 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:27:38.786 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:27:38.786 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:27:38.786 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:27:38.786 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:27:38.786 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:27:38.786 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:27:38.786 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:27:38.786 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:27:38.786 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:27:38.786 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:27:38.786 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:27:38.787 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:27:38.787 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:27:38.787 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:27:38.787 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:27:38.787 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:27:38.787 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:27:38.787 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:27:38.787 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:27:38.787 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:27:38.787 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:27:38.787 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:27:38.787 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:27:38.787 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:38.787 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:27:38.788 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:38.788 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:27:38.788 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:38.788 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:27:38.788 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:27:38.788 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:27:38.788 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:27:38.788 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:38.788 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:27:38.788 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:27:38.788 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:38.788 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:38.788 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:38.789 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:27:38.789 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:27:38.789 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:38.789 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:38.789 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:38.789 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:38.789 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:27:38.789 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:27:38.789 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:27:38.789 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:27:38.789 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:27:38.789 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:38.789 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:38.789 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:27:38.790 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:27:38.790 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:27:38.790 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:27:38.790 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:38.790 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:27:38.790 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:27:38.790 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:38.790 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:38.790 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:38.790 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:27:38.790 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:27:38.790 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:38.790 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:38.791 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:38.791 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:38.791 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:27:38.791 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:27:38.791 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:27:38.791 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:27:38.791 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:27:38.791 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:38.791 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:38.791 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:27:38.791 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:27:38.791 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:27:38.791 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:27:38.791 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:38.792 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:27:38.792 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:27:38.792 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:38.792 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:38.792 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:38.792 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:27:38.792 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:27:38.792 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:38.792 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:38.792 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:38.792 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:38.792 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:27:38.792 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:27:38.793 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:27:38.793 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:27:38.793 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:27:38.793 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:38.793 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:38.793 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:38.793 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:27:38.793 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:27:38.793 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:38.793 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:27:38.793 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:27:38.793 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:27:38.793 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:38.794 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:38.794 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:27:38.794 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:38.794 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:27:38.794 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:27:38.794 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:27:38.794 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:38.794 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:38.794 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:27:38.794 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:38.794 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:27:38.794 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:27:38.794 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:27:38.794 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:38.794 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:38.795 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:27:38.795 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:38.795 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:27:38.795 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:27:38.795 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 3 2024-08-31T12:27:38.795 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:38.795 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:38.795 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:38.795 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:27:38.795 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:27:38.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.780+0000 7f0467fff640 1 -- 172.21.15.19:0/2810295672 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0448077a50 msgr2=0x7f0448079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:38.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.780+0000 7f0467fff640 1 --2- 172.21.15.19:0/2810295672 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0448077a50 0x7f0448079f60 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f046806fd10 tx=0x7f0460006ee0 comp rx=0 tx=0).stop 2024-08-31T12:27:38.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.780+0000 7f04677fe640 1 -- 172.21.15.19:0/2810295672 reap_dead start 2024-08-31T12:27:38.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.780+0000 7f0467fff640 1 -- 172.21.15.19:0/2810295672 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0454004ce0 msgr2=0x7f046815b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:38.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.780+0000 7f0467fff640 1 --2- 172.21.15.19:0/2810295672 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0454004ce0 0x7f046815b1a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0458001880 tx=0x7f0458006360 comp rx=0 tx=0).stop 2024-08-31T12:27:38.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.781+0000 7f0467fff640 1 -- 172.21.15.19:0/2810295672 shutdown_connections 2024-08-31T12:27:38.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.781+0000 7f0467fff640 1 --2- 172.21.15.19:0/2810295672 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0454004ce0 0x7f046815b1a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:38.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.781+0000 7f0467fff640 1 -- 172.21.15.19:0/2810295672 >> 172.21.15.19:0/2810295672 conn(0x7f04680c99b0 msgr2=0x7f0468150880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:27:38.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.781+0000 7f0467fff640 1 -- 172.21.15.19:0/2810295672 shutdown_connections 2024-08-31T12:27:38.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.781+0000 7f0467fff640 1 -- 172.21.15.19:0/2810295672 wait complete. 2024-08-31T12:27:38.799 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:27:38.799 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-08-31T12:27:38.799 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-08-31T12:27:38.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.972+0000 7f2306100640 1 -- 172.21.15.19:0/687299258 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f22f8008960 msgr2=0x7f22f8008d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:38.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.972+0000 7f2306100640 1 --2- 172.21.15.19:0/687299258 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f22f8008960 0x7f22f8008d70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f22f8051ea0 tx=0x7f22f402a460 comp rx=0 tx=0).stop 2024-08-31T12:27:38.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.973+0000 7f2306100640 1 -- 172.21.15.19:0/687299258 shutdown_connections 2024-08-31T12:27:38.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.973+0000 7f2306100640 1 --2- 172.21.15.19:0/687299258 >> v2:172.21.15.43:3300/0 conn(0x7f22f80def00 0x7f22f80df310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:38.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.973+0000 7f2306100640 1 --2- 172.21.15.19:0/687299258 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f22f8008960 0x7f22f8008d70 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:38.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.973+0000 7f2306100640 1 --2- 172.21.15.19:0/687299258 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f22f80d2580 0x7f22f8034b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:38.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.973+0000 7f2306100640 1 -- 172.21.15.19:0/687299258 >> 172.21.15.19:0/687299258 conn(0x7f22f8050f40 msgr2=0x7f22f8051350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:27:38.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.973+0000 7f2306100640 1 -- 172.21.15.19:0/687299258 shutdown_connections 2024-08-31T12:27:38.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.974+0000 7f2306100640 1 -- 172.21.15.19:0/687299258 wait complete. 2024-08-31T12:27:38.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.977+0000 7f2306100640 1 Processor -- start 2024-08-31T12:27:38.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.977+0000 7f2306100640 1 -- start start 2024-08-31T12:27:38.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.979+0000 7f2306100640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f22f8008960 0x7f22f8049680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:38.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.979+0000 7f2306100640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f22f80d2580 0x7f22f80d3150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:38.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.979+0000 7f2306100640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f22f80def00 0x7f22f804a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:38.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.979+0000 7f2306100640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f22f80dfe60 con 0x7f22f80def00 2024-08-31T12:27:38.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.979+0000 7f2306100640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f22f80084d0 con 0x7f22f8008960 2024-08-31T12:27:38.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.979+0000 7f2306100640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f22f80df850 con 0x7f22f80d2580 2024-08-31T12:27:38.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.979+0000 7f22ff7fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f22f8008960 0x7f22f8049680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:38.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.979+0000 7f22ff7fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f22f8008960 0x7f22f8049680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:35050/0 (socket says 172.21.15.19:35050) 2024-08-31T12:27:38.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.979+0000 7f22ff7fe640 1 -- 172.21.15.19:0/2891940911 learned_addr learned my addr 172.21.15.19:0/2891940911 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:27:38.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.979+0000 7f22feffd640 1 --2- 172.21.15.19:0/2891940911 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f22f80d2580 0x7f22f80d3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:38.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.980+0000 7f22fffff640 1 --2- 172.21.15.19:0/2891940911 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f22f80def00 0x7f22f804a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:38.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.980+0000 7f22fffff640 1 -- 172.21.15.19:0/2891940911 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f22f80d2580 msgr2=0x7f22f80d3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:38.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.980+0000 7f22fffff640 1 --2- 172.21.15.19:0/2891940911 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f22f80d2580 0x7f22f80d3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:38.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.980+0000 7f22fffff640 1 -- 172.21.15.19:0/2891940911 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f22f8008960 msgr2=0x7f22f8049680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:38.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.980+0000 7f22fffff640 1 --2- 172.21.15.19:0/2891940911 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f22f8008960 0x7f22f8049680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:38.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.980+0000 7f22fffff640 1 -- 172.21.15.19:0/2891940911 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22f8049e30 con 0x7f22f80def00 2024-08-31T12:27:38.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.980+0000 7f22feffd640 1 --2- 172.21.15.19:0/2891940911 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f22f80d2580 0x7f22f80d3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:27:38.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.980+0000 7f22fffff640 1 --2- 172.21.15.19:0/2891940911 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f22f80def00 0x7f22f804a6f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f230006beb0 tx=0x7f2300051400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:38.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.981+0000 7f22ebfff640 1 -- 172.21.15.19:0/2891940911 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2300099070 con 0x7f22f80def00 2024-08-31T12:27:38.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.981+0000 7f22ebfff640 1 -- 172.21.15.19:0/2891940911 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2300094720 con 0x7f22f80def00 2024-08-31T12:27:38.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.981+0000 7f22ebfff640 1 -- 172.21.15.19:0/2891940911 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2300094af0 con 0x7f22f80def00 2024-08-31T12:27:38.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.981+0000 7f2306100640 1 -- 172.21.15.19:0/2891940911 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22f8175fc0 con 0x7f22f80def00 2024-08-31T12:27:38.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.981+0000 7f2306100640 1 -- 172.21.15.19:0/2891940911 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22f80d29f0 con 0x7f22f80def00 2024-08-31T12:27:38.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.983+0000 7f22ebfff640 1 -- 172.21.15.19:0/2891940911 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f230009c070 con 0x7f22f80def00 2024-08-31T12:27:38.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.985+0000 7f22ebfff640 1 --2- 172.21.15.19:0/2891940911 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f22e4077a50 0x7f22e4079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:38.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.985+0000 7f22ebfff640 1 -- 172.21.15.19:0/2891940911 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f2300115ca0 con 0x7f22f80def00 2024-08-31T12:27:38.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.985+0000 7f22ff7fe640 1 --2- 172.21.15.19:0/2891940911 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f22e4077a50 0x7f22e4079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:38.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.985+0000 7f2306100640 1 -- 172.21.15.19:0/2891940911 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22f8048620 con 0x7f22f80def00 2024-08-31T12:27:38.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.985+0000 7f22ff7fe640 1 --2- 172.21.15.19:0/2891940911 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f22e4077a50 0x7f22e4079f60 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f22ec009b70 tx=0x7f22ec005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:38.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:38.993+0000 7f22ebfff640 1 -- 172.21.15.19:0/2891940911 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f23000de330 con 0x7f22f80def00 2024-08-31T12:27:39.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:39.445+0000 7f2306100640 1 -- 172.21.15.19:0/2891940911 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f22f804d410 con 0x7f22f80def00 2024-08-31T12:27:39.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:39.446+0000 7f22ebfff640 1 -- 172.21.15.19:0/2891940911 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f23000e2af0 con 0x7f22f80def00 2024-08-31T12:27:39.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:39.449+0000 7f2306100640 1 -- 172.21.15.19:0/2891940911 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f22e4077a50 msgr2=0x7f22e4079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:39.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:39.449+0000 7f2306100640 1 --2- 172.21.15.19:0/2891940911 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f22e4077a50 0x7f22e4079f60 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f22ec009b70 tx=0x7f22ec005f90 comp rx=0 tx=0).stop 2024-08-31T12:27:39.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:39.449+0000 7f2306100640 1 -- 172.21.15.19:0/2891940911 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f22f80def00 msgr2=0x7f22f804a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:39.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:39.449+0000 7f2306100640 1 --2- 172.21.15.19:0/2891940911 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f22f80def00 0x7f22f804a6f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f230006beb0 tx=0x7f2300051400 comp rx=0 tx=0).stop 2024-08-31T12:27:39.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:39.450+0000 7f2306100640 1 -- 172.21.15.19:0/2891940911 shutdown_connections 2024-08-31T12:27:39.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:39.450+0000 7f2306100640 1 --2- 172.21.15.19:0/2891940911 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f22e4077a50 0x7f22e4079f60 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:39.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:39.450+0000 7f2306100640 1 --2- 172.21.15.19:0/2891940911 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f22f80def00 0x7f22f804a6f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:39.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:39.450+0000 7f2306100640 1 --2- 172.21.15.19:0/2891940911 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f22f80d2580 0x7f22f80d3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:39.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:39.450+0000 7f2306100640 1 --2- 172.21.15.19:0/2891940911 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f22f8008960 0x7f22f8049680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:39.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:39.450+0000 7f2306100640 1 -- 172.21.15.19:0/2891940911 >> 172.21.15.19:0/2891940911 conn(0x7f22f8050f40 msgr2=0x7f22f80dd210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:27:39.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:39.450+0000 7f2306100640 1 -- 172.21.15.19:0/2891940911 shutdown_connections 2024-08-31T12:27:39.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:39.450+0000 7f2306100640 1 -- 172.21.15.19:0/2891940911 wait complete. 2024-08-31T12:27:39.470 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2024-08-31T12:27:49.471 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-08-31T12:27:49.471 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:27:49.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:49.645+0000 7f13111c1640 1 -- 172.21.15.19:0/250718881 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f130c157c20 msgr2=0x7f130c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:49.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:49.645+0000 7f13111c1640 1 --2- 172.21.15.19:0/250718881 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f130c157c20 0x7f130c06d0f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f130c0716b0 tx=0x7f12fc02a560 comp rx=0 tx=0).stop 2024-08-31T12:27:49.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:49.645+0000 7f13111c1640 1 -- 172.21.15.19:0/250718881 shutdown_connections 2024-08-31T12:27:49.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:49.645+0000 7f13111c1640 1 --2- 172.21.15.19:0/250718881 >> v2:172.21.15.43:3300/0 conn(0x7f130c150d50 0x7f130c151160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:49.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:49.645+0000 7f13111c1640 1 --2- 172.21.15.19:0/250718881 >> v2:172.21.15.19:3300/0 conn(0x7f130c070940 0x7f130c070d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:49.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:49.645+0000 7f13111c1640 1 --2- 172.21.15.19:0/250718881 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f130c157c20 0x7f130c06d0f0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:49.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:49.645+0000 7f13111c1640 1 -- 172.21.15.19:0/250718881 >> 172.21.15.19:0/250718881 conn(0x7f130c0cc770 msgr2=0x7f130c0ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:27:49.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:49.646+0000 7f13111c1640 1 -- 172.21.15.19:0/250718881 shutdown_connections 2024-08-31T12:27:49.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:49.646+0000 7f13111c1640 1 -- 172.21.15.19:0/250718881 wait complete. 2024-08-31T12:27:49.650 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:49.649+0000 7f13111c1640 1 Processor -- start 2024-08-31T12:27:49.650 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:49.649+0000 7f13111c1640 1 -- start start 2024-08-31T12:27:49.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:49.651+0000 7f13111c1640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f130c070940 0x7f130c149f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:49.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:49.651+0000 7f13111c1640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f130c150d50 0x7f130c146ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:49.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:49.651+0000 7f13111c1640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f130c157c20 0x7f130c147420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:49.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:49.651+0000 7f13111c1640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f130c070140 con 0x7f130c070940 2024-08-31T12:27:49.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:49.651+0000 7f13111c1640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f130c073db0 con 0x7f130c150d50 2024-08-31T12:27:49.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:49.651+0000 7f13111c1640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f130c070420 con 0x7f130c157c20 2024-08-31T12:27:49.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:49.651+0000 7f130a59c640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f130c150d50 0x7f130c146ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:49.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:49.651+0000 7f130a59c640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f130c150d50 0x7f130c146ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:51222/0 (socket says 172.21.15.19:51222) 2024-08-31T12:27:49.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:49.651+0000 7f130a59c640 1 -- 172.21.15.19:0/2893031027 learned_addr learned my addr 172.21.15.19:0/2893031027 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:27:49.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:49.652+0000 7f130ad9d640 1 --2- 172.21.15.19:0/2893031027 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f130c070940 0x7f130c149f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:49.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:49.652+0000 7f130ad9d640 1 -- 172.21.15.19:0/2893031027 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f130c157c20 msgr2=0x7f130c147420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:27:49.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:49.652+0000 7f130ad9d640 1 --2- 172.21.15.19:0/2893031027 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f130c157c20 0x7f130c147420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:49.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:49.652+0000 7f130ad9d640 1 -- 172.21.15.19:0/2893031027 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f130c150d50 msgr2=0x7f130c146ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:49.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:49.652+0000 7f130ad9d640 1 --2- 172.21.15.19:0/2893031027 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f130c150d50 0x7f130c146ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:49.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:49.652+0000 7f130ad9d640 1 -- 172.21.15.19:0/2893031027 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f130c0cc2f0 con 0x7f130c070940 2024-08-31T12:27:49.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:49.653+0000 7f130ad9d640 1 --2- 172.21.15.19:0/2893031027 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f130c070940 0x7f130c149f60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f12fc031000 tx=0x7f12fc033ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:49.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:49.653+0000 7f12f37fe640 1 -- 172.21.15.19:0/2893031027 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12fc031b50 con 0x7f130c070940 2024-08-31T12:27:49.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:49.653+0000 7f12f37fe640 1 -- 172.21.15.19:0/2893031027 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f12fc031e00 con 0x7f130c070940 2024-08-31T12:27:49.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:49.653+0000 7f12f37fe640 1 -- 172.21.15.19:0/2893031027 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12fc005280 con 0x7f130c070940 2024-08-31T12:27:49.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:49.653+0000 7f13111c1640 1 -- 172.21.15.19:0/2893031027 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f130c14ebd0 con 0x7f130c070940 2024-08-31T12:27:49.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:49.654+0000 7f13111c1640 1 -- 172.21.15.19:0/2893031027 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f130c2012a0 con 0x7f130c070940 2024-08-31T12:27:49.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:49.655+0000 7f13111c1640 1 -- 172.21.15.19:0/2893031027 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f130c200dd0 con 0x7f130c070940 2024-08-31T12:27:49.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:49.656+0000 7f12f37fe640 1 -- 172.21.15.19:0/2893031027 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f12fc005420 con 0x7f130c070940 2024-08-31T12:27:49.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:49.657+0000 7f12f37fe640 1 --2- 172.21.15.19:0/2893031027 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f12f8077ad0 0x7f12f8079fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:49.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:49.657+0000 7f12f37fe640 1 -- 172.21.15.19:0/2893031027 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f12fc0ba5f0 con 0x7f130c070940 2024-08-31T12:27:49.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:49.657+0000 7f130a59c640 1 --2- 172.21.15.19:0/2893031027 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f12f8077ad0 0x7f12f8079fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:49.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:49.658+0000 7f130a59c640 1 --2- 172.21.15.19:0/2893031027 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f12f8077ad0 0x7f12f8079fe0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f130c154d80 tx=0x7f1300009260 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:49.666 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:49.665+0000 7f12f37fe640 1 -- 172.21.15.19:0/2893031027 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f12fc03f030 con 0x7f130c070940 2024-08-31T12:27:50.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.130+0000 7f13111c1640 1 -- 172.21.15.19:0/2893031027 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f130c0733d0 con 0x7f130c070940 2024-08-31T12:27:50.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.131+0000 7f12f37fe640 1 -- 172.21.15.19:0/2893031027 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f12fc0833c0 con 0x7f130c070940 2024-08-31T12:27:50.132 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":92,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:27:50.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.134+0000 7f13111c1640 1 -- 172.21.15.19:0/2893031027 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f12f8077ad0 msgr2=0x7f12f8079fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:50.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.134+0000 7f13111c1640 1 --2- 172.21.15.19:0/2893031027 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f12f8077ad0 0x7f12f8079fe0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f130c154d80 tx=0x7f1300009260 comp rx=0 tx=0).stop 2024-08-31T12:27:50.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.134+0000 7f13111c1640 1 -- 172.21.15.19:0/2893031027 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f130c070940 msgr2=0x7f130c149f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:50.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.134+0000 7f13111c1640 1 --2- 172.21.15.19:0/2893031027 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f130c070940 0x7f130c149f60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f12fc031000 tx=0x7f12fc033ba0 comp rx=0 tx=0).stop 2024-08-31T12:27:50.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.135+0000 7f13111c1640 1 -- 172.21.15.19:0/2893031027 shutdown_connections 2024-08-31T12:27:50.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.135+0000 7f13111c1640 1 --2- 172.21.15.19:0/2893031027 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f12f8077ad0 0x7f12f8079fe0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:50.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.135+0000 7f13111c1640 1 --2- 172.21.15.19:0/2893031027 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f130c157c20 0x7f130c147420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:50.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.135+0000 7f13111c1640 1 --2- 172.21.15.19:0/2893031027 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f130c150d50 0x7f130c146ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:50.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.135+0000 7f13111c1640 1 --2- 172.21.15.19:0/2893031027 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f130c070940 0x7f130c149f60 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:50.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.135+0000 7f13111c1640 1 -- 172.21.15.19:0/2893031027 >> 172.21.15.19:0/2893031027 conn(0x7f130c0cc770 msgr2=0x7f130c0ccb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:27:50.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.135+0000 7f13111c1640 1 -- 172.21.15.19:0/2893031027 shutdown_connections 2024-08-31T12:27:50.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.135+0000 7f13111c1640 1 -- 172.21.15.19:0/2893031027 wait complete. 2024-08-31T12:27:50.152 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-08-31T12:27:50.152 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-08-31T12:27:50.152 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-08-31T12:27:50.152 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-31T12:27:50.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.328+0000 7fcd21e7e640 1 -- 172.21.15.19:0/1712400717 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcd040052a0 msgr2=0x7fcd0400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:50.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.328+0000 7fcd21e7e640 1 --2- 172.21.15.19:0/1712400717 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcd040052a0 0x7fcd0400b9c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fcd14001f30 tx=0x7fcd04005830 comp rx=0 tx=0).stop 2024-08-31T12:27:50.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.328+0000 7fcd21e7e640 1 -- 172.21.15.19:0/1712400717 shutdown_connections 2024-08-31T12:27:50.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.328+0000 7fcd21e7e640 1 --2- 172.21.15.19:0/1712400717 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcd040052a0 0x7fcd0400b9c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:50.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.328+0000 7fcd21e7e640 1 -- 172.21.15.19:0/1712400717 >> 172.21.15.19:0/1712400717 conn(0x7fcd1c0cc950 msgr2=0x7fcd1c0ccd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:27:50.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.328+0000 7fcd21e7e640 1 -- 172.21.15.19:0/1712400717 shutdown_connections 2024-08-31T12:27:50.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.328+0000 7fcd21e7e640 1 -- 172.21.15.19:0/1712400717 wait complete. 2024-08-31T12:27:50.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.332+0000 7fcd21e7e640 1 Processor -- start 2024-08-31T12:27:50.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.332+0000 7fcd21e7e640 1 -- start start 2024-08-31T12:27:50.336 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.334+0000 7fcd21e7e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcd040052a0 0x7fcd1c156bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:50.336 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.334+0000 7fcd21e7e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcd1c143080 0x7fcd1c1f06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:50.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.335+0000 7fcd1b7fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcd040052a0 0x7fcd1c156bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:50.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.335+0000 7fcd21e7e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd1c143490 0x7fcd1c1f0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:50.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.335+0000 7fcd1b7fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcd040052a0 0x7fcd1c156bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:41546/0 (socket says 172.21.15.19:41546) 2024-08-31T12:27:50.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.335+0000 7fcd1affd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcd1c143080 0x7fcd1c1f06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:50.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.335+0000 7fcd1affd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcd1c143080 0x7fcd1c1f06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.19:35832/0 (socket says 172.21.15.19:35832) 2024-08-31T12:27:50.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.335+0000 7fcd21e7e640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fcd1c0c65c0 con 0x7fcd040052a0 2024-08-31T12:27:50.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.335+0000 7fcd1b7fe640 1 -- 172.21.15.19:0/1027073115 learned_addr learned my addr 172.21.15.19:0/1027073115 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:27:50.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.335+0000 7fcd21e7e640 1 -- 172.21.15.19:0/1027073115 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fcd1c0c62e0 con 0x7fcd1c143490 2024-08-31T12:27:50.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.335+0000 7fcd1bfff640 1 --2- 172.21.15.19:0/1027073115 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd1c143490 0x7fcd1c1f0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:50.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.335+0000 7fcd21e7e640 1 -- 172.21.15.19:0/1027073115 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fcd1c0c4470 con 0x7fcd1c143080 2024-08-31T12:27:50.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.336+0000 7fcd1b7fe640 1 -- 172.21.15.19:0/1027073115 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcd1c143080 msgr2=0x7fcd1c1f06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:50.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.336+0000 7fcd1b7fe640 1 --2- 172.21.15.19:0/1027073115 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcd1c143080 0x7fcd1c1f06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:50.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.336+0000 7fcd1b7fe640 1 -- 172.21.15.19:0/1027073115 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd1c143490 msgr2=0x7fcd1c1f0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:50.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.336+0000 7fcd1b7fe640 1 --2- 172.21.15.19:0/1027073115 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd1c143490 0x7fcd1c1f0c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:50.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.336+0000 7fcd1b7fe640 1 -- 172.21.15.19:0/1027073115 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd1c1f14b0 con 0x7fcd040052a0 2024-08-31T12:27:50.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.336+0000 7fcd1affd640 1 --2- 172.21.15.19:0/1027073115 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcd1c143080 0x7fcd1c1f06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:27:50.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.336+0000 7fcd1b7fe640 1 --2- 172.21.15.19:0/1027073115 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcd040052a0 0x7fcd1c156bd0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c007330 tx=0x7fcd0c00f840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:50.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.336+0000 7fcd03fff640 1 -- 172.21.15.19:0/1027073115 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd0c007630 con 0x7fcd040052a0 2024-08-31T12:27:50.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.336+0000 7fcd03fff640 1 -- 172.21.15.19:0/1027073115 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcd0c019420 con 0x7fcd040052a0 2024-08-31T12:27:50.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.336+0000 7fcd21e7e640 1 -- 172.21.15.19:0/1027073115 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd1c157110 con 0x7fcd040052a0 2024-08-31T12:27:50.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.336+0000 7fcd03fff640 1 -- 172.21.15.19:0/1027073115 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd0c019830 con 0x7fcd040052a0 2024-08-31T12:27:50.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.337+0000 7fcd21e7e640 1 -- 172.21.15.19:0/1027073115 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd1c155290 con 0x7fcd040052a0 2024-08-31T12:27:50.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.338+0000 7fcd21e7e640 1 --2- 172.21.15.19:0/1027073115 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcd1c153fc0 0x7fcd1c1543d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:50.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.338+0000 7fcd21e7e640 1 -- 172.21.15.19:0/1027073115 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fcd1c152980 con 0x7fcd1c153fc0 2024-08-31T12:27:50.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.338+0000 7fcd1affd640 1 --2- 172.21.15.19:0/1027073115 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcd1c153fc0 0x7fcd1c1543d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:50.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.338+0000 7fcd1affd640 1 -- 172.21.15.19:0/1027073115 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcd1c151d90 con 0x7fcd1c153fc0 2024-08-31T12:27:50.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.339+0000 7fcd03fff640 1 -- 172.21.15.19:0/1027073115 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fcd0c0199d0 con 0x7fcd040052a0 2024-08-31T12:27:50.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.339+0000 7fcd1affd640 1 --2- 172.21.15.19:0/1027073115 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcd1c153fc0 0x7fcd1c1543d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c155cc0 tx=0x7fcd10011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:50.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.340+0000 7fcd03fff640 1 --2- 172.21.15.19:0/1027073115 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fcd08077a50 0x7fcd08079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:50.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.340+0000 7fcd03fff640 1 -- 172.21.15.19:0/1027073115 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fcd0c09b7e0 con 0x7fcd040052a0 2024-08-31T12:27:50.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.340+0000 7fcd03fff640 1 -- 172.21.15.19:0/1027073115 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd10004260 con 0x7fcd1c153fc0 2024-08-31T12:27:50.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.341+0000 7fcd1bfff640 1 --2- 172.21.15.19:0/1027073115 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fcd08077a50 0x7fcd08079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:50.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.341+0000 7fcd03fff640 1 -- 172.21.15.19:0/1027073115 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcd10004260 con 0x7fcd1c153fc0 2024-08-31T12:27:50.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.341+0000 7fcd03fff640 1 -- 172.21.15.19:0/1027073115 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcd1c153fc0 msgr2=0x7fcd1c1543d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:50.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.341+0000 7fcd03fff640 1 --2- 172.21.15.19:0/1027073115 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcd1c153fc0 0x7fcd1c1543d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c155cc0 tx=0x7fcd10011810 comp rx=0 tx=0).stop 2024-08-31T12:27:50.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.341+0000 7fcd03fff640 1 -- 172.21.15.19:0/1027073115 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcd1c153fc0 msgr2=0x7fcd1c1543d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:27:50.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.341+0000 7fcd03fff640 1 --2- 172.21.15.19:0/1027073115 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcd1c153fc0 0x7fcd1c1543d0 secure :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c155cc0 tx=0x7fcd10011810 comp rx=0 tx=0).stop 2024-08-31T12:27:50.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.342+0000 7fcd1bfff640 1 --2- 172.21.15.19:0/1027073115 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fcd08077a50 0x7fcd08079f60 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fcd0400c160 tx=0x7fcd14006040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:50.354 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.353+0000 7fcd21e7e640 1 --2- 172.21.15.19:0/1027073115 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcd1c155460 0x7fcd1c152b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:50.354 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.353+0000 7fcd21e7e640 1 -- 172.21.15.19:0/1027073115 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fcd1c14e050 con 0x7fcd1c155460 2024-08-31T12:27:50.354 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.353+0000 7fcd1affd640 1 --2- 172.21.15.19:0/1027073115 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcd1c155460 0x7fcd1c152b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:50.355 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.354+0000 7fcd1affd640 1 -- 172.21.15.19:0/1027073115 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcd1c0c7a10 con 0x7fcd1c155460 2024-08-31T12:27:50.355 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.354+0000 7fcd1affd640 1 --2- 172.21.15.19:0/1027073115 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcd1c155460 0x7fcd1c152b30 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fcd1001b000 tx=0x7fcd10004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:50.356 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.355+0000 7fcd03fff640 1 -- 172.21.15.19:0/1027073115 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd10003a80 con 0x7fcd1c155460 2024-08-31T12:27:50.356 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.355+0000 7fcd03fff640 1 -- 172.21.15.19:0/1027073115 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5045 (secure 0 0 0) 0x7fcd10003c20 con 0x7fcd1c155460 2024-08-31T12:27:50.356 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.355+0000 7fcd03fff640 1 -- 172.21.15.19:0/1027073115 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcd1c155460 msgr2=0x7fcd1c152b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:50.356 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.355+0000 7fcd03fff640 1 --2- 172.21.15.19:0/1027073115 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcd1c155460 0x7fcd1c152b30 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fcd1001b000 tx=0x7fcd10004730 comp rx=0 tx=0).stop 2024-08-31T12:27:50.356 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.355+0000 7fcd03fff640 1 -- 172.21.15.19:0/1027073115 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcd1c155460 msgr2=0x7fcd1c152b30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:27:50.356 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.355+0000 7fcd03fff640 1 --2- 172.21.15.19:0/1027073115 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcd1c155460 0x7fcd1c152b30 secure :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fcd1001b000 tx=0x7fcd10004730 comp rx=0 tx=0).stop 2024-08-31T12:27:50.356 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:27:50.356 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:27:50.356 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:27:50.357 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "leader", 2024-08-31T12:27:50.357 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 92, 2024-08-31T12:27:50.357 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 204909, 2024-08-31T12:27:50.357 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:27:50.357 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:27:50.357 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:27:50.357 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:27:50.357 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:50.357 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 14, 2024-08-31T12:27:50.357 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:27:50.358 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:27:50.358 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:27:50.358 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:27:50.358 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:27:50.358 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:27:50.358 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:27:50.358 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:27:50.358 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:27:50.358 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:27:50.358 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:27:50.358 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:27:50.358 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:27:50.359 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:27:50.359 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:50.359 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:27:50.359 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:27:50.359 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:27:50.359 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:27:50.359 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:27:50.359 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:27:50.359 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:27:50.359 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:27:50.359 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:27:50.359 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:27:50.359 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:27:50.359 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:27:50.360 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:27:50.360 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:50.360 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:50.360 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:27:50.360 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:27:50.360 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:27:50.360 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:27:50.360 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:27:50.360 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:27:50.360 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:27:50.360 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:27:50.360 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:27:50.360 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:27:50.361 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:27:50.361 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:27:50.361 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:27:50.361 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:27:50.361 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:27:50.361 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:27:50.361 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:27:50.361 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:27:50.361 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:27:50.361 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:27:50.361 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:27:50.361 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:27:50.361 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:27:50.361 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:27:50.362 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:27:50.362 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:27:50.362 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:27:50.362 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:27:50.362 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:50.362 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:27:50.362 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:50.362 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:27:50.362 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:50.362 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:27:50.362 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:27:50.362 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:27:50.363 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:27:50.363 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:50.363 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:27:50.363 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:27:50.363 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:50.363 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:50.363 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:50.363 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:27:50.363 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:27:50.363 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:50.363 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:50.363 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:50.363 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:50.363 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:27:50.364 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:27:50.364 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:27:50.364 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:27:50.364 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:27:50.364 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:50.364 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:50.364 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:27:50.364 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:27:50.364 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:27:50.364 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:27:50.364 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:50.364 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:27:50.364 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:27:50.364 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:50.365 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:50.365 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:50.365 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:27:50.365 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:27:50.365 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:50.365 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:50.365 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:50.365 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:50.365 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:27:50.365 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:27:50.365 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:27:50.365 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:27:50.365 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:27:50.365 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:50.366 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:50.366 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:27:50.366 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:27:50.366 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:27:50.366 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:27:50.366 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:50.366 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:27:50.366 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:27:50.366 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:50.366 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:50.366 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:50.366 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:27:50.366 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:27:50.367 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:50.367 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:50.367 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:50.367 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:50.367 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:27:50.367 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:27:50.367 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:27:50.367 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:27:50.367 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:27:50.367 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:50.367 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:50.368 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:50.368 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:27:50.368 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:27:50.368 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:50.368 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:27:50.368 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:27:50.368 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:27:50.368 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:50.368 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:50.368 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:27:50.368 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:50.368 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:27:50.369 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:27:50.369 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 5 2024-08-31T12:27:50.369 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:50.369 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:50.369 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:27:50.369 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:50.369 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:27:50.369 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:27:50.369 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 7 2024-08-31T12:27:50.369 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:50.369 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:50.370 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:27:50.370 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:50.370 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:27:50.370 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:27:50.370 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 5 2024-08-31T12:27:50.370 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:50.370 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:50.370 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:27:50.370 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:50.370 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:27:50.370 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:27:50.370 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:27:50.370 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:50.370 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:50.371 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:50.371 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:27:50.371 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:27:50.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.355+0000 7fcd21e7e640 1 -- 172.21.15.19:0/1027073115 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fcd08077a50 msgr2=0x7fcd08079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:50.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.355+0000 7fcd21e7e640 1 --2- 172.21.15.19:0/1027073115 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fcd08077a50 0x7fcd08079f60 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fcd0400c160 tx=0x7fcd14006040 comp rx=0 tx=0).stop 2024-08-31T12:27:50.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.355+0000 7fcd1bfff640 1 -- 172.21.15.19:0/1027073115 reap_dead start 2024-08-31T12:27:50.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.355+0000 7fcd21e7e640 1 -- 172.21.15.19:0/1027073115 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcd040052a0 msgr2=0x7fcd1c156bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:50.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.355+0000 7fcd21e7e640 1 --2- 172.21.15.19:0/1027073115 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcd040052a0 0x7fcd1c156bd0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c007330 tx=0x7fcd0c00f840 comp rx=0 tx=0).stop 2024-08-31T12:27:50.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.356+0000 7fcd21e7e640 1 -- 172.21.15.19:0/1027073115 shutdown_connections 2024-08-31T12:27:50.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.356+0000 7fcd21e7e640 1 --2- 172.21.15.19:0/1027073115 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcd040052a0 0x7fcd1c156bd0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:50.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.356+0000 7fcd21e7e640 1 -- 172.21.15.19:0/1027073115 >> 172.21.15.19:0/1027073115 conn(0x7fcd1c0cc950 msgr2=0x7fcd1c1462f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:27:50.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.356+0000 7fcd21e7e640 1 -- 172.21.15.19:0/1027073115 shutdown_connections 2024-08-31T12:27:50.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.356+0000 7fcd21e7e640 1 -- 172.21.15.19:0/1027073115 wait complete. 2024-08-31T12:27:50.374 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:27:50.374 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-08-31T12:27:50.374 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-31T12:27:50.551 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.550+0000 7f2745d9b640 1 -- 172.21.15.19:0/3341698522 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2740036400 con 0x7f274815b0a0 2024-08-31T12:27:50.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.551+0000 7f274d07f640 1 -- 172.21.15.19:0/3341698522 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f274815b0a0 msgr2=0x7f274806d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:50.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.551+0000 7f274d07f640 1 --2- 172.21.15.19:0/3341698522 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f274815b0a0 0x7f274806d8e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2740005fd0 tx=0x7f274002aaa0 comp rx=0 tx=0).stop 2024-08-31T12:27:50.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.552+0000 7f274d07f640 1 -- 172.21.15.19:0/3341698522 shutdown_connections 2024-08-31T12:27:50.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.552+0000 7f274d07f640 1 --2- 172.21.15.19:0/3341698522 >> v2:172.21.15.19:3300/0 conn(0x7f274806eb00 0x7f274806de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:50.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.552+0000 7f274d07f640 1 --2- 172.21.15.19:0/3341698522 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f274815b0a0 0x7f274806d8e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:50.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.552+0000 7f274d07f640 1 -- 172.21.15.19:0/3341698522 >> 172.21.15.19:0/3341698522 conn(0x7f2748069260 msgr2=0x7f27480d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:27:50.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.552+0000 7f274d07f640 1 -- 172.21.15.19:0/3341698522 shutdown_connections 2024-08-31T12:27:50.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.552+0000 7f274d07f640 1 -- 172.21.15.19:0/3341698522 wait complete. 2024-08-31T12:27:50.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.554+0000 7f274d07f640 1 Processor -- start 2024-08-31T12:27:50.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.554+0000 7f274d07f640 1 -- start start 2024-08-31T12:27:50.557 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.556+0000 7f274d07f640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f274806eb00 0x7f27481f1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:50.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.556+0000 7f274d07f640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f27480cc820 0x7f27480c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:50.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.556+0000 7f274d07f640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f274815b0a0 0x7f274815e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:50.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.556+0000 7f274d07f640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f27480ccf90 con 0x7f27480cc820 2024-08-31T12:27:50.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.556+0000 7f274d07f640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f27480b4a40 con 0x7f274806eb00 2024-08-31T12:27:50.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.556+0000 7f274d07f640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f27480b5670 con 0x7f274815b0a0 2024-08-31T12:27:50.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.556+0000 7f2746d9d640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f274806eb00 0x7f27481f1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:50.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.556+0000 7f2746d9d640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f274806eb00 0x7f27481f1a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:51246/0 (socket says 172.21.15.19:51246) 2024-08-31T12:27:50.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.556+0000 7f2746d9d640 1 -- 172.21.15.19:0/129700506 learned_addr learned my addr 172.21.15.19:0/129700506 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:27:50.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.556+0000 7f274759e640 1 --2- 172.21.15.19:0/129700506 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f274815b0a0 0x7f274815e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:50.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.556+0000 7f274659c640 1 --2- 172.21.15.19:0/129700506 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f27480cc820 0x7f27480c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:50.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.557+0000 7f2746d9d640 1 -- 172.21.15.19:0/129700506 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f274815b0a0 msgr2=0x7f274815e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:50.559 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.557+0000 7f2746d9d640 1 --2- 172.21.15.19:0/129700506 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f274815b0a0 0x7f274815e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:50.559 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.557+0000 7f2746d9d640 1 -- 172.21.15.19:0/129700506 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f27480cc820 msgr2=0x7f27480c5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:50.559 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.557+0000 7f2746d9d640 1 --2- 172.21.15.19:0/129700506 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f27480cc820 0x7f27480c5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:50.559 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.557+0000 7f2746d9d640 1 -- 172.21.15.19:0/129700506 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27481f1f50 con 0x7f274806eb00 2024-08-31T12:27:50.559 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.557+0000 7f274659c640 1 --2- 172.21.15.19:0/129700506 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f27480cc820 0x7f27480c5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:27:50.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.557+0000 7f2746d9d640 1 --2- 172.21.15.19:0/129700506 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f274806eb00 0x7f27481f1a10 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f273800c490 tx=0x7f273800c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:50.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.557+0000 7f272f7fe640 1 -- 172.21.15.19:0/129700506 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2738017070 con 0x7f274806eb00 2024-08-31T12:27:50.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.557+0000 7f272f7fe640 1 -- 172.21.15.19:0/129700506 <== mon.1 v2:172.21.15.43:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f27380053e0 con 0x7f274806eb00 2024-08-31T12:27:50.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.557+0000 7f272f7fe640 1 -- 172.21.15.19:0/129700506 <== mon.1 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2738003850 con 0x7f274806eb00 2024-08-31T12:27:50.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.557+0000 7f274d07f640 1 -- 172.21.15.19:0/129700506 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f274815ed70 con 0x7f274806eb00 2024-08-31T12:27:50.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.557+0000 7f274d07f640 1 -- 172.21.15.19:0/129700506 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f274815f330 con 0x7f274806eb00 2024-08-31T12:27:50.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.559+0000 7f274d07f640 1 --2- 172.21.15.19:0/129700506 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2748152b50 0x7f2748152f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:50.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.559+0000 7f274d07f640 1 -- 172.21.15.19:0/129700506 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f27481f2120 con 0x7f2748152b50 2024-08-31T12:27:50.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.559+0000 7f274659c640 1 --2- 172.21.15.19:0/129700506 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2748152b50 0x7f2748152f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:50.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.559+0000 7f274659c640 1 -- 172.21.15.19:0/129700506 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f27480ca860 con 0x7f2748152b50 2024-08-31T12:27:50.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.560+0000 7f274659c640 1 --2- 172.21.15.19:0/129700506 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2748152b50 0x7f2748152f60 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f27480d10c0 tx=0x7f274002e890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:50.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.560+0000 7f272f7fe640 1 -- 172.21.15.19:0/129700506 <== mon.1 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f273801a070 con 0x7f274806eb00 2024-08-31T12:27:50.562 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.561+0000 7f272f7fe640 1 --2- 172.21.15.19:0/129700506 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2734077a80 0x7f2734079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:50.562 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.561+0000 7f272f7fe640 1 -- 172.21.15.19:0/129700506 <== mon.1 v2:172.21.15.43:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f2738098800 con 0x7f274806eb00 2024-08-31T12:27:50.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.561+0000 7f272f7fe640 1 -- 172.21.15.19:0/129700506 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f274002ed00 con 0x7f2748152b50 2024-08-31T12:27:50.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.562+0000 7f272f7fe640 1 -- 172.21.15.19:0/129700506 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2740004040 con 0x7f2748152b50 2024-08-31T12:27:50.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.562+0000 7f272f7fe640 1 -- 172.21.15.19:0/129700506 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2748152b50 msgr2=0x7f2748152f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:50.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.562+0000 7f272f7fe640 1 --2- 172.21.15.19:0/129700506 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2748152b50 0x7f2748152f60 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f27480d10c0 tx=0x7f274002e890 comp rx=0 tx=0).stop 2024-08-31T12:27:50.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.562+0000 7f272f7fe640 1 -- 172.21.15.19:0/129700506 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2748152b50 msgr2=0x7f2748152f60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:27:50.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.562+0000 7f272f7fe640 1 --2- 172.21.15.19:0/129700506 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2748152b50 0x7f2748152f60 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:50.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.562+0000 7f274759e640 1 --2- 172.21.15.19:0/129700506 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2734077a80 0x7f2734079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:50.564 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.563+0000 7f274759e640 1 --2- 172.21.15.19:0/129700506 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2734077a80 0x7f2734079f90 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f273c009630 tx=0x7f273c005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:50.571 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.570+0000 7f274d07f640 1 --2- 172.21.15.19:0/129700506 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27481f4780 0x7f27481f4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:50.572 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.570+0000 7f274d07f640 1 -- 172.21.15.19:0/129700506 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f27480b39a0 con 0x7f27481f4780 2024-08-31T12:27:50.572 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.571+0000 7f274659c640 1 --2- 172.21.15.19:0/129700506 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27481f4780 0x7f27481f4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:50.573 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.572+0000 7f274659c640 1 -- 172.21.15.19:0/129700506 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f27480cc2e0 con 0x7f27481f4780 2024-08-31T12:27:50.574 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.573+0000 7f274659c640 1 --2- 172.21.15.19:0/129700506 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27481f4780 0x7f27481f4b90 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2740009bf0 tx=0x7f27400095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:50.574 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.573+0000 7f272f7fe640 1 -- 172.21.15.19:0/129700506 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27400300d0 con 0x7f27481f4780 2024-08-31T12:27:50.574 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.573+0000 7f272f7fe640 1 -- 172.21.15.19:0/129700506 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4541 (secure 0 0 0) 0x7f2740030270 con 0x7f27481f4780 2024-08-31T12:27:50.574 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.573+0000 7f272f7fe640 1 -- 172.21.15.19:0/129700506 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27481f4780 msgr2=0x7f27481f4b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:50.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.573+0000 7f272f7fe640 1 --2- 172.21.15.19:0/129700506 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27481f4780 0x7f27481f4b90 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2740009bf0 tx=0x7f27400095c0 comp rx=0 tx=0).stop 2024-08-31T12:27:50.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.573+0000 7f272f7fe640 1 -- 172.21.15.19:0/129700506 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27481f4780 msgr2=0x7f27481f4b90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:27:50.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.573+0000 7f272f7fe640 1 --2- 172.21.15.19:0/129700506 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27481f4780 0x7f27481f4b90 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2740009bf0 tx=0x7f27400095c0 comp rx=0 tx=0).stop 2024-08-31T12:27:50.575 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:27:50.575 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:27:50.576 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:27:50.576 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:27:50.576 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 92, 2024-08-31T12:27:50.576 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 20394, 2024-08-31T12:27:50.576 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:27:50.576 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:27:50.576 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:27:50.576 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:27:50.576 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:50.576 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 15, 2024-08-31T12:27:50.576 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:27:50.576 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:27:50.576 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:27:50.577 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:27:50.577 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:27:50.577 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:27:50.577 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:27:50.577 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:27:50.577 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:27:50.577 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:27:50.577 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:27:50.577 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:27:50.577 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:27:50.577 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:27:50.577 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:50.577 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:27:50.578 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:27:50.578 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:27:50.578 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:27:50.578 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:27:50.578 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:27:50.578 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:27:50.578 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:27:50.578 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:27:50.578 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:27:50.578 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:27:50.578 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:27:50.578 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:27:50.578 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:50.578 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:50.579 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:27:50.579 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:27:50.579 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:27:50.579 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:27:50.579 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:27:50.579 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:27:50.579 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:27:50.579 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:27:50.579 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:27:50.579 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:27:50.579 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:27:50.579 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:27:50.580 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:27:50.580 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:27:50.580 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:27:50.580 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:27:50.580 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:27:50.580 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:27:50.580 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:27:50.580 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:27:50.580 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:27:50.580 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:27:50.580 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:27:50.580 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:27:50.580 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:27:50.580 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:27:50.580 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:27:50.581 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:27:50.581 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:50.581 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:27:50.581 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:50.581 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:27:50.581 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:50.581 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:27:50.581 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:27:50.581 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:27:50.581 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:27:50.581 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:50.581 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:27:50.581 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:27:50.582 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:50.582 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:50.582 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:50.582 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:27:50.582 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:27:50.582 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:50.582 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:50.582 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:50.582 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:50.582 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:27:50.582 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:27:50.582 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:27:50.582 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:27:50.582 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:27:50.583 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:50.583 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:50.583 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:27:50.583 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:27:50.583 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:27:50.583 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:27:50.583 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:50.583 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:27:50.583 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:27:50.583 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:50.583 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:50.584 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:50.584 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:27:50.584 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:27:50.584 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:50.584 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:50.584 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:50.584 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:50.584 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:27:50.584 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:27:50.584 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:27:50.584 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:27:50.584 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:27:50.585 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:50.585 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:50.585 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:27:50.585 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:27:50.585 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:27:50.585 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:27:50.585 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:50.585 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:27:50.585 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:27:50.585 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:50.585 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:50.585 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:50.585 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:27:50.585 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:27:50.586 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:50.586 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:50.586 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:50.586 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:50.586 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:27:50.586 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:27:50.586 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:27:50.586 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:27:50.586 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:27:50.586 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:50.586 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:50.586 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:50.587 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:27:50.587 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:27:50.587 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:50.587 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:27:50.587 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:27:50.587 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:27:50.587 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:50.587 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:50.587 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:27:50.587 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:50.587 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:27:50.587 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:27:50.587 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:27:50.587 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:50.587 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:50.588 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:50.588 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:27:50.588 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:27:50.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.575+0000 7f274d07f640 1 -- 172.21.15.19:0/129700506 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2734077a80 msgr2=0x7f2734079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:50.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.575+0000 7f274d07f640 1 --2- 172.21.15.19:0/129700506 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2734077a80 0x7f2734079f90 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f273c009630 tx=0x7f273c005f90 comp rx=0 tx=0).stop 2024-08-31T12:27:50.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.575+0000 7f274d07f640 1 -- 172.21.15.19:0/129700506 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f274806eb00 msgr2=0x7f27481f1a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:50.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.575+0000 7f274d07f640 1 --2- 172.21.15.19:0/129700506 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f274806eb00 0x7f27481f1a10 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f273800c490 tx=0x7f273800c890 comp rx=0 tx=0).stop 2024-08-31T12:27:50.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.575+0000 7f274759e640 1 -- 172.21.15.19:0/129700506 reap_dead start 2024-08-31T12:27:50.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.575+0000 7f274759e640 1 -- 172.21.15.19:0/129700506 reap_dead start 2024-08-31T12:27:50.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.575+0000 7f274d07f640 1 -- 172.21.15.19:0/129700506 shutdown_connections 2024-08-31T12:27:50.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.575+0000 7f274d07f640 1 -- 172.21.15.19:0/129700506 >> 172.21.15.19:0/129700506 conn(0x7f2748069260 msgr2=0x7f2748154d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:27:50.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.575+0000 7f274d07f640 1 -- 172.21.15.19:0/129700506 shutdown_connections 2024-08-31T12:27:50.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.575+0000 7f274d07f640 1 -- 172.21.15.19:0/129700506 wait complete. 2024-08-31T12:27:50.592 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:27:50.592 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-08-31T12:27:50.592 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-31T12:27:50.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.775+0000 7f863d11a640 1 -- 172.21.15.19:0/727078569 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8638140c10 msgr2=0x7f86380c7d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:50.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.775+0000 7f863d11a640 1 --2- 172.21.15.19:0/727078569 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8638140c10 0x7f86380c7d60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8628009830 tx=0x7f862802a510 comp rx=0 tx=0).stop 2024-08-31T12:27:50.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.776+0000 7f863d11a640 1 -- 172.21.15.19:0/727078569 shutdown_connections 2024-08-31T12:27:50.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.776+0000 7f863d11a640 1 --2- 172.21.15.19:0/727078569 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8638140c10 0x7f86380c7d60 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:50.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.776+0000 7f863d11a640 1 -- 172.21.15.19:0/727078569 >> 172.21.15.19:0/727078569 conn(0x7f86380cd7d0 msgr2=0x7f86380cdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:27:50.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.776+0000 7f863d11a640 1 -- 172.21.15.19:0/727078569 shutdown_connections 2024-08-31T12:27:50.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.776+0000 7f863d11a640 1 -- 172.21.15.19:0/727078569 wait complete. 2024-08-31T12:27:50.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.779+0000 7f863d11a640 1 Processor -- start 2024-08-31T12:27:50.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.779+0000 7f863d11a640 1 -- start start 2024-08-31T12:27:50.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.780+0000 7f863d11a640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f86380719b0 0x7f86381ee910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:50.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.780+0000 7f863d11a640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8638140c10 0x7f8638154690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:50.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.781+0000 7f863d11a640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f863815bd10 0x7f86381531d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:50.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.781+0000 7f863d11a640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f86380c6920 con 0x7f86380719b0 2024-08-31T12:27:50.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.781+0000 7f863d11a640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f86380c6c20 con 0x7f863815bd10 2024-08-31T12:27:50.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.781+0000 7f863d11a640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f863815b2b0 con 0x7f8638140c10 2024-08-31T12:27:50.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.781+0000 7f863659c640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8638140c10 0x7f8638154690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:50.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.781+0000 7f8636d9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f86380719b0 0x7f86381ee910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:50.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.781+0000 7f8636d9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f86380719b0 0x7f86381ee910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:41614/0 (socket says 172.21.15.19:41614) 2024-08-31T12:27:50.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.781+0000 7f8636d9d640 1 -- 172.21.15.19:0/265284525 learned_addr learned my addr 172.21.15.19:0/265284525 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:27:50.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.781+0000 7f863759e640 1 --2- 172.21.15.19:0/265284525 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f863815bd10 0x7f86381531d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:50.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.781+0000 7f8636d9d640 1 -- 172.21.15.19:0/265284525 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8638140c10 msgr2=0x7f8638154690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:50.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.781+0000 7f8636d9d640 1 --2- 172.21.15.19:0/265284525 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8638140c10 0x7f8638154690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:50.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.781+0000 7f8636d9d640 1 -- 172.21.15.19:0/265284525 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f863815bd10 msgr2=0x7f86381531d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:50.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.781+0000 7f8636d9d640 1 --2- 172.21.15.19:0/265284525 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f863815bd10 0x7f86381531d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:50.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.781+0000 7f8636d9d640 1 -- 172.21.15.19:0/265284525 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86381552d0 con 0x7f86380719b0 2024-08-31T12:27:50.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.782+0000 7f8636d9d640 1 --2- 172.21.15.19:0/265284525 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f86380719b0 0x7f86381ee910 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8628009780 tx=0x7f8628033f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:50.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.782+0000 7f861f7fe640 1 -- 172.21.15.19:0/265284525 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8628031d60 con 0x7f86380719b0 2024-08-31T12:27:50.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.782+0000 7f863d11a640 1 -- 172.21.15.19:0/265284525 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86380ca7e0 con 0x7f86380719b0 2024-08-31T12:27:50.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.782+0000 7f861f7fe640 1 -- 172.21.15.19:0/265284525 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f86280330d0 con 0x7f86380719b0 2024-08-31T12:27:50.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.782+0000 7f861f7fe640 1 -- 172.21.15.19:0/265284525 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8628033520 con 0x7f86380719b0 2024-08-31T12:27:50.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.782+0000 7f863d11a640 1 -- 172.21.15.19:0/265284525 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86381ee110 con 0x7f86380719b0 2024-08-31T12:27:50.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.784+0000 7f863d11a640 1 --2- 172.21.15.19:0/265284525 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f863815a0c0 0x7f863813db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:50.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.784+0000 7f863d11a640 1 -- 172.21.15.19:0/265284525 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f86380ca590 con 0x7f863815a0c0 2024-08-31T12:27:50.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.784+0000 7f861f7fe640 1 -- 172.21.15.19:0/265284525 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f86280036f0 con 0x7f86380719b0 2024-08-31T12:27:50.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.785+0000 7f861f7fe640 1 --2- 172.21.15.19:0/265284525 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8624077a80 0x7f8624079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:50.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.785+0000 7f861f7fe640 1 -- 172.21.15.19:0/265284525 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f86280bac40 con 0x7f86380719b0 2024-08-31T12:27:50.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.786+0000 7f863759e640 1 --2- 172.21.15.19:0/265284525 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8624077a80 0x7f8624079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:50.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.786+0000 7f863659c640 1 --2- 172.21.15.19:0/265284525 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f863815a0c0 0x7f863813db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:50.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.786+0000 7f863659c640 1 -- 172.21.15.19:0/265284525 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8638155050 con 0x7f863815a0c0 2024-08-31T12:27:50.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.787+0000 7f863659c640 1 --2- 172.21.15.19:0/265284525 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f863815a0c0 0x7f863813db80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f86380c8880 tx=0x7f862c00ec00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:50.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.787+0000 7f863759e640 1 --2- 172.21.15.19:0/265284525 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8624077a80 0x7f8624079f90 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f8630004a30 tx=0x7f86300019f0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:50.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.787+0000 7f861f7fe640 1 -- 172.21.15.19:0/265284525 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f862c017070 con 0x7f863815a0c0 2024-08-31T12:27:50.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.787+0000 7f861f7fe640 1 -- 172.21.15.19:0/265284525 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f862c00c770 con 0x7f863815a0c0 2024-08-31T12:27:50.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.787+0000 7f861f7fe640 1 -- 172.21.15.19:0/265284525 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f863815a0c0 msgr2=0x7f863813db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:50.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.787+0000 7f861f7fe640 1 --2- 172.21.15.19:0/265284525 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f863815a0c0 0x7f863813db80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f86380c8880 tx=0x7f862c00ec00 comp rx=0 tx=0).stop 2024-08-31T12:27:50.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.788+0000 7f861f7fe640 1 -- 172.21.15.19:0/265284525 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f863815a0c0 msgr2=0x7f863813db80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:27:50.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.788+0000 7f861f7fe640 1 --2- 172.21.15.19:0/265284525 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f863815a0c0 0x7f863813db80 secure :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f86380c8880 tx=0x7f862c00ec00 comp rx=0 tx=0).stop 2024-08-31T12:27:50.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.803+0000 7f863d11a640 1 --2- 172.21.15.19:0/265284525 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f863813e0c0 0x7f86381f13f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:50.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.803+0000 7f863d11a640 1 -- 172.21.15.19:0/265284525 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f86381f6770 con 0x7f863813e0c0 2024-08-31T12:27:50.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.804+0000 7f863659c640 1 --2- 172.21.15.19:0/265284525 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f863813e0c0 0x7f86381f13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:50.805 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.804+0000 7f863659c640 1 -- 172.21.15.19:0/265284525 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8638155730 con 0x7f863813e0c0 2024-08-31T12:27:50.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.804+0000 7f863659c640 1 --2- 172.21.15.19:0/265284525 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f863813e0c0 0x7f86381f13f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f862c019000 tx=0x7f862c0046b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:50.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.805+0000 7f861f7fe640 1 -- 172.21.15.19:0/265284525 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f862c017070 con 0x7f863813e0c0 2024-08-31T12:27:50.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.807+0000 7f861f7fe640 1 -- 172.21.15.19:0/265284525 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4875 (secure 0 0 0) 0x7f862c006250 con 0x7f863813e0c0 2024-08-31T12:27:50.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.807+0000 7f861f7fe640 1 -- 172.21.15.19:0/265284525 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f863813e0c0 msgr2=0x7f86381f13f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:50.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.807+0000 7f861f7fe640 1 --2- 172.21.15.19:0/265284525 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f863813e0c0 0x7f86381f13f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f862c019000 tx=0x7f862c0046b0 comp rx=0 tx=0).stop 2024-08-31T12:27:50.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.807+0000 7f861f7fe640 1 -- 172.21.15.19:0/265284525 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f863813e0c0 msgr2=0x7f86381f13f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:27:50.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.807+0000 7f861f7fe640 1 --2- 172.21.15.19:0/265284525 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f863813e0c0 0x7f86381f13f0 secure :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f862c019000 tx=0x7f862c0046b0 comp rx=0 tx=0).stop 2024-08-31T12:27:50.808 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:27:50.809 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:27:50.809 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:27:50.809 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:27:50.809 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 92, 2024-08-31T12:27:50.809 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 76673, 2024-08-31T12:27:50.809 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:27:50.809 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:27:50.809 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:27:50.809 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:27:50.809 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:50.809 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 15, 2024-08-31T12:27:50.809 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:27:50.809 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:27:50.809 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:27:50.810 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:27:50.810 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:27:50.810 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:27:50.810 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:27:50.810 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:27:50.810 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:27:50.810 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:27:50.810 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:27:50.810 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:27:50.810 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:27:50.810 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:27:50.810 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:50.810 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:27:50.810 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:27:50.811 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:27:50.811 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:27:50.811 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:27:50.811 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:27:50.811 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:27:50.812 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:27:50.812 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:27:50.812 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:27:50.812 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:27:50.812 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:27:50.812 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:27:50.812 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:50.812 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:50.812 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:27:50.812 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:27:50.812 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:27:50.813 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:27:50.813 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:27:50.813 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:27:50.813 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:27:50.813 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:27:50.813 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:27:50.813 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:27:50.813 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:27:50.813 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:27:50.813 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:27:50.813 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:27:50.813 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:27:50.813 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:27:50.813 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:27:50.814 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:27:50.814 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:27:50.814 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:27:50.814 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:27:50.814 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:27:50.814 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:27:50.814 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:27:50.814 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:27:50.814 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:27:50.814 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:27:50.814 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:27:50.814 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:50.815 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:27:50.815 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:50.815 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:27:50.815 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:50.815 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:27:50.815 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:27:50.815 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:27:50.815 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:27:50.815 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:50.815 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:27:50.815 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:27:50.815 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:50.815 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:50.816 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:50.816 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:27:50.816 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:27:50.816 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:50.816 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:50.816 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:50.816 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:50.817 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:27:50.817 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:27:50.817 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:27:50.817 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:27:50.817 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:27:50.817 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:50.817 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:50.818 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:27:50.818 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:27:50.818 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:27:50.818 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:27:50.818 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:50.818 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:27:50.818 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:27:50.819 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:50.819 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:50.819 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:50.819 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:27:50.819 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:27:50.819 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:50.820 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:50.820 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:50.820 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:50.820 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:27:50.820 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:27:50.820 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:27:50.821 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:27:50.821 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:27:50.821 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:50.821 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:50.821 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:27:50.822 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:27:50.822 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:27:50.822 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:27:50.822 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:50.822 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:27:50.822 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:27:50.823 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:50.823 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:50.823 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:50.823 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:27:50.823 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:27:50.823 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:27:50.824 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:50.824 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:50.824 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:50.824 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:27:50.824 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:27:50.824 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:27:50.825 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:27:50.825 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:27:50.825 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:50.825 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:50.825 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:50.825 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:27:50.826 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:27:50.826 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:50.826 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:27:50.826 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:27:50.826 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:27:50.826 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:50.826 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:50.826 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:27:50.826 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:50.826 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:27:50.826 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:27:50.826 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:27:50.826 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:50.826 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:50.827 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:27:50.827 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:50.827 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:27:50.827 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:27:50.827 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:27:50.827 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:50.827 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:27:50.827 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:27:50.827 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:27:50.827 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:27:50.827 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:27:50.827 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 3 2024-08-31T12:27:50.827 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:27:50.827 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:27:50.827 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:27:50.828 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:27:50.828 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:27:50.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.808+0000 7f863d11a640 1 -- 172.21.15.19:0/265284525 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8624077a80 msgr2=0x7f8624079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:50.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.808+0000 7f863d11a640 1 --2- 172.21.15.19:0/265284525 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8624077a80 0x7f8624079f90 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f8630004a30 tx=0x7f86300019f0 comp rx=0 tx=0).stop 2024-08-31T12:27:50.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.808+0000 7f863759e640 1 -- 172.21.15.19:0/265284525 reap_dead start 2024-08-31T12:27:50.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.809+0000 7f863d11a640 1 -- 172.21.15.19:0/265284525 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f86380719b0 msgr2=0x7f86381ee910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:50.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.809+0000 7f863d11a640 1 --2- 172.21.15.19:0/265284525 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f86380719b0 0x7f86381ee910 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8628009780 tx=0x7f8628033f90 comp rx=0 tx=0).stop 2024-08-31T12:27:50.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.810+0000 7f863d11a640 1 -- 172.21.15.19:0/265284525 shutdown_connections 2024-08-31T12:27:50.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.810+0000 7f863d11a640 1 --2- 172.21.15.19:0/265284525 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f86380719b0 0x7f86381ee910 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:50.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.810+0000 7f863d11a640 1 -- 172.21.15.19:0/265284525 >> 172.21.15.19:0/265284525 conn(0x7f86380cd7d0 msgr2=0x7f863813fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:27:50.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.810+0000 7f863d11a640 1 -- 172.21.15.19:0/265284525 shutdown_connections 2024-08-31T12:27:50.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:50.810+0000 7f863d11a640 1 -- 172.21.15.19:0/265284525 wait complete. 2024-08-31T12:27:50.830 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:27:50.831 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-08-31T12:27:50.831 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-08-31T12:27:50.831 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-08-31T12:27:50.831 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-08-31T12:27:50.831 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-08-31T12:27:50.831 INFO:teuthology.orchestra.run:waiting for 300 2024-08-31T12:27:50.846 INFO:tasks.ceph.mon.c:Stopped 2024-08-31T12:27:50.846 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-08-31T12:27:50.846 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:27:51.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.015+0000 7f3b15dcc640 1 -- 172.21.15.19:0/2780293604 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3b100c9ed0 msgr2=0x7f3b100c8f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:51.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.015+0000 7f3b15dcc640 1 --2- 172.21.15.19:0/2780293604 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3b100c9ed0 0x7f3b100c8f40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3b04005fd0 tx=0x7f3b0402a560 comp rx=0 tx=0).stop 2024-08-31T12:27:51.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.015+0000 7f3b15dcc640 1 -- 172.21.15.19:0/2780293604 shutdown_connections 2024-08-31T12:27:51.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.015+0000 7f3b15dcc640 1 --2- 172.21.15.19:0/2780293604 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3b100c9ed0 0x7f3b100c8f40 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:51.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.015+0000 7f3b15dcc640 1 --2- 172.21.15.19:0/2780293604 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3b100cefe0 0x7f3b1015ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:51.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.015+0000 7f3b15dcc640 1 -- 172.21.15.19:0/2780293604 >> 172.21.15.19:0/2780293604 conn(0x7f3b10068ec0 msgr2=0x7f3b100692d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:27:51.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.015+0000 7f3b15dcc640 1 -- 172.21.15.19:0/2780293604 shutdown_connections 2024-08-31T12:27:51.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.015+0000 7f3b15dcc640 1 -- 172.21.15.19:0/2780293604 wait complete. 2024-08-31T12:27:51.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.018+0000 7f3b15dcc640 1 Processor -- start 2024-08-31T12:27:51.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.018+0000 7f3b15dcc640 1 -- start start 2024-08-31T12:27:51.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.020+0000 7f3b15dcc640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3b100c9ed0 0x7f3b1014e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:51.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.021+0000 7f3b15dcc640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3b100cefe0 0x7f3b10149fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:51.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.021+0000 7f3b15dcc640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b10152cd0 0x7f3b101f6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:51.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.021+0000 7f3b0f7fe640 1 -- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3b100c9ed0 msgr2=0x7f3b1014e820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3301/0 2024-08-31T12:27:51.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.021+0000 7f3b0f7fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3b100c9ed0 0x7f3b1014e820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:27:51.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.021+0000 7f3b0ffff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b10152cd0 0x7f3b101f6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:51.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.021+0000 7f3b0effd640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3b100cefe0 0x7f3b10149fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:51.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.021+0000 7f3b0effd640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3b100cefe0 0x7f3b10149fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:51286/0 (socket says 172.21.15.19:51286) 2024-08-31T12:27:51.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.021+0000 7f3b0ffff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b10152cd0 0x7f3b101f6a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:41642/0 (socket says 172.21.15.19:41642) 2024-08-31T12:27:51.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.021+0000 7f3b15dcc640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f3b1015c970 con 0x7f3b10152cd0 2024-08-31T12:27:51.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.021+0000 7f3b15dcc640 1 -- 172.21.15.19:0/291094802 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3b1015c690 con 0x7f3b100cefe0 2024-08-31T12:27:51.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.021+0000 7f3b0effd640 1 -- 172.21.15.19:0/291094802 learned_addr learned my addr 172.21.15.19:0/291094802 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:27:51.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.021+0000 7f3b15dcc640 1 -- 172.21.15.19:0/291094802 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f3b101530e0 con 0x7f3b100c9ed0 2024-08-31T12:27:51.024 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.022+0000 7f3b0ffff640 1 -- 172.21.15.19:0/291094802 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3b100c9ed0 msgr2=0x7f3b1014e820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:27:51.024 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.022+0000 7f3b0ffff640 1 --2- 172.21.15.19:0/291094802 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3b100c9ed0 0x7f3b1014e820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:51.024 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.022+0000 7f3b0ffff640 1 -- 172.21.15.19:0/291094802 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3b100cefe0 msgr2=0x7f3b10149fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:51.024 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.022+0000 7f3b0ffff640 1 --2- 172.21.15.19:0/291094802 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3b100cefe0 0x7f3b10149fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:51.024 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.022+0000 7f3b0ffff640 1 -- 172.21.15.19:0/291094802 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b1014b380 con 0x7f3b10152cd0 2024-08-31T12:27:51.024 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.022+0000 7f3b0effd640 1 --2- 172.21.15.19:0/291094802 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3b100cefe0 0x7f3b10149fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:27:51.025 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.022+0000 7f3b0ffff640 1 --2- 172.21.15.19:0/291094802 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b10152cd0 0x7f3b101f6a00 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3b0800c730 tx=0x7f3b0800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:51.025 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.022+0000 7f3af7fff640 1 -- 172.21.15.19:0/291094802 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b08003b70 con 0x7f3b10152cd0 2024-08-31T12:27:51.025 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.022+0000 7f3af7fff640 1 -- 172.21.15.19:0/291094802 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3b08003d10 con 0x7f3b10152cd0 2024-08-31T12:27:51.025 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.022+0000 7f3b15dcc640 1 -- 172.21.15.19:0/291094802 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b10149510 con 0x7f3b10152cd0 2024-08-31T12:27:51.025 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.022+0000 7f3af7fff640 1 -- 172.21.15.19:0/291094802 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b08005330 con 0x7f3b10152cd0 2024-08-31T12:27:51.025 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.022+0000 7f3b15dcc640 1 -- 172.21.15.19:0/291094802 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b101497b0 con 0x7f3b10152cd0 2024-08-31T12:27:51.026 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.024+0000 7f3b15dcc640 1 -- 172.21.15.19:0/291094802 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b100b1520 con 0x7f3b10152cd0 2024-08-31T12:27:51.026 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.025+0000 7f3af7fff640 1 -- 172.21.15.19:0/291094802 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f3b080054d0 con 0x7f3b10152cd0 2024-08-31T12:27:51.026 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.025+0000 7f3af7fff640 1 --2- 172.21.15.19:0/291094802 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3afc077a00 0x7f3afc079f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:51.027 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.026+0000 7f3af7fff640 1 -- 172.21.15.19:0/291094802 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f3b080993f0 con 0x7f3b10152cd0 2024-08-31T12:27:51.027 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.026+0000 7f3b0f7fe640 1 --2- 172.21.15.19:0/291094802 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3afc077a00 0x7f3afc079f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:51.028 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.027+0000 7f3b0f7fe640 1 --2- 172.21.15.19:0/291094802 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3afc077a00 0x7f3afc079f10 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f3b00003850 tx=0x7f3b00006ca0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:51.036 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.035+0000 7f3af7fff640 1 -- 172.21.15.19:0/291094802 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f3b08062040 con 0x7f3b10152cd0 2024-08-31T12:27:51.496 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.495+0000 7f3b15dcc640 1 -- 172.21.15.19:0/291094802 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3b10072b20 con 0x7f3b10152cd0 2024-08-31T12:27:51.496 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.495+0000 7f3af7fff640 1 -- 172.21.15.19:0/291094802 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f3b08061940 con 0x7f3b10152cd0 2024-08-31T12:27:51.496 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":92,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:27:51.499 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.498+0000 7f3b15dcc640 1 -- 172.21.15.19:0/291094802 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3afc077a00 msgr2=0x7f3afc079f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:51.499 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.498+0000 7f3b15dcc640 1 --2- 172.21.15.19:0/291094802 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3afc077a00 0x7f3afc079f10 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f3b00003850 tx=0x7f3b00006ca0 comp rx=0 tx=0).stop 2024-08-31T12:27:51.500 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.498+0000 7f3b15dcc640 1 -- 172.21.15.19:0/291094802 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b10152cd0 msgr2=0x7f3b101f6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:51.500 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.498+0000 7f3b15dcc640 1 --2- 172.21.15.19:0/291094802 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b10152cd0 0x7f3b101f6a00 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3b0800c730 tx=0x7f3b0800cbf0 comp rx=0 tx=0).stop 2024-08-31T12:27:51.500 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.498+0000 7f3b15dcc640 1 -- 172.21.15.19:0/291094802 shutdown_connections 2024-08-31T12:27:51.500 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.498+0000 7f3b15dcc640 1 --2- 172.21.15.19:0/291094802 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3afc077a00 0x7f3afc079f10 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:51.500 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.498+0000 7f3b15dcc640 1 --2- 172.21.15.19:0/291094802 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b10152cd0 0x7f3b101f6a00 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:51.500 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.498+0000 7f3b15dcc640 1 --2- 172.21.15.19:0/291094802 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3b100cefe0 0x7f3b10149fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:51.500 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.498+0000 7f3b15dcc640 1 --2- 172.21.15.19:0/291094802 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3b100c9ed0 0x7f3b1014e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:51.500 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.498+0000 7f3b15dcc640 1 -- 172.21.15.19:0/291094802 >> 172.21.15.19:0/291094802 conn(0x7f3b10068ec0 msgr2=0x7f3b10073040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:27:51.500 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.498+0000 7f3b15dcc640 1 -- 172.21.15.19:0/291094802 shutdown_connections 2024-08-31T12:27:51.500 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:51.499+0000 7f3b15dcc640 1 -- 172.21.15.19:0/291094802 wait complete. 2024-08-31T12:27:54.516 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:27:54.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:54.696+0000 7efc52e3f640 1 -- 172.21.15.19:0/582226697 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc4c070d50 msgr2=0x7efc4c161b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:54.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:54.696+0000 7efc52e3f640 1 --2- 172.21.15.19:0/582226697 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc4c070d50 0x7efc4c161b20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7efc40009830 tx=0x7efc4002a5c0 comp rx=0 tx=0).stop 2024-08-31T12:27:54.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:54.696+0000 7efc52e3f640 1 -- 172.21.15.19:0/582226697 shutdown_connections 2024-08-31T12:27:54.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:54.696+0000 7efc52e3f640 1 --2- 172.21.15.19:0/582226697 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc4c070d50 0x7efc4c161b20 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:54.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:54.696+0000 7efc52e3f640 1 --2- 172.21.15.19:0/582226697 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efc4c157c20 0x7efc4c06d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:54.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:54.696+0000 7efc52e3f640 1 -- 172.21.15.19:0/582226697 >> 172.21.15.19:0/582226697 conn(0x7efc4c0cc770 msgr2=0x7efc4c0ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:27:54.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:54.696+0000 7efc52e3f640 1 -- 172.21.15.19:0/582226697 shutdown_connections 2024-08-31T12:27:54.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:54.696+0000 7efc52e3f640 1 -- 172.21.15.19:0/582226697 wait complete. 2024-08-31T12:27:54.699 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:54.698+0000 7efc52e3f640 1 Processor -- start 2024-08-31T12:27:54.699 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:54.698+0000 7efc52e3f640 1 -- start start 2024-08-31T12:27:54.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:54.700+0000 7efc52e3f640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efc4c070940 0x7efc4c152d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:54.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:54.700+0000 7efc52e3f640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efc4c070d50 0x7efc4c14e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:54.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:54.700+0000 7efc52e3f640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc4c157c20 0x7efc4c1515b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:54.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:54.700+0000 7efc52e3f640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7efc4c070140 con 0x7efc4c070940 2024-08-31T12:27:54.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:54.700+0000 7efc4bfff640 1 -- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efc4c070d50 msgr2=0x7efc4c14e0c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3301/0 2024-08-31T12:27:54.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:54.700+0000 7efc52e3f640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7efc4c073db0 con 0x7efc4c157c20 2024-08-31T12:27:54.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:54.700+0000 7efc52e3f640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7efc4c070420 con 0x7efc4c070d50 2024-08-31T12:27:54.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:54.700+0000 7efc4bfff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efc4c070d50 0x7efc4c14e0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:27:54.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:54.700+0000 7efc50bdb640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efc4c070940 0x7efc4c152d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:54.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:54.700+0000 7efc513dc640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc4c157c20 0x7efc4c1515b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:54.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:54.700+0000 7efc50bdb640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efc4c070940 0x7efc4c152d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:41654/0 (socket says 172.21.15.19:41654) 2024-08-31T12:27:54.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:54.700+0000 7efc50bdb640 1 -- 172.21.15.19:0/3576798179 learned_addr learned my addr 172.21.15.19:0/3576798179 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:27:54.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:54.701+0000 7efc50bdb640 1 -- 172.21.15.19:0/3576798179 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efc4c070d50 msgr2=0x7efc4c14e0c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:27:54.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:54.701+0000 7efc50bdb640 1 --2- 172.21.15.19:0/3576798179 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efc4c070d50 0x7efc4c14e0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:54.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:54.701+0000 7efc50bdb640 1 -- 172.21.15.19:0/3576798179 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc4c157c20 msgr2=0x7efc4c1515b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:54.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:54.701+0000 7efc50bdb640 1 --2- 172.21.15.19:0/3576798179 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc4c157c20 0x7efc4c1515b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:54.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:54.701+0000 7efc50bdb640 1 -- 172.21.15.19:0/3576798179 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc4c154930 con 0x7efc4c070940 2024-08-31T12:27:54.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:54.701+0000 7efc50bdb640 1 --2- 172.21.15.19:0/3576798179 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efc4c070940 0x7efc4c152d00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7efc3c00c9b0 tx=0x7efc3c00ce70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:54.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:54.701+0000 7efc497fa640 1 -- 172.21.15.19:0/3576798179 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc3c0089e0 con 0x7efc4c070940 2024-08-31T12:27:54.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:54.701+0000 7efc497fa640 1 -- 172.21.15.19:0/3576798179 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efc3c008c90 con 0x7efc4c070940 2024-08-31T12:27:54.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:54.701+0000 7efc497fa640 1 -- 172.21.15.19:0/3576798179 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc3c00f460 con 0x7efc4c070940 2024-08-31T12:27:54.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:54.701+0000 7efc52e3f640 1 -- 172.21.15.19:0/3576798179 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc4c152540 con 0x7efc4c070940 2024-08-31T12:27:54.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:54.701+0000 7efc52e3f640 1 -- 172.21.15.19:0/3576798179 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc4c1527e0 con 0x7efc4c070940 2024-08-31T12:27:54.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:54.704+0000 7efc52e3f640 1 -- 172.21.15.19:0/3576798179 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc4c071380 con 0x7efc4c070940 2024-08-31T12:27:54.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:54.705+0000 7efc497fa640 1 -- 172.21.15.19:0/3576798179 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7efc3c00f600 con 0x7efc4c070940 2024-08-31T12:27:54.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:54.706+0000 7efc497fa640 1 --2- 172.21.15.19:0/3576798179 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7efc38077a80 0x7efc38079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:54.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:54.706+0000 7efc497fa640 1 -- 172.21.15.19:0/3576798179 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7efc3c099c80 con 0x7efc4c070940 2024-08-31T12:27:54.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:54.706+0000 7efc4bfff640 1 --2- 172.21.15.19:0/3576798179 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7efc38077a80 0x7efc38079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:54.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:54.707+0000 7efc4bfff640 1 --2- 172.21.15.19:0/3576798179 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7efc38077a80 0x7efc38079f90 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7efc4c154f00 tx=0x7efc44001700 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:54.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:54.713+0000 7efc497fa640 1 -- 172.21.15.19:0/3576798179 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7efc4c071380 con 0x7efc4c070940 2024-08-31T12:27:55.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:55.192+0000 7efc52e3f640 1 -- 172.21.15.19:0/3576798179 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7efc4c0c3c90 con 0x7efc4c070940 2024-08-31T12:27:55.194 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:55.193+0000 7efc497fa640 1 -- 172.21.15.19:0/3576798179 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7efc4c0c3c90 con 0x7efc4c070940 2024-08-31T12:27:55.194 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":92,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":19,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:27:55.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:55.196+0000 7efc52e3f640 1 -- 172.21.15.19:0/3576798179 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7efc38077a80 msgr2=0x7efc38079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:55.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:55.196+0000 7efc52e3f640 1 --2- 172.21.15.19:0/3576798179 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7efc38077a80 0x7efc38079f90 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7efc4c154f00 tx=0x7efc44001700 comp rx=0 tx=0).stop 2024-08-31T12:27:55.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:55.196+0000 7efc52e3f640 1 -- 172.21.15.19:0/3576798179 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efc4c070940 msgr2=0x7efc4c152d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:55.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:55.196+0000 7efc52e3f640 1 --2- 172.21.15.19:0/3576798179 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efc4c070940 0x7efc4c152d00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7efc3c00c9b0 tx=0x7efc3c00ce70 comp rx=0 tx=0).stop 2024-08-31T12:27:55.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:55.197+0000 7efc52e3f640 1 -- 172.21.15.19:0/3576798179 shutdown_connections 2024-08-31T12:27:55.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:55.197+0000 7efc52e3f640 1 --2- 172.21.15.19:0/3576798179 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7efc38077a80 0x7efc38079f90 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:55.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:55.197+0000 7efc52e3f640 1 --2- 172.21.15.19:0/3576798179 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc4c157c20 0x7efc4c1515b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:55.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:55.197+0000 7efc52e3f640 1 --2- 172.21.15.19:0/3576798179 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efc4c070d50 0x7efc4c14e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:55.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:55.197+0000 7efc52e3f640 1 --2- 172.21.15.19:0/3576798179 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efc4c070940 0x7efc4c152d00 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:55.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:55.197+0000 7efc52e3f640 1 -- 172.21.15.19:0/3576798179 >> 172.21.15.19:0/3576798179 conn(0x7efc4c0cc770 msgr2=0x7efc4c0ca360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:27:55.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:55.197+0000 7efc52e3f640 1 -- 172.21.15.19:0/3576798179 shutdown_connections 2024-08-31T12:27:55.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:55.197+0000 7efc52e3f640 1 -- 172.21.15.19:0/3576798179 wait complete. 2024-08-31T12:27:58.216 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:27:58.232 DEBUG:teuthology.orchestra.run.smithi019:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:27:58.266 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:27:58.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.394+0000 7f9fef7fe640 1 -- 172.21.15.19:0/1793360508 <== mon.1 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe00047a0 con 0x7f9ff0151c50 2024-08-31T12:27:58.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.394+0000 7f9ff6b62640 1 -- 172.21.15.19:0/1793360508 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ff0151c50 msgr2=0x7f9ff00c7ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:58.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.394+0000 7f9ff6b62640 1 --2- 172.21.15.19:0/1793360508 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ff0151c50 0x7f9ff00c7ac0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9ff0072b10 tx=0x7f9fe002a610 comp rx=0 tx=0).stop 2024-08-31T12:27:58.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.394+0000 7f9ff6b62640 1 -- 172.21.15.19:0/1793360508 shutdown_connections 2024-08-31T12:27:58.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.394+0000 7f9ff6b62640 1 --2- 172.21.15.19:0/1793360508 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ff0151c50 0x7f9ff00c7ac0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:58.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.394+0000 7f9ff6b62640 1 --2- 172.21.15.19:0/1793360508 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9ff0151840 0x7f9ff006ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:58.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.394+0000 7f9ff6b62640 1 --2- 172.21.15.19:0/1793360508 >> v2:172.21.15.19:3300/0 conn(0x7f9ff00c3a60 0x7f9ff00c6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:58.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.394+0000 7f9ff6b62640 1 -- 172.21.15.19:0/1793360508 >> 172.21.15.19:0/1793360508 conn(0x7f9ff00cc770 msgr2=0x7f9ff00ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:27:58.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.394+0000 7f9ff6b62640 1 -- 172.21.15.19:0/1793360508 shutdown_connections 2024-08-31T12:27:58.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.394+0000 7f9ff6b62640 1 -- 172.21.15.19:0/1793360508 wait complete. 2024-08-31T12:27:58.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.397+0000 7f9ff6b62640 1 Processor -- start 2024-08-31T12:27:58.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.397+0000 7f9ff6b62640 1 -- start start 2024-08-31T12:27:58.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.399+0000 7f9ff6b62640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ff00c3a60 0x7f9ff0072a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:58.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.399+0000 7f9ff6b62640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9ff0151840 0x7f9ff00cb780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:58.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.399+0000 7f9ff6b62640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9ff0151c50 0x7f9ff0144c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:58.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.399+0000 7f9ff6b62640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff0070b20 con 0x7f9ff0151c50 2024-08-31T12:27:58.401 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.399+0000 7f9ff6b62640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff0070840 con 0x7f9ff00c3a60 2024-08-31T12:27:58.401 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.399+0000 7f9ff6b62640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f9ff0071400 con 0x7f9ff0151840 2024-08-31T12:27:58.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.399+0000 7f9feffff640 1 -- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9ff0151840 msgr2=0x7f9ff00cb780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3301/0 2024-08-31T12:27:58.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.400+0000 7f9ff50ff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9ff0151c50 0x7f9ff0144c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:58.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.400+0000 7f9ff50ff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9ff0151c50 0x7f9ff0144c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:35070/0 (socket says 172.21.15.19:35070) 2024-08-31T12:27:58.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.400+0000 7f9ff50ff640 1 -- 172.21.15.19:0/3588025965 learned_addr learned my addr 172.21.15.19:0/3588025965 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:27:58.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.400+0000 7f9feffff640 1 --2- 172.21.15.19:0/3588025965 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9ff0151840 0x7f9ff00cb780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:27:58.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.400+0000 7f9ff48fe640 1 --2- 172.21.15.19:0/3588025965 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ff00c3a60 0x7f9ff0072a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:58.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.400+0000 7f9ff50ff640 1 -- 172.21.15.19:0/3588025965 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9ff0151840 msgr2=0x7f9ff00cb780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:27:58.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.400+0000 7f9ff50ff640 1 --2- 172.21.15.19:0/3588025965 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9ff0151840 0x7f9ff00cb780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:58.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.400+0000 7f9ff50ff640 1 -- 172.21.15.19:0/3588025965 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ff00c3a60 msgr2=0x7f9ff0072a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:58.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.400+0000 7f9ff50ff640 1 --2- 172.21.15.19:0/3588025965 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ff00c3a60 0x7f9ff0072a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:58.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.400+0000 7f9ff50ff640 1 -- 172.21.15.19:0/3588025965 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ff0162a50 con 0x7f9ff0151c50 2024-08-31T12:27:58.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.400+0000 7f9ff50ff640 1 --2- 172.21.15.19:0/3588025965 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9ff0151c50 0x7f9ff0144c90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9fe0009230 tx=0x7f9fe0003960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:58.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.401+0000 7f9fed7fa640 1 -- 172.21.15.19:0/3588025965 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe003e030 con 0x7f9ff0151c50 2024-08-31T12:27:58.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.401+0000 7f9ff6b62640 1 -- 172.21.15.19:0/3588025965 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ff01f6180 con 0x7f9ff0151c50 2024-08-31T12:27:58.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.401+0000 7f9ff6b62640 1 -- 172.21.15.19:0/3588025965 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ff0147820 con 0x7f9ff0151c50 2024-08-31T12:27:58.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.401+0000 7f9fed7fa640 1 -- 172.21.15.19:0/3588025965 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9fe0003dd0 con 0x7f9ff0151c50 2024-08-31T12:27:58.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.401+0000 7f9fed7fa640 1 -- 172.21.15.19:0/3588025965 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe0035040 con 0x7f9ff0151c50 2024-08-31T12:27:58.404 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.403+0000 7f9ff6b62640 1 -- 172.21.15.19:0/3588025965 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ff01f63c0 con 0x7f9ff0151c50 2024-08-31T12:27:58.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.404+0000 7f9fed7fa640 1 -- 172.21.15.19:0/3588025965 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f9fe0041030 con 0x7f9ff0151c50 2024-08-31T12:27:58.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.405+0000 7f9fed7fa640 1 --2- 172.21.15.19:0/3588025965 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9fdc077a00 0x7f9fdc079f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:27:58.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.405+0000 7f9fed7fa640 1 -- 172.21.15.19:0/3588025965 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f9fe00b9db0 con 0x7f9ff0151c50 2024-08-31T12:27:58.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.405+0000 7f9ff48fe640 1 --2- 172.21.15.19:0/3588025965 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9fdc077a00 0x7f9fdc079f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:27:58.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.406+0000 7f9ff48fe640 1 --2- 172.21.15.19:0/3588025965 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9fdc077a00 0x7f9fdc079f10 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f9fe400bb50 tx=0x7f9fe400b2e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:27:58.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.412+0000 7f9fed7fa640 1 -- 172.21.15.19:0/3588025965 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f9ff01f63c0 con 0x7f9ff0151c50 2024-08-31T12:27:58.876 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.875+0000 7f9ff6b62640 1 -- 172.21.15.19:0/3588025965 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9ff01f5f70 con 0x7f9ff0151c50 2024-08-31T12:27:58.877 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.876+0000 7f9fed7fa640 1 -- 172.21.15.19:0/3588025965 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f9fe00823c0 con 0x7f9ff0151c50 2024-08-31T12:27:58.877 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":92,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":23,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:27:58.880 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.879+0000 7f9ff6b62640 1 -- 172.21.15.19:0/3588025965 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9fdc077a00 msgr2=0x7f9fdc079f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:58.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.879+0000 7f9ff6b62640 1 --2- 172.21.15.19:0/3588025965 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9fdc077a00 0x7f9fdc079f10 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f9fe400bb50 tx=0x7f9fe400b2e0 comp rx=0 tx=0).stop 2024-08-31T12:27:58.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.879+0000 7f9ff6b62640 1 -- 172.21.15.19:0/3588025965 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9ff0151c50 msgr2=0x7f9ff0144c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:27:58.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.879+0000 7f9ff6b62640 1 --2- 172.21.15.19:0/3588025965 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9ff0151c50 0x7f9ff0144c90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9fe0009230 tx=0x7f9fe0003960 comp rx=0 tx=0).stop 2024-08-31T12:27:58.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.879+0000 7f9ff6b62640 1 -- 172.21.15.19:0/3588025965 shutdown_connections 2024-08-31T12:27:58.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.879+0000 7f9ff6b62640 1 --2- 172.21.15.19:0/3588025965 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9fdc077a00 0x7f9fdc079f10 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:58.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.879+0000 7f9ff6b62640 1 --2- 172.21.15.19:0/3588025965 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9ff0151c50 0x7f9ff0144c90 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:58.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.879+0000 7f9ff6b62640 1 --2- 172.21.15.19:0/3588025965 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9ff0151840 0x7f9ff00cb780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:58.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.879+0000 7f9ff6b62640 1 --2- 172.21.15.19:0/3588025965 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ff00c3a60 0x7f9ff0072a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:27:58.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.879+0000 7f9ff6b62640 1 -- 172.21.15.19:0/3588025965 >> 172.21.15.19:0/3588025965 conn(0x7f9ff00cc770 msgr2=0x7f9ff014f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:27:58.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.880+0000 7f9ff6b62640 1 -- 172.21.15.19:0/3588025965 shutdown_connections 2024-08-31T12:27:58.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:27:58.880+0000 7f9ff6b62640 1 -- 172.21.15.19:0/3588025965 wait complete. 2024-08-31T12:28:01.900 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:28:06.478 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.477+0000 7fea98779640 1 -- 172.21.15.19:0/2282367433 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fea74006290 msgr2=0x7fea740087a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:06.478 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.477+0000 7fea98779640 1 --2- 172.21.15.19:0/2282367433 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fea74006290 0x7fea740087a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fea900c7e30 tx=0x7fea8002c410 comp rx=0 tx=0).stop 2024-08-31T12:28:06.478 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.477+0000 7fea96d16640 1 -- 172.21.15.19:0/2282367433 reap_dead start 2024-08-31T12:28:06.478 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.477+0000 7fea98779640 1 -- 172.21.15.19:0/2282367433 shutdown_connections 2024-08-31T12:28:06.479 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.477+0000 7fea98779640 1 --2- 172.21.15.19:0/2282367433 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fea74006290 0x7fea740087a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:06.479 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.477+0000 7fea98779640 1 -- 172.21.15.19:0/2282367433 >> 172.21.15.19:0/2282367433 conn(0x7fea900c99a0 msgr2=0x7fea900c9db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:06.479 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.477+0000 7fea98779640 1 -- 172.21.15.19:0/2282367433 shutdown_connections 2024-08-31T12:28:06.479 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.477+0000 7fea98779640 1 -- 172.21.15.19:0/2282367433 wait complete. 2024-08-31T12:28:06.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.480+0000 7fea98779640 1 Processor -- start 2024-08-31T12:28:06.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.480+0000 7fea98779640 1 -- start start 2024-08-31T12:28:06.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.482+0000 7fea98779640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fea90156440 0x7fea901557a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:06.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.483+0000 7fea98779640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fea9013d9f0 0x7fea9013de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:06.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.483+0000 7fea98779640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fea9014ee00 0x7fea9014f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:06.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.483+0000 7fea95d14640 1 -- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fea9013d9f0 msgr2=0x7fea9013de00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3301/0 2024-08-31T12:28:06.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.483+0000 7fea95d14640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fea9013d9f0 0x7fea9013de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:28:06.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.483+0000 7fea98779640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fea900714b0 con 0x7fea9014ee00 2024-08-31T12:28:06.485 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.483+0000 7fea98779640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fea900711d0 con 0x7fea90156440 2024-08-31T12:28:06.485 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.483+0000 7fea98779640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fea900cb5e0 con 0x7fea9013d9f0 2024-08-31T12:28:06.485 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.483+0000 7fea96515640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fea90156440 0x7fea901557a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:06.485 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.483+0000 7fea96515640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fea90156440 0x7fea901557a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.19:34902/0 (socket says 172.21.15.19:34902) 2024-08-31T12:28:06.485 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.483+0000 7fea96515640 1 -- 172.21.15.19:0/2149505298 learned_addr learned my addr 172.21.15.19:0/2149505298 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:28:06.485 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.483+0000 7fea96d16640 1 --2- 172.21.15.19:0/2149505298 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fea9014ee00 0x7fea9014f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:06.486 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.483+0000 7fea96515640 1 -- 172.21.15.19:0/2149505298 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fea9013d9f0 msgr2=0x7fea9013de00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:28:06.486 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.483+0000 7fea96515640 1 --2- 172.21.15.19:0/2149505298 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fea9013d9f0 0x7fea9013de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:06.486 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.483+0000 7fea96515640 1 -- 172.21.15.19:0/2149505298 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fea9014ee00 msgr2=0x7fea9014f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:06.486 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.483+0000 7fea96515640 1 --2- 172.21.15.19:0/2149505298 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fea9014ee00 0x7fea9014f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:06.486 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.483+0000 7fea96515640 1 -- 172.21.15.19:0/2149505298 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea9015c7e0 con 0x7fea90156440 2024-08-31T12:28:06.486 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.484+0000 7fea96515640 1 --2- 172.21.15.19:0/2149505298 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fea90156440 0x7fea901557a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fea8c00c500 tx=0x7fea8c00c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:06.486 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.484+0000 7fea86ffd640 1 -- 172.21.15.19:0/2149505298 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8c0052d0 con 0x7fea90156440 2024-08-31T12:28:06.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.484+0000 7fea86ffd640 1 -- 172.21.15.19:0/2149505298 <== mon.1 v2:172.21.15.43:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fea8c005470 con 0x7fea90156440 2024-08-31T12:28:06.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.484+0000 7fea98779640 1 -- 172.21.15.19:0/2149505298 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea900741a0 con 0x7fea90156440 2024-08-31T12:28:06.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.484+0000 7fea86ffd640 1 -- 172.21.15.19:0/2149505298 <== mon.1 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8c012830 con 0x7fea90156440 2024-08-31T12:28:06.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.484+0000 7fea98779640 1 -- 172.21.15.19:0/2149505298 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea9015aea0 con 0x7fea90156440 2024-08-31T12:28:06.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.485+0000 7fea98779640 1 -- 172.21.15.19:0/2149505298 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea9015b0e0 con 0x7fea90156440 2024-08-31T12:28:06.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.486+0000 7fea86ffd640 1 -- 172.21.15.19:0/2149505298 <== mon.1 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fea8c0129d0 con 0x7fea90156440 2024-08-31T12:28:06.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.487+0000 7fea86ffd640 1 --2- 172.21.15.19:0/2149505298 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fea74078910 0x7fea7407ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:06.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.487+0000 7fea86ffd640 1 -- 172.21.15.19:0/2149505298 <== mon.1 v2:172.21.15.43:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fea8c099300 con 0x7fea90156440 2024-08-31T12:28:06.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.489+0000 7fea95d14640 1 --2- 172.21.15.19:0/2149505298 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fea74078910 0x7fea7407ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:06.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.490+0000 7fea95d14640 1 --2- 172.21.15.19:0/2149505298 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fea74078910 0x7fea7407ae20 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fea80034000 tx=0x7fea800046c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:06.497 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.495+0000 7fea86ffd640 1 -- 172.21.15.19:0/2149505298 <== mon.1 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fea9015b0e0 con 0x7fea90156440 2024-08-31T12:28:06.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.977+0000 7fea98779640 1 -- 172.21.15.19:0/2149505298 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fea900c7980 con 0x7fea90156440 2024-08-31T12:28:06.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.978+0000 7fea86ffd640 1 -- 172.21.15.19:0/2149505298 <== mon.1 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7fea8c062020 con 0x7fea90156440 2024-08-31T12:28:06.979 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":96,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:28:06.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.981+0000 7fea98779640 1 -- 172.21.15.19:0/2149505298 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fea74078910 msgr2=0x7fea7407ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:06.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.981+0000 7fea98779640 1 --2- 172.21.15.19:0/2149505298 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fea74078910 0x7fea7407ae20 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fea80034000 tx=0x7fea800046c0 comp rx=0 tx=0).stop 2024-08-31T12:28:06.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.981+0000 7fea98779640 1 -- 172.21.15.19:0/2149505298 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fea90156440 msgr2=0x7fea901557a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:06.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.981+0000 7fea98779640 1 --2- 172.21.15.19:0/2149505298 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fea90156440 0x7fea901557a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fea8c00c500 tx=0x7fea8c00c9c0 comp rx=0 tx=0).stop 2024-08-31T12:28:06.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.981+0000 7fea98779640 1 -- 172.21.15.19:0/2149505298 shutdown_connections 2024-08-31T12:28:06.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.981+0000 7fea98779640 1 --2- 172.21.15.19:0/2149505298 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fea74078910 0x7fea7407ae20 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:06.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.981+0000 7fea98779640 1 --2- 172.21.15.19:0/2149505298 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fea9014ee00 0x7fea9014f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:06.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.981+0000 7fea98779640 1 --2- 172.21.15.19:0/2149505298 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fea9013d9f0 0x7fea9013de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:06.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.981+0000 7fea98779640 1 --2- 172.21.15.19:0/2149505298 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fea90156440 0x7fea901557a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:06.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.981+0000 7fea98779640 1 -- 172.21.15.19:0/2149505298 >> 172.21.15.19:0/2149505298 conn(0x7fea900c99a0 msgr2=0x7fea90074620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:06.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.982+0000 7fea98779640 1 -- 172.21.15.19:0/2149505298 shutdown_connections 2024-08-31T12:28:06.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:06.982+0000 7fea98779640 1 -- 172.21.15.19:0/2149505298 wait complete. 2024-08-31T12:28:07.000 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-08-31T12:28:07.000 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-08-31T12:28:07.000 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-31T12:28:07.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.186+0000 7f79be5f8640 1 -- 172.21.15.19:0/1728831333 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f79b806ebe0 msgr2=0x7f79b80ca6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:07.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.186+0000 7f79be5f8640 1 --2- 172.21.15.19:0/1728831333 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f79b806ebe0 0x7f79b80ca6d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f79ac009830 tx=0x7f79ac02a530 comp rx=0 tx=0).stop 2024-08-31T12:28:07.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.186+0000 7f79be5f8640 1 -- 172.21.15.19:0/1728831333 shutdown_connections 2024-08-31T12:28:07.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.186+0000 7f79be5f8640 1 --2- 172.21.15.19:0/1728831333 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f79b80cb5b0 0x7f79b80cb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:07.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.187+0000 7f79be5f8640 1 --2- 172.21.15.19:0/1728831333 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f79b806ebe0 0x7f79b80ca6d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:07.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.187+0000 7f79be5f8640 1 -- 172.21.15.19:0/1728831333 >> 172.21.15.19:0/1728831333 conn(0x7f79b8069350 msgr2=0x7f79b80d07a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:07.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.187+0000 7f79be5f8640 1 -- 172.21.15.19:0/1728831333 shutdown_connections 2024-08-31T12:28:07.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.187+0000 7f79be5f8640 1 -- 172.21.15.19:0/1728831333 wait complete. 2024-08-31T12:28:07.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.189+0000 7f79be5f8640 1 Processor -- start 2024-08-31T12:28:07.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.189+0000 7f79be5f8640 1 -- start start 2024-08-31T12:28:07.194 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.193+0000 7f79be5f8640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79b806ebe0 0x7f79b80c6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:07.194 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.193+0000 7f79be5f8640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f79b80b3630 0x7f79b81f1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:07.194 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.193+0000 7f79be5f8640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f79b80cb5b0 0x7f79b81e4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:07.194 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.193+0000 7f79be5f8640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f79b815b970 con 0x7f79b806ebe0 2024-08-31T12:28:07.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.193+0000 7f79b7fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79b806ebe0 0x7f79b80c6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:07.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.193+0000 7f79b7fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79b806ebe0 0x7f79b80c6d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:35116/0 (socket says 172.21.15.19:35116) 2024-08-31T12:28:07.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.193+0000 7f79b7fff640 1 -- 172.21.15.19:0/80325686 learned_addr learned my addr 172.21.15.19:0/80325686 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:28:07.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.193+0000 7f79b77fe640 1 -- 172.21.15.19:0/80325686 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f79b80b3630 msgr2=0x7f79b81f1b60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3301/0 2024-08-31T12:28:07.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.193+0000 7f79b77fe640 1 --2- 172.21.15.19:0/80325686 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f79b80b3630 0x7f79b81f1b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:28:07.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.193+0000 7f79be5f8640 1 -- 172.21.15.19:0/80325686 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f79b80cac10 con 0x7f79b80cb5b0 2024-08-31T12:28:07.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.193+0000 7f79be5f8640 1 -- 172.21.15.19:0/80325686 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f79b815bc50 con 0x7f79b80b3630 2024-08-31T12:28:07.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.193+0000 7f79bcb95640 1 --2- 172.21.15.19:0/80325686 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f79b80cb5b0 0x7f79b81e4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:07.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.194+0000 7f79b7fff640 1 -- 172.21.15.19:0/80325686 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f79b80b3630 msgr2=0x7f79b81f1b60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:28:07.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.194+0000 7f79b7fff640 1 --2- 172.21.15.19:0/80325686 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f79b80b3630 0x7f79b81f1b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:07.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.194+0000 7f79b7fff640 1 -- 172.21.15.19:0/80325686 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f79b80cb5b0 msgr2=0x7f79b81e4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:07.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.194+0000 7f79b7fff640 1 --2- 172.21.15.19:0/80325686 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f79b80cb5b0 0x7f79b81e4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:07.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.194+0000 7f79b7fff640 1 -- 172.21.15.19:0/80325686 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79b81f20a0 con 0x7f79b806ebe0 2024-08-31T12:28:07.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.194+0000 7f79bcb95640 1 --2- 172.21.15.19:0/80325686 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f79b80cb5b0 0x7f79b81e4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:28:07.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.194+0000 7f79b7fff640 1 --2- 172.21.15.19:0/80325686 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79b806ebe0 0x7f79b80c6d00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f79b0002220 tx=0x7f79b000a4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:07.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.195+0000 7f79b4ff9640 1 -- 172.21.15.19:0/80325686 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b0003b70 con 0x7f79b806ebe0 2024-08-31T12:28:07.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.195+0000 7f79b4ff9640 1 -- 172.21.15.19:0/80325686 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f79b0003e40 con 0x7f79b806ebe0 2024-08-31T12:28:07.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.195+0000 7f79be5f8640 1 -- 172.21.15.19:0/80325686 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79b81e4fa0 con 0x7f79b806ebe0 2024-08-31T12:28:07.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.195+0000 7f79b4ff9640 1 -- 172.21.15.19:0/80325686 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b0013b20 con 0x7f79b806ebe0 2024-08-31T12:28:07.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.195+0000 7f79be5f8640 1 -- 172.21.15.19:0/80325686 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79b8159550 con 0x7f79b806ebe0 2024-08-31T12:28:07.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.196+0000 7f79be5f8640 1 --2- 172.21.15.19:0/80325686 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79b81e5170 0x7f79b81e35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:07.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.196+0000 7f79be5f8640 1 -- 172.21.15.19:0/80325686 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f79b806e330 con 0x7f79b81e5170 2024-08-31T12:28:07.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.197+0000 7f79b77fe640 1 --2- 172.21.15.19:0/80325686 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79b81e5170 0x7f79b81e35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:07.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.197+0000 7f79b77fe640 1 -- 172.21.15.19:0/80325686 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f79b80c74d0 con 0x7f79b81e5170 2024-08-31T12:28:07.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.197+0000 7f79b77fe640 1 --2- 172.21.15.19:0/80325686 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79b81e5170 0x7f79b81e35a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f79b80ce290 tx=0x7f79ac0303a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:07.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.198+0000 7f79b4ff9640 1 -- 172.21.15.19:0/80325686 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79ac03f070 con 0x7f79b81e5170 2024-08-31T12:28:07.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.198+0000 7f79b4ff9640 1 -- 172.21.15.19:0/80325686 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f79b00050d0 con 0x7f79b806ebe0 2024-08-31T12:28:07.200 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.199+0000 7f79b4ff9640 1 --2- 172.21.15.19:0/80325686 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f79a4077950 0x7f79a4079e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:07.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.199+0000 7f79b4ff9640 1 -- 172.21.15.19:0/80325686 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f79b00a1bd0 con 0x7f79b806ebe0 2024-08-31T12:28:07.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.199+0000 7f79bcb95640 1 --2- 172.21.15.19:0/80325686 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f79a4077950 0x7f79a4079e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:07.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.199+0000 7f79b4ff9640 1 -- 172.21.15.19:0/80325686 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f79ac030ad0 con 0x7f79b81e5170 2024-08-31T12:28:07.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.199+0000 7f79b4ff9640 1 -- 172.21.15.19:0/80325686 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79b81e5170 msgr2=0x7f79b81e35a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:07.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.199+0000 7f79b4ff9640 1 --2- 172.21.15.19:0/80325686 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79b81e5170 0x7f79b81e35a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f79b80ce290 tx=0x7f79ac0303a0 comp rx=0 tx=0).stop 2024-08-31T12:28:07.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.200+0000 7f79b4ff9640 1 -- 172.21.15.19:0/80325686 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79b81e5170 msgr2=0x7f79b81e35a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:28:07.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.200+0000 7f79b4ff9640 1 --2- 172.21.15.19:0/80325686 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79b81e5170 0x7f79b81e35a0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:07.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.200+0000 7f79bcb95640 1 --2- 172.21.15.19:0/80325686 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f79a4077950 0x7f79a4079e60 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f79b80cd020 tx=0x7f79a8006f60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:07.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.215+0000 7f79be5f8640 1 --2- 172.21.15.19:0/80325686 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79b81e3ae0 0x7f79b8152a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:07.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.215+0000 7f79be5f8640 1 -- 172.21.15.19:0/80325686 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f79b81e3ef0 con 0x7f79b81e3ae0 2024-08-31T12:28:07.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.215+0000 7f79b77fe640 1 --2- 172.21.15.19:0/80325686 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79b81e3ae0 0x7f79b8152a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:07.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.216+0000 7f79b77fe640 1 -- 172.21.15.19:0/80325686 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f79b80d0a80 con 0x7f79b81e3ae0 2024-08-31T12:28:07.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.216+0000 7f79b77fe640 1 --2- 172.21.15.19:0/80325686 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79b81e3ae0 0x7f79b8152a60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f79ac009830 tx=0x7f79ac02a530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:07.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.217+0000 7f79b4ff9640 1 -- 172.21.15.19:0/80325686 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79ac005220 con 0x7f79b81e3ae0 2024-08-31T12:28:07.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.217+0000 7f79b4ff9640 1 -- 172.21.15.19:0/80325686 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5033 (secure 0 0 0) 0x7f79ac030e10 con 0x7f79b81e3ae0 2024-08-31T12:28:07.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.217+0000 7f79b4ff9640 1 -- 172.21.15.19:0/80325686 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79b81e3ae0 msgr2=0x7f79b8152a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:07.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.217+0000 7f79b4ff9640 1 --2- 172.21.15.19:0/80325686 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79b81e3ae0 0x7f79b8152a60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f79ac009830 tx=0x7f79ac02a530 comp rx=0 tx=0).stop 2024-08-31T12:28:07.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.217+0000 7f79b4ff9640 1 -- 172.21.15.19:0/80325686 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79b81e3ae0 msgr2=0x7f79b8152a60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:28:07.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.217+0000 7f79b4ff9640 1 --2- 172.21.15.19:0/80325686 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79b81e3ae0 0x7f79b8152a60 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:07.219 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:28:07.219 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:28:07.219 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:28:07.219 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "leader", 2024-08-31T12:28:07.219 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 96, 2024-08-31T12:28:07.219 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 221770, 2024-08-31T12:28:07.219 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:28:07.219 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:28:07.219 INFO:teuthology.orchestra.run.smithi019.stdout: 1 2024-08-31T12:28:07.219 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:28:07.219 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 1, 2024-08-31T12:28:07.220 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:28:07.220 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:28:07.220 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:28:07.220 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:28:07.220 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:28:07.220 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:28:07.220 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:28:07.220 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:28:07.220 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:28:07.220 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:28:07.220 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:28:07.220 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:28:07.220 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:28:07.220 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:28:07.221 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:28:07.221 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:28:07.221 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:28:07.221 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:28:07.221 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:28:07.221 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:28:07.221 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:28:07.221 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:28:07.221 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:28:07.221 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:28:07.221 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:28:07.221 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:28:07.221 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:28:07.222 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:28:07.222 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:28:07.222 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:07.222 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:28:07.222 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:28:07.222 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:28:07.222 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:28:07.222 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:28:07.222 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:28:07.222 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:28:07.222 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:28:07.222 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:28:07.222 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:28:07.222 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:28:07.222 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:28:07.223 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:28:07.223 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:28:07.223 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:28:07.223 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:28:07.223 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:28:07.223 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:28:07.223 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:28:07.223 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:28:07.223 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:28:07.223 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:28:07.223 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:28:07.223 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:28:07.223 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:28:07.223 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:28:07.223 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:28:07.224 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:28:07.224 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:28:07.224 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:28:07.224 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:07.224 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:28:07.224 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:07.224 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:28:07.224 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:28:07.224 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:28:07.224 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:28:07.224 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:07.224 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:28:07.224 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:28:07.224 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:28:07.224 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:07.225 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:07.225 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:28:07.225 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:28:07.225 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:28:07.225 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:28:07.225 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:28:07.225 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:07.225 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:28:07.225 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:28:07.225 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:28:07.225 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:28:07.225 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:28:07.225 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:07.226 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:07.226 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:28:07.226 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:28:07.226 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:28:07.226 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:28:07.226 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:07.226 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:28:07.226 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:28:07.226 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:28:07.226 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:07.226 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:07.226 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:28:07.226 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:28:07.226 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:28:07.226 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:28:07.227 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:28:07.227 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:07.227 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:28:07.227 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:28:07.227 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:28:07.227 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:28:07.227 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:28:07.227 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:07.227 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:07.227 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:28:07.227 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:28:07.227 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:28:07.227 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:28:07.228 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:07.228 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:28:07.228 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:28:07.228 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:28:07.228 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:07.228 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:07.228 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:28:07.228 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:28:07.228 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:28:07.228 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:28:07.228 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:28:07.228 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:07.228 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:28:07.228 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:28:07.229 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:28:07.229 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:28:07.229 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:28:07.229 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:28:07.229 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:28:07.229 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:07.229 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:28:07.229 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:28:07.229 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:07.229 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:28:07.229 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:28:07.229 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:28:07.229 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:28:07.230 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:28:07.230 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:28:07.230 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:07.230 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:28:07.230 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:28:07.230 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 5 2024-08-31T12:28:07.230 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:28:07.230 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:28:07.230 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:28:07.230 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:07.230 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:28:07.230 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:28:07.230 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 7 2024-08-31T12:28:07.230 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:28:07.230 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:28:07.231 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:28:07.231 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:07.231 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:28:07.231 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:28:07.231 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 7 2024-08-31T12:28:07.231 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:28:07.231 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:28:07.231 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:28:07.231 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:07.231 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:28:07.231 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:28:07.231 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:28:07.231 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:28:07.231 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:28:07.231 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:07.232 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:28:07.232 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:28:07.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.218+0000 7f79be5f8640 1 -- 172.21.15.19:0/80325686 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f79a4077950 msgr2=0x7f79a4079e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:07.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.218+0000 7f79be5f8640 1 --2- 172.21.15.19:0/80325686 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f79a4077950 0x7f79a4079e60 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f79b80cd020 tx=0x7f79a8006f60 comp rx=0 tx=0).stop 2024-08-31T12:28:07.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.218+0000 7f79be5f8640 1 -- 172.21.15.19:0/80325686 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79b806ebe0 msgr2=0x7f79b80c6d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:07.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.218+0000 7f79be5f8640 1 --2- 172.21.15.19:0/80325686 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79b806ebe0 0x7f79b80c6d00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f79b0002220 tx=0x7f79b000a4f0 comp rx=0 tx=0).stop 2024-08-31T12:28:07.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.218+0000 7f79bcb95640 1 -- 172.21.15.19:0/80325686 reap_dead start 2024-08-31T12:28:07.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.218+0000 7f79bcb95640 1 -- 172.21.15.19:0/80325686 reap_dead start 2024-08-31T12:28:07.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.218+0000 7f79be5f8640 1 -- 172.21.15.19:0/80325686 shutdown_connections 2024-08-31T12:28:07.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.218+0000 7f79be5f8640 1 -- 172.21.15.19:0/80325686 >> 172.21.15.19:0/80325686 conn(0x7f79b8069350 msgr2=0x7f79b8154ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:07.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.218+0000 7f79be5f8640 1 -- 172.21.15.19:0/80325686 shutdown_connections 2024-08-31T12:28:07.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.218+0000 7f79be5f8640 1 -- 172.21.15.19:0/80325686 wait complete. 2024-08-31T12:28:07.236 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-08-31T12:28:07.236 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-08-31T12:28:07.236 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-31T12:28:07.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.427+0000 7f31a2df8640 1 -- 172.21.15.19:0/1320801037 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3184004800 msgr2=0x7f3184004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:07.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.427+0000 7f31a2df8640 1 --2- 172.21.15.19:0/1320801037 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3184004800 0x7f3184004c10 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f319000c470 tx=0x7f3184005830 comp rx=0 tx=0).stop 2024-08-31T12:28:07.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.428+0000 7f31a1395640 1 -- 172.21.15.19:0/1320801037 reap_dead start 2024-08-31T12:28:07.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.428+0000 7f31a2df8640 1 -- 172.21.15.19:0/1320801037 shutdown_connections 2024-08-31T12:28:07.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.428+0000 7f31a2df8640 1 -- 172.21.15.19:0/1320801037 >> 172.21.15.19:0/1320801037 conn(0x7f319c0cc950 msgr2=0x7f319c0ccd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:07.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.428+0000 7f31a2df8640 1 -- 172.21.15.19:0/1320801037 shutdown_connections 2024-08-31T12:28:07.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.428+0000 7f31a2df8640 1 -- 172.21.15.19:0/1320801037 wait complete. 2024-08-31T12:28:07.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.431+0000 7f31a2df8640 1 Processor -- start 2024-08-31T12:28:07.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.432+0000 7f31a2df8640 1 -- start start 2024-08-31T12:28:07.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.434+0000 7f31a2df8640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f319c1fee70 0x7f319c15d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:07.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.434+0000 7f31a2df8640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f319c166160 0x7f319c15fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:07.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.434+0000 7f31a2df8640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f319c166570 0x7f319c160020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:07.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.434+0000 7f31a2df8640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f319c0c6e60 con 0x7f319c166160 2024-08-31T12:28:07.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.434+0000 7f31a2df8640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f319c0c7da0 con 0x7f319c1fee70 2024-08-31T12:28:07.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.434+0000 7f31a2df8640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f319c074610 con 0x7f319c166570 2024-08-31T12:28:07.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.434+0000 7f31a1395640 1 -- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f319c166570 msgr2=0x7f319c160020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3301/0 2024-08-31T12:28:07.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.434+0000 7f31a1395640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f319c166570 0x7f319c160020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:28:07.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.435+0000 7f319bfff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f319c166160 0x7f319c15fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:07.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.435+0000 7f319bfff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f319c166160 0x7f319c15fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:35148/0 (socket says 172.21.15.19:35148) 2024-08-31T12:28:07.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.435+0000 7f319bfff640 1 -- 172.21.15.19:0/3450379931 learned_addr learned my addr 172.21.15.19:0/3450379931 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:28:07.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.435+0000 7f31a0b94640 1 --2- 172.21.15.19:0/3450379931 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f319c1fee70 0x7f319c15d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:07.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.435+0000 7f319bfff640 1 -- 172.21.15.19:0/3450379931 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f319c166570 msgr2=0x7f319c160020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:28:07.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.435+0000 7f319bfff640 1 --2- 172.21.15.19:0/3450379931 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f319c166570 0x7f319c160020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:07.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.435+0000 7f319bfff640 1 -- 172.21.15.19:0/3450379931 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f319c1fee70 msgr2=0x7f319c15d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:07.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.435+0000 7f319bfff640 1 --2- 172.21.15.19:0/3450379931 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f319c1fee70 0x7f319c15d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:07.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.435+0000 7f319bfff640 1 -- 172.21.15.19:0/3450379931 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f319c0c2ec0 con 0x7f319c166160 2024-08-31T12:28:07.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.435+0000 7f319bfff640 1 --2- 172.21.15.19:0/3450379931 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f319c166160 0x7f319c15fae0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f319000c590 tx=0x7f31900065d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:07.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.435+0000 7f31997fa640 1 -- 172.21.15.19:0/3450379931 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3190042070 con 0x7f319c166160 2024-08-31T12:28:07.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.435+0000 7f31997fa640 1 -- 172.21.15.19:0/3450379931 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f31900039d0 con 0x7f319c166160 2024-08-31T12:28:07.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.435+0000 7f31a2df8640 1 -- 172.21.15.19:0/3450379931 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f319c0c3180 con 0x7f319c166160 2024-08-31T12:28:07.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.435+0000 7f31997fa640 1 -- 172.21.15.19:0/3450379931 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3190039070 con 0x7f319c166160 2024-08-31T12:28:07.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.435+0000 7f31a2df8640 1 -- 172.21.15.19:0/3450379931 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f319c1fe390 con 0x7f319c166160 2024-08-31T12:28:07.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.436+0000 7f31a2df8640 1 --2- 172.21.15.19:0/3450379931 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f319c15ec40 0x7f319c15f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:07.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.436+0000 7f31a2df8640 1 -- 172.21.15.19:0/3450379931 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f319c0cb690 con 0x7f319c15ec40 2024-08-31T12:28:07.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.437+0000 7f31a0b94640 1 --2- 172.21.15.19:0/3450379931 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f319c15ec40 0x7f319c15f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:07.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.437+0000 7f31a0b94640 1 -- 172.21.15.19:0/3450379931 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f319c0c3ac0 con 0x7f319c15ec40 2024-08-31T12:28:07.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.437+0000 7f31a0b94640 1 --2- 172.21.15.19:0/3450379931 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f319c15ec40 0x7f319c15f050 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f318c001020 tx=0x7f318c011c40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:07.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.437+0000 7f31997fa640 1 -- 172.21.15.19:0/3450379931 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f319002b610 con 0x7f319c166160 2024-08-31T12:28:07.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.438+0000 7f31997fa640 1 --2- 172.21.15.19:0/3450379931 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3188077a00 0x7f3188079f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:07.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.438+0000 7f31997fa640 1 -- 172.21.15.19:0/3450379931 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f31900b9280 con 0x7f319c166160 2024-08-31T12:28:07.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.438+0000 7f31a1395640 1 --2- 172.21.15.19:0/3450379931 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3188077a00 0x7f3188079f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:07.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.438+0000 7f31997fa640 1 -- 172.21.15.19:0/3450379931 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f318c01e070 con 0x7f319c15ec40 2024-08-31T12:28:07.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.438+0000 7f31997fa640 1 -- 172.21.15.19:0/3450379931 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f318c00fe50 con 0x7f319c15ec40 2024-08-31T12:28:07.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.438+0000 7f31997fa640 1 -- 172.21.15.19:0/3450379931 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f319c15ec40 msgr2=0x7f319c15f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:07.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.438+0000 7f31997fa640 1 --2- 172.21.15.19:0/3450379931 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f319c15ec40 0x7f319c15f050 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f318c001020 tx=0x7f318c011c40 comp rx=0 tx=0).stop 2024-08-31T12:28:07.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.438+0000 7f31997fa640 1 -- 172.21.15.19:0/3450379931 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f319c15ec40 msgr2=0x7f319c15f050 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:28:07.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.438+0000 7f31997fa640 1 --2- 172.21.15.19:0/3450379931 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f319c15ec40 0x7f319c15f050 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:07.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.438+0000 7f31a1395640 1 --2- 172.21.15.19:0/3450379931 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3188077a00 0x7f3188079f10 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f319c06fd30 tx=0x7f3194009500 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:07.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.444+0000 7f31a2df8640 1 --2- 172.21.15.19:0/3450379931 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f319c2017f0 0x7f319c201c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:07.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.444+0000 7f31a2df8640 1 -- 172.21.15.19:0/3450379931 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f319c08cce0 con 0x7f319c2017f0 2024-08-31T12:28:07.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.445+0000 7f31a0b94640 1 --2- 172.21.15.19:0/3450379931 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f319c2017f0 0x7f319c201c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:07.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.445+0000 7f31a0b94640 1 -- 172.21.15.19:0/3450379931 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f319c0741e0 con 0x7f319c2017f0 2024-08-31T12:28:07.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.445+0000 7f31a0b94640 1 --2- 172.21.15.19:0/3450379931 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f319c2017f0 0x7f319c201c00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f319c1fc0c0 tx=0x7f318c004840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:07.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.445+0000 7f31997fa640 1 -- 172.21.15.19:0/3450379931 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f318c01e040 con 0x7f319c2017f0 2024-08-31T12:28:07.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.446+0000 7f31997fa640 1 -- 172.21.15.19:0/3450379931 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4863 (secure 0 0 0) 0x7f318c0039f0 con 0x7f319c2017f0 2024-08-31T12:28:07.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.446+0000 7f31997fa640 1 -- 172.21.15.19:0/3450379931 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f319c2017f0 msgr2=0x7f319c201c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:07.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.446+0000 7f31997fa640 1 --2- 172.21.15.19:0/3450379931 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f319c2017f0 0x7f319c201c00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f319c1fc0c0 tx=0x7f318c004840 comp rx=0 tx=0).stop 2024-08-31T12:28:07.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.446+0000 7f31997fa640 1 -- 172.21.15.19:0/3450379931 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f319c2017f0 msgr2=0x7f319c201c00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:28:07.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.446+0000 7f31997fa640 1 --2- 172.21.15.19:0/3450379931 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f319c2017f0 0x7f319c201c00 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:07.447 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:28:07.448 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:28:07.448 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:28:07.448 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:28:07.448 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 96, 2024-08-31T12:28:07.448 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 37268, 2024-08-31T12:28:07.448 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:28:07.448 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:28:07.448 INFO:teuthology.orchestra.run.smithi019.stdout: 1 2024-08-31T12:28:07.448 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:28:07.448 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 1, 2024-08-31T12:28:07.448 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:28:07.448 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:28:07.448 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:28:07.449 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:28:07.449 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:28:07.449 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:28:07.449 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:28:07.449 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:28:07.449 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:28:07.449 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:28:07.449 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:28:07.449 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:28:07.449 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:28:07.449 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:28:07.449 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:28:07.449 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:28:07.450 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:28:07.450 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:28:07.450 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:28:07.450 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:28:07.450 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:28:07.450 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:28:07.450 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:28:07.450 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:28:07.450 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:28:07.450 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:28:07.450 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:28:07.450 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:28:07.451 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:28:07.451 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:07.451 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:28:07.451 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:28:07.451 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:28:07.451 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:28:07.451 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:28:07.451 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:28:07.451 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:28:07.451 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:28:07.451 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:28:07.451 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:28:07.451 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:28:07.451 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:28:07.452 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:28:07.452 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:28:07.452 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:28:07.452 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:28:07.452 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:28:07.452 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:28:07.452 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:28:07.452 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:28:07.452 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:28:07.452 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:28:07.452 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:28:07.453 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:28:07.453 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:28:07.453 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:28:07.453 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:28:07.453 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:28:07.453 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:28:07.453 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:28:07.453 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:07.453 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:28:07.453 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:07.453 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:28:07.453 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:28:07.454 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:28:07.454 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:28:07.454 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:07.454 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:28:07.454 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:28:07.454 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:28:07.454 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:07.454 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:07.454 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:28:07.454 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:28:07.454 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:28:07.454 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:28:07.454 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:28:07.454 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:07.455 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:28:07.455 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:28:07.455 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:28:07.455 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:28:07.455 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:28:07.455 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:07.455 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:07.455 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:28:07.455 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:28:07.455 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:28:07.455 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:28:07.455 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:07.455 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:28:07.455 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:28:07.456 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:28:07.456 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:07.456 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:07.456 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:28:07.456 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:28:07.456 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:28:07.456 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:28:07.456 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:28:07.456 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:07.456 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:28:07.456 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:28:07.456 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:28:07.456 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:28:07.457 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:28:07.457 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:07.457 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:07.457 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:28:07.457 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:28:07.457 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:28:07.457 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:28:07.457 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:07.457 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:28:07.457 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:28:07.457 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:28:07.457 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:07.457 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:07.458 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:28:07.458 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:28:07.458 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:28:07.458 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:28:07.458 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:28:07.458 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:07.458 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:28:07.458 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:28:07.458 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:28:07.458 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:28:07.458 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:28:07.458 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:28:07.458 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:28:07.458 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:07.459 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:28:07.459 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:28:07.459 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:07.459 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:28:07.459 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:28:07.459 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:28:07.459 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:28:07.459 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:28:07.459 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:28:07.459 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:07.459 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:28:07.459 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:28:07.459 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:28:07.459 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:28:07.460 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:28:07.460 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:28:07.460 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:07.460 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:28:07.460 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:28:07.460 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:28:07.460 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:28:07.460 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:28:07.460 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:28:07.460 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:07.460 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:28:07.460 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:28:07.460 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:28:07.461 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:28:07.461 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:28:07.461 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:07.461 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:28:07.461 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:28:07.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.446+0000 7f31a2df8640 1 -- 172.21.15.19:0/3450379931 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3188077a00 msgr2=0x7f3188079f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:07.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.446+0000 7f31a2df8640 1 --2- 172.21.15.19:0/3450379931 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f3188077a00 0x7f3188079f10 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f319c06fd30 tx=0x7f3194009500 comp rx=0 tx=0).stop 2024-08-31T12:28:07.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.446+0000 7f31a2df8640 1 -- 172.21.15.19:0/3450379931 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f319c166160 msgr2=0x7f319c15fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:07.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.446+0000 7f31a2df8640 1 --2- 172.21.15.19:0/3450379931 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f319c166160 0x7f319c15fae0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f319000c590 tx=0x7f31900065d0 comp rx=0 tx=0).stop 2024-08-31T12:28:07.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.446+0000 7f31a1395640 1 -- 172.21.15.19:0/3450379931 reap_dead start 2024-08-31T12:28:07.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.446+0000 7f31a2df8640 1 -- 172.21.15.19:0/3450379931 shutdown_connections 2024-08-31T12:28:07.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.446+0000 7f31a2df8640 1 --2- 172.21.15.19:0/3450379931 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f319c166160 0x7f319c15fae0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:07.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.446+0000 7f31a2df8640 1 -- 172.21.15.19:0/3450379931 >> 172.21.15.19:0/3450379931 conn(0x7f319c0cc950 msgr2=0x7f319c1652b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:07.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.446+0000 7f31a2df8640 1 -- 172.21.15.19:0/3450379931 shutdown_connections 2024-08-31T12:28:07.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:07.446+0000 7f31a2df8640 1 -- 172.21.15.19:0/3450379931 wait complete. 2024-08-31T12:28:07.463 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-08-31T12:28:07.463 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-08-31T12:28:18.997 INFO:tasks.workunit.client.0.smithi019.stdout:Wrote -1 instead of 4096 bytes. 2024-08-31T12:28:18.997 INFO:tasks.workunit.client.0.smithi019.stdout:Probably out of disk space 2024-08-31T12:28:18.998 INFO:tasks.workunit.client.0.smithi019.stderr:write: Bad file descriptor 2024-08-31T12:28:19.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T12:28:19.009 INFO:tasks.workunit:Stopping ['suites/ffsb.sh'] on client.0... 2024-08-31T12:28:19.009 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-08-31T12:28:19.387 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d19e6118f7015f0047264006cb0c873528eb3ae6/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_d19e6118f7015f0047264006cb0c873528eb3ae6/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d19e6118f7015f0047264006cb0c873528eb3ae6/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/ffsb.sh) on smithi019 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d19e6118f7015f0047264006cb0c873528eb3ae6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' 2024-08-31T12:28:19.603 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=9c6858758b7441539ce94ced4a3719f4 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d19e6118f7015f0047264006cb0c873528eb3ae6/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_d19e6118f7015f0047264006cb0c873528eb3ae6/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d19e6118f7015f0047264006cb0c873528eb3ae6/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/ffsb.sh) on smithi019 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d19e6118f7015f0047264006cb0c873528eb3ae6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' 2024-08-31T12:28:19.606 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-08-31T12:28:19.614 INFO:tasks.mon_thrash:joining mon_thrasher 2024-08-31T12:28:27.464 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-08-31T12:28:27.465 INFO:tasks.ceph.mon.c:Restarting daemon 2024-08-31T12:28:27.465 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-08-31T12:28:27.468 INFO:tasks.ceph.mon.c:Started 2024-08-31T12:28:27.468 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-08-31T12:28:27.468 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:28:27.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.595+0000 7f4f8ba7e640 1 -- 172.21.15.19:0/647707266 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4f8415b120 msgr2=0x7f4f8415b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:27.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.595+0000 7f4f8ba7e640 1 --2- 172.21.15.19:0/647707266 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4f8415b120 0x7f4f8415b530 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4f84069960 tx=0x7f4f7c02a5f0 comp rx=0 tx=0).stop 2024-08-31T12:28:27.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.595+0000 7f4f8ba7e640 1 -- 172.21.15.19:0/647707266 shutdown_connections 2024-08-31T12:28:27.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.595+0000 7f4f8ba7e640 1 --2- 172.21.15.19:0/647707266 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4f8415b120 0x7f4f8415b530 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:27.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.595+0000 7f4f8ba7e640 1 --2- 172.21.15.19:0/647707266 >> v2:172.21.15.43:3300/0 conn(0x7f4f840cc070 0x7f4f840cc480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:27.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.595+0000 7f4f8ba7e640 1 --2- 172.21.15.19:0/647707266 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4f840ccbb0 0x7f4f840cb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:27.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.595+0000 7f4f8ba7e640 1 -- 172.21.15.19:0/647707266 >> 172.21.15.19:0/647707266 conn(0x7f4f84074810 msgr2=0x7f4f84068aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:27.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.595+0000 7f4f8ba7e640 1 -- 172.21.15.19:0/647707266 shutdown_connections 2024-08-31T12:28:27.598 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.595+0000 7f4f8ba7e640 1 -- 172.21.15.19:0/647707266 wait complete. 2024-08-31T12:28:27.598 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.597+0000 7f4f8ba7e640 1 Processor -- start 2024-08-31T12:28:27.598 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.597+0000 7f4f8ba7e640 1 -- start start 2024-08-31T12:28:27.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.598+0000 7f4f8ba7e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f840cc070 0x7f4f841f2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:27.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.598+0000 7f4f8ba7e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4f840ccbb0 0x7f4f84154140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:27.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.598+0000 7f4f8ba7e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4f8415b120 0x7f4f84145fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:27.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.598+0000 7f4f8ba7e640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f4f84155660 con 0x7f4f8415b120 2024-08-31T12:28:27.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.598+0000 7f4f8ba7e640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4f8415ba70 con 0x7f4f840cc070 2024-08-31T12:28:27.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.598+0000 7f4f8ba7e640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f4f841557e0 con 0x7f4f840ccbb0 2024-08-31T12:28:27.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.598+0000 7f4f8a01b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4f8415b120 0x7f4f84145fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:27.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.598+0000 7f4f8a01b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4f8415b120 0x7f4f84145fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:33692/0 (socket says 172.21.15.19:33692) 2024-08-31T12:28:27.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.598+0000 7f4f8a01b640 1 -- 172.21.15.19:0/1842413991 learned_addr learned my addr 172.21.15.19:0/1842413991 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:28:27.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.598+0000 7f4f8981a640 1 --2- 172.21.15.19:0/1842413991 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f840cc070 0x7f4f841f2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:27.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.598+0000 7f4f89019640 1 --2- 172.21.15.19:0/1842413991 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4f840ccbb0 0x7f4f84154140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:27.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.598+0000 7f4f89019640 1 -- 172.21.15.19:0/1842413991 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4f840ccbb0 msgr2=0x7f4f84154140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-08-31T12:28:27.603 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.598+0000 7f4f89019640 1 -- 172.21.15.19:0/1842413991 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4f840ccbb0 msgr2=0x7f4f84154140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-31T12:28:27.603 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.598+0000 7f4f89019640 1 --2- 172.21.15.19:0/1842413991 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4f840ccbb0 0x7f4f84154140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-31T12:28:27.603 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.598+0000 7f4f89019640 1 --2- 172.21.15.19:0/1842413991 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4f840ccbb0 0x7f4f84154140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-31T12:28:27.603 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.599+0000 7f4f8a01b640 1 -- 172.21.15.19:0/1842413991 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4f840ccbb0 msgr2=0x7f4f84154140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-31T12:28:27.603 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.599+0000 7f4f8a01b640 1 --2- 172.21.15.19:0/1842413991 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4f840ccbb0 0x7f4f84154140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:27.603 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.599+0000 7f4f8a01b640 1 -- 172.21.15.19:0/1842413991 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f840cc070 msgr2=0x7f4f841f2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:27.603 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.599+0000 7f4f8a01b640 1 --2- 172.21.15.19:0/1842413991 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f840cc070 0x7f4f841f2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:27.603 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.599+0000 7f4f8a01b640 1 -- 172.21.15.19:0/1842413991 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f840cf650 con 0x7f4f8415b120 2024-08-31T12:28:27.603 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.599+0000 7f4f8a01b640 1 --2- 172.21.15.19:0/1842413991 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4f8415b120 0x7f4f84145fd0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c034040 tx=0x7f4f7c0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:27.604 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.599+0000 7f4f7a7fc640 1 -- 172.21.15.19:0/1842413991 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f7c03e030 con 0x7f4f8415b120 2024-08-31T12:28:27.604 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.599+0000 7f4f7a7fc640 1 -- 172.21.15.19:0/1842413991 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4f7c003d90 con 0x7f4f8415b120 2024-08-31T12:28:27.604 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.599+0000 7f4f8ba7e640 1 -- 172.21.15.19:0/1842413991 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f840cfb90 con 0x7f4f8415b120 2024-08-31T12:28:27.604 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.599+0000 7f4f7a7fc640 1 -- 172.21.15.19:0/1842413991 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f7c035040 con 0x7f4f8415b120 2024-08-31T12:28:27.604 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.599+0000 7f4f8ba7e640 1 -- 172.21.15.19:0/1842413991 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f8415df00 con 0x7f4f8415b120 2024-08-31T12:28:27.604 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.600+0000 7f4f8ba7e640 1 -- 172.21.15.19:0/1842413991 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f840cfd60 con 0x7f4f8415b120 2024-08-31T12:28:27.605 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.601+0000 7f4f7a7fc640 1 -- 172.21.15.19:0/1842413991 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f4f7c041030 con 0x7f4f8415b120 2024-08-31T12:28:27.605 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.601+0000 7f4f7a7fc640 1 --2- 172.21.15.19:0/1842413991 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4f68077a80 0x7f4f68079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:27.605 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.602+0000 7f4f8981a640 1 --2- 172.21.15.19:0/1842413991 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4f68077a80 0x7f4f68079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:27.605 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.602+0000 7f4f7a7fc640 1 -- 172.21.15.19:0/1842413991 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f4f7c0b9d00 con 0x7f4f8415b120 2024-08-31T12:28:27.605 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.602+0000 7f4f8981a640 1 --2- 172.21.15.19:0/1842413991 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4f68077a80 0x7f4f68079f90 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f4f80005fd0 tx=0x7f4f80009680 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:27.607 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.606+0000 7f4f7a7fc640 1 -- 172.21.15.19:0/1842413991 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f4f7c0823a0 con 0x7f4f8415b120 2024-08-31T12:28:27.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.862+0000 7f4f8ba7e640 1 -- 172.21.15.19:0/1842413991 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4f8406e1a0 con 0x7f4f8415b120 2024-08-31T12:28:27.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.863+0000 7f4f7a7fc640 1 -- 172.21.15.19:0/1842413991 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f4f7c086b60 con 0x7f4f8415b120 2024-08-31T12:28:27.865 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":96,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:28:27.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.865+0000 7f4f8ba7e640 1 -- 172.21.15.19:0/1842413991 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4f68077a80 msgr2=0x7f4f68079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:27.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.865+0000 7f4f8ba7e640 1 --2- 172.21.15.19:0/1842413991 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4f68077a80 0x7f4f68079f90 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f4f80005fd0 tx=0x7f4f80009680 comp rx=0 tx=0).stop 2024-08-31T12:28:27.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.865+0000 7f4f8ba7e640 1 -- 172.21.15.19:0/1842413991 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4f8415b120 msgr2=0x7f4f84145fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:27.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.865+0000 7f4f8ba7e640 1 --2- 172.21.15.19:0/1842413991 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4f8415b120 0x7f4f84145fd0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c034040 tx=0x7f4f7c0038a0 comp rx=0 tx=0).stop 2024-08-31T12:28:27.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.865+0000 7f4f8ba7e640 1 -- 172.21.15.19:0/1842413991 shutdown_connections 2024-08-31T12:28:27.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.865+0000 7f4f8ba7e640 1 --2- 172.21.15.19:0/1842413991 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4f68077a80 0x7f4f68079f90 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:27.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.865+0000 7f4f8ba7e640 1 --2- 172.21.15.19:0/1842413991 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4f8415b120 0x7f4f84145fd0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:27.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.865+0000 7f4f8ba7e640 1 --2- 172.21.15.19:0/1842413991 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4f840ccbb0 0x7f4f84154140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:27.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.865+0000 7f4f8ba7e640 1 --2- 172.21.15.19:0/1842413991 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f840cc070 0x7f4f841f2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:27.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.865+0000 7f4f8ba7e640 1 -- 172.21.15.19:0/1842413991 >> 172.21.15.19:0/1842413991 conn(0x7f4f84074810 msgr2=0x7f4f840c6530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:27.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.865+0000 7f4f8ba7e640 1 -- 172.21.15.19:0/1842413991 shutdown_connections 2024-08-31T12:28:27.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:27.865+0000 7f4f8ba7e640 1 -- 172.21.15.19:0/1842413991 wait complete. 2024-08-31T12:28:28.340 DEBUG:teuthology.orchestra.run.smithi019:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:28:28.343 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:28:30.878 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:28:31.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.006+0000 7fd2912b6640 1 -- 172.21.15.19:0/1841748622 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd278004d90 msgr2=0x7fd278005260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:31.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.006+0000 7fd2912b6640 1 --2- 172.21.15.19:0/1841748622 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd278004d90 0x7fd278005260 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd284005fd0 tx=0x7fd278005830 comp rx=0 tx=0).stop 2024-08-31T12:28:31.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.006+0000 7fd28b7fe640 1 -- 172.21.15.19:0/1841748622 reap_dead start 2024-08-31T12:28:31.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.007+0000 7fd2912b6640 1 -- 172.21.15.19:0/1841748622 shutdown_connections 2024-08-31T12:28:31.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.007+0000 7fd2912b6640 1 -- 172.21.15.19:0/1841748622 >> 172.21.15.19:0/1841748622 conn(0x7fd28c074810 msgr2=0x7fd28c068aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:31.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.007+0000 7fd2912b6640 1 -- 172.21.15.19:0/1841748622 shutdown_connections 2024-08-31T12:28:31.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.007+0000 7fd2912b6640 1 -- 172.21.15.19:0/1841748622 wait complete. 2024-08-31T12:28:31.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.008+0000 7fd2912b6640 1 Processor -- start 2024-08-31T12:28:31.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.009+0000 7fd2912b6640 1 -- start start 2024-08-31T12:28:31.012 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.010+0000 7fd2912b6640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd278004d90 0x7fd28c15e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:31.012 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.010+0000 7fd2912b6640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd28c1f4f00 0x7fd28c1502c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:31.012 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.010+0000 7fd2912b6640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd28c0ca300 0x7fd28c150800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:31.012 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.010+0000 7fd2912b6640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fd28c0cc780 con 0x7fd278004d90 2024-08-31T12:28:31.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.010+0000 7fd2912b6640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd28c15f050 con 0x7fd28c1f4f00 2024-08-31T12:28:31.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.010+0000 7fd2912b6640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fd28c0cc900 con 0x7fd28c0ca300 2024-08-31T12:28:31.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.010+0000 7fd28affd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd278004d90 0x7fd28c15e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:31.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.010+0000 7fd28affd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd278004d90 0x7fd28c15e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:58306/0 (socket says 172.21.15.19:58306) 2024-08-31T12:28:31.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.010+0000 7fd28affd640 1 -- 172.21.15.19:0/2947454046 learned_addr learned my addr 172.21.15.19:0/2947454046 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:28:31.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.010+0000 7fd28a7fc640 1 --2- 172.21.15.19:0/2947454046 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd28c1f4f00 0x7fd28c1502c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:31.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.010+0000 7fd28affd640 1 -- 172.21.15.19:0/2947454046 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd28c0ca300 msgr2=0x7fd28c150800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:31.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.010+0000 7fd28b7fe640 1 --2- 172.21.15.19:0/2947454046 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd28c0ca300 0x7fd28c150800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:31.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.010+0000 7fd28affd640 1 --2- 172.21.15.19:0/2947454046 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd28c0ca300 0x7fd28c150800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:31.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.010+0000 7fd28affd640 1 -- 172.21.15.19:0/2947454046 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd28c1f4f00 msgr2=0x7fd28c1502c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:31.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.010+0000 7fd28affd640 1 --2- 172.21.15.19:0/2947454046 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd28c1f4f00 0x7fd28c1502c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:31.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.010+0000 7fd28affd640 1 -- 172.21.15.19:0/2947454046 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd28c15d040 con 0x7fd278004d90 2024-08-31T12:28:31.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.010+0000 7fd28a7fc640 1 --2- 172.21.15.19:0/2947454046 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd28c1f4f00 0x7fd28c1502c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-31T12:28:31.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.010+0000 7fd28affd640 1 --2- 172.21.15.19:0/2947454046 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd278004d90 0x7fd28c15e080 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd27c001650 tx=0x7fd27c0039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:31.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.011+0000 7fd2837fe640 1 -- 172.21.15.19:0/2947454046 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd27c015070 con 0x7fd278004d90 2024-08-31T12:28:31.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.011+0000 7fd2837fe640 1 -- 172.21.15.19:0/2947454046 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd27c011840 con 0x7fd278004d90 2024-08-31T12:28:31.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.011+0000 7fd2912b6640 1 -- 172.21.15.19:0/2947454046 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd28c1f1a20 con 0x7fd278004d90 2024-08-31T12:28:31.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.011+0000 7fd2837fe640 1 -- 172.21.15.19:0/2947454046 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd27c011c70 con 0x7fd278004d90 2024-08-31T12:28:31.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.011+0000 7fd2912b6640 1 -- 172.21.15.19:0/2947454046 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd28c1f1fe0 con 0x7fd278004d90 2024-08-31T12:28:31.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.012+0000 7fd2912b6640 1 -- 172.21.15.19:0/2947454046 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd28c0cebd0 con 0x7fd278004d90 2024-08-31T12:28:31.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.012+0000 7fd2837fe640 1 -- 172.21.15.19:0/2947454046 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fd27c018070 con 0x7fd278004d90 2024-08-31T12:28:31.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.013+0000 7fd2837fe640 1 --2- 172.21.15.19:0/2947454046 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd26c077a80 0x7fd26c079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:31.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.013+0000 7fd2837fe640 1 -- 172.21.15.19:0/2947454046 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fd27c09a360 con 0x7fd278004d90 2024-08-31T12:28:31.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.013+0000 7fd28a7fc640 1 --2- 172.21.15.19:0/2947454046 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd26c077a80 0x7fd26c079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:31.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.014+0000 7fd28a7fc640 1 --2- 172.21.15.19:0/2947454046 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd26c077a80 0x7fd26c079f90 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd274006930 tx=0x7fd2740068a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:31.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.018+0000 7fd2837fe640 1 -- 172.21.15.19:0/2947454046 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fd28c0cebd0 con 0x7fd278004d90 2024-08-31T12:28:31.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.275+0000 7fd2912b6640 1 -- 172.21.15.19:0/2947454046 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd28c155660 con 0x7fd278004d90 2024-08-31T12:28:31.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.275+0000 7fd2837fe640 1 -- 172.21.15.19:0/2947454046 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7fd28c0cb690 con 0x7fd278004d90 2024-08-31T12:28:31.277 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":96,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":25,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:28:31.279 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.278+0000 7fd2912b6640 1 -- 172.21.15.19:0/2947454046 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd26c077a80 msgr2=0x7fd26c079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:31.279 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.278+0000 7fd2912b6640 1 --2- 172.21.15.19:0/2947454046 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd26c077a80 0x7fd26c079f90 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd274006930 tx=0x7fd2740068a0 comp rx=0 tx=0).stop 2024-08-31T12:28:31.279 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.278+0000 7fd2912b6640 1 -- 172.21.15.19:0/2947454046 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd278004d90 msgr2=0x7fd28c15e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:31.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.278+0000 7fd2912b6640 1 --2- 172.21.15.19:0/2947454046 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd278004d90 0x7fd28c15e080 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd27c001650 tx=0x7fd27c0039c0 comp rx=0 tx=0).stop 2024-08-31T12:28:31.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.278+0000 7fd2912b6640 1 -- 172.21.15.19:0/2947454046 shutdown_connections 2024-08-31T12:28:31.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.278+0000 7fd2912b6640 1 --2- 172.21.15.19:0/2947454046 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd26c077a80 0x7fd26c079f90 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:31.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.278+0000 7fd2912b6640 1 --2- 172.21.15.19:0/2947454046 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd28c0ca300 0x7fd28c150800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:31.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.278+0000 7fd2912b6640 1 --2- 172.21.15.19:0/2947454046 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd28c1f4f00 0x7fd28c1502c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:31.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.278+0000 7fd2912b6640 1 --2- 172.21.15.19:0/2947454046 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd278004d90 0x7fd28c15e080 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:31.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.278+0000 7fd2912b6640 1 -- 172.21.15.19:0/2947454046 >> 172.21.15.19:0/2947454046 conn(0x7fd28c074810 msgr2=0x7fd28c152530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:31.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.278+0000 7fd2912b6640 1 -- 172.21.15.19:0/2947454046 shutdown_connections 2024-08-31T12:28:31.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:31.278+0000 7fd2912b6640 1 -- 172.21.15.19:0/2947454046 wait complete. 2024-08-31T12:28:34.291 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:28:34.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.418+0000 7f6902607640 1 -- 172.21.15.19:0/2408348473 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f68fc0b3af0 msgr2=0x7f68fc0ccbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:34.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.418+0000 7f6902607640 1 --2- 172.21.15.19:0/2408348473 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f68fc0b3af0 0x7f68fc0ccbd0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f68ec005fd0 tx=0x7f68ec02a670 comp rx=0 tx=0).stop 2024-08-31T12:28:34.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.418+0000 7f6902607640 1 -- 172.21.15.19:0/2408348473 shutdown_connections 2024-08-31T12:28:34.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.418+0000 7f6902607640 1 --2- 172.21.15.19:0/2408348473 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f68fc0b3af0 0x7f68fc0ccbd0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:34.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.418+0000 7f6902607640 1 -- 172.21.15.19:0/2408348473 >> 172.21.15.19:0/2408348473 conn(0x7f68fc06db80 msgr2=0x7f68fc06e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:34.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.418+0000 7f6902607640 1 -- 172.21.15.19:0/2408348473 shutdown_connections 2024-08-31T12:28:34.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.418+0000 7f6902607640 1 -- 172.21.15.19:0/2408348473 wait complete. 2024-08-31T12:28:34.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.419+0000 7f6902607640 1 Processor -- start 2024-08-31T12:28:34.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.420+0000 7f6902607640 1 -- start start 2024-08-31T12:28:34.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.421+0000 7f6902607640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f68fc0b3af0 0x7f68fc1f2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:34.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.421+0000 7f6902607640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f68fc0cb340 0x7f68fc152300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:34.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.421+0000 7f6902607640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f68fc155600 0x7f68fc1e54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:34.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.421+0000 7f6902607640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f68fc15b860 con 0x7f68fc155600 2024-08-31T12:28:34.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.421+0000 7f6902607640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f68fc15bb40 con 0x7f68fc0cb340 2024-08-31T12:28:34.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.421+0000 7f6902607640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f68fc0cbcb0 con 0x7f68fc0b3af0 2024-08-31T12:28:34.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.421+0000 7f6900ba4640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f68fc155600 0x7f68fc1e54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:34.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.421+0000 7f6900ba4640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f68fc155600 0x7f68fc1e54c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:58326/0 (socket says 172.21.15.19:58326) 2024-08-31T12:28:34.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.421+0000 7f6900ba4640 1 -- 172.21.15.19:0/3885648606 learned_addr learned my addr 172.21.15.19:0/3885648606 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:28:34.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.421+0000 7f68fbfff640 1 --2- 172.21.15.19:0/3885648606 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f68fc0b3af0 0x7f68fc1f2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:34.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.421+0000 7f68fb7fe640 1 --2- 172.21.15.19:0/3885648606 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f68fc0cb340 0x7f68fc152300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:34.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.422+0000 7f6900ba4640 1 -- 172.21.15.19:0/3885648606 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f68fc0b3af0 msgr2=0x7f68fc1f2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:34.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.422+0000 7f6900ba4640 1 --2- 172.21.15.19:0/3885648606 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f68fc0b3af0 0x7f68fc1f2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:34.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.422+0000 7f6900ba4640 1 -- 172.21.15.19:0/3885648606 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f68fc0cb340 msgr2=0x7f68fc152300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:34.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.422+0000 7f6900ba4640 1 --2- 172.21.15.19:0/3885648606 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f68fc0cb340 0x7f68fc152300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:34.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.422+0000 7f6900ba4640 1 -- 172.21.15.19:0/3885648606 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68fc152840 con 0x7f68fc155600 2024-08-31T12:28:34.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.422+0000 7f6900ba4640 1 --2- 172.21.15.19:0/3885648606 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f68fc155600 0x7f68fc1e54c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f68f400be90 tx=0x7f68f400bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:34.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.422+0000 7f68f8ff9640 1 -- 172.21.15.19:0/3885648606 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f4019070 con 0x7f68fc155600 2024-08-31T12:28:34.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.422+0000 7f68f8ff9640 1 -- 172.21.15.19:0/3885648606 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f68f4014440 con 0x7f68fc155600 2024-08-31T12:28:34.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.422+0000 7f6902607640 1 -- 172.21.15.19:0/3885648606 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68fc151260 con 0x7f68fc155600 2024-08-31T12:28:34.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.422+0000 7f68f8ff9640 1 -- 172.21.15.19:0/3885648606 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f4014890 con 0x7f68fc155600 2024-08-31T12:28:34.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.422+0000 7f6902607640 1 -- 172.21.15.19:0/3885648606 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68fc151820 con 0x7f68fc155600 2024-08-31T12:28:34.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.423+0000 7f6902607640 1 -- 172.21.15.19:0/3885648606 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68fc0c7920 con 0x7f68fc155600 2024-08-31T12:28:34.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.424+0000 7f68f8ff9640 1 -- 172.21.15.19:0/3885648606 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f68f401c070 con 0x7f68fc155600 2024-08-31T12:28:34.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.424+0000 7f68f8ff9640 1 --2- 172.21.15.19:0/3885648606 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f68e8077a50 0x7f68e8079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:34.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.424+0000 7f68f8ff9640 1 -- 172.21.15.19:0/3885648606 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f68f4099640 con 0x7f68fc155600 2024-08-31T12:28:34.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.424+0000 7f68fbfff640 1 --2- 172.21.15.19:0/3885648606 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f68e8077a50 0x7f68e8079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:34.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.425+0000 7f68fbfff640 1 --2- 172.21.15.19:0/3885648606 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f68e8077a50 0x7f68e8079f60 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f68ec033000 tx=0x7f68ec004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:34.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.429+0000 7f68f8ff9640 1 -- 172.21.15.19:0/3885648606 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f68f4061ce0 con 0x7f68fc155600 2024-08-31T12:28:34.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.690+0000 7f6902607640 1 -- 172.21.15.19:0/3885648606 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f68fc0c6dc0 con 0x7f68fc155600 2024-08-31T12:28:34.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.690+0000 7f68f8ff9640 1 -- 172.21.15.19:0/3885648606 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f68f40664a0 con 0x7f68fc155600 2024-08-31T12:28:34.692 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":98,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:28:34.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.693+0000 7f6902607640 1 -- 172.21.15.19:0/3885648606 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f68e8077a50 msgr2=0x7f68e8079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:34.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.693+0000 7f6902607640 1 --2- 172.21.15.19:0/3885648606 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f68e8077a50 0x7f68e8079f60 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f68ec033000 tx=0x7f68ec004730 comp rx=0 tx=0).stop 2024-08-31T12:28:34.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.693+0000 7f6902607640 1 -- 172.21.15.19:0/3885648606 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f68fc155600 msgr2=0x7f68fc1e54c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:34.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.693+0000 7f6902607640 1 --2- 172.21.15.19:0/3885648606 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f68fc155600 0x7f68fc1e54c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f68f400be90 tx=0x7f68f400bf80 comp rx=0 tx=0).stop 2024-08-31T12:28:34.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.693+0000 7f6902607640 1 -- 172.21.15.19:0/3885648606 shutdown_connections 2024-08-31T12:28:34.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.693+0000 7f6902607640 1 --2- 172.21.15.19:0/3885648606 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f68e8077a50 0x7f68e8079f60 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:34.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.693+0000 7f6902607640 1 --2- 172.21.15.19:0/3885648606 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f68fc155600 0x7f68fc1e54c0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:34.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.693+0000 7f6902607640 1 --2- 172.21.15.19:0/3885648606 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f68fc0cb340 0x7f68fc152300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:34.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.693+0000 7f6902607640 1 --2- 172.21.15.19:0/3885648606 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f68fc0b3af0 0x7f68fc1f2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:34.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.693+0000 7f6902607640 1 -- 172.21.15.19:0/3885648606 >> 172.21.15.19:0/3885648606 conn(0x7f68fc06db80 msgr2=0x7f68fc0b33b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:34.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.693+0000 7f6902607640 1 -- 172.21.15.19:0/3885648606 shutdown_connections 2024-08-31T12:28:34.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.693+0000 7f6902607640 1 -- 172.21.15.19:0/3885648606 wait complete. 2024-08-31T12:28:34.705 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-08-31T12:28:34.706 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-08-31T12:28:34.706 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-31T12:28:34.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.823+0000 7f796790b640 1 -- 172.21.15.19:0/1101111885 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f794c003a30 msgr2=0x7f794c003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:34.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.823+0000 7f796790b640 1 --2- 172.21.15.19:0/1101111885 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f794c003a30 0x7f794c003ea0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f795c0016c0 tx=0x7f794c005830 comp rx=0 tx=0).stop 2024-08-31T12:28:34.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.823+0000 7f7965ea8640 1 -- 172.21.15.19:0/1101111885 reap_dead start 2024-08-31T12:28:34.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.823+0000 7f796790b640 1 -- 172.21.15.19:0/1101111885 shutdown_connections 2024-08-31T12:28:34.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.823+0000 7f796790b640 1 -- 172.21.15.19:0/1101111885 >> 172.21.15.19:0/1101111885 conn(0x7f79600728b0 msgr2=0x7f7960072cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:34.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.823+0000 7f796790b640 1 -- 172.21.15.19:0/1101111885 shutdown_connections 2024-08-31T12:28:34.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.823+0000 7f796790b640 1 -- 172.21.15.19:0/1101111885 wait complete. 2024-08-31T12:28:34.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.825+0000 7f796790b640 1 Processor -- start 2024-08-31T12:28:34.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.825+0000 7f796790b640 1 -- start start 2024-08-31T12:28:34.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.826+0000 7f796790b640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f794c003a30 0x7f79601e3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:34.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.826+0000 7f796790b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7960153760 0x7f7960155fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:34.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.826+0000 7f796790b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f79601e3c80 0x7f7960150070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:34.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.826+0000 7f796790b640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f79601535a0 con 0x7f7960153760 2024-08-31T12:28:34.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.826+0000 7f796790b640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f79600c4db0 con 0x7f794c003a30 2024-08-31T12:28:34.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.826+0000 7f796790b640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f7960159c30 con 0x7f79601e3c80 2024-08-31T12:28:34.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.826+0000 7f7964ea6640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7960153760 0x7f7960155fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:34.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.826+0000 7f7964ea6640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7960153760 0x7f7960155fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:58358/0 (socket says 172.21.15.19:58358) 2024-08-31T12:28:34.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.826+0000 7f7964ea6640 1 -- 172.21.15.19:0/3705448254 learned_addr learned my addr 172.21.15.19:0/3705448254 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:28:34.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.826+0000 7f7965ea8640 1 --2- 172.21.15.19:0/3705448254 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f79601e3c80 0x7f7960150070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:34.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.826+0000 7f79656a7640 1 --2- 172.21.15.19:0/3705448254 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f794c003a30 0x7f79601e3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:34.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.826+0000 7f7964ea6640 1 -- 172.21.15.19:0/3705448254 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f79601e3c80 msgr2=0x7f7960150070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:34.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.826+0000 7f7964ea6640 1 --2- 172.21.15.19:0/3705448254 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f79601e3c80 0x7f7960150070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:34.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.826+0000 7f7964ea6640 1 -- 172.21.15.19:0/3705448254 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f794c003a30 msgr2=0x7f79601e3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:34.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.826+0000 7f7964ea6640 1 --2- 172.21.15.19:0/3705448254 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f794c003a30 0x7f79601e3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:34.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.826+0000 7f7964ea6640 1 -- 172.21.15.19:0/3705448254 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79600ceb40 con 0x7f7960153760 2024-08-31T12:28:34.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.827+0000 7f7964ea6640 1 --2- 172.21.15.19:0/3705448254 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7960153760 0x7f7960155fd0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f795000fa00 tx=0x7f795000fec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:34.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.827+0000 7f7955ffb640 1 -- 172.21.15.19:0/3705448254 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f795000bcb0 con 0x7f7960153760 2024-08-31T12:28:34.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.827+0000 7f7955ffb640 1 -- 172.21.15.19:0/3705448254 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7950011440 con 0x7f7960153760 2024-08-31T12:28:34.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.827+0000 7f796790b640 1 -- 172.21.15.19:0/3705448254 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79600cf090 con 0x7f7960153760 2024-08-31T12:28:34.831 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.827+0000 7f7955ffb640 1 -- 172.21.15.19:0/3705448254 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7950011870 con 0x7f7960153760 2024-08-31T12:28:34.831 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.827+0000 7f796790b640 1 -- 172.21.15.19:0/3705448254 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79600cedb0 con 0x7f7960153760 2024-08-31T12:28:34.832 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.828+0000 7f796790b640 1 --2- 172.21.15.19:0/3705448254 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79601505b0 0x7f796002d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:34.832 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.828+0000 7f796790b640 1 -- 172.21.15.19:0/3705448254 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f79601e41d0 con 0x7f79601505b0 2024-08-31T12:28:34.832 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.828+0000 7f79656a7640 1 --2- 172.21.15.19:0/3705448254 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79601505b0 0x7f796002d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:34.833 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.828+0000 7f79656a7640 1 -- 172.21.15.19:0/3705448254 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f79601509c0 con 0x7f79601505b0 2024-08-31T12:28:34.833 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.828+0000 7f79656a7640 1 --2- 172.21.15.19:0/3705448254 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79601505b0 0x7f796002d2b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f795c001690 tx=0x7f795c006290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:34.833 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.829+0000 7f7955ffb640 1 -- 172.21.15.19:0/3705448254 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f7950019020 con 0x7f7960153760 2024-08-31T12:28:34.833 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.829+0000 7f7955ffb640 1 --2- 172.21.15.19:0/3705448254 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7944077a50 0x7f7944079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:34.833 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.829+0000 7f7955ffb640 1 -- 172.21.15.19:0/3705448254 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f795009a180 con 0x7f7960153760 2024-08-31T12:28:34.833 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.829+0000 7f7955ffb640 1 -- 172.21.15.19:0/3705448254 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f795c042070 con 0x7f79601505b0 2024-08-31T12:28:34.833 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.829+0000 7f7955ffb640 1 -- 172.21.15.19:0/3705448254 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f795c039070 con 0x7f79601505b0 2024-08-31T12:28:34.834 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.829+0000 7f7955ffb640 1 -- 172.21.15.19:0/3705448254 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79601505b0 msgr2=0x7f796002d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:34.834 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.829+0000 7f7965ea8640 1 --2- 172.21.15.19:0/3705448254 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7944077a50 0x7f7944079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:34.834 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.829+0000 7f7955ffb640 1 --2- 172.21.15.19:0/3705448254 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79601505b0 0x7f796002d2b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f795c001690 tx=0x7f795c006290 comp rx=0 tx=0).stop 2024-08-31T12:28:34.834 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.829+0000 7f7955ffb640 1 -- 172.21.15.19:0/3705448254 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79601505b0 msgr2=0x7f796002d2b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:28:34.834 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.829+0000 7f7955ffb640 1 --2- 172.21.15.19:0/3705448254 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79601505b0 0x7f796002d2b0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:34.834 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.830+0000 7f7965ea8640 1 --2- 172.21.15.19:0/3705448254 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7944077a50 0x7f7944079f60 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f794c00c0d0 tx=0x7f79580018c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:34.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.836+0000 7f796790b640 1 --2- 172.21.15.19:0/3705448254 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79601e3050 0x7f796002d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:34.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.836+0000 7f796790b640 1 -- 172.21.15.19:0/3705448254 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f79601e4630 con 0x7f79601e3050 2024-08-31T12:28:34.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.836+0000 7f79656a7640 1 --2- 172.21.15.19:0/3705448254 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79601e3050 0x7f796002d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:34.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.836+0000 7f79656a7640 1 -- 172.21.15.19:0/3705448254 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7960071350 con 0x7f79601e3050 2024-08-31T12:28:34.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.836+0000 7f79656a7640 1 --2- 172.21.15.19:0/3705448254 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79601e3050 0x7f796002d7f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f795c0016c0 tx=0x7f795c006b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:34.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.836+0000 7f7955ffb640 1 -- 172.21.15.19:0/3705448254 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f795c042030 con 0x7f79601e3050 2024-08-31T12:28:34.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.837+0000 7f7955ffb640 1 -- 172.21.15.19:0/3705448254 <== mon.0 v2:172.21.15.19:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5044 (secure 0 0 0) 0x7f795c02e040 con 0x7f79601e3050 2024-08-31T12:28:34.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.837+0000 7f7955ffb640 1 -- 172.21.15.19:0/3705448254 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79601e3050 msgr2=0x7f796002d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:34.839 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.837+0000 7f7955ffb640 1 --2- 172.21.15.19:0/3705448254 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79601e3050 0x7f796002d7f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f795c0016c0 tx=0x7f795c006b70 comp rx=0 tx=0).stop 2024-08-31T12:28:34.839 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.837+0000 7f7955ffb640 1 -- 172.21.15.19:0/3705448254 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79601e3050 msgr2=0x7f796002d7f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:28:34.839 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.837+0000 7f7955ffb640 1 --2- 172.21.15.19:0/3705448254 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79601e3050 0x7f796002d7f0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:34.839 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:28:34.839 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:28:34.839 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:28:34.840 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "leader", 2024-08-31T12:28:34.840 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 98, 2024-08-31T12:28:34.840 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 249390, 2024-08-31T12:28:34.840 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:28:34.840 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:28:34.840 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:28:34.840 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:28:34.840 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:28:34.841 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 3, 2024-08-31T12:28:34.841 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:28:34.841 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:28:34.841 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:28:34.841 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:28:34.841 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:28:34.841 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:28:34.841 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:28:34.842 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:28:34.842 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:28:34.842 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:28:34.842 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:28:34.842 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:28:34.842 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:28:34.842 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:28:34.842 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:28:34.842 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:28:34.843 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:28:34.843 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:28:34.843 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:28:34.843 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:28:34.843 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:28:34.843 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:28:34.843 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:28:34.843 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:28:34.843 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:28:34.844 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:28:34.844 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:28:34.844 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:28:34.844 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:28:34.844 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:34.844 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:28:34.844 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:28:34.844 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:28:34.844 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:28:34.845 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:28:34.845 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:28:34.845 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:28:34.845 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:28:34.845 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:28:34.845 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:28:34.845 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:28:34.845 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:28:34.846 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:28:34.846 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:28:34.846 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:28:34.846 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:28:34.846 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:28:34.846 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:28:34.846 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:28:34.846 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:28:34.846 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:28:34.847 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:28:34.847 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:28:34.847 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:28:34.847 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:28:34.847 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:28:34.847 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:28:34.847 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:28:34.847 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:28:34.847 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:28:34.847 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:34.848 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:28:34.848 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:34.848 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:28:34.848 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:28:34.848 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:28:34.848 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:28:34.848 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:34.848 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:28:34.849 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:28:34.849 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:28:34.849 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:34.849 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:34.849 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:28:34.849 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:28:34.849 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:28:34.849 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:28:34.850 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:28:34.850 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:34.850 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:28:34.850 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:28:34.850 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:28:34.850 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:28:34.850 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:28:34.850 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:34.850 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:34.851 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:28:34.851 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:28:34.851 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:28:34.851 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:28:34.851 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:34.851 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:28:34.851 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:28:34.851 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:28:34.852 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:34.852 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:34.852 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:28:34.852 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:28:34.852 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:28:34.852 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:28:34.852 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:28:34.852 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:34.852 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:28:34.853 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:28:34.853 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:28:34.853 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:28:34.853 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:28:34.853 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:34.853 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:34.853 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:28:34.853 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:28:34.854 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:28:34.854 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:28:34.854 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:34.854 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:28:34.854 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:28:34.854 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:28:34.854 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:34.854 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:34.855 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:28:34.855 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:28:34.855 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:28:34.855 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:28:34.855 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:28:34.855 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:34.855 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:28:34.855 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:28:34.856 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:28:34.856 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:28:34.856 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:28:34.856 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:28:34.856 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:28:34.856 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:34.856 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:28:34.856 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:28:34.856 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:34.857 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:28:34.857 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:28:34.857 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:28:34.857 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:28:34.857 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:28:34.857 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:28:34.857 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:34.857 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:28:34.857 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:28:34.858 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 5 2024-08-31T12:28:34.858 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:28:34.858 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:28:34.858 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:28:34.858 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:34.858 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:28:34.858 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:28:34.858 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 7 2024-08-31T12:28:34.858 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:28:34.859 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:28:34.859 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:28:34.859 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:34.859 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:28:34.859 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:28:34.859 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 7 2024-08-31T12:28:34.859 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:28:34.859 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:28:34.859 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": [ 2024-08-31T12:28:34.860 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:34.860 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:28:34.860 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:28:34.860 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 2 2024-08-31T12:28:34.860 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:28:34.860 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:28:34.860 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:34.860 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:28:34.860 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:28:34.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.837+0000 7f796790b640 1 -- 172.21.15.19:0/3705448254 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7944077a50 msgr2=0x7f7944079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:34.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.837+0000 7f796790b640 1 --2- 172.21.15.19:0/3705448254 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7944077a50 0x7f7944079f60 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f794c00c0d0 tx=0x7f79580018c0 comp rx=0 tx=0).stop 2024-08-31T12:28:34.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.837+0000 7f796790b640 1 -- 172.21.15.19:0/3705448254 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7960153760 msgr2=0x7f7960155fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:34.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.837+0000 7f796790b640 1 --2- 172.21.15.19:0/3705448254 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7960153760 0x7f7960155fd0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f795000fa00 tx=0x7f795000fec0 comp rx=0 tx=0).stop 2024-08-31T12:28:34.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.837+0000 7f7965ea8640 1 -- 172.21.15.19:0/3705448254 reap_dead start 2024-08-31T12:28:34.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.837+0000 7f7965ea8640 1 -- 172.21.15.19:0/3705448254 reap_dead start 2024-08-31T12:28:34.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.837+0000 7f796790b640 1 -- 172.21.15.19:0/3705448254 shutdown_connections 2024-08-31T12:28:34.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.837+0000 7f796790b640 1 -- 172.21.15.19:0/3705448254 >> 172.21.15.19:0/3705448254 conn(0x7f79600728b0 msgr2=0x7f7960072cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:34.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.837+0000 7f796790b640 1 -- 172.21.15.19:0/3705448254 shutdown_connections 2024-08-31T12:28:34.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.837+0000 7f796790b640 1 -- 172.21.15.19:0/3705448254 wait complete. 2024-08-31T12:28:34.863 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:28:34.863 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-08-31T12:28:34.864 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-31T12:28:34.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.978+0000 7f9521b62640 1 -- 172.21.15.19:0/4157861240 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f951c155300 msgr2=0x7f951c06d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:34.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.978+0000 7f9521b62640 1 --2- 172.21.15.19:0/4157861240 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f951c155300 0x7f951c06d000 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f951c15b0f0 tx=0x7f951402aa90 comp rx=0 tx=0).stop 2024-08-31T12:28:34.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.978+0000 7f9521b62640 1 -- 172.21.15.19:0/4157861240 shutdown_connections 2024-08-31T12:28:34.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.978+0000 7f9521b62640 1 --2- 172.21.15.19:0/4157861240 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f951c155300 0x7f951c06d000 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:34.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.978+0000 7f9521b62640 1 --2- 172.21.15.19:0/4157861240 >> v2:172.21.15.43:3300/0 conn(0x7f951c0c8a40 0x7f951c0c9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:34.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.978+0000 7f9521b62640 1 -- 172.21.15.19:0/4157861240 >> 172.21.15.19:0/4157861240 conn(0x7f951c0690d0 msgr2=0x7f951c0d0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:34.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.978+0000 7f9521b62640 1 -- 172.21.15.19:0/4157861240 shutdown_connections 2024-08-31T12:28:34.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.978+0000 7f9521b62640 1 -- 172.21.15.19:0/4157861240 wait complete. 2024-08-31T12:28:34.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.979+0000 7f9521b62640 1 Processor -- start 2024-08-31T12:28:34.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.980+0000 7f9521b62640 1 -- start start 2024-08-31T12:28:34.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.981+0000 7f9521b62640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f951c06e0e0 0x7f951c1541d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:34.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.981+0000 7f9521b62640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f951c0c8a40 0x7f951c1f4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:34.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.981+0000 7f9521b62640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f951c155300 0x7f951c1e50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:34.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.981+0000 7f9521b62640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f951c0c8330 con 0x7f951c155300 2024-08-31T12:28:34.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.981+0000 7f9521b62640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f951c15b250 con 0x7f951c06e0e0 2024-08-31T12:28:34.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.981+0000 7f9521b62640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f951c0c8610 con 0x7f951c0c8a40 2024-08-31T12:28:34.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.981+0000 7f951bfff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f951c155300 0x7f951c1e50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:34.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.981+0000 7f951bfff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f951c155300 0x7f951c1e50a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:58392/0 (socket says 172.21.15.19:58392) 2024-08-31T12:28:34.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.981+0000 7f951bfff640 1 -- 172.21.15.19:0/3258799340 learned_addr learned my addr 172.21.15.19:0/3258799340 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:28:34.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.981+0000 7f951b7fe640 1 --2- 172.21.15.19:0/3258799340 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f951c06e0e0 0x7f951c1541d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:34.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.981+0000 7f951affd640 1 --2- 172.21.15.19:0/3258799340 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f951c0c8a40 0x7f951c1f4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:34.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.981+0000 7f951bfff640 1 -- 172.21.15.19:0/3258799340 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f951c0c8a40 msgr2=0x7f951c1f4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:34.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.981+0000 7f951bfff640 1 --2- 172.21.15.19:0/3258799340 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f951c0c8a40 0x7f951c1f4f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:34.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.981+0000 7f951bfff640 1 -- 172.21.15.19:0/3258799340 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f951c06e0e0 msgr2=0x7f951c1541d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:34.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.981+0000 7f951bfff640 1 --2- 172.21.15.19:0/3258799340 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f951c06e0e0 0x7f951c1541d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:34.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.981+0000 7f951bfff640 1 -- 172.21.15.19:0/3258799340 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f951c0c4a60 con 0x7f951c155300 2024-08-31T12:28:34.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.981+0000 7f951b7fe640 1 --2- 172.21.15.19:0/3258799340 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f951c06e0e0 0x7f951c1541d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-31T12:28:34.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.981+0000 7f951affd640 1 --2- 172.21.15.19:0/3258799340 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f951c0c8a40 0x7f951c1f4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-31T12:28:34.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.982+0000 7f951bfff640 1 --2- 172.21.15.19:0/3258799340 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f951c155300 0x7f951c1e50a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f951402b080 tx=0x7f9514003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:34.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.982+0000 7f9513fff640 1 -- 172.21.15.19:0/3258799340 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9514004250 con 0x7f951c155300 2024-08-31T12:28:34.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.982+0000 7f9513fff640 1 -- 172.21.15.19:0/3258799340 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f95140043f0 con 0x7f951c155300 2024-08-31T12:28:34.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.982+0000 7f9521b62640 1 -- 172.21.15.19:0/3258799340 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f951c0c4d40 con 0x7f951c155300 2024-08-31T12:28:34.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.982+0000 7f9513fff640 1 -- 172.21.15.19:0/3258799340 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9514004820 con 0x7f951c155300 2024-08-31T12:28:34.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.982+0000 7f9521b62640 1 -- 172.21.15.19:0/3258799340 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f951c0c5300 con 0x7f951c155300 2024-08-31T12:28:34.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.983+0000 7f9521b62640 1 --2- 172.21.15.19:0/3258799340 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f951c1e55e0 0x7f951c0c2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:34.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.983+0000 7f9521b62640 1 -- 172.21.15.19:0/3258799340 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f951c1e4660 con 0x7f951c1e55e0 2024-08-31T12:28:34.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.983+0000 7f951b7fe640 1 --2- 172.21.15.19:0/3258799340 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f951c1e55e0 0x7f951c0c2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:34.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.983+0000 7f951b7fe640 1 -- 172.21.15.19:0/3258799340 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f951c0d1220 con 0x7f951c1e55e0 2024-08-31T12:28:34.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.983+0000 7f951b7fe640 1 --2- 172.21.15.19:0/3258799340 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f951c1e55e0 0x7f951c0c2860 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f950c00d8e0 tx=0x7f950c00dce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:34.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.983+0000 7f9513fff640 1 -- 172.21.15.19:0/3258799340 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f9514040420 con 0x7f951c155300 2024-08-31T12:28:34.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.984+0000 7f9513fff640 1 --2- 172.21.15.19:0/3258799340 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f94fc077ab0 0x7f94fc079fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:34.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.984+0000 7f9513fff640 1 -- 172.21.15.19:0/3258799340 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f95140bab60 con 0x7f951c155300 2024-08-31T12:28:34.989 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.984+0000 7f951affd640 1 --2- 172.21.15.19:0/3258799340 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f94fc077ab0 0x7f94fc079fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:34.989 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.984+0000 7f9513fff640 1 -- 172.21.15.19:0/3258799340 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f950c008c90 con 0x7f951c1e55e0 2024-08-31T12:28:34.989 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.984+0000 7f9513fff640 1 -- 172.21.15.19:0/3258799340 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f950c008e30 con 0x7f951c1e55e0 2024-08-31T12:28:34.989 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.984+0000 7f9513fff640 1 -- 172.21.15.19:0/3258799340 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f951c1e55e0 msgr2=0x7f951c0c2860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:34.989 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.984+0000 7f9513fff640 1 --2- 172.21.15.19:0/3258799340 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f951c1e55e0 0x7f951c0c2860 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f950c00d8e0 tx=0x7f950c00dce0 comp rx=0 tx=0).stop 2024-08-31T12:28:34.989 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.984+0000 7f9513fff640 1 -- 172.21.15.19:0/3258799340 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f951c1e55e0 msgr2=0x7f951c0c2860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:28:34.989 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.984+0000 7f9513fff640 1 --2- 172.21.15.19:0/3258799340 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f951c1e55e0 0x7f951c0c2860 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:34.989 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.985+0000 7f951affd640 1 --2- 172.21.15.19:0/3258799340 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f94fc077ab0 0x7f94fc079fc0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f951c1f5fd0 tx=0x7f9504008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:34.991 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.990+0000 7f9521b62640 1 --2- 172.21.15.19:0/3258799340 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f951c1f5450 0x7f951c0c2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:34.992 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.990+0000 7f9521b62640 1 -- 172.21.15.19:0/3258799340 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f951c0c54d0 con 0x7f951c1f5450 2024-08-31T12:28:34.992 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.991+0000 7f951b7fe640 1 --2- 172.21.15.19:0/3258799340 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f951c1f5450 0x7f951c0c2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:34.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.991+0000 7f951b7fe640 1 -- 172.21.15.19:0/3258799340 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f951c15a210 con 0x7f951c1f5450 2024-08-31T12:28:34.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.991+0000 7f951b7fe640 1 --2- 172.21.15.19:0/3258799340 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f951c1f5450 0x7f951c0c2da0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f950c00d8e0 tx=0x7f950c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:34.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.991+0000 7f9513fff640 1 -- 172.21.15.19:0/3258799340 <== mon.1 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f950c017db0 con 0x7f951c1f5450 2024-08-31T12:28:34.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.991+0000 7f9513fff640 1 -- 172.21.15.19:0/3258799340 <== mon.1 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4874 (secure 0 0 0) 0x7f950c01e2b0 con 0x7f951c1f5450 2024-08-31T12:28:34.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.991+0000 7f9513fff640 1 -- 172.21.15.19:0/3258799340 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f951c1f5450 msgr2=0x7f951c0c2da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:34.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.991+0000 7f9513fff640 1 --2- 172.21.15.19:0/3258799340 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f951c1f5450 0x7f951c0c2da0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f950c00d8e0 tx=0x7f950c0046c0 comp rx=0 tx=0).stop 2024-08-31T12:28:34.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.992+0000 7f9513fff640 1 -- 172.21.15.19:0/3258799340 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f951c1f5450 msgr2=0x7f951c0c2da0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:28:34.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.992+0000 7f9513fff640 1 --2- 172.21.15.19:0/3258799340 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f951c1f5450 0x7f951c0c2da0 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f950c00d8e0 tx=0x7f950c0046c0 comp rx=0 tx=0).stop 2024-08-31T12:28:34.994 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:28:34.994 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:28:34.994 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:28:34.994 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:28:34.995 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 98, 2024-08-31T12:28:34.995 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 64814, 2024-08-31T12:28:34.995 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:28:34.995 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:28:34.995 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:28:34.995 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:28:34.995 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:28:34.995 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 3, 2024-08-31T12:28:34.996 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:28:34.996 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:28:34.996 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:28:34.996 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:28:34.996 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:28:34.996 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:28:34.996 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:28:34.996 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:28:34.997 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:28:34.997 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:28:34.997 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:28:34.997 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:28:34.997 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:28:34.997 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:28:34.997 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:28:34.997 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:28:34.998 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:28:34.998 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:28:34.998 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:28:34.998 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:28:34.998 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:28:34.998 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:28:34.998 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:28:34.998 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:28:34.999 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:28:34.999 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:28:34.999 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:28:34.999 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:28:34.999 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:28:34.999 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:34.999 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:28:34.999 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:28:34.999 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:28:35.000 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:28:35.000 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:28:35.000 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:28:35.000 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:28:35.000 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:28:35.000 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:28:35.000 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:28:35.000 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:28:35.000 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:28:35.001 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:28:35.001 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:28:35.001 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:28:35.001 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:28:35.001 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:28:35.001 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:28:35.001 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:28:35.001 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:28:35.001 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:28:35.002 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:28:35.002 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:28:35.002 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:28:35.002 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:28:35.002 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:28:35.002 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:28:35.002 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:28:35.002 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:28:35.002 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:28:35.003 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:35.003 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:28:35.003 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:35.003 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:28:35.003 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:28:35.003 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:28:35.003 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:28:35.003 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:35.003 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:28:35.004 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:28:35.004 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:28:35.004 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:35.004 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:35.004 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:28:35.004 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:28:35.004 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:28:35.004 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:28:35.005 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:28:35.005 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:35.005 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:28:35.005 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:28:35.005 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:28:35.005 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:28:35.005 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:28:35.005 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:35.006 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:35.006 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:28:35.006 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:28:35.006 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:28:35.006 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:28:35.006 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:35.006 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:28:35.006 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:28:35.006 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:28:35.007 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:35.007 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:35.007 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:28:35.007 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:28:35.007 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:28:35.007 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:28:35.007 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:28:35.007 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:35.008 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:28:35.008 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:28:35.008 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:28:35.008 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:28:35.008 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:28:35.008 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:35.008 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:35.009 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:28:35.009 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:28:35.009 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:28:35.009 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:28:35.009 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:35.009 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:28:35.009 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:28:35.009 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:28:35.010 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:35.010 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:35.010 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:28:35.010 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:28:35.010 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:28:35.010 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:28:35.010 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:28:35.010 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:35.011 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:28:35.011 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:28:35.011 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:28:35.011 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:28:35.011 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:28:35.011 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:28:35.011 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:28:35.011 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:35.012 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:28:35.012 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:28:35.012 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:35.012 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:28:35.012 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:28:35.012 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:28:35.012 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:28:35.012 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:28:35.013 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": [ 2024-08-31T12:28:35.013 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:35.013 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:28:35.013 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:28:35.013 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:28:35.013 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:28:35.013 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:28:35.013 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": [ 2024-08-31T12:28:35.014 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:35.014 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:28:35.014 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:28:35.014 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:28:35.014 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:28:35.014 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:28:35.015 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:28:35.015 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:35.015 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:28:35.015 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:28:35.015 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:28:35.015 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:28:35.015 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:28:35.015 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:35.015 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:28:35.016 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:28:35.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.992+0000 7f9521b62640 1 -- 172.21.15.19:0/3258799340 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f94fc077ab0 msgr2=0x7f94fc079fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:35.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.992+0000 7f9521b62640 1 --2- 172.21.15.19:0/3258799340 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f94fc077ab0 0x7f94fc079fc0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f951c1f5fd0 tx=0x7f9504008040 comp rx=0 tx=0).stop 2024-08-31T12:28:35.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.992+0000 7f9521b62640 1 -- 172.21.15.19:0/3258799340 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f951c155300 msgr2=0x7f951c1e50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:35.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.992+0000 7f9521b62640 1 --2- 172.21.15.19:0/3258799340 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f951c155300 0x7f951c1e50a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f951402b080 tx=0x7f9514003f10 comp rx=0 tx=0).stop 2024-08-31T12:28:35.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.992+0000 7f951bfff640 1 -- 172.21.15.19:0/3258799340 reap_dead start 2024-08-31T12:28:35.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.992+0000 7f9521b62640 1 -- 172.21.15.19:0/3258799340 shutdown_connections 2024-08-31T12:28:35.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.992+0000 7f9521b62640 1 --2- 172.21.15.19:0/3258799340 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f951c155300 0x7f951c1e50a0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:35.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.992+0000 7f9521b62640 1 -- 172.21.15.19:0/3258799340 >> 172.21.15.19:0/3258799340 conn(0x7f951c0690d0 msgr2=0x7f951c06d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:35.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.992+0000 7f9521b62640 1 -- 172.21.15.19:0/3258799340 shutdown_connections 2024-08-31T12:28:35.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:34.992+0000 7f9521b62640 1 -- 172.21.15.19:0/3258799340 wait complete. 2024-08-31T12:28:35.018 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:28:35.018 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-08-31T12:28:35.018 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-31T12:28:35.134 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.132+0000 7f041029b640 1 -- 172.21.15.19:0/2278689810 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f040806e0e0 msgr2=0x7f04080c9850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:35.134 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.132+0000 7f041029b640 1 --2- 172.21.15.19:0/2278689810 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f040806e0e0 0x7f04080c9850 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f040815b0f0 tx=0x7f03f802a670 comp rx=0 tx=0).stop 2024-08-31T12:28:35.134 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.132+0000 7f041029b640 1 -- 172.21.15.19:0/2278689810 shutdown_connections 2024-08-31T12:28:35.134 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.132+0000 7f041029b640 1 --2- 172.21.15.19:0/2278689810 >> v2:172.21.15.43:3300/0 conn(0x7f0408155300 0x7f040806d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:35.134 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.132+0000 7f041029b640 1 --2- 172.21.15.19:0/2278689810 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f040806e0e0 0x7f04080c9850 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:35.134 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.132+0000 7f041029b640 1 -- 172.21.15.19:0/2278689810 >> 172.21.15.19:0/2278689810 conn(0x7f04080690d0 msgr2=0x7f04080d0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:35.134 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.132+0000 7f041029b640 1 -- 172.21.15.19:0/2278689810 shutdown_connections 2024-08-31T12:28:35.135 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.132+0000 7f041029b640 1 -- 172.21.15.19:0/2278689810 wait complete. 2024-08-31T12:28:35.135 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.134+0000 7f041029b640 1 Processor -- start 2024-08-31T12:28:35.135 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.134+0000 7f041029b640 1 -- start start 2024-08-31T12:28:35.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.135+0000 7f041029b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f040806e0e0 0x7f04080c2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:35.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.135+0000 7f041029b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f04080c8a40 0x7f04080c55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:35.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.135+0000 7f041029b640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0408155300 0x7f0408153f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:35.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.135+0000 7f041029b640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f04080c8330 con 0x7f040806e0e0 2024-08-31T12:28:35.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.135+0000 7f041029b640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f040815b250 con 0x7f0408155300 2024-08-31T12:28:35.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.135+0000 7f041029b640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f04080c8610 con 0x7f04080c8a40 2024-08-31T12:28:35.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.135+0000 7f040e037640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f040806e0e0 0x7f04080c2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:35.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.136+0000 7f040e037640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f040806e0e0 0x7f04080c2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:58414/0 (socket says 172.21.15.19:58414) 2024-08-31T12:28:35.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.136+0000 7f040e037640 1 -- 172.21.15.19:0/3197849119 learned_addr learned my addr 172.21.15.19:0/3197849119 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:28:35.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.136+0000 7f040e838640 1 --2- 172.21.15.19:0/3197849119 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0408155300 0x7f0408153f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:35.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.136+0000 7f040d836640 1 --2- 172.21.15.19:0/3197849119 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f04080c8a40 0x7f04080c55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:35.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.136+0000 7f040e037640 1 -- 172.21.15.19:0/3197849119 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f04080c8a40 msgr2=0x7f04080c55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:35.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.136+0000 7f040e037640 1 --2- 172.21.15.19:0/3197849119 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f04080c8a40 0x7f04080c55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:35.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.136+0000 7f040e037640 1 -- 172.21.15.19:0/3197849119 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0408155300 msgr2=0x7f0408153f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:35.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.136+0000 7f040e037640 1 --2- 172.21.15.19:0/3197849119 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0408155300 0x7f0408153f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:35.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.136+0000 7f040e037640 1 -- 172.21.15.19:0/3197849119 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f040815a310 con 0x7f040806e0e0 2024-08-31T12:28:35.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.136+0000 7f040e037640 1 --2- 172.21.15.19:0/3197849119 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f040806e0e0 0x7f04080c2a40 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f040400c3a0 tx=0x7f040400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:35.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.136+0000 7f03feffd640 1 -- 172.21.15.19:0/3197849119 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0404017070 con 0x7f040806e0e0 2024-08-31T12:28:35.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.136+0000 7f041029b640 1 -- 172.21.15.19:0/3197849119 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04080c4a60 con 0x7f040806e0e0 2024-08-31T12:28:35.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.136+0000 7f03feffd640 1 -- 172.21.15.19:0/3197849119 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f04040053c0 con 0x7f040806e0e0 2024-08-31T12:28:35.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.136+0000 7f03feffd640 1 -- 172.21.15.19:0/3197849119 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0404003850 con 0x7f040806e0e0 2024-08-31T12:28:35.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.136+0000 7f041029b640 1 -- 172.21.15.19:0/3197849119 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04081f1820 con 0x7f040806e0e0 2024-08-31T12:28:35.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.137+0000 7f041029b640 1 --2- 172.21.15.19:0/3197849119 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f04081f2020 0x7f04081544c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:35.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.137+0000 7f041029b640 1 -- 172.21.15.19:0/3197849119 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f04080c4810 con 0x7f04081f2020 2024-08-31T12:28:35.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.138+0000 7f040d836640 1 --2- 172.21.15.19:0/3197849119 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f04081f2020 0x7f04081544c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:35.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.138+0000 7f040d836640 1 -- 172.21.15.19:0/3197849119 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f04080c9850 con 0x7f04081f2020 2024-08-31T12:28:35.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.138+0000 7f03feffd640 1 -- 172.21.15.19:0/3197849119 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f040401a070 con 0x7f040806e0e0 2024-08-31T12:28:35.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.138+0000 7f040d836640 1 --2- 172.21.15.19:0/3197849119 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f04081f2020 0x7f04081544c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f04080d0ea0 tx=0x7f03f8003820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:35.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.138+0000 7f03feffd640 1 --2- 172.21.15.19:0/3197849119 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f03ec077a80 0x7f03ec079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:35.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.139+0000 7f03feffd640 1 -- 172.21.15.19:0/3197849119 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f0404098aa0 con 0x7f040806e0e0 2024-08-31T12:28:35.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.139+0000 7f040e838640 1 --2- 172.21.15.19:0/3197849119 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f03ec077a80 0x7f03ec079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:35.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.139+0000 7f03feffd640 1 -- 172.21.15.19:0/3197849119 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03f8040070 con 0x7f04081f2020 2024-08-31T12:28:35.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.139+0000 7f03feffd640 1 -- 172.21.15.19:0/3197849119 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f03f8003df0 con 0x7f04081f2020 2024-08-31T12:28:35.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.139+0000 7f03feffd640 1 -- 172.21.15.19:0/3197849119 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f04081f2020 msgr2=0x7f04081544c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:35.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.139+0000 7f03feffd640 1 --2- 172.21.15.19:0/3197849119 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f04081f2020 0x7f04081544c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f04080d0ea0 tx=0x7f03f8003820 comp rx=0 tx=0).stop 2024-08-31T12:28:35.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.139+0000 7f03feffd640 1 -- 172.21.15.19:0/3197849119 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f04081f2020 msgr2=0x7f04081544c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:28:35.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.139+0000 7f03feffd640 1 --2- 172.21.15.19:0/3197849119 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f04081f2020 0x7f04081544c0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:35.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.139+0000 7f040e838640 1 --2- 172.21.15.19:0/3197849119 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f03ec077a80 0x7f03ec079f90 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f04081e6070 tx=0x7f0400009560 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:35.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.145+0000 7f041029b640 1 --2- 172.21.15.19:0/3197849119 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f04081f1a20 0x7f04081f48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:35.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.145+0000 7f041029b640 1 -- 172.21.15.19:0/3197849119 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f04081e6810 con 0x7f04081f1a20 2024-08-31T12:28:35.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.145+0000 7f040d836640 1 --2- 172.21.15.19:0/3197849119 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f04081f1a20 0x7f04081f48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:35.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.146+0000 7f040d836640 1 -- 172.21.15.19:0/3197849119 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0408063ad0 con 0x7f04081f1a20 2024-08-31T12:28:35.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.146+0000 7f040d836640 1 --2- 172.21.15.19:0/3197849119 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f04081f1a20 0x7f04081f48a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f03f8000c00 tx=0x7f03f802a670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:35.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.146+0000 7f03feffd640 1 -- 172.21.15.19:0/3197849119 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03f8035040 con 0x7f04081f1a20 2024-08-31T12:28:35.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.146+0000 7f03feffd640 1 -- 172.21.15.19:0/3197849119 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4539 (secure 0 0 0) 0x7f03f8049980 con 0x7f04081f1a20 2024-08-31T12:28:35.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.146+0000 7f03feffd640 1 -- 172.21.15.19:0/3197849119 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f04081f1a20 msgr2=0x7f04081f48a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:35.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.146+0000 7f03feffd640 1 --2- 172.21.15.19:0/3197849119 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f04081f1a20 0x7f04081f48a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f03f8000c00 tx=0x7f03f802a670 comp rx=0 tx=0).stop 2024-08-31T12:28:35.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.146+0000 7f03feffd640 1 -- 172.21.15.19:0/3197849119 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f04081f1a20 msgr2=0x7f04081f48a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-31T12:28:35.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.146+0000 7f03feffd640 1 --2- 172.21.15.19:0/3197849119 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f04081f1a20 0x7f04081f48a0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:35.149 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:28:35.149 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:28:35.149 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:28:35.149 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "peon", 2024-08-31T12:28:35.149 INFO:teuthology.orchestra.run.smithi019.stdout: "election_epoch": 98, 2024-08-31T12:28:35.149 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 7577, 2024-08-31T12:28:35.150 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum": [ 2024-08-31T12:28:35.150 INFO:teuthology.orchestra.run.smithi019.stdout: 0, 2024-08-31T12:28:35.150 INFO:teuthology.orchestra.run.smithi019.stdout: 1, 2024-08-31T12:28:35.150 INFO:teuthology.orchestra.run.smithi019.stdout: 2 2024-08-31T12:28:35.150 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:28:35.150 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_age": 3, 2024-08-31T12:28:35.150 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:28:35.150 INFO:teuthology.orchestra.run.smithi019.stdout: "required_con": "2450521708007866372", 2024-08-31T12:28:35.151 INFO:teuthology.orchestra.run.smithi019.stdout: "required_mon": [ 2024-08-31T12:28:35.151 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:28:35.151 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:28:35.151 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:28:35.151 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:28:35.151 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:28:35.151 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:28:35.151 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:28:35.151 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:28:35.152 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:28:35.152 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:28:35.152 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:28:35.152 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:28:35.152 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_con": "4540701547738038271", 2024-08-31T12:28:35.152 INFO:teuthology.orchestra.run.smithi019.stdout: "quorum_mon": [ 2024-08-31T12:28:35.152 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:28:35.152 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:28:35.153 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:28:35.153 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:28:35.153 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:28:35.153 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:28:35.153 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:28:35.153 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:28:35.153 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:28:35.153 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:28:35.154 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:28:35.154 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:28:35.154 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:35.154 INFO:teuthology.orchestra.run.smithi019.stdout: "outside_quorum": [], 2024-08-31T12:28:35.154 INFO:teuthology.orchestra.run.smithi019.stdout: "extra_probe_peers": [], 2024-08-31T12:28:35.154 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_provider": [], 2024-08-31T12:28:35.154 INFO:teuthology.orchestra.run.smithi019.stdout: "monmap": { 2024-08-31T12:28:35.154 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-08-31T12:28:35.154 INFO:teuthology.orchestra.run.smithi019.stdout: "fsid": "65c40ef5-9e6a-4782-8f93-4032c6800123", 2024-08-31T12:28:35.155 INFO:teuthology.orchestra.run.smithi019.stdout: "modified": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:28:35.155 INFO:teuthology.orchestra.run.smithi019.stdout: "created": "2024-08-31T12:14:57.374518Z", 2024-08-31T12:28:35.155 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release": 19, 2024-08-31T12:28:35.155 INFO:teuthology.orchestra.run.smithi019.stdout: "min_mon_release_name": "squid", 2024-08-31T12:28:35.155 INFO:teuthology.orchestra.run.smithi019.stdout: "election_strategy": 1, 2024-08-31T12:28:35.155 INFO:teuthology.orchestra.run.smithi019.stdout: "disallowed_leaders: ": "", 2024-08-31T12:28:35.155 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false, 2024-08-31T12:28:35.155 INFO:teuthology.orchestra.run.smithi019.stdout: "tiebreaker_mon": "", 2024-08-31T12:28:35.155 INFO:teuthology.orchestra.run.smithi019.stdout: "removed_ranks: ": "", 2024-08-31T12:28:35.156 INFO:teuthology.orchestra.run.smithi019.stdout: "features": { 2024-08-31T12:28:35.156 INFO:teuthology.orchestra.run.smithi019.stdout: "persistent": [ 2024-08-31T12:28:35.156 INFO:teuthology.orchestra.run.smithi019.stdout: "kraken", 2024-08-31T12:28:35.156 INFO:teuthology.orchestra.run.smithi019.stdout: "luminous", 2024-08-31T12:28:35.156 INFO:teuthology.orchestra.run.smithi019.stdout: "mimic", 2024-08-31T12:28:35.156 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap-prune", 2024-08-31T12:28:35.156 INFO:teuthology.orchestra.run.smithi019.stdout: "nautilus", 2024-08-31T12:28:35.156 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus", 2024-08-31T12:28:35.156 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific", 2024-08-31T12:28:35.157 INFO:teuthology.orchestra.run.smithi019.stdout: "elector-pinging", 2024-08-31T12:28:35.157 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy", 2024-08-31T12:28:35.157 INFO:teuthology.orchestra.run.smithi019.stdout: "reef", 2024-08-31T12:28:35.157 INFO:teuthology.orchestra.run.smithi019.stdout: "squid" 2024-08-31T12:28:35.157 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:28:35.157 INFO:teuthology.orchestra.run.smithi019.stdout: "optional": [] 2024-08-31T12:28:35.157 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:35.157 INFO:teuthology.orchestra.run.smithi019.stdout: "mons": [ 2024-08-31T12:28:35.158 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:35.158 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 0, 2024-08-31T12:28:35.158 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "a", 2024-08-31T12:28:35.158 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:28:35.158 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:28:35.158 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:35.158 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:28:35.158 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:3300", 2024-08-31T12:28:35.158 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:28:35.159 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:35.159 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:35.159 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:28:35.159 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789", 2024-08-31T12:28:35.159 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:28:35.159 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:28:35.159 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:28:35.159 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:35.159 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6789/0", 2024-08-31T12:28:35.160 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.19:6789/0", 2024-08-31T12:28:35.160 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:28:35.160 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:28:35.160 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:28:35.160 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:35.160 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:35.160 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 1, 2024-08-31T12:28:35.160 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "b", 2024-08-31T12:28:35.160 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:28:35.161 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:28:35.161 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:35.161 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:28:35.161 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3300", 2024-08-31T12:28:35.161 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:28:35.161 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:35.161 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:35.161 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:28:35.161 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789", 2024-08-31T12:28:35.162 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:28:35.162 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:28:35.162 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:28:35.162 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:35.162 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6789/0", 2024-08-31T12:28:35.162 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6789/0", 2024-08-31T12:28:35.162 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:28:35.162 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:28:35.162 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:28:35.163 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:35.163 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:35.163 INFO:teuthology.orchestra.run.smithi019.stdout: "rank": 2, 2024-08-31T12:28:35.163 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "c", 2024-08-31T12:28:35.163 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addrs": { 2024-08-31T12:28:35.163 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-08-31T12:28:35.163 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:35.163 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-08-31T12:28:35.164 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:3301", 2024-08-31T12:28:35.164 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:28:35.164 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:35.164 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:35.164 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-08-31T12:28:35.164 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790", 2024-08-31T12:28:35.165 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 0 2024-08-31T12:28:35.165 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:28:35.165 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:28:35.165 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:35.165 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.43:6790/0", 2024-08-31T12:28:35.165 INFO:teuthology.orchestra.run.smithi019.stdout: "public_addr": "172.21.15.43:6790/0", 2024-08-31T12:28:35.165 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-08-31T12:28:35.165 INFO:teuthology.orchestra.run.smithi019.stdout: "weight": 0, 2024-08-31T12:28:35.165 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_location": "{}" 2024-08-31T12:28:35.166 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:28:35.166 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:28:35.166 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:35.166 INFO:teuthology.orchestra.run.smithi019.stdout: "feature_map": { 2024-08-31T12:28:35.166 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": [ 2024-08-31T12:28:35.166 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:35.166 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:28:35.166 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:28:35.166 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:28:35.167 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:28:35.167 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-08-31T12:28:35.167 INFO:teuthology.orchestra.run.smithi019.stdout: "client": [ 2024-08-31T12:28:35.167 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-08-31T12:28:35.167 INFO:teuthology.orchestra.run.smithi019.stdout: "features": "0x3f03cffffffdffff", 2024-08-31T12:28:35.167 INFO:teuthology.orchestra.run.smithi019.stdout: "release": "squid", 2024-08-31T12:28:35.167 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 1 2024-08-31T12:28:35.167 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-08-31T12:28:35.167 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-08-31T12:28:35.168 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-08-31T12:28:35.168 INFO:teuthology.orchestra.run.smithi019.stdout: "stretch_mode": false 2024-08-31T12:28:35.168 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:28:35.169 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.147+0000 7f041029b640 1 -- 172.21.15.19:0/3197849119 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f03ec077a80 msgr2=0x7f03ec079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:35.169 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.147+0000 7f041029b640 1 --2- 172.21.15.19:0/3197849119 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f03ec077a80 0x7f03ec079f90 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f04081e6070 tx=0x7f0400009560 comp rx=0 tx=0).stop 2024-08-31T12:28:35.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.147+0000 7f041029b640 1 -- 172.21.15.19:0/3197849119 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f040806e0e0 msgr2=0x7f04080c2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:35.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.147+0000 7f041029b640 1 --2- 172.21.15.19:0/3197849119 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f040806e0e0 0x7f04080c2a40 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f040400c3a0 tx=0x7f040400c860 comp rx=0 tx=0).stop 2024-08-31T12:28:35.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.147+0000 7f040e838640 1 -- 172.21.15.19:0/3197849119 reap_dead start 2024-08-31T12:28:35.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.147+0000 7f040e838640 1 -- 172.21.15.19:0/3197849119 reap_dead start 2024-08-31T12:28:35.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.147+0000 7f041029b640 1 -- 172.21.15.19:0/3197849119 shutdown_connections 2024-08-31T12:28:35.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.147+0000 7f041029b640 1 -- 172.21.15.19:0/3197849119 >> 172.21.15.19:0/3197849119 conn(0x7f04080690d0 msgr2=0x7f04080cb2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:35.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.147+0000 7f041029b640 1 -- 172.21.15.19:0/3197849119 shutdown_connections 2024-08-31T12:28:35.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.147+0000 7f041029b640 1 -- 172.21.15.19:0/3197849119 wait complete. 2024-08-31T12:28:35.171 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-08-31T12:28:35.171 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-08-31T12:28:35.171 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-08-31T12:28:35.285 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.284+0000 7efe5e3ad640 1 -- 172.21.15.19:0/351079896 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efe580cc620 msgr2=0x7efe580cdab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:35.285 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.284+0000 7efe5e3ad640 1 --2- 172.21.15.19:0/351079896 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efe580cc620 0x7efe580cdab0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7efe580d1820 tx=0x7efe4802a5f0 comp rx=0 tx=0).stop 2024-08-31T12:28:35.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.284+0000 7efe5e3ad640 1 -- 172.21.15.19:0/351079896 shutdown_connections 2024-08-31T12:28:35.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.284+0000 7efe5e3ad640 1 --2- 172.21.15.19:0/351079896 >> v2:172.21.15.43:3300/0 conn(0x7efe580d1210 0x7efe5815d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:35.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.284+0000 7efe5e3ad640 1 --2- 172.21.15.19:0/351079896 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efe580ca770 0x7efe580d0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:35.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.284+0000 7efe5e3ad640 1 --2- 172.21.15.19:0/351079896 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efe580cc620 0x7efe580cdab0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:35.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.284+0000 7efe5e3ad640 1 -- 172.21.15.19:0/351079896 >> 172.21.15.19:0/351079896 conn(0x7efe5806e6e0 msgr2=0x7efe5806eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:35.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.284+0000 7efe5e3ad640 1 -- 172.21.15.19:0/351079896 shutdown_connections 2024-08-31T12:28:35.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.284+0000 7efe5e3ad640 1 -- 172.21.15.19:0/351079896 wait complete. 2024-08-31T12:28:35.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.286+0000 7efe5e3ad640 1 Processor -- start 2024-08-31T12:28:35.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.286+0000 7efe5e3ad640 1 -- start start 2024-08-31T12:28:35.289 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.287+0000 7efe5e3ad640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe580ca770 0x7efe5815b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:35.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.287+0000 7efe5e3ad640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efe580cc620 0x7efe58157b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:35.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.287+0000 7efe5e3ad640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efe580d1210 0x7efe58156730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:35.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.287+0000 7efe5e3ad640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7efe5815d090 con 0x7efe580cc620 2024-08-31T12:28:35.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.287+0000 7efe5e3ad640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7efe5815cdb0 con 0x7efe580ca770 2024-08-31T12:28:35.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.287+0000 7efe5e3ad640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7efe580cc320 con 0x7efe580d1210 2024-08-31T12:28:35.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.287+0000 7efe577fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efe580cc620 0x7efe58157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:35.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.287+0000 7efe577fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efe580cc620 0x7efe58157b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:58440/0 (socket says 172.21.15.19:58440) 2024-08-31T12:28:35.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.287+0000 7efe577fe640 1 -- 172.21.15.19:0/285252099 learned_addr learned my addr 172.21.15.19:0/285252099 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:28:35.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.287+0000 7efe57fff640 1 --2- 172.21.15.19:0/285252099 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe580ca770 0x7efe5815b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:35.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.287+0000 7efe5c94a640 1 --2- 172.21.15.19:0/285252099 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efe580d1210 0x7efe58156730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:35.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.287+0000 7efe577fe640 1 -- 172.21.15.19:0/285252099 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efe580d1210 msgr2=0x7efe58156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:35.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.287+0000 7efe577fe640 1 --2- 172.21.15.19:0/285252099 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efe580d1210 0x7efe58156730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:35.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.287+0000 7efe577fe640 1 -- 172.21.15.19:0/285252099 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe580ca770 msgr2=0x7efe5815b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:35.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.287+0000 7efe577fe640 1 --2- 172.21.15.19:0/285252099 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe580ca770 0x7efe5815b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:35.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.287+0000 7efe577fe640 1 -- 172.21.15.19:0/285252099 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe5815b810 con 0x7efe580cc620 2024-08-31T12:28:35.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.288+0000 7efe577fe640 1 --2- 172.21.15.19:0/285252099 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efe580cc620 0x7efe58157b90 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7efe4c00e7c0 tx=0x7efe4c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:35.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.288+0000 7efe54ff9640 1 -- 172.21.15.19:0/285252099 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe4c00ccc0 con 0x7efe580cc620 2024-08-31T12:28:35.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.288+0000 7efe54ff9640 1 -- 172.21.15.19:0/285252099 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efe4c00ce60 con 0x7efe580cc620 2024-08-31T12:28:35.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.288+0000 7efe5e3ad640 1 -- 172.21.15.19:0/285252099 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe581588b0 con 0x7efe580cc620 2024-08-31T12:28:35.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.288+0000 7efe54ff9640 1 -- 172.21.15.19:0/285252099 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe4c005280 con 0x7efe580cc620 2024-08-31T12:28:35.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.288+0000 7efe5e3ad640 1 -- 172.21.15.19:0/285252099 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe58155bb0 con 0x7efe580cc620 2024-08-31T12:28:35.293 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.289+0000 7efe5e3ad640 1 -- 172.21.15.19:0/285252099 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe581590d0 con 0x7efe580cc620 2024-08-31T12:28:35.293 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.290+0000 7efe54ff9640 1 -- 172.21.15.19:0/285252099 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7efe4c005420 con 0x7efe580cc620 2024-08-31T12:28:35.294 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.290+0000 7efe54ff9640 1 --2- 172.21.15.19:0/285252099 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7efe44077a50 0x7efe44079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:35.294 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.290+0000 7efe54ff9640 1 -- 172.21.15.19:0/285252099 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7efe4c099e70 con 0x7efe580cc620 2024-08-31T12:28:35.294 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.290+0000 7efe57fff640 1 --2- 172.21.15.19:0/285252099 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7efe44077a50 0x7efe44079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:35.294 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.291+0000 7efe57fff640 1 --2- 172.21.15.19:0/285252099 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7efe44077a50 0x7efe44079f60 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7efe48033000 tx=0x7efe48004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:35.295 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.294+0000 7efe54ff9640 1 -- 172.21.15.19:0/285252099 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7efe581590d0 con 0x7efe580cc620 2024-08-31T12:28:35.552 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.551+0000 7efe5e3ad640 1 -- 172.21.15.19:0/285252099 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7efe580cb2c0 con 0x7efe580cc620 2024-08-31T12:28:35.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.552+0000 7efe54ff9640 1 -- 172.21.15.19:0/285252099 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7efe4c06b0a0 con 0x7efe580cc620 2024-08-31T12:28:35.556 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.555+0000 7efe5e3ad640 1 -- 172.21.15.19:0/285252099 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7efe44077a50 msgr2=0x7efe44079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:35.557 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.555+0000 7efe5e3ad640 1 --2- 172.21.15.19:0/285252099 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7efe44077a50 0x7efe44079f60 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7efe48033000 tx=0x7efe48004730 comp rx=0 tx=0).stop 2024-08-31T12:28:35.557 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.555+0000 7efe5e3ad640 1 -- 172.21.15.19:0/285252099 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efe580cc620 msgr2=0x7efe58157b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:35.557 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.555+0000 7efe5e3ad640 1 --2- 172.21.15.19:0/285252099 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efe580cc620 0x7efe58157b90 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7efe4c00e7c0 tx=0x7efe4c00ec80 comp rx=0 tx=0).stop 2024-08-31T12:28:35.557 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.555+0000 7efe5e3ad640 1 -- 172.21.15.19:0/285252099 shutdown_connections 2024-08-31T12:28:35.557 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.555+0000 7efe5e3ad640 1 --2- 172.21.15.19:0/285252099 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7efe44077a50 0x7efe44079f60 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:35.557 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.555+0000 7efe5e3ad640 1 --2- 172.21.15.19:0/285252099 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efe580d1210 0x7efe58156730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:35.557 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.555+0000 7efe5e3ad640 1 --2- 172.21.15.19:0/285252099 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efe580cc620 0x7efe58157b90 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:35.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.555+0000 7efe5e3ad640 1 --2- 172.21.15.19:0/285252099 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe580ca770 0x7efe5815b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:35.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.555+0000 7efe5e3ad640 1 -- 172.21.15.19:0/285252099 >> 172.21.15.19:0/285252099 conn(0x7efe5806e6e0 msgr2=0x7efe5815d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:35.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.555+0000 7efe5e3ad640 1 -- 172.21.15.19:0/285252099 shutdown_connections 2024-08-31T12:28:35.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:35.555+0000 7efe5e3ad640 1 -- 172.21.15.19:0/285252099 wait complete. 2024-08-31T12:28:35.572 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2024-08-31T12:28:45.572 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-31T12:28:45.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.700+0000 7f484bc8c640 1 -- 172.21.15.19:0/599846925 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4844071ea0 msgr2=0x7f484415c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:45.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.700+0000 7f484bc8c640 1 --2- 172.21.15.19:0/599846925 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4844071ea0 0x7f484415c4b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f48440cbaa0 tx=0x7f483c02aae0 comp rx=0 tx=0).stop 2024-08-31T12:28:45.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.700+0000 7f484bc8c640 1 -- 172.21.15.19:0/599846925 shutdown_connections 2024-08-31T12:28:45.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.700+0000 7f484bc8c640 1 --2- 172.21.15.19:0/599846925 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4844071ea0 0x7f484415c4b0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:45.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.700+0000 7f484bc8c640 1 --2- 172.21.15.19:0/599846925 >> v2:172.21.15.43:3300/0 conn(0x7f48440cac90 0x7f4844071960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:45.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.700+0000 7f484bc8c640 1 --2- 172.21.15.19:0/599846925 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f48440ca7d0 0x7f48441530f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:45.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.700+0000 7f484bc8c640 1 -- 172.21.15.19:0/599846925 >> 172.21.15.19:0/599846925 conn(0x7f48440cef50 msgr2=0x7f48440cb340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:45.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.700+0000 7f484bc8c640 1 -- 172.21.15.19:0/599846925 shutdown_connections 2024-08-31T12:28:45.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.700+0000 7f484bc8c640 1 -- 172.21.15.19:0/599846925 wait complete. 2024-08-31T12:28:45.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.701+0000 7f484bc8c640 1 Processor -- start 2024-08-31T12:28:45.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.701+0000 7f484bc8c640 1 -- start start 2024-08-31T12:28:45.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.703+0000 7f484bc8c640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4844071ea0 0x7f484415b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:45.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.703+0000 7f484bc8c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f48440ca7d0 0x7f48441558e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:45.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.703+0000 7f484bc8c640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f48440cac90 0x7f48441474a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:45.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.703+0000 7f484bc8c640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f484415d330 con 0x7f48440ca7d0 2024-08-31T12:28:45.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.703+0000 7f484bc8c640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4844071440 con 0x7f4844071ea0 2024-08-31T12:28:45.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.703+0000 7f484bc8c640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f48440c58d0 con 0x7f48440cac90 2024-08-31T12:28:45.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.703+0000 7f4849227640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f48440ca7d0 0x7f48441558e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:45.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.703+0000 7f4849227640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f48440ca7d0 0x7f48441558e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:45536/0 (socket says 172.21.15.19:45536) 2024-08-31T12:28:45.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.703+0000 7f4849227640 1 -- 172.21.15.19:0/1223351498 learned_addr learned my addr 172.21.15.19:0/1223351498 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:28:45.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.703+0000 7f4849a28640 1 --2- 172.21.15.19:0/1223351498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4844071ea0 0x7f484415b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:45.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.703+0000 7f484a229640 1 --2- 172.21.15.19:0/1223351498 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f48440cac90 0x7f48441474a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:45.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.703+0000 7f4849227640 1 -- 172.21.15.19:0/1223351498 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f48440cac90 msgr2=0x7f48441474a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:45.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.703+0000 7f4849227640 1 --2- 172.21.15.19:0/1223351498 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f48440cac90 0x7f48441474a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:45.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.703+0000 7f4849227640 1 -- 172.21.15.19:0/1223351498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4844071ea0 msgr2=0x7f484415b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:45.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.703+0000 7f4849227640 1 --2- 172.21.15.19:0/1223351498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4844071ea0 0x7f484415b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:45.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.703+0000 7f4849227640 1 -- 172.21.15.19:0/1223351498 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48441f3540 con 0x7f48440ca7d0 2024-08-31T12:28:45.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.703+0000 7f4849227640 1 --2- 172.21.15.19:0/1223351498 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f48440ca7d0 0x7f48441558e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f483400e790 tx=0x7f483400ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:45.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.703+0000 7f483a7fc640 1 -- 172.21.15.19:0/1223351498 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f483400eea0 con 0x7f48440ca7d0 2024-08-31T12:28:45.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.704+0000 7f483a7fc640 1 -- 172.21.15.19:0/1223351498 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f483400cd50 con 0x7f48440ca7d0 2024-08-31T12:28:45.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.704+0000 7f483a7fc640 1 -- 172.21.15.19:0/1223351498 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4834010690 con 0x7f48440ca7d0 2024-08-31T12:28:45.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.704+0000 7f484bc8c640 1 -- 172.21.15.19:0/1223351498 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48441f3260 con 0x7f48440ca7d0 2024-08-31T12:28:45.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.704+0000 7f484bc8c640 1 -- 172.21.15.19:0/1223351498 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4844153950 con 0x7f48440ca7d0 2024-08-31T12:28:45.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.705+0000 7f484bc8c640 1 -- 172.21.15.19:0/1223351498 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4844156850 con 0x7f48440ca7d0 2024-08-31T12:28:45.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.705+0000 7f483a7fc640 1 -- 172.21.15.19:0/1223351498 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f4834010830 con 0x7f48440ca7d0 2024-08-31T12:28:45.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.706+0000 7f483a7fc640 1 --2- 172.21.15.19:0/1223351498 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4828077a50 0x7f4828079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:45.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.706+0000 7f483a7fc640 1 -- 172.21.15.19:0/1223351498 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f4834099fa0 con 0x7f48440ca7d0 2024-08-31T12:28:45.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.706+0000 7f4849a28640 1 --2- 172.21.15.19:0/1223351498 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4828077a50 0x7f4828079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:45.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.707+0000 7f4849a28640 1 --2- 172.21.15.19:0/1223351498 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4828077a50 0x7f4828079f60 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f4840009ab0 tx=0x7f4840005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:45.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.710+0000 7f483a7fc640 1 -- 172.21.15.19:0/1223351498 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f48340626d0 con 0x7f48440ca7d0 2024-08-31T12:28:45.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.932+0000 7f484bc8c640 1 -- 172.21.15.19:0/1223351498 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f48440c4050 con 0x7f48440ca7d0 2024-08-31T12:28:45.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.933+0000 7f483a7fc640 1 -- 172.21.15.19:0/1223351498 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+6901 (secure 0 0 0) 0x7f4834066e90 con 0x7f48440ca7d0 2024-08-31T12:28:45.936 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:28:45.936 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":12,"btime":"2024-08-31T12:16:13:439874+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4371,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6833/4215834946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":4215834946},{"type":"v1","addr":"172.21.15.19:6833","nonce":4215834946}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/543344732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":543344732},{"type":"v1","addr":"172.21.15.19:6835","nonce":543344732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4380,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6837/699077015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6836","nonce":699077015},{"type":"v1","addr":"172.21.15.19:6837","nonce":699077015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-31T12:15:56.652936+0000","modified":"2024-08-31T12:16:13.439859+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4429,"mds_1":4426,"mds_2":4398},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4398":{"gid":4398,"name":"f","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.43:6837/2103490885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6835","nonce":2103490885},{"type":"v1","addr":"172.21.15.43:6837","nonce":2103490885}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4426":{"gid":4426,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.43:6839/1196858061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1196858061},{"type":"v1","addr":"172.21.15.43:6839","nonce":1196858061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4429":{"gid":4429,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.43:6838/3981040655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":3981040655},{"type":"v1","addr":"172.21.15.43:6838","nonce":3981040655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4429,"qdb_cluster":[4398,4426,4429]},"id":1}]} 2024-08-31T12:28:45.936 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 12 2024-08-31T12:28:45.938 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.935+0000 7f484bc8c640 1 -- 172.21.15.19:0/1223351498 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4828077a50 msgr2=0x7f4828079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:45.938 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.935+0000 7f484bc8c640 1 --2- 172.21.15.19:0/1223351498 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4828077a50 0x7f4828079f60 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f4840009ab0 tx=0x7f4840005f90 comp rx=0 tx=0).stop 2024-08-31T12:28:45.938 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.935+0000 7f484bc8c640 1 -- 172.21.15.19:0/1223351498 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f48440ca7d0 msgr2=0x7f48441558e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:45.938 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.935+0000 7f484bc8c640 1 --2- 172.21.15.19:0/1223351498 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f48440ca7d0 0x7f48441558e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f483400e790 tx=0x7f483400ec50 comp rx=0 tx=0).stop 2024-08-31T12:28:45.938 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.935+0000 7f484bc8c640 1 -- 172.21.15.19:0/1223351498 shutdown_connections 2024-08-31T12:28:45.939 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.935+0000 7f484bc8c640 1 --2- 172.21.15.19:0/1223351498 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4828077a50 0x7f4828079f60 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:45.939 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.935+0000 7f484bc8c640 1 --2- 172.21.15.19:0/1223351498 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f48440cac90 0x7f48441474a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:45.939 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.935+0000 7f484bc8c640 1 --2- 172.21.15.19:0/1223351498 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f48440ca7d0 0x7f48441558e0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:45.939 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.935+0000 7f484bc8c640 1 --2- 172.21.15.19:0/1223351498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4844071ea0 0x7f484415b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:45.939 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.936+0000 7f484bc8c640 1 -- 172.21.15.19:0/1223351498 >> 172.21.15.19:0/1223351498 conn(0x7f48440cef50 msgr2=0x7f48440c4850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:45.939 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.936+0000 7f484bc8c640 1 -- 172.21.15.19:0/1223351498 shutdown_connections 2024-08-31T12:28:45.939 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:45.936+0000 7f484bc8c640 1 -- 172.21.15.19:0/1223351498 wait complete. 2024-08-31T12:28:45.949 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-08-31T12:28:45.950 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-31T12:28:46.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.060+0000 7fc4c211a640 1 -- 172.21.15.19:0/3276085581 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc4a8003a00 msgr2=0x7fc4a8003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:46.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.060+0000 7fc4c211a640 1 --2- 172.21.15.19:0/3276085581 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc4a8003a00 0x7fc4a8003e70 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac009b50 tx=0x7fc4a8005830 comp rx=0 tx=0).stop 2024-08-31T12:28:46.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.060+0000 7fc4bbfff640 1 -- 172.21.15.19:0/3276085581 reap_dead start 2024-08-31T12:28:46.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.060+0000 7fc4c211a640 1 -- 172.21.15.19:0/3276085581 shutdown_connections 2024-08-31T12:28:46.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.060+0000 7fc4c211a640 1 -- 172.21.15.19:0/3276085581 >> 172.21.15.19:0/3276085581 conn(0x7fc4bc06f9d0 msgr2=0x7fc4bc06e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:46.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.060+0000 7fc4c211a640 1 -- 172.21.15.19:0/3276085581 shutdown_connections 2024-08-31T12:28:46.064 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.060+0000 7fc4c211a640 1 -- 172.21.15.19:0/3276085581 wait complete. 2024-08-31T12:28:46.064 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.062+0000 7fc4c211a640 1 Processor -- start 2024-08-31T12:28:46.064 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.062+0000 7fc4c211a640 1 -- start start 2024-08-31T12:28:46.067 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.063+0000 7fc4c211a640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc4a8003a00 0x7fc4bc1f1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:46.067 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.063+0000 7fc4c211a640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc4bc14ae70 0x7fc4bc14d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:46.067 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.063+0000 7fc4c211a640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc4bc14b280 0x7fc4bc14e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:46.067 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.063+0000 7fc4c211a640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fc4bc15ee20 con 0x7fc4bc14b280 2024-08-31T12:28:46.067 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.063+0000 7fc4c211a640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc4bc15eb40 con 0x7fc4bc14ae70 2024-08-31T12:28:46.067 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.063+0000 7fc4c211a640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fc4bc0b5110 con 0x7fc4a8003a00 2024-08-31T12:28:46.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.064+0000 7fc4bbfff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc4bc14b280 0x7fc4bc14e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:46.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.064+0000 7fc4bbfff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc4bc14b280 0x7fc4bc14e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:45548/0 (socket says 172.21.15.19:45548) 2024-08-31T12:28:46.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.064+0000 7fc4bbfff640 1 -- 172.21.15.19:0/2988294616 learned_addr learned my addr 172.21.15.19:0/2988294616 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:28:46.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.064+0000 7fc4bb7fe640 1 --2- 172.21.15.19:0/2988294616 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc4a8003a00 0x7fc4bc1f1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:46.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.064+0000 7fc4b2ffd640 1 --2- 172.21.15.19:0/2988294616 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc4bc14ae70 0x7fc4bc14d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:46.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.064+0000 7fc4bbfff640 1 -- 172.21.15.19:0/2988294616 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc4a8003a00 msgr2=0x7fc4bc1f1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:46.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.064+0000 7fc4bbfff640 1 --2- 172.21.15.19:0/2988294616 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc4a8003a00 0x7fc4bc1f1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:46.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.064+0000 7fc4bbfff640 1 -- 172.21.15.19:0/2988294616 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc4bc14ae70 msgr2=0x7fc4bc14d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:46.069 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.064+0000 7fc4bbfff640 1 --2- 172.21.15.19:0/2988294616 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc4bc14ae70 0x7fc4bc14d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:46.069 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.064+0000 7fc4bbfff640 1 -- 172.21.15.19:0/2988294616 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4bc1f21f0 con 0x7fc4bc14b280 2024-08-31T12:28:46.069 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.064+0000 7fc4bbfff640 1 --2- 172.21.15.19:0/2988294616 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc4bc14b280 0x7fc4bc14e0f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc4b4005fd0 tx=0x7fc4b4011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:46.069 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.064+0000 7fc4b8ff9640 1 -- 172.21.15.19:0/2988294616 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b400fcb0 con 0x7fc4bc14b280 2024-08-31T12:28:46.069 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.064+0000 7fc4b8ff9640 1 -- 172.21.15.19:0/2988294616 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc4b400fe50 con 0x7fc4bc14b280 2024-08-31T12:28:46.069 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.064+0000 7fc4c211a640 1 -- 172.21.15.19:0/2988294616 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4bc14b6c0 con 0x7fc4bc14b280 2024-08-31T12:28:46.069 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.064+0000 7fc4b8ff9640 1 -- 172.21.15.19:0/2988294616 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b4003bd0 con 0x7fc4bc14b280 2024-08-31T12:28:46.070 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.064+0000 7fc4c211a640 1 -- 172.21.15.19:0/2988294616 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4bc0d01a0 con 0x7fc4bc14b280 2024-08-31T12:28:46.070 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.065+0000 7fc4c211a640 1 -- 172.21.15.19:0/2988294616 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4bc155700 con 0x7fc4bc14b280 2024-08-31T12:28:46.070 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.066+0000 7fc4b8ff9640 1 -- 172.21.15.19:0/2988294616 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fc4b4019410 con 0x7fc4bc14b280 2024-08-31T12:28:46.071 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.066+0000 7fc4b8ff9640 1 --2- 172.21.15.19:0/2988294616 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc49c077a50 0x7fc49c079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:46.071 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.067+0000 7fc4b8ff9640 1 -- 172.21.15.19:0/2988294616 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fc4b409a030 con 0x7fc4bc14b280 2024-08-31T12:28:46.071 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.067+0000 7fc4bb7fe640 1 --2- 172.21.15.19:0/2988294616 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc49c077a50 0x7fc49c079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:46.071 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.067+0000 7fc4bb7fe640 1 --2- 172.21.15.19:0/2988294616 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc49c077a50 0x7fc49c079f60 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac009b50 tx=0x7fc4ac02d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:46.073 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.071+0000 7fc4b8ff9640 1 -- 172.21.15.19:0/2988294616 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fc4b40626d0 con 0x7fc4bc14b280 2024-08-31T12:28:46.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.327+0000 7fc4c211a640 1 -- 172.21.15.19:0/2988294616 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc4bc0b3390 con 0x7fc4bc14b280 2024-08-31T12:28:46.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.327+0000 7fc4b8ff9640 1 -- 172.21.15.19:0/2988294616 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fc4b4066e90 con 0x7fc4bc14b280 2024-08-31T12:28:46.330 INFO:teuthology.orchestra.run.smithi019.stdout:{"election_epoch":98,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","modified":"2024-08-31T12:14:57.374518Z","created":"2024-08-31T12:14:57.374518Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-31T12:28:46.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.329+0000 7fc4c211a640 1 -- 172.21.15.19:0/2988294616 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc49c077a50 msgr2=0x7fc49c079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:46.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.329+0000 7fc4c211a640 1 --2- 172.21.15.19:0/2988294616 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc49c077a50 0x7fc49c079f60 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac009b50 tx=0x7fc4ac02d040 comp rx=0 tx=0).stop 2024-08-31T12:28:46.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.330+0000 7fc4c211a640 1 -- 172.21.15.19:0/2988294616 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc4bc14b280 msgr2=0x7fc4bc14e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:46.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.330+0000 7fc4c211a640 1 --2- 172.21.15.19:0/2988294616 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc4bc14b280 0x7fc4bc14e0f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc4b4005fd0 tx=0x7fc4b4011ed0 comp rx=0 tx=0).stop 2024-08-31T12:28:46.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.330+0000 7fc4c211a640 1 -- 172.21.15.19:0/2988294616 shutdown_connections 2024-08-31T12:28:46.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.330+0000 7fc4c211a640 1 --2- 172.21.15.19:0/2988294616 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc49c077a50 0x7fc49c079f60 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:46.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.330+0000 7fc4c211a640 1 --2- 172.21.15.19:0/2988294616 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc4bc14b280 0x7fc4bc14e0f0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:46.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.330+0000 7fc4c211a640 1 --2- 172.21.15.19:0/2988294616 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc4bc14ae70 0x7fc4bc14d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:46.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.330+0000 7fc4c211a640 1 --2- 172.21.15.19:0/2988294616 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc4a8003a00 0x7fc4bc1f1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:46.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.330+0000 7fc4c211a640 1 -- 172.21.15.19:0/2988294616 >> 172.21.15.19:0/2988294616 conn(0x7fc4bc06f9d0 msgr2=0x7fc4bc1597a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:46.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.330+0000 7fc4c211a640 1 -- 172.21.15.19:0/2988294616 shutdown_connections 2024-08-31T12:28:46.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.330+0000 7fc4c211a640 1 -- 172.21.15.19:0/2988294616 wait complete. 2024-08-31T12:28:46.343 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-08-31T12:28:46.344 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-08-31T12:28:46.353 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-08-31T12:28:46.354 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-08-31T12:28:46.354 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-31T12:28:46.379 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-08-31T12:28:46.379 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-31T12:28:46.435 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-08-31T12:28:46.592 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.589+0000 7fd4dba69640 1 -- 172.21.15.19:0/971527377 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd4d40cc620 msgr2=0x7fd4d40cdab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:46.592 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.589+0000 7fd4dba69640 1 --2- 172.21.15.19:0/971527377 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd4d40cc620 0x7fd4d40cdab0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd4c4009be0 tx=0x7fd4c402a9b0 comp rx=0 tx=0).stop 2024-08-31T12:28:46.592 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.589+0000 7fd4dba69640 1 -- 172.21.15.19:0/971527377 shutdown_connections 2024-08-31T12:28:46.592 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.589+0000 7fd4dba69640 1 --2- 172.21.15.19:0/971527377 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd4d40cc620 0x7fd4d40cdab0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:46.592 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.589+0000 7fd4dba69640 1 -- 172.21.15.19:0/971527377 >> 172.21.15.19:0/971527377 conn(0x7fd4d406e6e0 msgr2=0x7fd4d406eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:46.592 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.589+0000 7fd4dba69640 1 -- 172.21.15.19:0/971527377 shutdown_connections 2024-08-31T12:28:46.593 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.589+0000 7fd4dba69640 1 -- 172.21.15.19:0/971527377 wait complete. 2024-08-31T12:28:46.593 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.591+0000 7fd4dba69640 1 Processor -- start 2024-08-31T12:28:46.593 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.591+0000 7fd4dba69640 1 -- start start 2024-08-31T12:28:46.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.592+0000 7fd4dba69640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4d40ca770 0x7fd4d4156b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:46.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.592+0000 7fd4dba69640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd4d40cc620 0x7fd4d40cf5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:46.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.592+0000 7fd4dba69640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd4d415d890 0x7fd4d41583b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:46.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.592+0000 7fd4dba69640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fd4d415d090 con 0x7fd4d40cc620 2024-08-31T12:28:46.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.592+0000 7fd4dba69640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd4d415cdb0 con 0x7fd4d40ca770 2024-08-31T12:28:46.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.592+0000 7fd4dba69640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fd4d40cc320 con 0x7fd4d415d890 2024-08-31T12:28:46.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.592+0000 7fd4d9004640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd4d40cc620 0x7fd4d40cf5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:46.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.592+0000 7fd4d9004640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd4d40cc620 0x7fd4d40cf5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:45558/0 (socket says 172.21.15.19:45558) 2024-08-31T12:28:46.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.592+0000 7fd4d9004640 1 -- 172.21.15.19:0/350742698 learned_addr learned my addr 172.21.15.19:0/350742698 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:28:46.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.592+0000 7fd4d9805640 1 --2- 172.21.15.19:0/350742698 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4d40ca770 0x7fd4d4156b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:46.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.592+0000 7fd4da006640 1 --2- 172.21.15.19:0/350742698 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd4d415d890 0x7fd4d41583b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:46.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.593+0000 7fd4d9004640 1 -- 172.21.15.19:0/350742698 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd4d415d890 msgr2=0x7fd4d41583b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:46.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.593+0000 7fd4d9004640 1 --2- 172.21.15.19:0/350742698 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd4d415d890 0x7fd4d41583b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:46.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.593+0000 7fd4d9004640 1 -- 172.21.15.19:0/350742698 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4d40ca770 msgr2=0x7fd4d4156b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:46.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.593+0000 7fd4d9004640 1 --2- 172.21.15.19:0/350742698 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4d40ca770 0x7fd4d4156b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:46.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.593+0000 7fd4d9004640 1 -- 172.21.15.19:0/350742698 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4d41567f0 con 0x7fd4d40cc620 2024-08-31T12:28:46.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.593+0000 7fd4d9004640 1 --2- 172.21.15.19:0/350742698 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd4d40cc620 0x7fd4d40cf5a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd4c800c3f0 tx=0x7fd4c800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:46.598 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.593+0000 7fd4d27fc640 1 -- 172.21.15.19:0/350742698 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c8017070 con 0x7fd4d40cc620 2024-08-31T12:28:46.598 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.593+0000 7fd4d27fc640 1 -- 172.21.15.19:0/350742698 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd4c80035f0 con 0x7fd4d40cc620 2024-08-31T12:28:46.598 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.593+0000 7fd4d27fc640 1 -- 172.21.15.19:0/350742698 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c8003a40 con 0x7fd4d40cc620 2024-08-31T12:28:46.598 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.593+0000 7fd4dba69640 1 -- 172.21.15.19:0/350742698 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4d40c6650 con 0x7fd4d40cc620 2024-08-31T12:28:46.598 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.593+0000 7fd4dba69640 1 -- 172.21.15.19:0/350742698 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4d40c6950 con 0x7fd4d40cc620 2024-08-31T12:28:46.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.594+0000 7fd4dba69640 1 -- 172.21.15.19:0/350742698 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4d40b61b0 con 0x7fd4d40cc620 2024-08-31T12:28:46.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.595+0000 7fd4d27fc640 1 -- 172.21.15.19:0/350742698 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fd4c801a070 con 0x7fd4d40cc620 2024-08-31T12:28:46.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.595+0000 7fd4d27fc640 1 --2- 172.21.15.19:0/350742698 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd4c0077a50 0x7fd4c0079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:46.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.595+0000 7fd4d27fc640 1 -- 172.21.15.19:0/350742698 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fd4c8098970 con 0x7fd4d40cc620 2024-08-31T12:28:46.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.596+0000 7fd4d9805640 1 --2- 172.21.15.19:0/350742698 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd4c0077a50 0x7fd4c0079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:46.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.596+0000 7fd4d9805640 1 --2- 172.21.15.19:0/350742698 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd4c0077a50 0x7fd4c0079f60 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fd4c4034000 tx=0x7fd4c4039040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:46.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.600+0000 7fd4d27fc640 1 -- 172.21.15.19:0/350742698 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fd4d40b61b0 con 0x7fd4d40cc620 2024-08-31T12:28:46.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.798+0000 7fd4dba69640 1 -- 172.21.15.19:0/350742698 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd4d40b5970 con 0x7fd4d40cc620 2024-08-31T12:28:46.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.798+0000 7fd4d27fc640 1 -- 172.21.15.19:0/350742698 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) ==== 80+0+0 (secure 0 0 0) 0x7fd4d40b5970 con 0x7fd4d40cc620 2024-08-31T12:28:46.800 INFO:teuthology.orchestra.run.smithi019.stderr:listed 0 entries 2024-08-31T12:28:46.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.800+0000 7fd4dba69640 1 -- 172.21.15.19:0/350742698 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd4c0077a50 msgr2=0x7fd4c0079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:46.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.800+0000 7fd4dba69640 1 --2- 172.21.15.19:0/350742698 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd4c0077a50 0x7fd4c0079f60 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fd4c4034000 tx=0x7fd4c4039040 comp rx=0 tx=0).stop 2024-08-31T12:28:46.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.800+0000 7fd4dba69640 1 -- 172.21.15.19:0/350742698 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd4d40cc620 msgr2=0x7fd4d40cf5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:46.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.800+0000 7fd4dba69640 1 --2- 172.21.15.19:0/350742698 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd4d40cc620 0x7fd4d40cf5a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd4c800c3f0 tx=0x7fd4c800c8b0 comp rx=0 tx=0).stop 2024-08-31T12:28:46.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.800+0000 7fd4dba69640 1 -- 172.21.15.19:0/350742698 shutdown_connections 2024-08-31T12:28:46.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.800+0000 7fd4dba69640 1 --2- 172.21.15.19:0/350742698 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd4c0077a50 0x7fd4c0079f60 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:46.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.800+0000 7fd4dba69640 1 --2- 172.21.15.19:0/350742698 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd4d415d890 0x7fd4d41583b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:46.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.800+0000 7fd4dba69640 1 --2- 172.21.15.19:0/350742698 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd4d40cc620 0x7fd4d40cf5a0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:46.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.800+0000 7fd4dba69640 1 --2- 172.21.15.19:0/350742698 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd4d40ca770 0x7fd4d4156b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:46.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.800+0000 7fd4dba69640 1 -- 172.21.15.19:0/350742698 >> 172.21.15.19:0/350742698 conn(0x7fd4d406e6e0 msgr2=0x7fd4d40638a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:46.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.800+0000 7fd4dba69640 1 -- 172.21.15.19:0/350742698 shutdown_connections 2024-08-31T12:28:46.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.800+0000 7fd4dba69640 1 -- 172.21.15.19:0/350742698 wait complete. 2024-08-31T12:28:46.814 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-08-31T12:28:46.814 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/proc/self/mounts of=/dev/stdout 2024-08-31T12:28:46.829 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-08-31T12:28:46.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.982+0000 7fe69aefa640 1 -- 172.21.15.19:0/2362187325 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe6940cc620 msgr2=0x7fe6940cdab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:46.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.982+0000 7fe69aefa640 1 --2- 172.21.15.19:0/2362187325 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe6940cc620 0x7fe6940cdab0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe6940d1820 tx=0x7fe68402aac0 comp rx=0 tx=0).stop 2024-08-31T12:28:46.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.983+0000 7fe69aefa640 1 -- 172.21.15.19:0/2362187325 shutdown_connections 2024-08-31T12:28:46.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.983+0000 7fe69aefa640 1 --2- 172.21.15.19:0/2362187325 >> v2:172.21.15.43:3300/0 conn(0x7fe6940d1210 0x7fe69415d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:46.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.983+0000 7fe69aefa640 1 --2- 172.21.15.19:0/2362187325 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe6940ca770 0x7fe6940d0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:46.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.983+0000 7fe69aefa640 1 --2- 172.21.15.19:0/2362187325 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe6940cc620 0x7fe6940cdab0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:46.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.983+0000 7fe69aefa640 1 -- 172.21.15.19:0/2362187325 >> 172.21.15.19:0/2362187325 conn(0x7fe69406e6e0 msgr2=0x7fe69406eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:46.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.983+0000 7fe69aefa640 1 -- 172.21.15.19:0/2362187325 shutdown_connections 2024-08-31T12:28:46.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.983+0000 7fe69aefa640 1 -- 172.21.15.19:0/2362187325 wait complete. 2024-08-31T12:28:46.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.984+0000 7fe69aefa640 1 Processor -- start 2024-08-31T12:28:46.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.984+0000 7fe69aefa640 1 -- start start 2024-08-31T12:28:46.989 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.986+0000 7fe69aefa640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe6940ca770 0x7fe6941e6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:46.989 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.986+0000 7fe69aefa640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe6940cc620 0x7fe6941f5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:46.989 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.986+0000 7fe69aefa640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe6940d1210 0x7fe694158450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:46.990 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.986+0000 7fe69aefa640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fe69415d090 con 0x7fe6940d1210 2024-08-31T12:28:46.990 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.986+0000 7fe69aefa640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe69415cdb0 con 0x7fe6940ca770 2024-08-31T12:28:46.990 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.986+0000 7fe69aefa640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fe6940cc320 con 0x7fe6940cc620 2024-08-31T12:28:46.990 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.986+0000 7fe699497640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe6940d1210 0x7fe694158450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:46.990 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.986+0000 7fe699497640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe6940d1210 0x7fe694158450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:45576/0 (socket says 172.21.15.19:45576) 2024-08-31T12:28:46.990 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.986+0000 7fe699497640 1 -- 172.21.15.19:0/3285332847 learned_addr learned my addr 172.21.15.19:0/3285332847 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:28:46.990 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.986+0000 7fe698c96640 1 --2- 172.21.15.19:0/3285332847 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe6940ca770 0x7fe6941e6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:46.990 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.986+0000 7fe693fff640 1 --2- 172.21.15.19:0/3285332847 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe6940cc620 0x7fe6941f5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:46.990 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.986+0000 7fe699497640 1 -- 172.21.15.19:0/3285332847 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe6940cc620 msgr2=0x7fe6941f5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:46.991 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.986+0000 7fe699497640 1 --2- 172.21.15.19:0/3285332847 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe6940cc620 0x7fe6941f5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:46.991 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.986+0000 7fe699497640 1 -- 172.21.15.19:0/3285332847 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe6940ca770 msgr2=0x7fe6941e6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:46.991 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.986+0000 7fe699497640 1 --2- 172.21.15.19:0/3285332847 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe6940ca770 0x7fe6941e6360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:46.991 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.986+0000 7fe699497640 1 -- 172.21.15.19:0/3285332847 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6940b4d20 con 0x7fe6940d1210 2024-08-31T12:28:46.991 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.986+0000 7fe699497640 1 --2- 172.21.15.19:0/3285332847 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe6940d1210 0x7fe694158450 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe68c00e6d0 tx=0x7fe68c00eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:46.991 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.986+0000 7fe6917fa640 1 -- 172.21.15.19:0/3285332847 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe68c00cab0 con 0x7fe6940d1210 2024-08-31T12:28:46.991 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.987+0000 7fe69aefa640 1 -- 172.21.15.19:0/3285332847 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6940b4fc0 con 0x7fe6940d1210 2024-08-31T12:28:46.991 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.987+0000 7fe6917fa640 1 -- 172.21.15.19:0/3285332847 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe68c00cc50 con 0x7fe6940d1210 2024-08-31T12:28:46.991 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.987+0000 7fe6917fa640 1 -- 172.21.15.19:0/3285332847 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe68c005020 con 0x7fe6940d1210 2024-08-31T12:28:46.992 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.987+0000 7fe69aefa640 1 -- 172.21.15.19:0/3285332847 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6940b5280 con 0x7fe6940d1210 2024-08-31T12:28:46.992 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.988+0000 7fe69aefa640 1 -- 172.21.15.19:0/3285332847 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6940cb2c0 con 0x7fe6940d1210 2024-08-31T12:28:46.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.988+0000 7fe6917fa640 1 -- 172.21.15.19:0/3285332847 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fe68c0051e0 con 0x7fe6940d1210 2024-08-31T12:28:46.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.989+0000 7fe6917fa640 1 --2- 172.21.15.19:0/3285332847 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe674077a80 0x7fe674079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:46.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.989+0000 7fe6917fa640 1 -- 172.21.15.19:0/3285332847 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fe68c099f20 con 0x7fe6940d1210 2024-08-31T12:28:46.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.989+0000 7fe698c96640 1 --2- 172.21.15.19:0/3285332847 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe674077a80 0x7fe674079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:46.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.990+0000 7fe698c96640 1 --2- 172.21.15.19:0/3285332847 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe674077a80 0x7fe674079f90 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fe684035000 tx=0x7fe684038040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:46.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:46.993+0000 7fe6917fa640 1 -- 172.21.15.19:0/3285332847 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fe68c062540 con 0x7fe6940d1210 2024-08-31T12:28:47.192 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:47.190+0000 7fe69aefa640 1 -- 172.21.15.19:0/3285332847 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe6941e7010 con 0x7fe6940d1210 2024-08-31T12:28:47.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:47.190+0000 7fe6917fa640 1 -- 172.21.15.19:0/3285332847 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) ==== 80+0+0 (secure 0 0 0) 0x7fe68c06b040 con 0x7fe6940d1210 2024-08-31T12:28:47.193 INFO:teuthology.orchestra.run.smithi019.stderr:listed 0 entries 2024-08-31T12:28:47.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:47.193+0000 7fe69aefa640 1 -- 172.21.15.19:0/3285332847 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe674077a80 msgr2=0x7fe674079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:47.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:47.193+0000 7fe69aefa640 1 --2- 172.21.15.19:0/3285332847 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe674077a80 0x7fe674079f90 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fe684035000 tx=0x7fe684038040 comp rx=0 tx=0).stop 2024-08-31T12:28:47.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:47.193+0000 7fe69aefa640 1 -- 172.21.15.19:0/3285332847 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe6940d1210 msgr2=0x7fe694158450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:47.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:47.193+0000 7fe69aefa640 1 --2- 172.21.15.19:0/3285332847 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe6940d1210 0x7fe694158450 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe68c00e6d0 tx=0x7fe68c00eb90 comp rx=0 tx=0).stop 2024-08-31T12:28:47.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:47.193+0000 7fe69aefa640 1 -- 172.21.15.19:0/3285332847 shutdown_connections 2024-08-31T12:28:47.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:47.193+0000 7fe69aefa640 1 --2- 172.21.15.19:0/3285332847 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe674077a80 0x7fe674079f90 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:47.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:47.193+0000 7fe69aefa640 1 --2- 172.21.15.19:0/3285332847 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe6940d1210 0x7fe694158450 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:47.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:47.193+0000 7fe69aefa640 1 --2- 172.21.15.19:0/3285332847 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe6940cc620 0x7fe6941f5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:47.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:47.193+0000 7fe69aefa640 1 --2- 172.21.15.19:0/3285332847 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe6940ca770 0x7fe6941e6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:47.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:47.193+0000 7fe69aefa640 1 -- 172.21.15.19:0/3285332847 >> 172.21.15.19:0/3285332847 conn(0x7fe69406e6e0 msgr2=0x7fe694063710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:47.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:47.193+0000 7fe69aefa640 1 -- 172.21.15.19:0/3285332847 shutdown_connections 2024-08-31T12:28:47.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:47.193+0000 7fe69aefa640 1 -- 172.21.15.19:0/3285332847 wait complete. 2024-08-31T12:28:47.206 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi019.front.sepia.ceph.com... 2024-08-31T12:28:47.207 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-31T12:28:47.207 DEBUG:teuthology.orchestra.run.smithi019:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-08-31T12:28:47.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi019.stderr:ceph-fuse[33817]: fuse finished with error 0 and tester_r 0 2024-08-31T12:28:47.954 INFO:teuthology.orchestra.run:waiting for 300 2024-08-31T12:28:47.955 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi019.front.sepia.ceph.com 2024-08-31T12:28:47.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-31T12:28:47.955 DEBUG:teuthology.orchestra.run.smithi019:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-08-31T12:28:47.973 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-31T12:28:47.974 DEBUG:teuthology.orchestra.run.smithi019:> ip netns list 2024-08-31T12:28:48.037 INFO:teuthology.orchestra.run.smithi019.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-08-31T12:28:48.038 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-31T12:28:48.038 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-08-31T12:28:48.106 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-08-31T12:28:48.106 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip link delete ceph-brx 2024-08-31T12:28:48.214 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-08-31T12:28:48.225 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-08-31T12:28:48.225 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-31T12:28:48.351 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.349+0000 7f80ab78b640 1 -- 172.21.15.19:0/2424909285 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8090004d40 msgr2=0x7f8090005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:48.352 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.349+0000 7f80ab78b640 1 --2- 172.21.15.19:0/2424909285 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8090004d40 0x7f8090005150 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f80a40710d0 tx=0x7f8090005830 comp rx=0 tx=0).stop 2024-08-31T12:28:48.352 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.349+0000 7f80a9d28640 1 -- 172.21.15.19:0/2424909285 reap_dead start 2024-08-31T12:28:48.352 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.349+0000 7f80ab78b640 1 -- 172.21.15.19:0/2424909285 shutdown_connections 2024-08-31T12:28:48.352 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.349+0000 7f80ab78b640 1 -- 172.21.15.19:0/2424909285 >> 172.21.15.19:0/2424909285 conn(0x7f80a40ca670 msgr2=0x7f80a40cd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:48.352 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.349+0000 7f80ab78b640 1 -- 172.21.15.19:0/2424909285 shutdown_connections 2024-08-31T12:28:48.352 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.349+0000 7f80ab78b640 1 -- 172.21.15.19:0/2424909285 wait complete. 2024-08-31T12:28:48.353 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.351+0000 7f80ab78b640 1 Processor -- start 2024-08-31T12:28:48.353 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.351+0000 7f80ab78b640 1 -- start start 2024-08-31T12:28:48.356 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.352+0000 7f80ab78b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f80a413e0f0 0x7f80a4155c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:48.356 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.352+0000 7f80ab78b640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80a413e500 0x7f80a41561c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:48.356 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.352+0000 7f80ab78b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f80a414c2d0 0x7f80a414c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:48.356 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.353+0000 7f80ab78b640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f80a40c7650 con 0x7f80a414c2d0 2024-08-31T12:28:48.356 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.353+0000 7f80ab78b640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f80a40ce140 con 0x7f80a413e500 2024-08-31T12:28:48.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.353+0000 7f80ab78b640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f80a40c72e0 con 0x7f80a413e0f0 2024-08-31T12:28:48.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.353+0000 7f80a9d28640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f80a414c2d0 0x7f80a414c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:48.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.353+0000 7f80a9d28640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f80a414c2d0 0x7f80a414c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:43818/0 (socket says 172.21.15.19:43818) 2024-08-31T12:28:48.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.353+0000 7f80a9d28640 1 -- 172.21.15.19:0/314105577 learned_addr learned my addr 172.21.15.19:0/314105577 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:28:48.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.353+0000 7f80a8d26640 1 --2- 172.21.15.19:0/314105577 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80a413e500 0x7f80a41561c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:48.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.353+0000 7f80a9527640 1 --2- 172.21.15.19:0/314105577 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f80a413e0f0 0x7f80a4155c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:48.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.353+0000 7f80a9d28640 1 -- 172.21.15.19:0/314105577 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f80a413e0f0 msgr2=0x7f80a4155c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:48.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.353+0000 7f80a9d28640 1 --2- 172.21.15.19:0/314105577 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f80a413e0f0 0x7f80a4155c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:48.358 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.353+0000 7f80a9d28640 1 -- 172.21.15.19:0/314105577 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80a413e500 msgr2=0x7f80a41561c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:48.358 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.353+0000 7f80a9d28640 1 --2- 172.21.15.19:0/314105577 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80a413e500 0x7f80a41561c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:48.358 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.353+0000 7f80a9d28640 1 -- 172.21.15.19:0/314105577 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80a40c9bd0 con 0x7f80a414c2d0 2024-08-31T12:28:48.358 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.353+0000 7f80a9527640 1 --2- 172.21.15.19:0/314105577 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f80a413e0f0 0x7f80a4155c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:28:48.358 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.353+0000 7f80a9d28640 1 --2- 172.21.15.19:0/314105577 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f80a414c2d0 0x7f80a414c6e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f809c036040 tx=0x7f809c033da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:48.358 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.354+0000 7f8099ffb640 1 -- 172.21.15.19:0/314105577 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f809c043070 con 0x7f80a414c2d0 2024-08-31T12:28:48.358 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.354+0000 7f8099ffb640 1 -- 172.21.15.19:0/314105577 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f809c003b90 con 0x7f80a414c2d0 2024-08-31T12:28:48.358 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.354+0000 7f8099ffb640 1 -- 172.21.15.19:0/314105577 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f809c038040 con 0x7f80a414c2d0 2024-08-31T12:28:48.359 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.354+0000 7f80ab78b640 1 -- 172.21.15.19:0/314105577 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80a4156800 con 0x7f80a414c2d0 2024-08-31T12:28:48.359 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.354+0000 7f80ab78b640 1 -- 172.21.15.19:0/314105577 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80a40c87a0 con 0x7f80a414c2d0 2024-08-31T12:28:48.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.355+0000 7f80ab78b640 1 -- 172.21.15.19:0/314105577 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80a414ad00 con 0x7f80a414c2d0 2024-08-31T12:28:48.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.356+0000 7f8099ffb640 1 -- 172.21.15.19:0/314105577 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f809c004030 con 0x7f80a414c2d0 2024-08-31T12:28:48.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.356+0000 7f8099ffb640 1 --2- 172.21.15.19:0/314105577 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8088077a50 0x7f8088079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:48.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.356+0000 7f8099ffb640 1 -- 172.21.15.19:0/314105577 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f809c0b9a10 con 0x7f80a414c2d0 2024-08-31T12:28:48.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.356+0000 7f80a9527640 1 --2- 172.21.15.19:0/314105577 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8088077a50 0x7f8088079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:48.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.357+0000 7f80a9527640 1 --2- 172.21.15.19:0/314105577 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8088077a50 0x7f8088079f60 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f80a406d390 tx=0x7f80a00067a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:48.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.365+0000 7f8099ffb640 1 -- 172.21.15.19:0/314105577 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f80a414ad00 con 0x7f80a414c2d0 2024-08-31T12:28:48.548 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.546+0000 7f80ab78b640 1 -- 172.21.15.19:0/314105577 --> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f80a406e760 con 0x7f8088077a50 2024-08-31T12:28:48.554 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.552+0000 7f8099ffb640 1 -- 172.21.15.19:0/314105577 <== mgr.4100 v2:172.21.15.43:6800/2824336655 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+254489 (secure 0 0 0) 0x7f80a406e760 con 0x7f8088077a50 2024-08-31T12:28:48.557 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:28:48.558 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-08-31T12:28:48.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.555+0000 7f80ab78b640 1 -- 172.21.15.19:0/314105577 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8088077a50 msgr2=0x7f8088079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:48.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.555+0000 7f80ab78b640 1 --2- 172.21.15.19:0/314105577 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8088077a50 0x7f8088079f60 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f80a406d390 tx=0x7f80a00067a0 comp rx=0 tx=0).stop 2024-08-31T12:28:48.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.555+0000 7f80ab78b640 1 -- 172.21.15.19:0/314105577 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f80a414c2d0 msgr2=0x7f80a414c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:48.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.555+0000 7f80ab78b640 1 --2- 172.21.15.19:0/314105577 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f80a414c2d0 0x7f80a414c6e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f809c036040 tx=0x7f809c033da0 comp rx=0 tx=0).stop 2024-08-31T12:28:48.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.555+0000 7f80ab78b640 1 -- 172.21.15.19:0/314105577 shutdown_connections 2024-08-31T12:28:48.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.555+0000 7f80ab78b640 1 --2- 172.21.15.19:0/314105577 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8088077a50 0x7f8088079f60 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:48.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.555+0000 7f80ab78b640 1 --2- 172.21.15.19:0/314105577 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f80a414c2d0 0x7f80a414c6e0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:48.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.555+0000 7f80ab78b640 1 --2- 172.21.15.19:0/314105577 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80a413e500 0x7f80a41561c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:48.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.555+0000 7f80ab78b640 1 --2- 172.21.15.19:0/314105577 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f80a413e0f0 0x7f80a4155c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:48.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.555+0000 7f80ab78b640 1 -- 172.21.15.19:0/314105577 >> 172.21.15.19:0/314105577 conn(0x7f80a40ca670 msgr2=0x7f80a4157e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:48.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.555+0000 7f80ab78b640 1 -- 172.21.15.19:0/314105577 shutdown_connections 2024-08-31T12:28:48.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.555+0000 7f80ab78b640 1 -- 172.21.15.19:0/314105577 wait complete. 2024-08-31T12:28:48.569 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":408,"stamp":"2024-08-31T12:28:46.706392+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":13438780938,"num_objects":3466,"num_object_clones":0,"num_object_copies":6932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3466,"num_whiteouts":0,"num_read":96,"num_read_kb":119,"num_write":921562,"num_write_kb":16804360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":186483,"ondisk_log_size":186483,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":28675632,"kb_used_data":26250480,"kb_used_omap":12,"kb_used_meta":2425011,"kb_avail":726299088,"statfs":{"total":773094113280,"available":743730266112,"internally_reserved":0,"allocated":26880491520,"data_stored":26878225604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":2483211825},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5476,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003676"},"pg_stats":[{"pgid":"3.3f","version":"23'2724","reported_seq":2747,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.201520+0000","last_change":"2024-08-31T12:15:57.306560+0000","last_active":"2024-08-31T12:28:19.201520+0000","last_peered":"2024-08-31T12:28:19.201520+0000","last_clean":"2024-08-31T12:28:19.201520+0000","last_became_active":"2024-08-31T12:15:57.306148+0000","last_became_peered":"2024-08-31T12:15:57.306148+0000","last_unstale":"2024-08-31T12:28:19.201520+0000","last_undegraded":"2024-08-31T12:28:19.201520+0000","last_fullsized":"2024-08-31T12:28:19.201520+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2724,"log_dups_size":0,"ondisk_log_size":2724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:35:53.831742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180377297,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13788,"num_write_kb":231441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"23'3448","reported_seq":3471,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.169946+0000","last_change":"2024-08-31T12:15:57.307314+0000","last_active":"2024-08-31T12:28:19.169946+0000","last_peered":"2024-08-31T12:28:19.169946+0000","last_clean":"2024-08-31T12:28:19.169946+0000","last_became_active":"2024-08-31T12:15:57.306940+0000","last_became_peered":"2024-08-31T12:15:57.306940+0000","last_unstale":"2024-08-31T12:28:19.169946+0000","last_undegraded":"2024-08-31T12:28:19.169946+0000","last_fullsized":"2024-08-31T12:28:19.169946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3448,"log_dups_size":0,"ondisk_log_size":3448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:38:30.321214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264252554,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17611,"num_write_kb":328696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"23'3134","reported_seq":3157,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.114777+0000","last_change":"2024-08-31T12:15:57.309372+0000","last_active":"2024-08-31T12:28:19.114777+0000","last_peered":"2024-08-31T12:28:19.114777+0000","last_clean":"2024-08-31T12:28:19.114777+0000","last_became_active":"2024-08-31T12:15:57.305795+0000","last_became_peered":"2024-08-31T12:15:57.305795+0000","last_unstale":"2024-08-31T12:28:19.114777+0000","last_undegraded":"2024-08-31T12:28:19.114777+0000","last_fullsized":"2024-08-31T12:28:19.114777+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3134,"log_dups_size":0,"ondisk_log_size":3134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:37:45.891394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213942725,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15777,"num_write_kb":272211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"23'2785","reported_seq":2808,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.158289+0000","last_change":"2024-08-31T12:15:57.307339+0000","last_active":"2024-08-31T12:28:19.158289+0000","last_peered":"2024-08-31T12:28:19.158289+0000","last_clean":"2024-08-31T12:28:19.158289+0000","last_became_active":"2024-08-31T12:15:57.306957+0000","last_became_peered":"2024-08-31T12:15:57.306957+0000","last_unstale":"2024-08-31T12:28:19.158289+0000","last_undegraded":"2024-08-31T12:28:19.158289+0000","last_fullsized":"2024-08-31T12:28:19.158289+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2785,"log_dups_size":0,"ondisk_log_size":2785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:43:33.705099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14240,"num_write_kb":241389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"23'3277","reported_seq":3299,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.203476+0000","last_change":"2024-08-31T12:15:57.317492+0000","last_active":"2024-08-31T12:28:19.203476+0000","last_peered":"2024-08-31T12:28:19.203476+0000","last_clean":"2024-08-31T12:28:19.203476+0000","last_became_active":"2024-08-31T12:15:57.317079+0000","last_became_peered":"2024-08-31T12:15:57.317079+0000","last_unstale":"2024-08-31T12:28:19.203476+0000","last_undegraded":"2024-08-31T12:28:19.203476+0000","last_fullsized":"2024-08-31T12:28:19.203476+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3277,"log_dups_size":0,"ondisk_log_size":3277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:44:40.712699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247474530,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15722,"num_write_kb":304743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"23'2512","reported_seq":2534,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.179506+0000","last_change":"2024-08-31T12:15:57.307505+0000","last_active":"2024-08-31T12:28:19.179506+0000","last_peered":"2024-08-31T12:28:19.179506+0000","last_clean":"2024-08-31T12:28:19.179506+0000","last_became_active":"2024-08-31T12:15:57.307010+0000","last_became_peered":"2024-08-31T12:15:57.307010+0000","last_unstale":"2024-08-31T12:28:19.179506+0000","last_undegraded":"2024-08-31T12:28:19.179506+0000","last_fullsized":"2024-08-31T12:28:19.179506+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2512,"log_dups_size":0,"ondisk_log_size":2512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:39:31.100565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180356168,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12564,"num_write_kb":226449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"23'2626","reported_seq":2648,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.143842+0000","last_change":"2024-08-31T12:15:57.316892+0000","last_active":"2024-08-31T12:28:19.143842+0000","last_peered":"2024-08-31T12:28:19.143842+0000","last_clean":"2024-08-31T12:28:19.143842+0000","last_became_active":"2024-08-31T12:15:57.316504+0000","last_became_peered":"2024-08-31T12:15:57.316504+0000","last_unstale":"2024-08-31T12:28:19.143842+0000","last_undegraded":"2024-08-31T12:28:19.143842+0000","last_fullsized":"2024-08-31T12:28:19.143842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2626,"log_dups_size":0,"ondisk_log_size":2626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:05:48.614918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201398355,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13306,"num_write_kb":250051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"23'3158","reported_seq":3173,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.134179+0000","last_change":"2024-08-31T12:16:40.694493+0000","last_active":"2024-08-31T12:28:19.134179+0000","last_peered":"2024-08-31T12:28:19.134179+0000","last_clean":"2024-08-31T12:28:19.134179+0000","last_became_active":"2024-08-31T12:16:40.694165+0000","last_became_peered":"2024-08-31T12:16:40.694165+0000","last_unstale":"2024-08-31T12:28:19.134179+0000","last_undegraded":"2024-08-31T12:28:19.134179+0000","last_fullsized":"2024-08-31T12:28:19.134179+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3158,"log_dups_size":0,"ondisk_log_size":3158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:55:57.961306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226497620,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15619,"num_write_kb":283880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.37","version":"23'3491","reported_seq":3513,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.201405+0000","last_change":"2024-08-31T12:15:57.316723+0000","last_active":"2024-08-31T12:28:19.201405+0000","last_peered":"2024-08-31T12:28:19.201405+0000","last_clean":"2024-08-31T12:28:19.201405+0000","last_became_active":"2024-08-31T12:15:57.316354+0000","last_became_peered":"2024-08-31T12:15:57.316354+0000","last_unstale":"2024-08-31T12:28:19.201405+0000","last_undegraded":"2024-08-31T12:28:19.201405+0000","last_fullsized":"2024-08-31T12:28:19.201405+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3491,"log_dups_size":0,"ondisk_log_size":3491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:31:19.860950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251662082,"num_objects":61,"num_object_clones":0,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18100,"num_write_kb":318223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"23'2954","reported_seq":2976,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.206584+0000","last_change":"2024-08-31T12:15:57.311161+0000","last_active":"2024-08-31T12:28:19.206584+0000","last_peered":"2024-08-31T12:28:19.206584+0000","last_clean":"2024-08-31T12:28:19.206584+0000","last_became_active":"2024-08-31T12:15:57.310807+0000","last_became_peered":"2024-08-31T12:15:57.310807+0000","last_unstale":"2024-08-31T12:28:19.206584+0000","last_undegraded":"2024-08-31T12:28:19.206584+0000","last_fullsized":"2024-08-31T12:28:19.206584+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2954,"log_dups_size":0,"ondisk_log_size":2954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:56:03.280339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234894029,"num_objects":59,"num_object_clones":0,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15073,"num_write_kb":289757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"23'2880","reported_seq":2902,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.165528+0000","last_change":"2024-08-31T12:15:57.300930+0000","last_active":"2024-08-31T12:28:19.165528+0000","last_peered":"2024-08-31T12:28:19.165528+0000","last_clean":"2024-08-31T12:28:19.165528+0000","last_became_active":"2024-08-31T12:15:57.300610+0000","last_became_peered":"2024-08-31T12:15:57.300610+0000","last_unstale":"2024-08-31T12:28:19.165528+0000","last_undegraded":"2024-08-31T12:28:19.165528+0000","last_fullsized":"2024-08-31T12:28:19.165528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2880,"log_dups_size":0,"ondisk_log_size":2880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:42:39.731742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201326592,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14139,"num_write_kb":253227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"23'2246","reported_seq":2268,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.123192+0000","last_change":"2024-08-31T12:15:57.303151+0000","last_active":"2024-08-31T12:28:19.123192+0000","last_peered":"2024-08-31T12:28:19.123192+0000","last_clean":"2024-08-31T12:28:19.123192+0000","last_became_active":"2024-08-31T12:15:57.302622+0000","last_became_peered":"2024-08-31T12:15:57.302622+0000","last_unstale":"2024-08-31T12:28:19.123192+0000","last_undegraded":"2024-08-31T12:28:19.123192+0000","last_fullsized":"2024-08-31T12:28:19.123192+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2246,"log_dups_size":0,"ondisk_log_size":2246,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:49:18.493629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176192645,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11403,"num_write_kb":217791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"23'2948","reported_seq":2970,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.215841+0000","last_change":"2024-08-31T12:15:57.305676+0000","last_active":"2024-08-31T12:28:19.215841+0000","last_peered":"2024-08-31T12:28:19.215841+0000","last_clean":"2024-08-31T12:28:19.215841+0000","last_became_active":"2024-08-31T12:15:57.305321+0000","last_became_peered":"2024-08-31T12:15:57.305321+0000","last_unstale":"2024-08-31T12:28:19.215841+0000","last_undegraded":"2024-08-31T12:28:19.215841+0000","last_fullsized":"2024-08-31T12:28:19.215841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2948,"log_dups_size":0,"ondisk_log_size":2948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:42:39.731742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188840677,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15214,"num_write_kb":245235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"23'3122","reported_seq":3144,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.194400+0000","last_change":"2024-08-31T12:15:57.309671+0000","last_active":"2024-08-31T12:28:19.194400+0000","last_peered":"2024-08-31T12:28:19.194400+0000","last_clean":"2024-08-31T12:28:19.194400+0000","last_became_active":"2024-08-31T12:15:57.307013+0000","last_became_peered":"2024-08-31T12:15:57.307013+0000","last_unstale":"2024-08-31T12:28:19.194400+0000","last_undegraded":"2024-08-31T12:28:19.194400+0000","last_fullsized":"2024-08-31T12:28:19.194400+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3122,"log_dups_size":0,"ondisk_log_size":3122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:18:55.451164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213986201,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15743,"num_write_kb":271989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"23'2952","reported_seq":2974,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.181372+0000","last_change":"2024-08-31T12:15:57.298150+0000","last_active":"2024-08-31T12:28:19.181372+0000","last_peered":"2024-08-31T12:28:19.181372+0000","last_clean":"2024-08-31T12:28:19.181372+0000","last_became_active":"2024-08-31T12:15:57.297182+0000","last_became_peered":"2024-08-31T12:15:57.297182+0000","last_unstale":"2024-08-31T12:28:19.181372+0000","last_undegraded":"2024-08-31T12:28:19.181372+0000","last_fullsized":"2024-08-31T12:28:19.181372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2952,"log_dups_size":0,"ondisk_log_size":2952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:16:42.625396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213913551,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14445,"num_write_kb":266758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"23'2537","reported_seq":2553,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.202231+0000","last_change":"2024-08-31T12:16:41.075080+0000","last_active":"2024-08-31T12:28:19.202231+0000","last_peered":"2024-08-31T12:28:19.202231+0000","last_clean":"2024-08-31T12:28:19.202231+0000","last_became_active":"2024-08-31T12:16:41.074773+0000","last_became_peered":"2024-08-31T12:16:41.074773+0000","last_unstale":"2024-08-31T12:28:19.202231+0000","last_undegraded":"2024-08-31T12:28:19.202231+0000","last_fullsized":"2024-08-31T12:28:19.202231+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2537,"log_dups_size":0,"ondisk_log_size":2537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:10:45.435137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184599153,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12917,"num_write_kb":231981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"23'3715","reported_seq":3737,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.196629+0000","last_change":"2024-08-31T12:15:57.308899+0000","last_active":"2024-08-31T12:28:19.196629+0000","last_peered":"2024-08-31T12:28:19.196629+0000","last_clean":"2024-08-31T12:28:19.196629+0000","last_became_active":"2024-08-31T12:15:57.308388+0000","last_became_peered":"2024-08-31T12:15:57.308388+0000","last_unstale":"2024-08-31T12:28:19.196629+0000","last_undegraded":"2024-08-31T12:28:19.196629+0000","last_fullsized":"2024-08-31T12:28:19.196629+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3715,"log_dups_size":0,"ondisk_log_size":3715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:54:03.871840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251747648,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18708,"num_write_kb":320814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"23'2138","reported_seq":2160,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.124811+0000","last_change":"2024-08-31T12:15:57.300201+0000","last_active":"2024-08-31T12:28:19.124811+0000","last_peered":"2024-08-31T12:28:19.124811+0000","last_clean":"2024-08-31T12:28:19.124811+0000","last_became_active":"2024-08-31T12:15:57.299826+0000","last_became_peered":"2024-08-31T12:15:57.299826+0000","last_unstale":"2024-08-31T12:28:19.124811+0000","last_undegraded":"2024-08-31T12:28:19.124811+0000","last_fullsized":"2024-08-31T12:28:19.124811+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2138,"log_dups_size":0,"ondisk_log_size":2138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:14:34.488102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167791563,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10514,"num_write_kb":205964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"23'2668","reported_seq":2690,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.054487+0000","last_change":"2024-08-31T12:15:57.317606+0000","last_active":"2024-08-31T12:28:19.054487+0000","last_peered":"2024-08-31T12:28:19.054487+0000","last_clean":"2024-08-31T12:28:19.054487+0000","last_became_active":"2024-08-31T12:15:57.317185+0000","last_became_peered":"2024-08-31T12:15:57.317185+0000","last_unstale":"2024-08-31T12:28:19.054487+0000","last_undegraded":"2024-08-31T12:28:19.054487+0000","last_fullsized":"2024-08-31T12:28:19.054487+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2668,"log_dups_size":0,"ondisk_log_size":2668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:25:46.346490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209762270,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13163,"num_write_kb":257524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"23'2441","reported_seq":2463,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.104779+0000","last_change":"2024-08-31T12:15:57.298778+0000","last_active":"2024-08-31T12:28:19.104779+0000","last_peered":"2024-08-31T12:28:19.104779+0000","last_clean":"2024-08-31T12:28:19.104779+0000","last_became_active":"2024-08-31T12:15:57.298430+0000","last_became_peered":"2024-08-31T12:15:57.298430+0000","last_unstale":"2024-08-31T12:28:19.104779+0000","last_undegraded":"2024-08-31T12:28:19.104779+0000","last_fullsized":"2024-08-31T12:28:19.104779+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2441,"log_dups_size":0,"ondisk_log_size":2441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:22:13.435300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205530518,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12230,"num_write_kb":249721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"23'2939","reported_seq":2961,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.228814+0000","last_change":"2024-08-31T12:15:57.317176+0000","last_active":"2024-08-31T12:28:19.228814+0000","last_peered":"2024-08-31T12:28:19.228814+0000","last_clean":"2024-08-31T12:28:19.228814+0000","last_became_active":"2024-08-31T12:15:57.316870+0000","last_became_peered":"2024-08-31T12:15:57.316870+0000","last_unstale":"2024-08-31T12:28:19.228814+0000","last_undegraded":"2024-08-31T12:28:19.228814+0000","last_fullsized":"2024-08-31T12:28:19.228814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2939,"log_dups_size":0,"ondisk_log_size":2939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:43:12.703971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213923005,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14700,"num_write_kb":267872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"23'3056","reported_seq":3072,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.122771+0000","last_change":"2024-08-31T12:16:40.784525+0000","last_active":"2024-08-31T12:28:19.122771+0000","last_peered":"2024-08-31T12:28:19.122771+0000","last_clean":"2024-08-31T12:28:19.122771+0000","last_became_active":"2024-08-31T12:16:40.784204+0000","last_became_peered":"2024-08-31T12:16:40.784204+0000","last_unstale":"2024-08-31T12:28:19.122771+0000","last_undegraded":"2024-08-31T12:28:19.122771+0000","last_fullsized":"2024-08-31T12:28:19.122771+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3056,"log_dups_size":0,"ondisk_log_size":3056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:51:38.173279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226493660,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15308,"num_write_kb":282531,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"23'2827","reported_seq":2849,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.207366+0000","last_change":"2024-08-31T12:15:57.304032+0000","last_active":"2024-08-31T12:28:19.207366+0000","last_peered":"2024-08-31T12:28:19.207366+0000","last_clean":"2024-08-31T12:28:19.207366+0000","last_became_active":"2024-08-31T12:15:57.303401+0000","last_became_peered":"2024-08-31T12:15:57.303401+0000","last_unstale":"2024-08-31T12:28:19.207366+0000","last_undegraded":"2024-08-31T12:28:19.207366+0000","last_fullsized":"2024-08-31T12:28:19.207366+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2827,"log_dups_size":0,"ondisk_log_size":2827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:56:03.280339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205588184,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14070,"num_write_kb":257066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"23'2468","reported_seq":2490,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.209009+0000","last_change":"2024-08-31T12:15:57.305412+0000","last_active":"2024-08-31T12:28:19.209009+0000","last_peered":"2024-08-31T12:28:19.209009+0000","last_clean":"2024-08-31T12:28:19.209009+0000","last_became_active":"2024-08-31T12:15:57.305075+0000","last_became_peered":"2024-08-31T12:15:57.305075+0000","last_unstale":"2024-08-31T12:28:19.209009+0000","last_undegraded":"2024-08-31T12:28:19.209009+0000","last_fullsized":"2024-08-31T12:28:19.209009+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:30:13.715505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184555785,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12290,"num_write_kb":229419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"23'3164","reported_seq":3186,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.223770+0000","last_change":"2024-08-31T12:15:57.300179+0000","last_active":"2024-08-31T12:28:19.223770+0000","last_peered":"2024-08-31T12:28:19.223770+0000","last_clean":"2024-08-31T12:28:19.223770+0000","last_became_active":"2024-08-31T12:15:57.299861+0000","last_became_peered":"2024-08-31T12:15:57.299861+0000","last_unstale":"2024-08-31T12:28:19.223770+0000","last_undegraded":"2024-08-31T12:28:19.223770+0000","last_fullsized":"2024-08-31T12:28:19.223770+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3164,"log_dups_size":0,"ondisk_log_size":3164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:47:51.267409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243322604,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15709,"num_write_kb":300530,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"23'2729","reported_seq":2751,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.205829+0000","last_change":"2024-08-31T12:15:57.308519+0000","last_active":"2024-08-31T12:28:19.205829+0000","last_peered":"2024-08-31T12:28:19.205829+0000","last_clean":"2024-08-31T12:28:19.205829+0000","last_became_active":"2024-08-31T12:15:57.308189+0000","last_became_peered":"2024-08-31T12:15:57.308189+0000","last_unstale":"2024-08-31T12:28:19.205829+0000","last_undegraded":"2024-08-31T12:28:19.205829+0000","last_fullsized":"2024-08-31T12:28:19.205829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2729,"log_dups_size":0,"ondisk_log_size":2729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:37:02.435678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189169648,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13404,"num_write_kb":238420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"23'3082","reported_seq":3104,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.211281+0000","last_change":"2024-08-31T12:15:57.308321+0000","last_active":"2024-08-31T12:28:19.211281+0000","last_peered":"2024-08-31T12:28:19.211281+0000","last_clean":"2024-08-31T12:28:19.211281+0000","last_became_active":"2024-08-31T12:15:57.304514+0000","last_became_peered":"2024-08-31T12:15:57.304514+0000","last_unstale":"2024-08-31T12:28:19.211281+0000","last_undegraded":"2024-08-31T12:28:19.211281+0000","last_fullsized":"2024-08-31T12:28:19.211281+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3082,"log_dups_size":0,"ondisk_log_size":3082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:09:49.617826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222298838,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15326,"num_write_kb":278485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"23'2570","reported_seq":2592,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.125425+0000","last_change":"2024-08-31T12:15:57.299268+0000","last_active":"2024-08-31T12:28:19.125425+0000","last_peered":"2024-08-31T12:28:19.125425+0000","last_clean":"2024-08-31T12:28:19.125425+0000","last_became_active":"2024-08-31T12:15:57.298766+0000","last_became_peered":"2024-08-31T12:15:57.298766+0000","last_unstale":"2024-08-31T12:28:19.125425+0000","last_undegraded":"2024-08-31T12:28:19.125425+0000","last_fullsized":"2024-08-31T12:28:19.125425+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2570,"log_dups_size":0,"ondisk_log_size":2570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:37:45.891394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188773348,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12878,"num_write_kb":235916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"23'2321","reported_seq":2344,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.234241+0000","last_change":"2024-08-31T12:15:57.310121+0000","last_active":"2024-08-31T12:28:19.234241+0000","last_peered":"2024-08-31T12:28:19.234241+0000","last_clean":"2024-08-31T12:28:19.234241+0000","last_became_active":"2024-08-31T12:15:57.309440+0000","last_became_peered":"2024-08-31T12:15:57.309440+0000","last_unstale":"2024-08-31T12:28:19.234241+0000","last_undegraded":"2024-08-31T12:28:19.234241+0000","last_fullsized":"2024-08-31T12:28:19.234241+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2321,"log_dups_size":0,"ondisk_log_size":2321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:53:19.523811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180357122,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11238,"num_write_kb":221098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"23'3439","reported_seq":3462,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.198926+0000","last_change":"2024-08-31T12:15:57.306214+0000","last_active":"2024-08-31T12:28:19.198926+0000","last_peered":"2024-08-31T12:28:19.198926+0000","last_clean":"2024-08-31T12:28:19.198926+0000","last_became_active":"2024-08-31T12:15:57.305878+0000","last_became_peered":"2024-08-31T12:15:57.305878+0000","last_unstale":"2024-08-31T12:28:19.198926+0000","last_undegraded":"2024-08-31T12:28:19.198926+0000","last_fullsized":"2024-08-31T12:28:19.198926+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3439,"log_dups_size":0,"ondisk_log_size":3439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:05:48.614918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251668879,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17281,"num_write_kb":314954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"23'2868","reported_seq":2891,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.151062+0000","last_change":"2024-08-31T12:15:57.307376+0000","last_active":"2024-08-31T12:28:19.151062+0000","last_peered":"2024-08-31T12:28:19.151062+0000","last_clean":"2024-08-31T12:28:19.151062+0000","last_became_active":"2024-08-31T12:15:57.307056+0000","last_became_peered":"2024-08-31T12:15:57.307056+0000","last_unstale":"2024-08-31T12:28:19.151062+0000","last_undegraded":"2024-08-31T12:28:19.151062+0000","last_fullsized":"2024-08-31T12:28:19.151062+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2868,"log_dups_size":0,"ondisk_log_size":2868,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:52:18.004608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222298112,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14386,"num_write_kb":274682,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"23'3113","reported_seq":3136,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.193855+0000","last_change":"2024-08-31T12:15:57.313798+0000","last_active":"2024-08-31T12:28:19.193855+0000","last_peered":"2024-08-31T12:28:19.193855+0000","last_clean":"2024-08-31T12:28:19.193855+0000","last_became_active":"2024-08-31T12:15:57.313463+0000","last_became_peered":"2024-08-31T12:15:57.313463+0000","last_unstale":"2024-08-31T12:28:19.193855+0000","last_undegraded":"2024-08-31T12:28:19.193855+0000","last_fullsized":"2024-08-31T12:28:19.193855+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3113,"log_dups_size":0,"ondisk_log_size":3113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:47:51.267409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260049613,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15263,"num_write_kb":315030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"23'2280","reported_seq":2302,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.088996+0000","last_change":"2024-08-31T12:15:57.293350+0000","last_active":"2024-08-31T12:28:19.088996+0000","last_peered":"2024-08-31T12:28:19.088996+0000","last_clean":"2024-08-31T12:28:19.088996+0000","last_became_active":"2024-08-31T12:15:57.292981+0000","last_became_peered":"2024-08-31T12:15:57.292981+0000","last_unstale":"2024-08-31T12:28:19.088996+0000","last_undegraded":"2024-08-31T12:28:19.088996+0000","last_fullsized":"2024-08-31T12:28:19.088996+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2280,"log_dups_size":0,"ondisk_log_size":2280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:52:00.312013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180377946,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11142,"num_write_kb":220690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"23'569","reported_seq":593,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:56.212722+0000","last_change":"2024-08-31T12:15:56.293497+0000","last_active":"2024-08-31T12:25:56.212722+0000","last_peered":"2024-08-31T12:25:56.212722+0000","last_clean":"2024-08-31T12:25:56.212722+0000","last_became_active":"2024-08-31T12:15:56.293164+0000","last_became_peered":"2024-08-31T12:15:56.293164+0000","last_unstale":"2024-08-31T12:25:56.212722+0000","last_undegraded":"2024-08-31T12:25:56.212722+0000","last_fullsized":"2024-08-31T12:25:56.212722+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":569,"log_dups_size":0,"ondisk_log_size":569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:43:17.443433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4195367,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":566,"num_write_kb":4210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"23'2388","reported_seq":2410,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.228535+0000","last_change":"2024-08-31T12:15:57.312738+0000","last_active":"2024-08-31T12:28:19.228535+0000","last_peered":"2024-08-31T12:28:19.228535+0000","last_clean":"2024-08-31T12:28:19.228535+0000","last_became_active":"2024-08-31T12:15:57.312360+0000","last_became_peered":"2024-08-31T12:15:57.312360+0000","last_unstale":"2024-08-31T12:28:19.228535+0000","last_undegraded":"2024-08-31T12:28:19.228535+0000","last_fullsized":"2024-08-31T12:28:19.228535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2388,"log_dups_size":0,"ondisk_log_size":2388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:22:13.435300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163584295,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11807,"num_write_kb":207035,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"23'5","reported_seq":29,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:56.196855+0000","last_change":"2024-08-31T12:15:56.283526+0000","last_active":"2024-08-31T12:25:56.196855+0000","last_peered":"2024-08-31T12:25:56.196855+0000","last_clean":"2024-08-31T12:25:56.196855+0000","last_became_active":"2024-08-31T12:15:56.283147+0000","last_became_peered":"2024-08-31T12:15:56.283147+0000","last_unstale":"2024-08-31T12:25:56.196855+0000","last_undegraded":"2024-08-31T12:25:56.196855+0000","last_fullsized":"2024-08-31T12:25:56.196855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:51:59.306912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"23'4","reported_seq":28,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:36.210659+0000","last_change":"2024-08-31T12:15:56.291170+0000","last_active":"2024-08-31T12:25:36.210659+0000","last_peered":"2024-08-31T12:25:36.210659+0000","last_clean":"2024-08-31T12:25:36.210659+0000","last_became_active":"2024-08-31T12:15:56.290835+0000","last_became_peered":"2024-08-31T12:15:56.290835+0000","last_unstale":"2024-08-31T12:25:36.210659+0000","last_undegraded":"2024-08-31T12:25:36.210659+0000","last_fullsized":"2024-08-31T12:25:36.210659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:15:51.561831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"23'2535","reported_seq":2557,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.224638+0000","last_change":"2024-08-31T12:15:57.308183+0000","last_active":"2024-08-31T12:28:19.224638+0000","last_peered":"2024-08-31T12:28:19.224638+0000","last_clean":"2024-08-31T12:28:19.224638+0000","last_became_active":"2024-08-31T12:15:57.306962+0000","last_became_peered":"2024-08-31T12:15:57.306962+0000","last_unstale":"2024-08-31T12:28:19.224638+0000","last_undegraded":"2024-08-31T12:28:19.224638+0000","last_fullsized":"2024-08-31T12:28:19.224638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2535,"log_dups_size":0,"ondisk_log_size":2535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:01:18.188264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205541461,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":251825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"23'2","reported_seq":26,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:40.196751+0000","last_change":"2024-08-31T12:15:56.288126+0000","last_active":"2024-08-31T12:25:40.196751+0000","last_peered":"2024-08-31T12:25:40.196751+0000","last_clean":"2024-08-31T12:25:40.196751+0000","last_became_active":"2024-08-31T12:15:56.287552+0000","last_became_peered":"2024-08-31T12:15:56.287552+0000","last_unstale":"2024-08-31T12:25:40.196751+0000","last_undegraded":"2024-08-31T12:25:40.196751+0000","last_fullsized":"2024-08-31T12:25:40.196751+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:54:29.988079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"23'3439","reported_seq":3461,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.208231+0000","last_change":"2024-08-31T12:15:57.305029+0000","last_active":"2024-08-31T12:28:19.208231+0000","last_peered":"2024-08-31T12:28:19.208231+0000","last_clean":"2024-08-31T12:28:19.208231+0000","last_became_active":"2024-08-31T12:15:57.304700+0000","last_became_peered":"2024-08-31T12:15:57.304700+0000","last_unstale":"2024-08-31T12:28:19.208231+0000","last_undegraded":"2024-08-31T12:28:19.208231+0000","last_fullsized":"2024-08-31T12:28:19.208231+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3439,"log_dups_size":0,"ondisk_log_size":3439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:58:17.482204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243316666,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17415,"num_write_kb":307366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"23'4","reported_seq":28,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:36.194403+0000","last_change":"2024-08-31T12:15:56.284907+0000","last_active":"2024-08-31T12:25:36.194403+0000","last_peered":"2024-08-31T12:25:36.194403+0000","last_clean":"2024-08-31T12:25:36.194403+0000","last_became_active":"2024-08-31T12:15:56.284558+0000","last_became_peered":"2024-08-31T12:15:56.284558+0000","last_unstale":"2024-08-31T12:25:36.194403+0000","last_undegraded":"2024-08-31T12:25:36.194403+0000","last_fullsized":"2024-08-31T12:25:36.194403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:29:53.117724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"23'2894","reported_seq":2916,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.203880+0000","last_change":"2024-08-31T12:15:57.308679+0000","last_active":"2024-08-31T12:28:19.203880+0000","last_peered":"2024-08-31T12:28:19.203880+0000","last_clean":"2024-08-31T12:28:19.203880+0000","last_became_active":"2024-08-31T12:15:57.308260+0000","last_became_peered":"2024-08-31T12:15:57.308260+0000","last_unstale":"2024-08-31T12:28:19.203880+0000","last_undegraded":"2024-08-31T12:28:19.203880+0000","last_fullsized":"2024-08-31T12:28:19.203880+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2894,"log_dups_size":0,"ondisk_log_size":2894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:31:19.860950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209955924,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14127,"num_write_kb":261546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"21'2","reported_seq":26,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:40.196825+0000","last_change":"2024-08-31T12:15:56.290763+0000","last_active":"2024-08-31T12:25:40.196825+0000","last_peered":"2024-08-31T12:25:40.196825+0000","last_clean":"2024-08-31T12:25:40.196825+0000","last_became_active":"2024-08-31T12:15:56.290400+0000","last_became_peered":"2024-08-31T12:15:56.290400+0000","last_unstale":"2024-08-31T12:25:40.196825+0000","last_undegraded":"2024-08-31T12:25:40.196825+0000","last_fullsized":"2024-08-31T12:25:40.196825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:53:08.929860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"23'2656","reported_seq":2678,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.161047+0000","last_change":"2024-08-31T12:15:57.311657+0000","last_active":"2024-08-31T12:28:19.161047+0000","last_peered":"2024-08-31T12:28:19.161047+0000","last_clean":"2024-08-31T12:28:19.161047+0000","last_became_active":"2024-08-31T12:15:57.310564+0000","last_became_peered":"2024-08-31T12:15:57.310564+0000","last_unstale":"2024-08-31T12:28:19.161047+0000","last_undegraded":"2024-08-31T12:28:19.161047+0000","last_fullsized":"2024-08-31T12:28:19.161047+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2656,"log_dups_size":0,"ondisk_log_size":2656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:37:02.435678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180410852,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13728,"num_write_kb":231123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"23'19","reported_seq":43,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:36.210981+0000","last_change":"2024-08-31T12:15:56.286792+0000","last_active":"2024-08-31T12:25:36.210981+0000","last_peered":"2024-08-31T12:25:36.210981+0000","last_clean":"2024-08-31T12:25:36.210981+0000","last_became_active":"2024-08-31T12:15:56.286458+0000","last_became_peered":"2024-08-31T12:15:56.286458+0000","last_unstale":"2024-08-31T12:25:36.210981+0000","last_undegraded":"2024-08-31T12:25:36.210981+0000","last_fullsized":"2024-08-31T12:25:36.210981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:49:17.488528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"23'3658","reported_seq":3680,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.159920+0000","last_change":"2024-08-31T12:15:57.309485+0000","last_active":"2024-08-31T12:28:19.159920+0000","last_peered":"2024-08-31T12:28:19.159920+0000","last_clean":"2024-08-31T12:28:19.159920+0000","last_became_active":"2024-08-31T12:15:57.309102+0000","last_became_peered":"2024-08-31T12:15:57.309102+0000","last_unstale":"2024-08-31T12:28:19.159920+0000","last_undegraded":"2024-08-31T12:28:19.159920+0000","last_fullsized":"2024-08-31T12:28:19.159920+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3658,"log_dups_size":0,"ondisk_log_size":3658,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:17:34.692296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":277024326,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17723,"num_write_kb":341816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"23'317","reported_seq":341,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:22.335219+0000","last_change":"2024-08-31T12:15:56.289460+0000","last_active":"2024-08-31T12:28:22.335219+0000","last_peered":"2024-08-31T12:28:22.335219+0000","last_clean":"2024-08-31T12:28:22.335219+0000","last_became_active":"2024-08-31T12:15:56.289096+0000","last_became_peered":"2024-08-31T12:15:56.289096+0000","last_unstale":"2024-08-31T12:28:22.335219+0000","last_undegraded":"2024-08-31T12:28:22.335219+0000","last_fullsized":"2024-08-31T12:28:22.335219+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:06:34.313772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":317,"num_write_kb":4236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"23'2741","reported_seq":2763,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.210058+0000","last_change":"2024-08-31T12:15:57.317968+0000","last_active":"2024-08-31T12:28:19.210058+0000","last_peered":"2024-08-31T12:28:19.210058+0000","last_clean":"2024-08-31T12:28:19.210058+0000","last_became_active":"2024-08-31T12:15:57.317632+0000","last_became_peered":"2024-08-31T12:15:57.317632+0000","last_unstale":"2024-08-31T12:28:19.210058+0000","last_undegraded":"2024-08-31T12:28:19.210058+0000","last_fullsized":"2024-08-31T12:28:19.210058+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2741,"log_dups_size":0,"ondisk_log_size":2741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:22:02.569106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197132288,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13667,"num_write_kb":247166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'6","reported_seq":30,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:16.208499+0000","last_change":"2024-08-31T12:15:56.284014+0000","last_active":"2024-08-31T12:25:16.208499+0000","last_peered":"2024-08-31T12:25:16.208499+0000","last_clean":"2024-08-31T12:25:16.208499+0000","last_became_active":"2024-08-31T12:15:56.283675+0000","last_became_peered":"2024-08-31T12:15:56.283675+0000","last_unstale":"2024-08-31T12:25:16.208499+0000","last_undegraded":"2024-08-31T12:25:16.208499+0000","last_fullsized":"2024-08-31T12:25:16.208499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:51:59.306912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"23'3052","reported_seq":3074,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.204958+0000","last_change":"2024-08-31T12:15:57.312440+0000","last_active":"2024-08-31T12:28:19.204958+0000","last_peered":"2024-08-31T12:28:19.204958+0000","last_clean":"2024-08-31T12:28:19.204958+0000","last_became_active":"2024-08-31T12:15:57.312007+0000","last_became_peered":"2024-08-31T12:15:57.312007+0000","last_unstale":"2024-08-31T12:28:19.204958+0000","last_undegraded":"2024-08-31T12:28:19.204958+0000","last_fullsized":"2024-08-31T12:28:19.204958+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3052,"log_dups_size":0,"ondisk_log_size":3052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:23:43.553625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197138377,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15100,"num_write_kb":253041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"23'4","reported_seq":28,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:20.194216+0000","last_change":"2024-08-31T12:15:56.290968+0000","last_active":"2024-08-31T12:25:20.194216+0000","last_peered":"2024-08-31T12:25:20.194216+0000","last_clean":"2024-08-31T12:25:20.194216+0000","last_became_active":"2024-08-31T12:15:56.290603+0000","last_became_peered":"2024-08-31T12:15:56.290603+0000","last_unstale":"2024-08-31T12:25:20.194216+0000","last_undegraded":"2024-08-31T12:25:20.194216+0000","last_fullsized":"2024-08-31T12:25:20.194216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:50:23.215239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"23'3279","reported_seq":3301,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.194687+0000","last_change":"2024-08-31T12:15:57.312790+0000","last_active":"2024-08-31T12:28:19.194687+0000","last_peered":"2024-08-31T12:28:19.194687+0000","last_clean":"2024-08-31T12:28:19.194687+0000","last_became_active":"2024-08-31T12:15:57.312383+0000","last_became_peered":"2024-08-31T12:15:57.312383+0000","last_unstale":"2024-08-31T12:28:19.194687+0000","last_undegraded":"2024-08-31T12:28:19.194687+0000","last_fullsized":"2024-08-31T12:28:19.194687+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3279,"log_dups_size":0,"ondisk_log_size":3279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:20:41.042375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234891518,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16277,"num_write_kb":294634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"23'5","reported_seq":29,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:16.191885+0000","last_change":"2024-08-31T12:15:56.289795+0000","last_active":"2024-08-31T12:25:16.191885+0000","last_peered":"2024-08-31T12:25:16.191885+0000","last_clean":"2024-08-31T12:25:16.191885+0000","last_became_active":"2024-08-31T12:15:56.289477+0000","last_became_peered":"2024-08-31T12:15:56.289477+0000","last_unstale":"2024-08-31T12:25:16.191885+0000","last_undegraded":"2024-08-31T12:25:16.191885+0000","last_fullsized":"2024-08-31T12:25:16.191885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:15:51.561831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":54,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"23'2828","reported_seq":2850,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.192699+0000","last_change":"2024-08-31T12:15:57.312865+0000","last_active":"2024-08-31T12:28:19.192699+0000","last_peered":"2024-08-31T12:28:19.192699+0000","last_clean":"2024-08-31T12:28:19.192699+0000","last_became_active":"2024-08-31T12:15:57.312462+0000","last_became_peered":"2024-08-31T12:15:57.312462+0000","last_unstale":"2024-08-31T12:28:19.192699+0000","last_undegraded":"2024-08-31T12:28:19.192699+0000","last_fullsized":"2024-08-31T12:28:19.192699+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2828,"log_dups_size":0,"ondisk_log_size":2828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:16:42.625396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222316815,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13883,"num_write_kb":272702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'15","reported_seq":39,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:18.179855+0000","last_change":"2024-08-31T12:15:56.285412+0000","last_active":"2024-08-31T12:25:18.179855+0000","last_peered":"2024-08-31T12:25:18.179855+0000","last_clean":"2024-08-31T12:25:18.179855+0000","last_became_active":"2024-08-31T12:15:56.285108+0000","last_became_peered":"2024-08-31T12:15:56.285108+0000","last_unstale":"2024-08-31T12:25:18.179855+0000","last_undegraded":"2024-08-31T12:25:18.179855+0000","last_fullsized":"2024-08-31T12:25:18.179855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:29:45.580957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"23'2843","reported_seq":2865,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.145892+0000","last_change":"2024-08-31T12:15:57.311701+0000","last_active":"2024-08-31T12:28:19.145892+0000","last_peered":"2024-08-31T12:28:19.145892+0000","last_clean":"2024-08-31T12:28:19.145892+0000","last_became_active":"2024-08-31T12:15:57.311368+0000","last_became_peered":"2024-08-31T12:15:57.311368+0000","last_unstale":"2024-08-31T12:28:19.145892+0000","last_undegraded":"2024-08-31T12:28:19.145892+0000","last_fullsized":"2024-08-31T12:28:19.145892+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2843,"log_dups_size":0,"ondisk_log_size":2843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:30:13.715505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205547202,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14167,"num_write_kb":257592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"23'3","reported_seq":27,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:00.192148+0000","last_change":"2024-08-31T12:15:56.289979+0000","last_active":"2024-08-31T12:25:00.192148+0000","last_peered":"2024-08-31T12:25:00.192148+0000","last_clean":"2024-08-31T12:25:00.192148+0000","last_became_active":"2024-08-31T12:15:56.288289+0000","last_became_peered":"2024-08-31T12:15:56.288289+0000","last_unstale":"2024-08-31T12:25:00.192148+0000","last_undegraded":"2024-08-31T12:25:00.192148+0000","last_fullsized":"2024-08-31T12:25:00.192148+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:46:55.057087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"23'2808","reported_seq":2830,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.206168+0000","last_change":"2024-08-31T12:15:57.311580+0000","last_active":"2024-08-31T12:28:19.206168+0000","last_peered":"2024-08-31T12:28:19.206168+0000","last_clean":"2024-08-31T12:28:19.206168+0000","last_became_active":"2024-08-31T12:15:57.311323+0000","last_became_peered":"2024-08-31T12:15:57.311323+0000","last_unstale":"2024-08-31T12:28:19.206168+0000","last_undegraded":"2024-08-31T12:28:19.206168+0000","last_fullsized":"2024-08-31T12:28:19.206168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2808,"log_dups_size":0,"ondisk_log_size":2808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:54:03.871840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201522906,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13808,"num_write_kb":252081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"23'3","reported_seq":27,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:24:58.177773+0000","last_change":"2024-08-31T12:15:56.290496+0000","last_active":"2024-08-31T12:24:58.177773+0000","last_peered":"2024-08-31T12:24:58.177773+0000","last_clean":"2024-08-31T12:24:58.177773+0000","last_became_active":"2024-08-31T12:15:56.290196+0000","last_became_peered":"2024-08-31T12:15:56.290196+0000","last_unstale":"2024-08-31T12:24:58.177773+0000","last_undegraded":"2024-08-31T12:24:58.177773+0000","last_fullsized":"2024-08-31T12:24:58.177773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:52:20.055757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"23'3282","reported_seq":3304,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.175690+0000","last_change":"2024-08-31T12:15:57.313317+0000","last_active":"2024-08-31T12:28:19.175690+0000","last_peered":"2024-08-31T12:28:19.175690+0000","last_clean":"2024-08-31T12:28:19.175690+0000","last_became_active":"2024-08-31T12:15:57.312145+0000","last_became_peered":"2024-08-31T12:15:57.312145+0000","last_unstale":"2024-08-31T12:28:19.175690+0000","last_undegraded":"2024-08-31T12:28:19.175690+0000","last_fullsized":"2024-08-31T12:28:19.175690+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3282,"log_dups_size":0,"ondisk_log_size":3282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:09:31.904512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243277619,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16586,"num_write_kb":304033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":25,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:24:56.207028+0000","last_change":"2024-08-31T12:15:56.281809+0000","last_active":"2024-08-31T12:24:56.207028+0000","last_peered":"2024-08-31T12:24:56.207028+0000","last_clean":"2024-08-31T12:24:56.207028+0000","last_became_active":"2024-08-31T12:15:56.281368+0000","last_became_peered":"2024-08-31T12:15:56.281368+0000","last_unstale":"2024-08-31T12:24:56.207028+0000","last_undegraded":"2024-08-31T12:24:56.207028+0000","last_fullsized":"2024-08-31T12:24:56.207028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:17:33.687196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"23'3289","reported_seq":3311,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.183134+0000","last_change":"2024-08-31T12:15:57.305874+0000","last_active":"2024-08-31T12:28:19.183134+0000","last_peered":"2024-08-31T12:28:19.183134+0000","last_clean":"2024-08-31T12:28:19.183134+0000","last_became_active":"2024-08-31T12:15:57.305507+0000","last_became_peered":"2024-08-31T12:15:57.305507+0000","last_unstale":"2024-08-31T12:28:19.183134+0000","last_undegraded":"2024-08-31T12:28:19.183134+0000","last_fullsized":"2024-08-31T12:28:19.183134+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3289,"log_dups_size":0,"ondisk_log_size":3289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:22:02.569106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218117314,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16432,"num_write_kb":278837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"23'4","reported_seq":28,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:24:58.177745+0000","last_change":"2024-08-31T12:15:56.291851+0000","last_active":"2024-08-31T12:24:58.177745+0000","last_peered":"2024-08-31T12:24:58.177745+0000","last_clean":"2024-08-31T12:24:58.177745+0000","last_became_active":"2024-08-31T12:15:56.291446+0000","last_became_peered":"2024-08-31T12:15:56.291446+0000","last_unstale":"2024-08-31T12:24:58.177745+0000","last_undegraded":"2024-08-31T12:24:58.177745+0000","last_fullsized":"2024-08-31T12:24:58.177745+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:54:29.988079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"23'2737","reported_seq":2759,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.167644+0000","last_change":"2024-08-31T12:15:57.316154+0000","last_active":"2024-08-31T12:28:19.167644+0000","last_peered":"2024-08-31T12:28:19.167644+0000","last_clean":"2024-08-31T12:28:19.167644+0000","last_became_active":"2024-08-31T12:15:57.315830+0000","last_became_peered":"2024-08-31T12:15:57.315830+0000","last_unstale":"2024-08-31T12:28:19.167644+0000","last_undegraded":"2024-08-31T12:28:19.167644+0000","last_fullsized":"2024-08-31T12:28:19.167644+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2737,"log_dups_size":0,"ondisk_log_size":2737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:05:53.039520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209813064,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13700,"num_write_kb":259755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":121,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:50.198157+0000","last_change":"2024-08-31T12:15:47.084912+0000","last_active":"2024-08-31T12:25:50.198157+0000","last_peered":"2024-08-31T12:25:50.198157+0000","last_clean":"2024-08-31T12:25:50.198157+0000","last_became_active":"2024-08-31T12:15:47.083771+0000","last_became_peered":"2024-08-31T12:15:47.083771+0000","last_unstale":"2024-08-31T12:25:50.198157+0000","last_undegraded":"2024-08-31T12:25:50.198157+0000","last_fullsized":"2024-08-31T12:25:50.198157+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:45.213688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:45.213688+0000","last_clean_scrub_stamp":"2024-08-31T12:15:45.213688+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:25:24.827530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"23'4","reported_seq":29,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:26:25.205795+0000","last_change":"2024-08-31T12:15:56.285355+0000","last_active":"2024-08-31T12:26:25.205795+0000","last_peered":"2024-08-31T12:26:25.205795+0000","last_clean":"2024-08-31T12:26:25.205795+0000","last_became_active":"2024-08-31T12:15:56.284470+0000","last_became_peered":"2024-08-31T12:15:56.284470+0000","last_unstale":"2024-08-31T12:26:25.205795+0000","last_undegraded":"2024-08-31T12:26:25.205795+0000","last_fullsized":"2024-08-31T12:26:25.205795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:52:20.055757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"23'2767","reported_seq":2790,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.201878+0000","last_change":"2024-08-31T12:15:57.309039+0000","last_active":"2024-08-31T12:28:19.201878+0000","last_peered":"2024-08-31T12:28:19.201878+0000","last_clean":"2024-08-31T12:28:19.201878+0000","last_became_active":"2024-08-31T12:15:57.308721+0000","last_became_peered":"2024-08-31T12:15:57.308721+0000","last_unstale":"2024-08-31T12:28:19.201878+0000","last_undegraded":"2024-08-31T12:28:19.201878+0000","last_fullsized":"2024-08-31T12:28:19.201878+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2767,"log_dups_size":0,"ondisk_log_size":2767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:43:12.703971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218112028,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13723,"num_write_kb":267984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"23'4","reported_seq":29,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:27:20.870746+0000","last_change":"2024-08-31T12:15:56.293221+0000","last_active":"2024-08-31T12:27:20.870746+0000","last_peered":"2024-08-31T12:27:20.870746+0000","last_clean":"2024-08-31T12:27:20.870746+0000","last_became_active":"2024-08-31T12:15:56.292910+0000","last_became_peered":"2024-08-31T12:15:56.292910+0000","last_unstale":"2024-08-31T12:27:20.870746+0000","last_undegraded":"2024-08-31T12:27:20.870746+0000","last_fullsized":"2024-08-31T12:27:20.870746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:14:33.483002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"23'2757","reported_seq":2780,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.182199+0000","last_change":"2024-08-31T12:15:57.307266+0000","last_active":"2024-08-31T12:28:19.182199+0000","last_peered":"2024-08-31T12:28:19.182199+0000","last_clean":"2024-08-31T12:28:19.182199+0000","last_became_active":"2024-08-31T12:15:57.306529+0000","last_became_peered":"2024-08-31T12:15:57.306529+0000","last_unstale":"2024-08-31T12:28:19.182199+0000","last_undegraded":"2024-08-31T12:28:19.182199+0000","last_fullsized":"2024-08-31T12:28:19.182199+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2757,"log_dups_size":0,"ondisk_log_size":2757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:42:39.731742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201337916,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14174,"num_write_kb":253444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"23'35","reported_seq":90,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:26:23.185903+0000","last_change":"2024-08-31T12:15:56.291822+0000","last_active":"2024-08-31T12:26:23.185903+0000","last_peered":"2024-08-31T12:26:23.185903+0000","last_clean":"2024-08-31T12:26:23.185903+0000","last_became_active":"2024-08-31T12:15:56.291064+0000","last_became_peered":"2024-08-31T12:15:56.291064+0000","last_unstale":"2024-08-31T12:26:23.185903+0000","last_undegraded":"2024-08-31T12:26:23.185903+0000","last_fullsized":"2024-08-31T12:26:23.185903+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:53:08.929860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47459,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":30,"num_read_kb":64,"num_write":34,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"23'2938","reported_seq":2961,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.167628+0000","last_change":"2024-08-31T12:15:57.299212+0000","last_active":"2024-08-31T12:28:19.167628+0000","last_peered":"2024-08-31T12:28:19.167628+0000","last_clean":"2024-08-31T12:28:19.167628+0000","last_became_active":"2024-08-31T12:15:57.298892+0000","last_became_peered":"2024-08-31T12:15:57.298892+0000","last_unstale":"2024-08-31T12:28:19.167628+0000","last_undegraded":"2024-08-31T12:28:19.167628+0000","last_fullsized":"2024-08-31T12:28:19.167628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2938,"log_dups_size":0,"ondisk_log_size":2938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:53:19.523811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197144226,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14947,"num_write_kb":252544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"23'2","reported_seq":27,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:26:25.205779+0000","last_change":"2024-08-31T12:15:56.286720+0000","last_active":"2024-08-31T12:26:25.205779+0000","last_peered":"2024-08-31T12:26:25.205779+0000","last_clean":"2024-08-31T12:26:25.205779+0000","last_became_active":"2024-08-31T12:15:56.285648+0000","last_became_peered":"2024-08-31T12:15:56.285648+0000","last_unstale":"2024-08-31T12:26:25.205779+0000","last_undegraded":"2024-08-31T12:26:25.205779+0000","last_fullsized":"2024-08-31T12:26:25.205779+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:39:31.946792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"23'2762","reported_seq":2785,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.152132+0000","last_change":"2024-08-31T12:15:57.311785+0000","last_active":"2024-08-31T12:28:19.152132+0000","last_peered":"2024-08-31T12:28:19.152132+0000","last_clean":"2024-08-31T12:28:19.152132+0000","last_became_active":"2024-08-31T12:15:57.311469+0000","last_became_peered":"2024-08-31T12:15:57.311469+0000","last_unstale":"2024-08-31T12:28:19.152132+0000","last_undegraded":"2024-08-31T12:28:19.152132+0000","last_fullsized":"2024-08-31T12:28:19.152132+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2762,"log_dups_size":0,"ondisk_log_size":2762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:29:54.122825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197139313,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14081,"num_write_kb":248914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"23'1","reported_seq":26,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:26:21.215756+0000","last_change":"2024-08-31T12:15:56.284302+0000","last_active":"2024-08-31T12:26:21.215756+0000","last_peered":"2024-08-31T12:26:21.215756+0000","last_clean":"2024-08-31T12:26:21.215756+0000","last_became_active":"2024-08-31T12:15:56.283979+0000","last_became_peered":"2024-08-31T12:15:56.283979+0000","last_unstale":"2024-08-31T12:26:21.215756+0000","last_undegraded":"2024-08-31T12:26:21.215756+0000","last_fullsized":"2024-08-31T12:26:21.215756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:51:59.306912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"23'2523","reported_seq":2539,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.196590+0000","last_change":"2024-08-31T12:16:41.195070+0000","last_active":"2024-08-31T12:28:19.196590+0000","last_peered":"2024-08-31T12:28:19.196590+0000","last_clean":"2024-08-31T12:28:19.196590+0000","last_became_active":"2024-08-31T12:16:41.194724+0000","last_became_peered":"2024-08-31T12:16:41.194724+0000","last_unstale":"2024-08-31T12:28:19.196590+0000","last_undegraded":"2024-08-31T12:28:19.196590+0000","last_fullsized":"2024-08-31T12:28:19.196590+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2523,"log_dups_size":0,"ondisk_log_size":2523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:24:46.945850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184558382,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13011,"num_write_kb":232344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"21'1","reported_seq":26,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:26:25.202953+0000","last_change":"2024-08-31T12:15:56.297648+0000","last_active":"2024-08-31T12:26:25.202953+0000","last_peered":"2024-08-31T12:26:25.202953+0000","last_clean":"2024-08-31T12:26:25.202953+0000","last_became_active":"2024-08-31T12:15:56.297306+0000","last_became_peered":"2024-08-31T12:15:56.297306+0000","last_unstale":"2024-08-31T12:26:25.202953+0000","last_undegraded":"2024-08-31T12:26:25.202953+0000","last_fullsized":"2024-08-31T12:26:25.202953+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:29:54.693819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"23'3164","reported_seq":3187,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.097415+0000","last_change":"2024-08-31T12:15:57.297230+0000","last_active":"2024-08-31T12:28:19.097415+0000","last_peered":"2024-08-31T12:28:19.097415+0000","last_clean":"2024-08-31T12:28:19.097415+0000","last_became_active":"2024-08-31T12:15:57.296892+0000","last_became_peered":"2024-08-31T12:15:57.296892+0000","last_unstale":"2024-08-31T12:28:19.097415+0000","last_undegraded":"2024-08-31T12:28:19.097415+0000","last_fullsized":"2024-08-31T12:28:19.097415+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3164,"log_dups_size":0,"ondisk_log_size":3164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:42:36.033586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213956769,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16152,"num_write_kb":273638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"23'427","reported_seq":451,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:23.458871+0000","last_change":"2024-08-31T12:15:56.286504+0000","last_active":"2024-08-31T12:28:23.458871+0000","last_peered":"2024-08-31T12:28:23.458871+0000","last_clean":"2024-08-31T12:28:23.458871+0000","last_became_active":"2024-08-31T12:15:56.286173+0000","last_became_peered":"2024-08-31T12:15:56.286173+0000","last_unstale":"2024-08-31T12:28:23.458871+0000","last_undegraded":"2024-08-31T12:28:23.458871+0000","last_fullsized":"2024-08-31T12:28:23.458871+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":427,"log_dups_size":0,"ondisk_log_size":427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:06:34.313772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":975,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"23'2916","reported_seq":2938,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.229676+0000","last_change":"2024-08-31T12:15:57.308013+0000","last_active":"2024-08-31T12:28:19.229676+0000","last_peered":"2024-08-31T12:28:19.229676+0000","last_clean":"2024-08-31T12:28:19.229676+0000","last_became_active":"2024-08-31T12:15:57.307701+0000","last_became_peered":"2024-08-31T12:15:57.307701+0000","last_unstale":"2024-08-31T12:28:19.229676+0000","last_undegraded":"2024-08-31T12:28:19.229676+0000","last_fullsized":"2024-08-31T12:28:19.229676+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2916,"log_dups_size":0,"ondisk_log_size":2916,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:06:15.161589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205521115,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14225,"num_write_kb":257747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'4","reported_seq":28,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:24:36.187735+0000","last_change":"2024-08-31T12:15:56.294852+0000","last_active":"2024-08-31T12:24:36.187735+0000","last_peered":"2024-08-31T12:24:36.187735+0000","last_clean":"2024-08-31T12:24:36.187735+0000","last_became_active":"2024-08-31T12:15:56.294496+0000","last_became_peered":"2024-08-31T12:15:56.294496+0000","last_unstale":"2024-08-31T12:24:36.187735+0000","last_undegraded":"2024-08-31T12:24:36.187735+0000","last_fullsized":"2024-08-31T12:24:36.187735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:49:17.488528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"23'2537","reported_seq":2559,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.168378+0000","last_change":"2024-08-31T12:15:57.305202+0000","last_active":"2024-08-31T12:28:19.168378+0000","last_peered":"2024-08-31T12:28:19.168378+0000","last_clean":"2024-08-31T12:28:19.168378+0000","last_became_active":"2024-08-31T12:15:57.304602+0000","last_became_peered":"2024-08-31T12:15:57.304602+0000","last_unstale":"2024-08-31T12:28:19.168378+0000","last_undegraded":"2024-08-31T12:28:19.168378+0000","last_fullsized":"2024-08-31T12:28:19.168378+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2537,"log_dups_size":0,"ondisk_log_size":2537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:56:03.280339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176167762,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12536,"num_write_kb":222149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"23'3","reported_seq":27,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:24:40.189661+0000","last_change":"2024-08-31T12:15:56.297542+0000","last_active":"2024-08-31T12:24:40.189661+0000","last_peered":"2024-08-31T12:24:40.189661+0000","last_clean":"2024-08-31T12:24:40.189661+0000","last_became_active":"2024-08-31T12:15:56.297168+0000","last_became_peered":"2024-08-31T12:15:56.297168+0000","last_unstale":"2024-08-31T12:24:40.189661+0000","last_undegraded":"2024-08-31T12:24:40.189661+0000","last_fullsized":"2024-08-31T12:24:40.189661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:56:41.924772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"23'2320","reported_seq":2342,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.187968+0000","last_change":"2024-08-31T12:15:57.298845+0000","last_active":"2024-08-31T12:28:19.187968+0000","last_peered":"2024-08-31T12:28:19.187968+0000","last_clean":"2024-08-31T12:28:19.187968+0000","last_became_active":"2024-08-31T12:15:57.298520+0000","last_became_peered":"2024-08-31T12:15:57.298520+0000","last_unstale":"2024-08-31T12:28:19.187968+0000","last_undegraded":"2024-08-31T12:28:19.187968+0000","last_fullsized":"2024-08-31T12:28:19.187968+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2320,"log_dups_size":0,"ondisk_log_size":2320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:06:35.318873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197144309,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11287,"num_write_kb":237691,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"21'3","reported_seq":27,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:24:36.205095+0000","last_change":"2024-08-31T12:15:56.290475+0000","last_active":"2024-08-31T12:24:36.205095+0000","last_peered":"2024-08-31T12:24:36.205095+0000","last_clean":"2024-08-31T12:24:36.205095+0000","last_became_active":"2024-08-31T12:15:56.289960+0000","last_became_peered":"2024-08-31T12:15:56.289960+0000","last_unstale":"2024-08-31T12:24:36.205095+0000","last_undegraded":"2024-08-31T12:24:36.205095+0000","last_fullsized":"2024-08-31T12:24:36.205095+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:29:53.117724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"23'2970","reported_seq":2992,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.225999+0000","last_change":"2024-08-31T12:15:57.301460+0000","last_active":"2024-08-31T12:28:19.225999+0000","last_peered":"2024-08-31T12:28:19.225999+0000","last_clean":"2024-08-31T12:28:19.225999+0000","last_became_active":"2024-08-31T12:15:57.301157+0000","last_became_peered":"2024-08-31T12:15:57.301157+0000","last_unstale":"2024-08-31T12:28:19.225999+0000","last_undegraded":"2024-08-31T12:28:19.225999+0000","last_fullsized":"2024-08-31T12:28:19.225999+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2970,"log_dups_size":0,"ondisk_log_size":2970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:22:52.173578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209728881,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15223,"num_write_kb":265883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"21'1","reported_seq":25,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:24:36.205125+0000","last_change":"2024-08-31T12:15:56.289720+0000","last_active":"2024-08-31T12:24:36.205125+0000","last_peered":"2024-08-31T12:24:36.205125+0000","last_clean":"2024-08-31T12:24:36.205125+0000","last_became_active":"2024-08-31T12:15:56.289332+0000","last_became_peered":"2024-08-31T12:15:56.289332+0000","last_unstale":"2024-08-31T12:24:36.205125+0000","last_undegraded":"2024-08-31T12:24:36.205125+0000","last_fullsized":"2024-08-31T12:24:36.205125+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:15:51.561831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"23'2415","reported_seq":2437,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.230233+0000","last_change":"2024-08-31T12:15:57.313802+0000","last_active":"2024-08-31T12:28:19.230233+0000","last_peered":"2024-08-31T12:28:19.230233+0000","last_clean":"2024-08-31T12:28:19.230233+0000","last_became_active":"2024-08-31T12:15:57.313428+0000","last_became_peered":"2024-08-31T12:15:57.313428+0000","last_unstale":"2024-08-31T12:28:19.230233+0000","last_undegraded":"2024-08-31T12:28:19.230233+0000","last_fullsized":"2024-08-31T12:28:19.230233+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2415,"log_dups_size":0,"ondisk_log_size":2415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:22:13.435300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176173175,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11447,"num_write_kb":217885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"23'1068","reported_seq":1092,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:27:20.763645+0000","last_change":"2024-08-31T12:15:56.296105+0000","last_active":"2024-08-31T12:27:20.763645+0000","last_peered":"2024-08-31T12:27:20.763645+0000","last_clean":"2024-08-31T12:27:20.763645+0000","last_became_active":"2024-08-31T12:15:56.295524+0000","last_became_peered":"2024-08-31T12:15:56.295524+0000","last_unstale":"2024-08-31T12:27:20.763645+0000","last_undegraded":"2024-08-31T12:27:20.763645+0000","last_fullsized":"2024-08-31T12:27:20.763645+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":1068,"log_dups_size":0,"ondisk_log_size":1068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:41:55.482044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1064,"num_write_kb":4401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"23'2502","reported_seq":2524,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.061863+0000","last_change":"2024-08-31T12:15:57.300155+0000","last_active":"2024-08-31T12:28:19.061863+0000","last_peered":"2024-08-31T12:28:19.061863+0000","last_clean":"2024-08-31T12:28:19.061863+0000","last_became_active":"2024-08-31T12:15:57.299436+0000","last_became_peered":"2024-08-31T12:15:57.299436+0000","last_unstale":"2024-08-31T12:28:19.061863+0000","last_undegraded":"2024-08-31T12:28:19.061863+0000","last_fullsized":"2024-08-31T12:28:19.061863+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2502,"log_dups_size":0,"ondisk_log_size":2502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:37:45.891394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188744397,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12696,"num_write_kb":235198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"23'3358","reported_seq":3380,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.189014+0000","last_change":"2024-08-31T12:15:57.300529+0000","last_active":"2024-08-31T12:28:19.189014+0000","last_peered":"2024-08-31T12:28:19.189014+0000","last_clean":"2024-08-31T12:28:19.189014+0000","last_became_active":"2024-08-31T12:15:57.300159+0000","last_became_peered":"2024-08-31T12:15:57.300159+0000","last_unstale":"2024-08-31T12:28:19.189014+0000","last_undegraded":"2024-08-31T12:28:19.189014+0000","last_fullsized":"2024-08-31T12:28:19.189014+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3358,"log_dups_size":0,"ondisk_log_size":3358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:35:53.831742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251730727,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16759,"num_write_kb":312885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":26,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:26:00.199883+0000","last_change":"2024-08-31T12:15:56.297791+0000","last_active":"2024-08-31T12:26:00.199883+0000","last_peered":"2024-08-31T12:26:00.199883+0000","last_clean":"2024-08-31T12:26:00.199883+0000","last_became_active":"2024-08-31T12:15:56.296976+0000","last_became_peered":"2024-08-31T12:15:56.296976+0000","last_unstale":"2024-08-31T12:26:00.199883+0000","last_undegraded":"2024-08-31T12:26:00.199883+0000","last_fullsized":"2024-08-31T12:26:00.199883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:07:54.963050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"23'2167","reported_seq":2189,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.184056+0000","last_change":"2024-08-31T12:15:57.301167+0000","last_active":"2024-08-31T12:28:19.184056+0000","last_peered":"2024-08-31T12:28:19.184056+0000","last_clean":"2024-08-31T12:28:19.184056+0000","last_became_active":"2024-08-31T12:15:57.300646+0000","last_became_peered":"2024-08-31T12:15:57.300646+0000","last_unstale":"2024-08-31T12:28:19.184056+0000","last_undegraded":"2024-08-31T12:28:19.184056+0000","last_fullsized":"2024-08-31T12:28:19.184056+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2167,"log_dups_size":0,"ondisk_log_size":2167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:05:53.039520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150996671,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10769,"num_write_kb":190633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"23'9","reported_seq":33,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:26:00.199906+0000","last_change":"2024-08-31T12:15:56.295139+0000","last_active":"2024-08-31T12:26:00.199906+0000","last_peered":"2024-08-31T12:26:00.199906+0000","last_clean":"2024-08-31T12:26:00.199906+0000","last_became_active":"2024-08-31T12:15:56.294560+0000","last_became_peered":"2024-08-31T12:15:56.294560+0000","last_unstale":"2024-08-31T12:26:00.199906+0000","last_undegraded":"2024-08-31T12:26:00.199906+0000","last_fullsized":"2024-08-31T12:26:00.199906+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:52:16.999507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"23'3348","reported_seq":3371,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.187804+0000","last_change":"2024-08-31T12:15:57.311074+0000","last_active":"2024-08-31T12:28:19.187804+0000","last_peered":"2024-08-31T12:28:19.187804+0000","last_clean":"2024-08-31T12:28:19.187804+0000","last_became_active":"2024-08-31T12:15:57.310262+0000","last_became_peered":"2024-08-31T12:15:57.310262+0000","last_unstale":"2024-08-31T12:28:19.187804+0000","last_undegraded":"2024-08-31T12:28:19.187804+0000","last_fullsized":"2024-08-31T12:28:19.187804+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3348,"log_dups_size":0,"ondisk_log_size":3348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:47:51.267409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251719163,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16721,"num_write_kb":312844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"23'3","reported_seq":28,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:56.212749+0000","last_change":"2024-08-31T12:15:56.285428+0000","last_active":"2024-08-31T12:25:56.212749+0000","last_peered":"2024-08-31T12:25:56.212749+0000","last_clean":"2024-08-31T12:25:56.212749+0000","last_became_active":"2024-08-31T12:15:56.284884+0000","last_became_peered":"2024-08-31T12:15:56.284884+0000","last_unstale":"2024-08-31T12:25:56.212749+0000","last_undegraded":"2024-08-31T12:25:56.212749+0000","last_fullsized":"2024-08-31T12:25:56.212749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:49:25.181432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"23'3167","reported_seq":3190,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.175794+0000","last_change":"2024-08-31T12:15:57.305772+0000","last_active":"2024-08-31T12:28:19.175794+0000","last_peered":"2024-08-31T12:28:19.175794+0000","last_clean":"2024-08-31T12:28:19.175794+0000","last_became_active":"2024-08-31T12:15:57.305409+0000","last_became_peered":"2024-08-31T12:15:57.305409+0000","last_unstale":"2024-08-31T12:28:19.175794+0000","last_undegraded":"2024-08-31T12:28:19.175794+0000","last_fullsized":"2024-08-31T12:28:19.175794+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3167,"log_dups_size":0,"ondisk_log_size":3167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:53:19.523811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222298112,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15945,"num_write_kb":281070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"23'229","reported_seq":254,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:37.337439+0000","last_change":"2024-08-31T12:15:56.294197+0000","last_active":"2024-08-31T12:28:37.337439+0000","last_peered":"2024-08-31T12:28:37.337439+0000","last_clean":"2024-08-31T12:28:37.337439+0000","last_became_active":"2024-08-31T12:15:56.293864+0000","last_became_peered":"2024-08-31T12:15:56.293864+0000","last_unstale":"2024-08-31T12:28:37.337439+0000","last_undegraded":"2024-08-31T12:28:37.337439+0000","last_fullsized":"2024-08-31T12:28:37.337439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":229,"log_dups_size":0,"ondisk_log_size":229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:43:17.443433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1325973,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":228,"num_write_kb":1426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":13424230175,"num_objects":3391,"num_object_clones":0,"num_object_copies":6782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":918154,"num_write_kb":16787942,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26849361920,"data_stored":26848441814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":183652,"ondisk_log_size":183652,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":13960395,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":30,"num_read_kb":64,"num_write":3307,"num_write_kb":14790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28073984,"data_stored":27920790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2759,"ondisk_log_size":2759,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509600,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2618576,"kb_used_data":2384832,"kb_used_omap":1,"kb_used_meta":233726,"kb_avail":91753264,"statfs":{"total":96636764160,"available":93955342336,"internally_reserved":0,"allocated":2442067968,"data_stored":2441803901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":239335878},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509599,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4206900,"kb_used_data":3851044,"kb_used_omap":1,"kb_used_meta":355838,"kb_avail":90164940,"statfs":{"total":96636764160,"available":92328898560,"internally_reserved":0,"allocated":3943469056,"data_stored":3943171530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":364378567},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509600,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2544520,"kb_used_data":2306616,"kb_used_omap":1,"kb_used_meta":237886,"kb_avail":91827320,"statfs":{"total":96636764160,"available":94031175680,"internally_reserved":0,"allocated":2361974784,"data_stored":2361727195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":243595717},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509600,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4141744,"kb_used_data":3795616,"kb_used_omap":1,"kb_used_meta":346110,"kb_avail":90230096,"statfs":{"total":96636764160,"available":92395618304,"internally_reserved":0,"allocated":3886710784,"data_stored":3886395634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354417093},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542303,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4422856,"kb_used_data":4066936,"kb_used_omap":1,"kb_used_meta":355902,"kb_avail":89948984,"statfs":{"total":96636764160,"available":92107759616,"internally_reserved":0,"allocated":4164542464,"data_stored":4164215175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":364444106},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542305,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4162248,"kb_used_data":3793912,"kb_used_omap":1,"kb_used_meta":368318,"kb_avail":90209592,"statfs":{"total":96636764160,"available":92374622208,"internally_reserved":0,"allocated":3884965888,"data_stored":3884692848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":377158085},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542303,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4136736,"kb_used_data":3839056,"kb_used_omap":1,"kb_used_meta":297662,"kb_avail":90235104,"statfs":{"total":96636764160,"available":92400746496,"internally_reserved":0,"allocated":3931193344,"data_stored":3930920211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":304806342},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575007,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2442052,"kb_used_data":2212468,"kb_used_omap":1,"kb_used_meta":229566,"kb_avail":91929788,"statfs":{"total":96636764160,"available":94136102912,"internally_reserved":0,"allocated":2265567232,"data_stored":2265299110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":235076037},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1343488,"data_stored":1326051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":48187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8458240,"data_stored":8437802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4210688,"data_stored":4195018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4210688,"data_stored":4195654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":9736192,"data_stored":9715756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3884539904,"data_stored":3884393932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2361716736,"data_stored":2361640968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2265264128,"data_stored":2265165640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3934777344,"data_stored":3934648445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3926749184,"data_stored":3926639910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3884711936,"data_stored":3884606187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2437623808,"data_stored":2437522964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4153978880,"data_stored":4153823768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-31T12:28:48.573 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-31T12:28:48.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.720+0000 7feea1f9e640 1 -- 172.21.15.19:0/1749130351 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fee9c153190 msgr2=0x7fee9c159820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:48.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.720+0000 7feea1f9e640 1 --2- 172.21.15.19:0/1749130351 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fee9c153190 0x7fee9c159820 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fee9c15d2b0 tx=0x7fee8c02a530 comp rx=0 tx=0).stop 2024-08-31T12:28:48.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.720+0000 7feea1f9e640 1 -- 172.21.15.19:0/1749130351 shutdown_connections 2024-08-31T12:28:48.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.721+0000 7feea1f9e640 1 --2- 172.21.15.19:0/1749130351 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fee9c153190 0x7fee9c159820 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:48.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.721+0000 7feea1f9e640 1 -- 172.21.15.19:0/1749130351 >> 172.21.15.19:0/1749130351 conn(0x7fee9c0728b0 msgr2=0x7fee9c072cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:48.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.721+0000 7feea1f9e640 1 -- 172.21.15.19:0/1749130351 shutdown_connections 2024-08-31T12:28:48.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.721+0000 7feea1f9e640 1 -- 172.21.15.19:0/1749130351 wait complete. 2024-08-31T12:28:48.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.722+0000 7feea1f9e640 1 Processor -- start 2024-08-31T12:28:48.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.722+0000 7feea1f9e640 1 -- start start 2024-08-31T12:28:48.727 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.724+0000 7feea1f9e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fee9c0c9030 0x7fee9c15a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:48.727 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.724+0000 7feea1f9e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee9c0c9440 0x7fee9c14de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:48.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.724+0000 7feea1f9e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fee9c153190 0x7fee9c151c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:48.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.724+0000 7feea1f9e640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fee9c0c4db0 con 0x7fee9c153190 2024-08-31T12:28:48.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.724+0000 7feea1f9e640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fee9c159d60 con 0x7fee9c0c9440 2024-08-31T12:28:48.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.724+0000 7feea1f9e640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fee9c1535a0 con 0x7fee9c0c9030 2024-08-31T12:28:48.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.724+0000 7fee9bfff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fee9c153190 0x7fee9c151c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:48.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.724+0000 7fee9bfff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fee9c153190 0x7fee9c151c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:43832/0 (socket says 172.21.15.19:43832) 2024-08-31T12:28:48.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.724+0000 7fee9bfff640 1 -- 172.21.15.19:0/3972827675 learned_addr learned my addr 172.21.15.19:0/3972827675 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:28:48.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.724+0000 7fee9affd640 1 --2- 172.21.15.19:0/3972827675 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee9c0c9440 0x7fee9c14de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:48.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.724+0000 7fee9b7fe640 1 --2- 172.21.15.19:0/3972827675 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fee9c0c9030 0x7fee9c15a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:48.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.724+0000 7fee9bfff640 1 -- 172.21.15.19:0/3972827675 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fee9c0c9030 msgr2=0x7fee9c15a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:48.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.724+0000 7fee9bfff640 1 --2- 172.21.15.19:0/3972827675 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fee9c0c9030 0x7fee9c15a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:48.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.724+0000 7fee9bfff640 1 -- 172.21.15.19:0/3972827675 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee9c0c9440 msgr2=0x7fee9c14de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:48.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.724+0000 7fee9bfff640 1 --2- 172.21.15.19:0/3972827675 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee9c0c9440 0x7fee9c14de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:48.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.724+0000 7fee9bfff640 1 -- 172.21.15.19:0/3972827675 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee9c15aec0 con 0x7fee9c153190 2024-08-31T12:28:48.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.724+0000 7fee9b7fe640 1 --2- 172.21.15.19:0/3972827675 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fee9c0c9030 0x7fee9c15a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:28:48.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.724+0000 7fee9bfff640 1 --2- 172.21.15.19:0/3972827675 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fee9c153190 0x7fee9c151c90 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fee9400c730 tx=0x7fee9400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:48.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.724+0000 7fee83fff640 1 -- 172.21.15.19:0/3972827675 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee94003b70 con 0x7fee9c153190 2024-08-31T12:28:48.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.724+0000 7fee83fff640 1 -- 172.21.15.19:0/3972827675 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fee94003d30 con 0x7fee9c153190 2024-08-31T12:28:48.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.725+0000 7feea1f9e640 1 -- 172.21.15.19:0/3972827675 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee9c15b0d0 con 0x7fee9c153190 2024-08-31T12:28:48.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.725+0000 7fee83fff640 1 -- 172.21.15.19:0/3972827675 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee940052d0 con 0x7fee9c153190 2024-08-31T12:28:48.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.725+0000 7feea1f9e640 1 -- 172.21.15.19:0/3972827675 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee9c06f940 con 0x7fee9c153190 2024-08-31T12:28:48.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.725+0000 7feea1f9e640 1 -- 172.21.15.19:0/3972827675 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee9c0c47d0 con 0x7fee9c153190 2024-08-31T12:28:48.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.726+0000 7fee83fff640 1 -- 172.21.15.19:0/3972827675 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fee94005470 con 0x7fee9c153190 2024-08-31T12:28:48.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.727+0000 7fee83fff640 1 --2- 172.21.15.19:0/3972827675 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fee88077a80 0x7fee88079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:48.732 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.727+0000 7fee83fff640 1 -- 172.21.15.19:0/3972827675 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fee94099340 con 0x7fee9c153190 2024-08-31T12:28:48.732 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.727+0000 7fee9b7fe640 1 --2- 172.21.15.19:0/3972827675 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fee88077a80 0x7fee88079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:48.732 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.727+0000 7fee9b7fe640 1 --2- 172.21.15.19:0/3972827675 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fee88077a80 0x7fee88079f90 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fee8c034000 tx=0x7fee8c004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:48.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.731+0000 7fee83fff640 1 -- 172.21.15.19:0/3972827675 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fee9c0c47d0 con 0x7fee9c153190 2024-08-31T12:28:48.914 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.912+0000 7feea1f9e640 1 -- 172.21.15.19:0/3972827675 --> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fee9c15a180 con 0x7fee88077a80 2024-08-31T12:28:48.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.917+0000 7fee83fff640 1 -- 172.21.15.19:0/3972827675 <== mgr.4100 v2:172.21.15.43:6800/2824336655 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+254513 (secure 0 0 0) 0x7fee9c15a180 con 0x7fee88077a80 2024-08-31T12:28:48.922 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:28:48.923 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-08-31T12:28:48.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.920+0000 7feea1f9e640 1 -- 172.21.15.19:0/3972827675 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fee88077a80 msgr2=0x7fee88079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:48.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.920+0000 7feea1f9e640 1 --2- 172.21.15.19:0/3972827675 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fee88077a80 0x7fee88079f90 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fee8c034000 tx=0x7fee8c004730 comp rx=0 tx=0).stop 2024-08-31T12:28:48.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.920+0000 7feea1f9e640 1 -- 172.21.15.19:0/3972827675 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fee9c153190 msgr2=0x7fee9c151c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:48.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.920+0000 7feea1f9e640 1 --2- 172.21.15.19:0/3972827675 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fee9c153190 0x7fee9c151c90 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fee9400c730 tx=0x7fee9400cbf0 comp rx=0 tx=0).stop 2024-08-31T12:28:48.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.920+0000 7feea1f9e640 1 -- 172.21.15.19:0/3972827675 shutdown_connections 2024-08-31T12:28:48.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.920+0000 7feea1f9e640 1 --2- 172.21.15.19:0/3972827675 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fee88077a80 0x7fee88079f90 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:48.926 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.920+0000 7feea1f9e640 1 --2- 172.21.15.19:0/3972827675 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fee9c153190 0x7fee9c151c90 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:48.926 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.920+0000 7feea1f9e640 1 --2- 172.21.15.19:0/3972827675 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee9c0c9440 0x7fee9c14de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:48.926 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.920+0000 7feea1f9e640 1 --2- 172.21.15.19:0/3972827675 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fee9c0c9030 0x7fee9c15a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:48.926 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.920+0000 7feea1f9e640 1 -- 172.21.15.19:0/3972827675 >> 172.21.15.19:0/3972827675 conn(0x7fee9c0728b0 msgr2=0x7fee9c0cae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:48.926 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.920+0000 7feea1f9e640 1 -- 172.21.15.19:0/3972827675 shutdown_connections 2024-08-31T12:28:48.926 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:48.920+0000 7feea1f9e640 1 -- 172.21.15.19:0/3972827675 wait complete. 2024-08-31T12:28:48.934 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":409,"stamp":"2024-08-31T12:28:48.707058+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":13438780938,"num_objects":3466,"num_object_clones":0,"num_object_copies":6932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3466,"num_whiteouts":0,"num_read":96,"num_read_kb":119,"num_write":921562,"num_write_kb":16804360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":186528,"ondisk_log_size":186528,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":28675632,"kb_used_data":26250480,"kb_used_omap":12,"kb_used_meta":2425011,"kb_avail":726299088,"statfs":{"total":773094113280,"available":743730266112,"internally_reserved":0,"allocated":26880491520,"data_stored":26878225604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":2483211825},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5476,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003706"},"pg_stats":[{"pgid":"3.3f","version":"23'2724","reported_seq":2747,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.201520+0000","last_change":"2024-08-31T12:15:57.306560+0000","last_active":"2024-08-31T12:28:19.201520+0000","last_peered":"2024-08-31T12:28:19.201520+0000","last_clean":"2024-08-31T12:28:19.201520+0000","last_became_active":"2024-08-31T12:15:57.306148+0000","last_became_peered":"2024-08-31T12:15:57.306148+0000","last_unstale":"2024-08-31T12:28:19.201520+0000","last_undegraded":"2024-08-31T12:28:19.201520+0000","last_fullsized":"2024-08-31T12:28:19.201520+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2724,"log_dups_size":0,"ondisk_log_size":2724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:35:53.831742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180377297,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13788,"num_write_kb":231441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"23'3448","reported_seq":3471,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.169946+0000","last_change":"2024-08-31T12:15:57.307314+0000","last_active":"2024-08-31T12:28:19.169946+0000","last_peered":"2024-08-31T12:28:19.169946+0000","last_clean":"2024-08-31T12:28:19.169946+0000","last_became_active":"2024-08-31T12:15:57.306940+0000","last_became_peered":"2024-08-31T12:15:57.306940+0000","last_unstale":"2024-08-31T12:28:19.169946+0000","last_undegraded":"2024-08-31T12:28:19.169946+0000","last_fullsized":"2024-08-31T12:28:19.169946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3448,"log_dups_size":0,"ondisk_log_size":3448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:38:30.321214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264252554,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17611,"num_write_kb":328696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"23'3134","reported_seq":3157,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.114777+0000","last_change":"2024-08-31T12:15:57.309372+0000","last_active":"2024-08-31T12:28:19.114777+0000","last_peered":"2024-08-31T12:28:19.114777+0000","last_clean":"2024-08-31T12:28:19.114777+0000","last_became_active":"2024-08-31T12:15:57.305795+0000","last_became_peered":"2024-08-31T12:15:57.305795+0000","last_unstale":"2024-08-31T12:28:19.114777+0000","last_undegraded":"2024-08-31T12:28:19.114777+0000","last_fullsized":"2024-08-31T12:28:19.114777+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3134,"log_dups_size":0,"ondisk_log_size":3134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:37:45.891394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213942725,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15777,"num_write_kb":272211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"23'2785","reported_seq":2808,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.158289+0000","last_change":"2024-08-31T12:15:57.307339+0000","last_active":"2024-08-31T12:28:19.158289+0000","last_peered":"2024-08-31T12:28:19.158289+0000","last_clean":"2024-08-31T12:28:19.158289+0000","last_became_active":"2024-08-31T12:15:57.306957+0000","last_became_peered":"2024-08-31T12:15:57.306957+0000","last_unstale":"2024-08-31T12:28:19.158289+0000","last_undegraded":"2024-08-31T12:28:19.158289+0000","last_fullsized":"2024-08-31T12:28:19.158289+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2785,"log_dups_size":0,"ondisk_log_size":2785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:43:33.705099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14240,"num_write_kb":241389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"23'3277","reported_seq":3299,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.203476+0000","last_change":"2024-08-31T12:15:57.317492+0000","last_active":"2024-08-31T12:28:19.203476+0000","last_peered":"2024-08-31T12:28:19.203476+0000","last_clean":"2024-08-31T12:28:19.203476+0000","last_became_active":"2024-08-31T12:15:57.317079+0000","last_became_peered":"2024-08-31T12:15:57.317079+0000","last_unstale":"2024-08-31T12:28:19.203476+0000","last_undegraded":"2024-08-31T12:28:19.203476+0000","last_fullsized":"2024-08-31T12:28:19.203476+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3277,"log_dups_size":0,"ondisk_log_size":3277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:44:40.712699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247474530,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15722,"num_write_kb":304743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"23'2512","reported_seq":2534,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.179506+0000","last_change":"2024-08-31T12:15:57.307505+0000","last_active":"2024-08-31T12:28:19.179506+0000","last_peered":"2024-08-31T12:28:19.179506+0000","last_clean":"2024-08-31T12:28:19.179506+0000","last_became_active":"2024-08-31T12:15:57.307010+0000","last_became_peered":"2024-08-31T12:15:57.307010+0000","last_unstale":"2024-08-31T12:28:19.179506+0000","last_undegraded":"2024-08-31T12:28:19.179506+0000","last_fullsized":"2024-08-31T12:28:19.179506+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2512,"log_dups_size":0,"ondisk_log_size":2512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:39:31.100565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180356168,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12564,"num_write_kb":226449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"23'2626","reported_seq":2648,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.143842+0000","last_change":"2024-08-31T12:15:57.316892+0000","last_active":"2024-08-31T12:28:19.143842+0000","last_peered":"2024-08-31T12:28:19.143842+0000","last_clean":"2024-08-31T12:28:19.143842+0000","last_became_active":"2024-08-31T12:15:57.316504+0000","last_became_peered":"2024-08-31T12:15:57.316504+0000","last_unstale":"2024-08-31T12:28:19.143842+0000","last_undegraded":"2024-08-31T12:28:19.143842+0000","last_fullsized":"2024-08-31T12:28:19.143842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2626,"log_dups_size":0,"ondisk_log_size":2626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:05:48.614918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201398355,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13306,"num_write_kb":250051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"23'3158","reported_seq":3173,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.134179+0000","last_change":"2024-08-31T12:16:40.694493+0000","last_active":"2024-08-31T12:28:19.134179+0000","last_peered":"2024-08-31T12:28:19.134179+0000","last_clean":"2024-08-31T12:28:19.134179+0000","last_became_active":"2024-08-31T12:16:40.694165+0000","last_became_peered":"2024-08-31T12:16:40.694165+0000","last_unstale":"2024-08-31T12:28:19.134179+0000","last_undegraded":"2024-08-31T12:28:19.134179+0000","last_fullsized":"2024-08-31T12:28:19.134179+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3158,"log_dups_size":0,"ondisk_log_size":3158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:55:57.961306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226497620,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15619,"num_write_kb":283880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.37","version":"23'3491","reported_seq":3513,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.201405+0000","last_change":"2024-08-31T12:15:57.316723+0000","last_active":"2024-08-31T12:28:19.201405+0000","last_peered":"2024-08-31T12:28:19.201405+0000","last_clean":"2024-08-31T12:28:19.201405+0000","last_became_active":"2024-08-31T12:15:57.316354+0000","last_became_peered":"2024-08-31T12:15:57.316354+0000","last_unstale":"2024-08-31T12:28:19.201405+0000","last_undegraded":"2024-08-31T12:28:19.201405+0000","last_fullsized":"2024-08-31T12:28:19.201405+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3491,"log_dups_size":0,"ondisk_log_size":3491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:31:19.860950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251662082,"num_objects":61,"num_object_clones":0,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18100,"num_write_kb":318223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"23'2954","reported_seq":2976,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.206584+0000","last_change":"2024-08-31T12:15:57.311161+0000","last_active":"2024-08-31T12:28:19.206584+0000","last_peered":"2024-08-31T12:28:19.206584+0000","last_clean":"2024-08-31T12:28:19.206584+0000","last_became_active":"2024-08-31T12:15:57.310807+0000","last_became_peered":"2024-08-31T12:15:57.310807+0000","last_unstale":"2024-08-31T12:28:19.206584+0000","last_undegraded":"2024-08-31T12:28:19.206584+0000","last_fullsized":"2024-08-31T12:28:19.206584+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2954,"log_dups_size":0,"ondisk_log_size":2954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:56:03.280339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234894029,"num_objects":59,"num_object_clones":0,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15073,"num_write_kb":289757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"23'2880","reported_seq":2902,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.165528+0000","last_change":"2024-08-31T12:15:57.300930+0000","last_active":"2024-08-31T12:28:19.165528+0000","last_peered":"2024-08-31T12:28:19.165528+0000","last_clean":"2024-08-31T12:28:19.165528+0000","last_became_active":"2024-08-31T12:15:57.300610+0000","last_became_peered":"2024-08-31T12:15:57.300610+0000","last_unstale":"2024-08-31T12:28:19.165528+0000","last_undegraded":"2024-08-31T12:28:19.165528+0000","last_fullsized":"2024-08-31T12:28:19.165528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2880,"log_dups_size":0,"ondisk_log_size":2880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:42:39.731742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201326592,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14139,"num_write_kb":253227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"23'2246","reported_seq":2268,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.123192+0000","last_change":"2024-08-31T12:15:57.303151+0000","last_active":"2024-08-31T12:28:19.123192+0000","last_peered":"2024-08-31T12:28:19.123192+0000","last_clean":"2024-08-31T12:28:19.123192+0000","last_became_active":"2024-08-31T12:15:57.302622+0000","last_became_peered":"2024-08-31T12:15:57.302622+0000","last_unstale":"2024-08-31T12:28:19.123192+0000","last_undegraded":"2024-08-31T12:28:19.123192+0000","last_fullsized":"2024-08-31T12:28:19.123192+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2246,"log_dups_size":0,"ondisk_log_size":2246,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:49:18.493629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176192645,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11403,"num_write_kb":217791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"23'2948","reported_seq":2970,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.215841+0000","last_change":"2024-08-31T12:15:57.305676+0000","last_active":"2024-08-31T12:28:19.215841+0000","last_peered":"2024-08-31T12:28:19.215841+0000","last_clean":"2024-08-31T12:28:19.215841+0000","last_became_active":"2024-08-31T12:15:57.305321+0000","last_became_peered":"2024-08-31T12:15:57.305321+0000","last_unstale":"2024-08-31T12:28:19.215841+0000","last_undegraded":"2024-08-31T12:28:19.215841+0000","last_fullsized":"2024-08-31T12:28:19.215841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2948,"log_dups_size":0,"ondisk_log_size":2948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:42:39.731742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188840677,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15214,"num_write_kb":245235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"23'3122","reported_seq":3144,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.194400+0000","last_change":"2024-08-31T12:15:57.309671+0000","last_active":"2024-08-31T12:28:19.194400+0000","last_peered":"2024-08-31T12:28:19.194400+0000","last_clean":"2024-08-31T12:28:19.194400+0000","last_became_active":"2024-08-31T12:15:57.307013+0000","last_became_peered":"2024-08-31T12:15:57.307013+0000","last_unstale":"2024-08-31T12:28:19.194400+0000","last_undegraded":"2024-08-31T12:28:19.194400+0000","last_fullsized":"2024-08-31T12:28:19.194400+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3122,"log_dups_size":0,"ondisk_log_size":3122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:18:55.451164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213986201,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15743,"num_write_kb":271989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"23'2952","reported_seq":2974,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.181372+0000","last_change":"2024-08-31T12:15:57.298150+0000","last_active":"2024-08-31T12:28:19.181372+0000","last_peered":"2024-08-31T12:28:19.181372+0000","last_clean":"2024-08-31T12:28:19.181372+0000","last_became_active":"2024-08-31T12:15:57.297182+0000","last_became_peered":"2024-08-31T12:15:57.297182+0000","last_unstale":"2024-08-31T12:28:19.181372+0000","last_undegraded":"2024-08-31T12:28:19.181372+0000","last_fullsized":"2024-08-31T12:28:19.181372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2952,"log_dups_size":0,"ondisk_log_size":2952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:16:42.625396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213913551,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14445,"num_write_kb":266758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"23'2546","reported_seq":2562,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.462905+0000","last_change":"2024-08-31T12:16:41.075080+0000","last_active":"2024-08-31T12:28:47.462905+0000","last_peered":"2024-08-31T12:28:47.462905+0000","last_clean":"2024-08-31T12:28:47.462905+0000","last_became_active":"2024-08-31T12:16:41.074773+0000","last_became_peered":"2024-08-31T12:16:41.074773+0000","last_unstale":"2024-08-31T12:28:47.462905+0000","last_undegraded":"2024-08-31T12:28:47.462905+0000","last_fullsized":"2024-08-31T12:28:47.462905+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2546,"log_dups_size":0,"ondisk_log_size":2546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:10:45.435137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184599153,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12917,"num_write_kb":231981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"23'3715","reported_seq":3737,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.196629+0000","last_change":"2024-08-31T12:15:57.308899+0000","last_active":"2024-08-31T12:28:19.196629+0000","last_peered":"2024-08-31T12:28:19.196629+0000","last_clean":"2024-08-31T12:28:19.196629+0000","last_became_active":"2024-08-31T12:15:57.308388+0000","last_became_peered":"2024-08-31T12:15:57.308388+0000","last_unstale":"2024-08-31T12:28:19.196629+0000","last_undegraded":"2024-08-31T12:28:19.196629+0000","last_fullsized":"2024-08-31T12:28:19.196629+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3715,"log_dups_size":0,"ondisk_log_size":3715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:54:03.871840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251747648,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18708,"num_write_kb":320814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"23'2148","reported_seq":2170,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.446094+0000","last_change":"2024-08-31T12:15:57.300201+0000","last_active":"2024-08-31T12:28:47.446094+0000","last_peered":"2024-08-31T12:28:47.446094+0000","last_clean":"2024-08-31T12:28:47.446094+0000","last_became_active":"2024-08-31T12:15:57.299826+0000","last_became_peered":"2024-08-31T12:15:57.299826+0000","last_unstale":"2024-08-31T12:28:47.446094+0000","last_undegraded":"2024-08-31T12:28:47.446094+0000","last_fullsized":"2024-08-31T12:28:47.446094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2148,"log_dups_size":0,"ondisk_log_size":2148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:14:34.488102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167791563,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10514,"num_write_kb":205964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"23'2668","reported_seq":2690,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.054487+0000","last_change":"2024-08-31T12:15:57.317606+0000","last_active":"2024-08-31T12:28:19.054487+0000","last_peered":"2024-08-31T12:28:19.054487+0000","last_clean":"2024-08-31T12:28:19.054487+0000","last_became_active":"2024-08-31T12:15:57.317185+0000","last_became_peered":"2024-08-31T12:15:57.317185+0000","last_unstale":"2024-08-31T12:28:19.054487+0000","last_undegraded":"2024-08-31T12:28:19.054487+0000","last_fullsized":"2024-08-31T12:28:19.054487+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2668,"log_dups_size":0,"ondisk_log_size":2668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:25:46.346490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209762270,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13163,"num_write_kb":257524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"23'2441","reported_seq":2463,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.104779+0000","last_change":"2024-08-31T12:15:57.298778+0000","last_active":"2024-08-31T12:28:19.104779+0000","last_peered":"2024-08-31T12:28:19.104779+0000","last_clean":"2024-08-31T12:28:19.104779+0000","last_became_active":"2024-08-31T12:15:57.298430+0000","last_became_peered":"2024-08-31T12:15:57.298430+0000","last_unstale":"2024-08-31T12:28:19.104779+0000","last_undegraded":"2024-08-31T12:28:19.104779+0000","last_fullsized":"2024-08-31T12:28:19.104779+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2441,"log_dups_size":0,"ondisk_log_size":2441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:22:13.435300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205530518,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12230,"num_write_kb":249721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"23'2939","reported_seq":2961,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.228814+0000","last_change":"2024-08-31T12:15:57.317176+0000","last_active":"2024-08-31T12:28:19.228814+0000","last_peered":"2024-08-31T12:28:19.228814+0000","last_clean":"2024-08-31T12:28:19.228814+0000","last_became_active":"2024-08-31T12:15:57.316870+0000","last_became_peered":"2024-08-31T12:15:57.316870+0000","last_unstale":"2024-08-31T12:28:19.228814+0000","last_undegraded":"2024-08-31T12:28:19.228814+0000","last_fullsized":"2024-08-31T12:28:19.228814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2939,"log_dups_size":0,"ondisk_log_size":2939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:43:12.703971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213923005,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14700,"num_write_kb":267872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"23'3066","reported_seq":3082,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.485438+0000","last_change":"2024-08-31T12:16:40.784525+0000","last_active":"2024-08-31T12:28:47.485438+0000","last_peered":"2024-08-31T12:28:47.485438+0000","last_clean":"2024-08-31T12:28:47.485438+0000","last_became_active":"2024-08-31T12:16:40.784204+0000","last_became_peered":"2024-08-31T12:16:40.784204+0000","last_unstale":"2024-08-31T12:28:47.485438+0000","last_undegraded":"2024-08-31T12:28:47.485438+0000","last_fullsized":"2024-08-31T12:28:47.485438+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3066,"log_dups_size":0,"ondisk_log_size":3066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:51:38.173279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226493660,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15308,"num_write_kb":282531,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"23'2827","reported_seq":2849,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.207366+0000","last_change":"2024-08-31T12:15:57.304032+0000","last_active":"2024-08-31T12:28:19.207366+0000","last_peered":"2024-08-31T12:28:19.207366+0000","last_clean":"2024-08-31T12:28:19.207366+0000","last_became_active":"2024-08-31T12:15:57.303401+0000","last_became_peered":"2024-08-31T12:15:57.303401+0000","last_unstale":"2024-08-31T12:28:19.207366+0000","last_undegraded":"2024-08-31T12:28:19.207366+0000","last_fullsized":"2024-08-31T12:28:19.207366+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2827,"log_dups_size":0,"ondisk_log_size":2827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:56:03.280339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205588184,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14070,"num_write_kb":257066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"23'2468","reported_seq":2490,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.209009+0000","last_change":"2024-08-31T12:15:57.305412+0000","last_active":"2024-08-31T12:28:19.209009+0000","last_peered":"2024-08-31T12:28:19.209009+0000","last_clean":"2024-08-31T12:28:19.209009+0000","last_became_active":"2024-08-31T12:15:57.305075+0000","last_became_peered":"2024-08-31T12:15:57.305075+0000","last_unstale":"2024-08-31T12:28:19.209009+0000","last_undegraded":"2024-08-31T12:28:19.209009+0000","last_fullsized":"2024-08-31T12:28:19.209009+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:30:13.715505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184555785,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12290,"num_write_kb":229419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"23'3164","reported_seq":3186,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.223770+0000","last_change":"2024-08-31T12:15:57.300179+0000","last_active":"2024-08-31T12:28:19.223770+0000","last_peered":"2024-08-31T12:28:19.223770+0000","last_clean":"2024-08-31T12:28:19.223770+0000","last_became_active":"2024-08-31T12:15:57.299861+0000","last_became_peered":"2024-08-31T12:15:57.299861+0000","last_unstale":"2024-08-31T12:28:19.223770+0000","last_undegraded":"2024-08-31T12:28:19.223770+0000","last_fullsized":"2024-08-31T12:28:19.223770+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3164,"log_dups_size":0,"ondisk_log_size":3164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:47:51.267409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243322604,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15709,"num_write_kb":300530,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"23'2729","reported_seq":2751,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.205829+0000","last_change":"2024-08-31T12:15:57.308519+0000","last_active":"2024-08-31T12:28:19.205829+0000","last_peered":"2024-08-31T12:28:19.205829+0000","last_clean":"2024-08-31T12:28:19.205829+0000","last_became_active":"2024-08-31T12:15:57.308189+0000","last_became_peered":"2024-08-31T12:15:57.308189+0000","last_unstale":"2024-08-31T12:28:19.205829+0000","last_undegraded":"2024-08-31T12:28:19.205829+0000","last_fullsized":"2024-08-31T12:28:19.205829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2729,"log_dups_size":0,"ondisk_log_size":2729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:37:02.435678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189169648,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13404,"num_write_kb":238420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"23'3082","reported_seq":3104,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.211281+0000","last_change":"2024-08-31T12:15:57.308321+0000","last_active":"2024-08-31T12:28:19.211281+0000","last_peered":"2024-08-31T12:28:19.211281+0000","last_clean":"2024-08-31T12:28:19.211281+0000","last_became_active":"2024-08-31T12:15:57.304514+0000","last_became_peered":"2024-08-31T12:15:57.304514+0000","last_unstale":"2024-08-31T12:28:19.211281+0000","last_undegraded":"2024-08-31T12:28:19.211281+0000","last_fullsized":"2024-08-31T12:28:19.211281+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3082,"log_dups_size":0,"ondisk_log_size":3082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:09:49.617826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222298838,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15326,"num_write_kb":278485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"23'2570","reported_seq":2592,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.125425+0000","last_change":"2024-08-31T12:15:57.299268+0000","last_active":"2024-08-31T12:28:19.125425+0000","last_peered":"2024-08-31T12:28:19.125425+0000","last_clean":"2024-08-31T12:28:19.125425+0000","last_became_active":"2024-08-31T12:15:57.298766+0000","last_became_peered":"2024-08-31T12:15:57.298766+0000","last_unstale":"2024-08-31T12:28:19.125425+0000","last_undegraded":"2024-08-31T12:28:19.125425+0000","last_fullsized":"2024-08-31T12:28:19.125425+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2570,"log_dups_size":0,"ondisk_log_size":2570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:37:45.891394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188773348,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12878,"num_write_kb":235916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"23'2321","reported_seq":2344,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.234241+0000","last_change":"2024-08-31T12:15:57.310121+0000","last_active":"2024-08-31T12:28:19.234241+0000","last_peered":"2024-08-31T12:28:19.234241+0000","last_clean":"2024-08-31T12:28:19.234241+0000","last_became_active":"2024-08-31T12:15:57.309440+0000","last_became_peered":"2024-08-31T12:15:57.309440+0000","last_unstale":"2024-08-31T12:28:19.234241+0000","last_undegraded":"2024-08-31T12:28:19.234241+0000","last_fullsized":"2024-08-31T12:28:19.234241+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2321,"log_dups_size":0,"ondisk_log_size":2321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:53:19.523811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180357122,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11238,"num_write_kb":221098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"23'3439","reported_seq":3462,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.198926+0000","last_change":"2024-08-31T12:15:57.306214+0000","last_active":"2024-08-31T12:28:19.198926+0000","last_peered":"2024-08-31T12:28:19.198926+0000","last_clean":"2024-08-31T12:28:19.198926+0000","last_became_active":"2024-08-31T12:15:57.305878+0000","last_became_peered":"2024-08-31T12:15:57.305878+0000","last_unstale":"2024-08-31T12:28:19.198926+0000","last_undegraded":"2024-08-31T12:28:19.198926+0000","last_fullsized":"2024-08-31T12:28:19.198926+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3439,"log_dups_size":0,"ondisk_log_size":3439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:05:48.614918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251668879,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17281,"num_write_kb":314954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"23'2880","reported_seq":2903,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.439514+0000","last_change":"2024-08-31T12:15:57.307376+0000","last_active":"2024-08-31T12:28:47.439514+0000","last_peered":"2024-08-31T12:28:47.439514+0000","last_clean":"2024-08-31T12:28:47.439514+0000","last_became_active":"2024-08-31T12:15:57.307056+0000","last_became_peered":"2024-08-31T12:15:57.307056+0000","last_unstale":"2024-08-31T12:28:47.439514+0000","last_undegraded":"2024-08-31T12:28:47.439514+0000","last_fullsized":"2024-08-31T12:28:47.439514+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2880,"log_dups_size":0,"ondisk_log_size":2880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:52:18.004608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222298112,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14386,"num_write_kb":274682,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"23'3113","reported_seq":3136,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.193855+0000","last_change":"2024-08-31T12:15:57.313798+0000","last_active":"2024-08-31T12:28:19.193855+0000","last_peered":"2024-08-31T12:28:19.193855+0000","last_clean":"2024-08-31T12:28:19.193855+0000","last_became_active":"2024-08-31T12:15:57.313463+0000","last_became_peered":"2024-08-31T12:15:57.313463+0000","last_unstale":"2024-08-31T12:28:19.193855+0000","last_undegraded":"2024-08-31T12:28:19.193855+0000","last_fullsized":"2024-08-31T12:28:19.193855+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3113,"log_dups_size":0,"ondisk_log_size":3113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:47:51.267409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260049613,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15263,"num_write_kb":315030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"23'2280","reported_seq":2302,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.088996+0000","last_change":"2024-08-31T12:15:57.293350+0000","last_active":"2024-08-31T12:28:19.088996+0000","last_peered":"2024-08-31T12:28:19.088996+0000","last_clean":"2024-08-31T12:28:19.088996+0000","last_became_active":"2024-08-31T12:15:57.292981+0000","last_became_peered":"2024-08-31T12:15:57.292981+0000","last_unstale":"2024-08-31T12:28:19.088996+0000","last_undegraded":"2024-08-31T12:28:19.088996+0000","last_fullsized":"2024-08-31T12:28:19.088996+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2280,"log_dups_size":0,"ondisk_log_size":2280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:52:00.312013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180377946,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11142,"num_write_kb":220690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"23'569","reported_seq":593,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:56.212722+0000","last_change":"2024-08-31T12:15:56.293497+0000","last_active":"2024-08-31T12:25:56.212722+0000","last_peered":"2024-08-31T12:25:56.212722+0000","last_clean":"2024-08-31T12:25:56.212722+0000","last_became_active":"2024-08-31T12:15:56.293164+0000","last_became_peered":"2024-08-31T12:15:56.293164+0000","last_unstale":"2024-08-31T12:25:56.212722+0000","last_undegraded":"2024-08-31T12:25:56.212722+0000","last_fullsized":"2024-08-31T12:25:56.212722+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":569,"log_dups_size":0,"ondisk_log_size":569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:43:17.443433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4195367,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":566,"num_write_kb":4210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"23'2388","reported_seq":2410,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.228535+0000","last_change":"2024-08-31T12:15:57.312738+0000","last_active":"2024-08-31T12:28:19.228535+0000","last_peered":"2024-08-31T12:28:19.228535+0000","last_clean":"2024-08-31T12:28:19.228535+0000","last_became_active":"2024-08-31T12:15:57.312360+0000","last_became_peered":"2024-08-31T12:15:57.312360+0000","last_unstale":"2024-08-31T12:28:19.228535+0000","last_undegraded":"2024-08-31T12:28:19.228535+0000","last_fullsized":"2024-08-31T12:28:19.228535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2388,"log_dups_size":0,"ondisk_log_size":2388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:22:13.435300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163584295,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11807,"num_write_kb":207035,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"23'5","reported_seq":29,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:56.196855+0000","last_change":"2024-08-31T12:15:56.283526+0000","last_active":"2024-08-31T12:25:56.196855+0000","last_peered":"2024-08-31T12:25:56.196855+0000","last_clean":"2024-08-31T12:25:56.196855+0000","last_became_active":"2024-08-31T12:15:56.283147+0000","last_became_peered":"2024-08-31T12:15:56.283147+0000","last_unstale":"2024-08-31T12:25:56.196855+0000","last_undegraded":"2024-08-31T12:25:56.196855+0000","last_fullsized":"2024-08-31T12:25:56.196855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:51:59.306912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"23'4","reported_seq":28,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:36.210659+0000","last_change":"2024-08-31T12:15:56.291170+0000","last_active":"2024-08-31T12:25:36.210659+0000","last_peered":"2024-08-31T12:25:36.210659+0000","last_clean":"2024-08-31T12:25:36.210659+0000","last_became_active":"2024-08-31T12:15:56.290835+0000","last_became_peered":"2024-08-31T12:15:56.290835+0000","last_unstale":"2024-08-31T12:25:36.210659+0000","last_undegraded":"2024-08-31T12:25:36.210659+0000","last_fullsized":"2024-08-31T12:25:36.210659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:15:51.561831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"23'2535","reported_seq":2557,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.224638+0000","last_change":"2024-08-31T12:15:57.308183+0000","last_active":"2024-08-31T12:28:19.224638+0000","last_peered":"2024-08-31T12:28:19.224638+0000","last_clean":"2024-08-31T12:28:19.224638+0000","last_became_active":"2024-08-31T12:15:57.306962+0000","last_became_peered":"2024-08-31T12:15:57.306962+0000","last_unstale":"2024-08-31T12:28:19.224638+0000","last_undegraded":"2024-08-31T12:28:19.224638+0000","last_fullsized":"2024-08-31T12:28:19.224638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2535,"log_dups_size":0,"ondisk_log_size":2535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:01:18.188264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205541461,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":251825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"23'2","reported_seq":26,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:40.196751+0000","last_change":"2024-08-31T12:15:56.288126+0000","last_active":"2024-08-31T12:25:40.196751+0000","last_peered":"2024-08-31T12:25:40.196751+0000","last_clean":"2024-08-31T12:25:40.196751+0000","last_became_active":"2024-08-31T12:15:56.287552+0000","last_became_peered":"2024-08-31T12:15:56.287552+0000","last_unstale":"2024-08-31T12:25:40.196751+0000","last_undegraded":"2024-08-31T12:25:40.196751+0000","last_fullsized":"2024-08-31T12:25:40.196751+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:54:29.988079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"23'3439","reported_seq":3461,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.208231+0000","last_change":"2024-08-31T12:15:57.305029+0000","last_active":"2024-08-31T12:28:19.208231+0000","last_peered":"2024-08-31T12:28:19.208231+0000","last_clean":"2024-08-31T12:28:19.208231+0000","last_became_active":"2024-08-31T12:15:57.304700+0000","last_became_peered":"2024-08-31T12:15:57.304700+0000","last_unstale":"2024-08-31T12:28:19.208231+0000","last_undegraded":"2024-08-31T12:28:19.208231+0000","last_fullsized":"2024-08-31T12:28:19.208231+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3439,"log_dups_size":0,"ondisk_log_size":3439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:58:17.482204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243316666,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17415,"num_write_kb":307366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"23'4","reported_seq":28,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:36.194403+0000","last_change":"2024-08-31T12:15:56.284907+0000","last_active":"2024-08-31T12:25:36.194403+0000","last_peered":"2024-08-31T12:25:36.194403+0000","last_clean":"2024-08-31T12:25:36.194403+0000","last_became_active":"2024-08-31T12:15:56.284558+0000","last_became_peered":"2024-08-31T12:15:56.284558+0000","last_unstale":"2024-08-31T12:25:36.194403+0000","last_undegraded":"2024-08-31T12:25:36.194403+0000","last_fullsized":"2024-08-31T12:25:36.194403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:29:53.117724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"23'2894","reported_seq":2916,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.203880+0000","last_change":"2024-08-31T12:15:57.308679+0000","last_active":"2024-08-31T12:28:19.203880+0000","last_peered":"2024-08-31T12:28:19.203880+0000","last_clean":"2024-08-31T12:28:19.203880+0000","last_became_active":"2024-08-31T12:15:57.308260+0000","last_became_peered":"2024-08-31T12:15:57.308260+0000","last_unstale":"2024-08-31T12:28:19.203880+0000","last_undegraded":"2024-08-31T12:28:19.203880+0000","last_fullsized":"2024-08-31T12:28:19.203880+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2894,"log_dups_size":0,"ondisk_log_size":2894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:31:19.860950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209955924,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14127,"num_write_kb":261546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"21'2","reported_seq":26,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:40.196825+0000","last_change":"2024-08-31T12:15:56.290763+0000","last_active":"2024-08-31T12:25:40.196825+0000","last_peered":"2024-08-31T12:25:40.196825+0000","last_clean":"2024-08-31T12:25:40.196825+0000","last_became_active":"2024-08-31T12:15:56.290400+0000","last_became_peered":"2024-08-31T12:15:56.290400+0000","last_unstale":"2024-08-31T12:25:40.196825+0000","last_undegraded":"2024-08-31T12:25:40.196825+0000","last_fullsized":"2024-08-31T12:25:40.196825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:53:08.929860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"23'2656","reported_seq":2678,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.161047+0000","last_change":"2024-08-31T12:15:57.311657+0000","last_active":"2024-08-31T12:28:19.161047+0000","last_peered":"2024-08-31T12:28:19.161047+0000","last_clean":"2024-08-31T12:28:19.161047+0000","last_became_active":"2024-08-31T12:15:57.310564+0000","last_became_peered":"2024-08-31T12:15:57.310564+0000","last_unstale":"2024-08-31T12:28:19.161047+0000","last_undegraded":"2024-08-31T12:28:19.161047+0000","last_fullsized":"2024-08-31T12:28:19.161047+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2656,"log_dups_size":0,"ondisk_log_size":2656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:37:02.435678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180410852,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13728,"num_write_kb":231123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"23'19","reported_seq":43,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:36.210981+0000","last_change":"2024-08-31T12:15:56.286792+0000","last_active":"2024-08-31T12:25:36.210981+0000","last_peered":"2024-08-31T12:25:36.210981+0000","last_clean":"2024-08-31T12:25:36.210981+0000","last_became_active":"2024-08-31T12:15:56.286458+0000","last_became_peered":"2024-08-31T12:15:56.286458+0000","last_unstale":"2024-08-31T12:25:36.210981+0000","last_undegraded":"2024-08-31T12:25:36.210981+0000","last_fullsized":"2024-08-31T12:25:36.210981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:49:17.488528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"23'3658","reported_seq":3680,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.159920+0000","last_change":"2024-08-31T12:15:57.309485+0000","last_active":"2024-08-31T12:28:19.159920+0000","last_peered":"2024-08-31T12:28:19.159920+0000","last_clean":"2024-08-31T12:28:19.159920+0000","last_became_active":"2024-08-31T12:15:57.309102+0000","last_became_peered":"2024-08-31T12:15:57.309102+0000","last_unstale":"2024-08-31T12:28:19.159920+0000","last_undegraded":"2024-08-31T12:28:19.159920+0000","last_fullsized":"2024-08-31T12:28:19.159920+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3658,"log_dups_size":0,"ondisk_log_size":3658,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:17:34.692296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":277024326,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17723,"num_write_kb":341816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"23'317","reported_seq":341,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:22.335219+0000","last_change":"2024-08-31T12:15:56.289460+0000","last_active":"2024-08-31T12:28:22.335219+0000","last_peered":"2024-08-31T12:28:22.335219+0000","last_clean":"2024-08-31T12:28:22.335219+0000","last_became_active":"2024-08-31T12:15:56.289096+0000","last_became_peered":"2024-08-31T12:15:56.289096+0000","last_unstale":"2024-08-31T12:28:22.335219+0000","last_undegraded":"2024-08-31T12:28:22.335219+0000","last_fullsized":"2024-08-31T12:28:22.335219+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:06:34.313772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":317,"num_write_kb":4236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"23'2741","reported_seq":2763,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.210058+0000","last_change":"2024-08-31T12:15:57.317968+0000","last_active":"2024-08-31T12:28:19.210058+0000","last_peered":"2024-08-31T12:28:19.210058+0000","last_clean":"2024-08-31T12:28:19.210058+0000","last_became_active":"2024-08-31T12:15:57.317632+0000","last_became_peered":"2024-08-31T12:15:57.317632+0000","last_unstale":"2024-08-31T12:28:19.210058+0000","last_undegraded":"2024-08-31T12:28:19.210058+0000","last_fullsized":"2024-08-31T12:28:19.210058+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2741,"log_dups_size":0,"ondisk_log_size":2741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:22:02.569106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197132288,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13667,"num_write_kb":247166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'6","reported_seq":30,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:16.208499+0000","last_change":"2024-08-31T12:15:56.284014+0000","last_active":"2024-08-31T12:25:16.208499+0000","last_peered":"2024-08-31T12:25:16.208499+0000","last_clean":"2024-08-31T12:25:16.208499+0000","last_became_active":"2024-08-31T12:15:56.283675+0000","last_became_peered":"2024-08-31T12:15:56.283675+0000","last_unstale":"2024-08-31T12:25:16.208499+0000","last_undegraded":"2024-08-31T12:25:16.208499+0000","last_fullsized":"2024-08-31T12:25:16.208499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:51:59.306912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"23'3052","reported_seq":3074,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.204958+0000","last_change":"2024-08-31T12:15:57.312440+0000","last_active":"2024-08-31T12:28:19.204958+0000","last_peered":"2024-08-31T12:28:19.204958+0000","last_clean":"2024-08-31T12:28:19.204958+0000","last_became_active":"2024-08-31T12:15:57.312007+0000","last_became_peered":"2024-08-31T12:15:57.312007+0000","last_unstale":"2024-08-31T12:28:19.204958+0000","last_undegraded":"2024-08-31T12:28:19.204958+0000","last_fullsized":"2024-08-31T12:28:19.204958+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3052,"log_dups_size":0,"ondisk_log_size":3052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:23:43.553625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197138377,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15100,"num_write_kb":253041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"23'4","reported_seq":28,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:20.194216+0000","last_change":"2024-08-31T12:15:56.290968+0000","last_active":"2024-08-31T12:25:20.194216+0000","last_peered":"2024-08-31T12:25:20.194216+0000","last_clean":"2024-08-31T12:25:20.194216+0000","last_became_active":"2024-08-31T12:15:56.290603+0000","last_became_peered":"2024-08-31T12:15:56.290603+0000","last_unstale":"2024-08-31T12:25:20.194216+0000","last_undegraded":"2024-08-31T12:25:20.194216+0000","last_fullsized":"2024-08-31T12:25:20.194216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:50:23.215239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"23'3279","reported_seq":3301,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.194687+0000","last_change":"2024-08-31T12:15:57.312790+0000","last_active":"2024-08-31T12:28:19.194687+0000","last_peered":"2024-08-31T12:28:19.194687+0000","last_clean":"2024-08-31T12:28:19.194687+0000","last_became_active":"2024-08-31T12:15:57.312383+0000","last_became_peered":"2024-08-31T12:15:57.312383+0000","last_unstale":"2024-08-31T12:28:19.194687+0000","last_undegraded":"2024-08-31T12:28:19.194687+0000","last_fullsized":"2024-08-31T12:28:19.194687+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3279,"log_dups_size":0,"ondisk_log_size":3279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:20:41.042375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234891518,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16277,"num_write_kb":294634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"23'5","reported_seq":29,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:16.191885+0000","last_change":"2024-08-31T12:15:56.289795+0000","last_active":"2024-08-31T12:25:16.191885+0000","last_peered":"2024-08-31T12:25:16.191885+0000","last_clean":"2024-08-31T12:25:16.191885+0000","last_became_active":"2024-08-31T12:15:56.289477+0000","last_became_peered":"2024-08-31T12:15:56.289477+0000","last_unstale":"2024-08-31T12:25:16.191885+0000","last_undegraded":"2024-08-31T12:25:16.191885+0000","last_fullsized":"2024-08-31T12:25:16.191885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:15:51.561831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":54,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"23'2828","reported_seq":2850,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.192699+0000","last_change":"2024-08-31T12:15:57.312865+0000","last_active":"2024-08-31T12:28:19.192699+0000","last_peered":"2024-08-31T12:28:19.192699+0000","last_clean":"2024-08-31T12:28:19.192699+0000","last_became_active":"2024-08-31T12:15:57.312462+0000","last_became_peered":"2024-08-31T12:15:57.312462+0000","last_unstale":"2024-08-31T12:28:19.192699+0000","last_undegraded":"2024-08-31T12:28:19.192699+0000","last_fullsized":"2024-08-31T12:28:19.192699+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2828,"log_dups_size":0,"ondisk_log_size":2828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:16:42.625396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222316815,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13883,"num_write_kb":272702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'15","reported_seq":39,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:18.179855+0000","last_change":"2024-08-31T12:15:56.285412+0000","last_active":"2024-08-31T12:25:18.179855+0000","last_peered":"2024-08-31T12:25:18.179855+0000","last_clean":"2024-08-31T12:25:18.179855+0000","last_became_active":"2024-08-31T12:15:56.285108+0000","last_became_peered":"2024-08-31T12:15:56.285108+0000","last_unstale":"2024-08-31T12:25:18.179855+0000","last_undegraded":"2024-08-31T12:25:18.179855+0000","last_fullsized":"2024-08-31T12:25:18.179855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:29:45.580957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"23'2843","reported_seq":2865,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.145892+0000","last_change":"2024-08-31T12:15:57.311701+0000","last_active":"2024-08-31T12:28:19.145892+0000","last_peered":"2024-08-31T12:28:19.145892+0000","last_clean":"2024-08-31T12:28:19.145892+0000","last_became_active":"2024-08-31T12:15:57.311368+0000","last_became_peered":"2024-08-31T12:15:57.311368+0000","last_unstale":"2024-08-31T12:28:19.145892+0000","last_undegraded":"2024-08-31T12:28:19.145892+0000","last_fullsized":"2024-08-31T12:28:19.145892+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2843,"log_dups_size":0,"ondisk_log_size":2843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:30:13.715505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205547202,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14167,"num_write_kb":257592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"23'3","reported_seq":27,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:00.192148+0000","last_change":"2024-08-31T12:15:56.289979+0000","last_active":"2024-08-31T12:25:00.192148+0000","last_peered":"2024-08-31T12:25:00.192148+0000","last_clean":"2024-08-31T12:25:00.192148+0000","last_became_active":"2024-08-31T12:15:56.288289+0000","last_became_peered":"2024-08-31T12:15:56.288289+0000","last_unstale":"2024-08-31T12:25:00.192148+0000","last_undegraded":"2024-08-31T12:25:00.192148+0000","last_fullsized":"2024-08-31T12:25:00.192148+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:46:55.057087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"23'2808","reported_seq":2830,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.206168+0000","last_change":"2024-08-31T12:15:57.311580+0000","last_active":"2024-08-31T12:28:19.206168+0000","last_peered":"2024-08-31T12:28:19.206168+0000","last_clean":"2024-08-31T12:28:19.206168+0000","last_became_active":"2024-08-31T12:15:57.311323+0000","last_became_peered":"2024-08-31T12:15:57.311323+0000","last_unstale":"2024-08-31T12:28:19.206168+0000","last_undegraded":"2024-08-31T12:28:19.206168+0000","last_fullsized":"2024-08-31T12:28:19.206168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2808,"log_dups_size":0,"ondisk_log_size":2808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:54:03.871840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201522906,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13808,"num_write_kb":252081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"23'3","reported_seq":27,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:24:58.177773+0000","last_change":"2024-08-31T12:15:56.290496+0000","last_active":"2024-08-31T12:24:58.177773+0000","last_peered":"2024-08-31T12:24:58.177773+0000","last_clean":"2024-08-31T12:24:58.177773+0000","last_became_active":"2024-08-31T12:15:56.290196+0000","last_became_peered":"2024-08-31T12:15:56.290196+0000","last_unstale":"2024-08-31T12:24:58.177773+0000","last_undegraded":"2024-08-31T12:24:58.177773+0000","last_fullsized":"2024-08-31T12:24:58.177773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:52:20.055757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"23'3282","reported_seq":3304,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.175690+0000","last_change":"2024-08-31T12:15:57.313317+0000","last_active":"2024-08-31T12:28:19.175690+0000","last_peered":"2024-08-31T12:28:19.175690+0000","last_clean":"2024-08-31T12:28:19.175690+0000","last_became_active":"2024-08-31T12:15:57.312145+0000","last_became_peered":"2024-08-31T12:15:57.312145+0000","last_unstale":"2024-08-31T12:28:19.175690+0000","last_undegraded":"2024-08-31T12:28:19.175690+0000","last_fullsized":"2024-08-31T12:28:19.175690+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3282,"log_dups_size":0,"ondisk_log_size":3282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:09:31.904512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243277619,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16586,"num_write_kb":304033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":25,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:24:56.207028+0000","last_change":"2024-08-31T12:15:56.281809+0000","last_active":"2024-08-31T12:24:56.207028+0000","last_peered":"2024-08-31T12:24:56.207028+0000","last_clean":"2024-08-31T12:24:56.207028+0000","last_became_active":"2024-08-31T12:15:56.281368+0000","last_became_peered":"2024-08-31T12:15:56.281368+0000","last_unstale":"2024-08-31T12:24:56.207028+0000","last_undegraded":"2024-08-31T12:24:56.207028+0000","last_fullsized":"2024-08-31T12:24:56.207028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:17:33.687196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"23'3289","reported_seq":3311,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.183134+0000","last_change":"2024-08-31T12:15:57.305874+0000","last_active":"2024-08-31T12:28:19.183134+0000","last_peered":"2024-08-31T12:28:19.183134+0000","last_clean":"2024-08-31T12:28:19.183134+0000","last_became_active":"2024-08-31T12:15:57.305507+0000","last_became_peered":"2024-08-31T12:15:57.305507+0000","last_unstale":"2024-08-31T12:28:19.183134+0000","last_undegraded":"2024-08-31T12:28:19.183134+0000","last_fullsized":"2024-08-31T12:28:19.183134+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3289,"log_dups_size":0,"ondisk_log_size":3289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:22:02.569106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218117314,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16432,"num_write_kb":278837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"23'4","reported_seq":28,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:24:58.177745+0000","last_change":"2024-08-31T12:15:56.291851+0000","last_active":"2024-08-31T12:24:58.177745+0000","last_peered":"2024-08-31T12:24:58.177745+0000","last_clean":"2024-08-31T12:24:58.177745+0000","last_became_active":"2024-08-31T12:15:56.291446+0000","last_became_peered":"2024-08-31T12:15:56.291446+0000","last_unstale":"2024-08-31T12:24:58.177745+0000","last_undegraded":"2024-08-31T12:24:58.177745+0000","last_fullsized":"2024-08-31T12:24:58.177745+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:54:29.988079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"23'2737","reported_seq":2759,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.167644+0000","last_change":"2024-08-31T12:15:57.316154+0000","last_active":"2024-08-31T12:28:19.167644+0000","last_peered":"2024-08-31T12:28:19.167644+0000","last_clean":"2024-08-31T12:28:19.167644+0000","last_became_active":"2024-08-31T12:15:57.315830+0000","last_became_peered":"2024-08-31T12:15:57.315830+0000","last_unstale":"2024-08-31T12:28:19.167644+0000","last_undegraded":"2024-08-31T12:28:19.167644+0000","last_fullsized":"2024-08-31T12:28:19.167644+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2737,"log_dups_size":0,"ondisk_log_size":2737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:05:53.039520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209813064,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13700,"num_write_kb":259755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":121,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:50.198157+0000","last_change":"2024-08-31T12:15:47.084912+0000","last_active":"2024-08-31T12:25:50.198157+0000","last_peered":"2024-08-31T12:25:50.198157+0000","last_clean":"2024-08-31T12:25:50.198157+0000","last_became_active":"2024-08-31T12:15:47.083771+0000","last_became_peered":"2024-08-31T12:15:47.083771+0000","last_unstale":"2024-08-31T12:25:50.198157+0000","last_undegraded":"2024-08-31T12:25:50.198157+0000","last_fullsized":"2024-08-31T12:25:50.198157+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:45.213688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:45.213688+0000","last_clean_scrub_stamp":"2024-08-31T12:15:45.213688+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:25:24.827530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"23'4","reported_seq":29,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:26:25.205795+0000","last_change":"2024-08-31T12:15:56.285355+0000","last_active":"2024-08-31T12:26:25.205795+0000","last_peered":"2024-08-31T12:26:25.205795+0000","last_clean":"2024-08-31T12:26:25.205795+0000","last_became_active":"2024-08-31T12:15:56.284470+0000","last_became_peered":"2024-08-31T12:15:56.284470+0000","last_unstale":"2024-08-31T12:26:25.205795+0000","last_undegraded":"2024-08-31T12:26:25.205795+0000","last_fullsized":"2024-08-31T12:26:25.205795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:52:20.055757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"23'2767","reported_seq":2790,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.201878+0000","last_change":"2024-08-31T12:15:57.309039+0000","last_active":"2024-08-31T12:28:19.201878+0000","last_peered":"2024-08-31T12:28:19.201878+0000","last_clean":"2024-08-31T12:28:19.201878+0000","last_became_active":"2024-08-31T12:15:57.308721+0000","last_became_peered":"2024-08-31T12:15:57.308721+0000","last_unstale":"2024-08-31T12:28:19.201878+0000","last_undegraded":"2024-08-31T12:28:19.201878+0000","last_fullsized":"2024-08-31T12:28:19.201878+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2767,"log_dups_size":0,"ondisk_log_size":2767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:43:12.703971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218112028,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13723,"num_write_kb":267984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"23'4","reported_seq":29,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:27:20.870746+0000","last_change":"2024-08-31T12:15:56.293221+0000","last_active":"2024-08-31T12:27:20.870746+0000","last_peered":"2024-08-31T12:27:20.870746+0000","last_clean":"2024-08-31T12:27:20.870746+0000","last_became_active":"2024-08-31T12:15:56.292910+0000","last_became_peered":"2024-08-31T12:15:56.292910+0000","last_unstale":"2024-08-31T12:27:20.870746+0000","last_undegraded":"2024-08-31T12:27:20.870746+0000","last_fullsized":"2024-08-31T12:27:20.870746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:14:33.483002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"23'2757","reported_seq":2780,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.182199+0000","last_change":"2024-08-31T12:15:57.307266+0000","last_active":"2024-08-31T12:28:19.182199+0000","last_peered":"2024-08-31T12:28:19.182199+0000","last_clean":"2024-08-31T12:28:19.182199+0000","last_became_active":"2024-08-31T12:15:57.306529+0000","last_became_peered":"2024-08-31T12:15:57.306529+0000","last_unstale":"2024-08-31T12:28:19.182199+0000","last_undegraded":"2024-08-31T12:28:19.182199+0000","last_fullsized":"2024-08-31T12:28:19.182199+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2757,"log_dups_size":0,"ondisk_log_size":2757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:42:39.731742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201337916,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14174,"num_write_kb":253444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"23'35","reported_seq":90,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:26:23.185903+0000","last_change":"2024-08-31T12:15:56.291822+0000","last_active":"2024-08-31T12:26:23.185903+0000","last_peered":"2024-08-31T12:26:23.185903+0000","last_clean":"2024-08-31T12:26:23.185903+0000","last_became_active":"2024-08-31T12:15:56.291064+0000","last_became_peered":"2024-08-31T12:15:56.291064+0000","last_unstale":"2024-08-31T12:26:23.185903+0000","last_undegraded":"2024-08-31T12:26:23.185903+0000","last_fullsized":"2024-08-31T12:26:23.185903+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:53:08.929860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47459,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":30,"num_read_kb":64,"num_write":34,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"23'2938","reported_seq":2961,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.167628+0000","last_change":"2024-08-31T12:15:57.299212+0000","last_active":"2024-08-31T12:28:19.167628+0000","last_peered":"2024-08-31T12:28:19.167628+0000","last_clean":"2024-08-31T12:28:19.167628+0000","last_became_active":"2024-08-31T12:15:57.298892+0000","last_became_peered":"2024-08-31T12:15:57.298892+0000","last_unstale":"2024-08-31T12:28:19.167628+0000","last_undegraded":"2024-08-31T12:28:19.167628+0000","last_fullsized":"2024-08-31T12:28:19.167628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2938,"log_dups_size":0,"ondisk_log_size":2938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:53:19.523811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197144226,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14947,"num_write_kb":252544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"23'2","reported_seq":27,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:26:25.205779+0000","last_change":"2024-08-31T12:15:56.286720+0000","last_active":"2024-08-31T12:26:25.205779+0000","last_peered":"2024-08-31T12:26:25.205779+0000","last_clean":"2024-08-31T12:26:25.205779+0000","last_became_active":"2024-08-31T12:15:56.285648+0000","last_became_peered":"2024-08-31T12:15:56.285648+0000","last_unstale":"2024-08-31T12:26:25.205779+0000","last_undegraded":"2024-08-31T12:26:25.205779+0000","last_fullsized":"2024-08-31T12:26:25.205779+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:39:31.946792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"23'2762","reported_seq":2785,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.152132+0000","last_change":"2024-08-31T12:15:57.311785+0000","last_active":"2024-08-31T12:28:19.152132+0000","last_peered":"2024-08-31T12:28:19.152132+0000","last_clean":"2024-08-31T12:28:19.152132+0000","last_became_active":"2024-08-31T12:15:57.311469+0000","last_became_peered":"2024-08-31T12:15:57.311469+0000","last_unstale":"2024-08-31T12:28:19.152132+0000","last_undegraded":"2024-08-31T12:28:19.152132+0000","last_fullsized":"2024-08-31T12:28:19.152132+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2762,"log_dups_size":0,"ondisk_log_size":2762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:29:54.122825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197139313,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14081,"num_write_kb":248914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"23'1","reported_seq":26,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:26:21.215756+0000","last_change":"2024-08-31T12:15:56.284302+0000","last_active":"2024-08-31T12:26:21.215756+0000","last_peered":"2024-08-31T12:26:21.215756+0000","last_clean":"2024-08-31T12:26:21.215756+0000","last_became_active":"2024-08-31T12:15:56.283979+0000","last_became_peered":"2024-08-31T12:15:56.283979+0000","last_unstale":"2024-08-31T12:26:21.215756+0000","last_undegraded":"2024-08-31T12:26:21.215756+0000","last_fullsized":"2024-08-31T12:26:21.215756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:51:59.306912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"23'2527","reported_seq":2543,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.473715+0000","last_change":"2024-08-31T12:16:41.195070+0000","last_active":"2024-08-31T12:28:47.473715+0000","last_peered":"2024-08-31T12:28:47.473715+0000","last_clean":"2024-08-31T12:28:47.473715+0000","last_became_active":"2024-08-31T12:16:41.194724+0000","last_became_peered":"2024-08-31T12:16:41.194724+0000","last_unstale":"2024-08-31T12:28:47.473715+0000","last_undegraded":"2024-08-31T12:28:47.473715+0000","last_fullsized":"2024-08-31T12:28:47.473715+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2527,"log_dups_size":0,"ondisk_log_size":2527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:24:46.945850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184558382,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13011,"num_write_kb":232344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"21'1","reported_seq":26,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:26:25.202953+0000","last_change":"2024-08-31T12:15:56.297648+0000","last_active":"2024-08-31T12:26:25.202953+0000","last_peered":"2024-08-31T12:26:25.202953+0000","last_clean":"2024-08-31T12:26:25.202953+0000","last_became_active":"2024-08-31T12:15:56.297306+0000","last_became_peered":"2024-08-31T12:15:56.297306+0000","last_unstale":"2024-08-31T12:26:25.202953+0000","last_undegraded":"2024-08-31T12:26:25.202953+0000","last_fullsized":"2024-08-31T12:26:25.202953+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:29:54.693819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"23'3164","reported_seq":3187,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.097415+0000","last_change":"2024-08-31T12:15:57.297230+0000","last_active":"2024-08-31T12:28:19.097415+0000","last_peered":"2024-08-31T12:28:19.097415+0000","last_clean":"2024-08-31T12:28:19.097415+0000","last_became_active":"2024-08-31T12:15:57.296892+0000","last_became_peered":"2024-08-31T12:15:57.296892+0000","last_unstale":"2024-08-31T12:28:19.097415+0000","last_undegraded":"2024-08-31T12:28:19.097415+0000","last_fullsized":"2024-08-31T12:28:19.097415+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3164,"log_dups_size":0,"ondisk_log_size":3164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:42:36.033586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213956769,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16152,"num_write_kb":273638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"23'427","reported_seq":451,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:23.458871+0000","last_change":"2024-08-31T12:15:56.286504+0000","last_active":"2024-08-31T12:28:23.458871+0000","last_peered":"2024-08-31T12:28:23.458871+0000","last_clean":"2024-08-31T12:28:23.458871+0000","last_became_active":"2024-08-31T12:15:56.286173+0000","last_became_peered":"2024-08-31T12:15:56.286173+0000","last_unstale":"2024-08-31T12:28:23.458871+0000","last_undegraded":"2024-08-31T12:28:23.458871+0000","last_fullsized":"2024-08-31T12:28:23.458871+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":427,"log_dups_size":0,"ondisk_log_size":427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:06:34.313772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":975,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"23'2916","reported_seq":2938,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.229676+0000","last_change":"2024-08-31T12:15:57.308013+0000","last_active":"2024-08-31T12:28:19.229676+0000","last_peered":"2024-08-31T12:28:19.229676+0000","last_clean":"2024-08-31T12:28:19.229676+0000","last_became_active":"2024-08-31T12:15:57.307701+0000","last_became_peered":"2024-08-31T12:15:57.307701+0000","last_unstale":"2024-08-31T12:28:19.229676+0000","last_undegraded":"2024-08-31T12:28:19.229676+0000","last_fullsized":"2024-08-31T12:28:19.229676+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2916,"log_dups_size":0,"ondisk_log_size":2916,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:06:15.161589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205521115,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14225,"num_write_kb":257747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'4","reported_seq":28,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:24:36.187735+0000","last_change":"2024-08-31T12:15:56.294852+0000","last_active":"2024-08-31T12:24:36.187735+0000","last_peered":"2024-08-31T12:24:36.187735+0000","last_clean":"2024-08-31T12:24:36.187735+0000","last_became_active":"2024-08-31T12:15:56.294496+0000","last_became_peered":"2024-08-31T12:15:56.294496+0000","last_unstale":"2024-08-31T12:24:36.187735+0000","last_undegraded":"2024-08-31T12:24:36.187735+0000","last_fullsized":"2024-08-31T12:24:36.187735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:49:17.488528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"23'2537","reported_seq":2559,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.168378+0000","last_change":"2024-08-31T12:15:57.305202+0000","last_active":"2024-08-31T12:28:19.168378+0000","last_peered":"2024-08-31T12:28:19.168378+0000","last_clean":"2024-08-31T12:28:19.168378+0000","last_became_active":"2024-08-31T12:15:57.304602+0000","last_became_peered":"2024-08-31T12:15:57.304602+0000","last_unstale":"2024-08-31T12:28:19.168378+0000","last_undegraded":"2024-08-31T12:28:19.168378+0000","last_fullsized":"2024-08-31T12:28:19.168378+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2537,"log_dups_size":0,"ondisk_log_size":2537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:56:03.280339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176167762,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12536,"num_write_kb":222149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"23'3","reported_seq":27,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:24:40.189661+0000","last_change":"2024-08-31T12:15:56.297542+0000","last_active":"2024-08-31T12:24:40.189661+0000","last_peered":"2024-08-31T12:24:40.189661+0000","last_clean":"2024-08-31T12:24:40.189661+0000","last_became_active":"2024-08-31T12:15:56.297168+0000","last_became_peered":"2024-08-31T12:15:56.297168+0000","last_unstale":"2024-08-31T12:24:40.189661+0000","last_undegraded":"2024-08-31T12:24:40.189661+0000","last_fullsized":"2024-08-31T12:24:40.189661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:56:41.924772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"23'2320","reported_seq":2342,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.187968+0000","last_change":"2024-08-31T12:15:57.298845+0000","last_active":"2024-08-31T12:28:19.187968+0000","last_peered":"2024-08-31T12:28:19.187968+0000","last_clean":"2024-08-31T12:28:19.187968+0000","last_became_active":"2024-08-31T12:15:57.298520+0000","last_became_peered":"2024-08-31T12:15:57.298520+0000","last_unstale":"2024-08-31T12:28:19.187968+0000","last_undegraded":"2024-08-31T12:28:19.187968+0000","last_fullsized":"2024-08-31T12:28:19.187968+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2320,"log_dups_size":0,"ondisk_log_size":2320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:06:35.318873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197144309,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11287,"num_write_kb":237691,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"21'3","reported_seq":27,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:24:36.205095+0000","last_change":"2024-08-31T12:15:56.290475+0000","last_active":"2024-08-31T12:24:36.205095+0000","last_peered":"2024-08-31T12:24:36.205095+0000","last_clean":"2024-08-31T12:24:36.205095+0000","last_became_active":"2024-08-31T12:15:56.289960+0000","last_became_peered":"2024-08-31T12:15:56.289960+0000","last_unstale":"2024-08-31T12:24:36.205095+0000","last_undegraded":"2024-08-31T12:24:36.205095+0000","last_fullsized":"2024-08-31T12:24:36.205095+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:29:53.117724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"23'2970","reported_seq":2992,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.225999+0000","last_change":"2024-08-31T12:15:57.301460+0000","last_active":"2024-08-31T12:28:19.225999+0000","last_peered":"2024-08-31T12:28:19.225999+0000","last_clean":"2024-08-31T12:28:19.225999+0000","last_became_active":"2024-08-31T12:15:57.301157+0000","last_became_peered":"2024-08-31T12:15:57.301157+0000","last_unstale":"2024-08-31T12:28:19.225999+0000","last_undegraded":"2024-08-31T12:28:19.225999+0000","last_fullsized":"2024-08-31T12:28:19.225999+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2970,"log_dups_size":0,"ondisk_log_size":2970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:22:52.173578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209728881,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15223,"num_write_kb":265883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"21'1","reported_seq":25,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:24:36.205125+0000","last_change":"2024-08-31T12:15:56.289720+0000","last_active":"2024-08-31T12:24:36.205125+0000","last_peered":"2024-08-31T12:24:36.205125+0000","last_clean":"2024-08-31T12:24:36.205125+0000","last_became_active":"2024-08-31T12:15:56.289332+0000","last_became_peered":"2024-08-31T12:15:56.289332+0000","last_unstale":"2024-08-31T12:24:36.205125+0000","last_undegraded":"2024-08-31T12:24:36.205125+0000","last_fullsized":"2024-08-31T12:24:36.205125+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:15:51.561831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"23'2415","reported_seq":2437,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.230233+0000","last_change":"2024-08-31T12:15:57.313802+0000","last_active":"2024-08-31T12:28:19.230233+0000","last_peered":"2024-08-31T12:28:19.230233+0000","last_clean":"2024-08-31T12:28:19.230233+0000","last_became_active":"2024-08-31T12:15:57.313428+0000","last_became_peered":"2024-08-31T12:15:57.313428+0000","last_unstale":"2024-08-31T12:28:19.230233+0000","last_undegraded":"2024-08-31T12:28:19.230233+0000","last_fullsized":"2024-08-31T12:28:19.230233+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2415,"log_dups_size":0,"ondisk_log_size":2415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:22:13.435300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176173175,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11447,"num_write_kb":217885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"23'1068","reported_seq":1092,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:27:20.763645+0000","last_change":"2024-08-31T12:15:56.296105+0000","last_active":"2024-08-31T12:27:20.763645+0000","last_peered":"2024-08-31T12:27:20.763645+0000","last_clean":"2024-08-31T12:27:20.763645+0000","last_became_active":"2024-08-31T12:15:56.295524+0000","last_became_peered":"2024-08-31T12:15:56.295524+0000","last_unstale":"2024-08-31T12:27:20.763645+0000","last_undegraded":"2024-08-31T12:27:20.763645+0000","last_fullsized":"2024-08-31T12:27:20.763645+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":1068,"log_dups_size":0,"ondisk_log_size":1068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:41:55.482044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1064,"num_write_kb":4401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"23'2502","reported_seq":2524,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.061863+0000","last_change":"2024-08-31T12:15:57.300155+0000","last_active":"2024-08-31T12:28:19.061863+0000","last_peered":"2024-08-31T12:28:19.061863+0000","last_clean":"2024-08-31T12:28:19.061863+0000","last_became_active":"2024-08-31T12:15:57.299436+0000","last_became_peered":"2024-08-31T12:15:57.299436+0000","last_unstale":"2024-08-31T12:28:19.061863+0000","last_undegraded":"2024-08-31T12:28:19.061863+0000","last_fullsized":"2024-08-31T12:28:19.061863+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2502,"log_dups_size":0,"ondisk_log_size":2502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:37:45.891394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188744397,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12696,"num_write_kb":235198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"23'3358","reported_seq":3380,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.189014+0000","last_change":"2024-08-31T12:15:57.300529+0000","last_active":"2024-08-31T12:28:19.189014+0000","last_peered":"2024-08-31T12:28:19.189014+0000","last_clean":"2024-08-31T12:28:19.189014+0000","last_became_active":"2024-08-31T12:15:57.300159+0000","last_became_peered":"2024-08-31T12:15:57.300159+0000","last_unstale":"2024-08-31T12:28:19.189014+0000","last_undegraded":"2024-08-31T12:28:19.189014+0000","last_fullsized":"2024-08-31T12:28:19.189014+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3358,"log_dups_size":0,"ondisk_log_size":3358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:35:53.831742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251730727,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16759,"num_write_kb":312885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":26,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:26:00.199883+0000","last_change":"2024-08-31T12:15:56.297791+0000","last_active":"2024-08-31T12:26:00.199883+0000","last_peered":"2024-08-31T12:26:00.199883+0000","last_clean":"2024-08-31T12:26:00.199883+0000","last_became_active":"2024-08-31T12:15:56.296976+0000","last_became_peered":"2024-08-31T12:15:56.296976+0000","last_unstale":"2024-08-31T12:26:00.199883+0000","last_undegraded":"2024-08-31T12:26:00.199883+0000","last_fullsized":"2024-08-31T12:26:00.199883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:07:54.963050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"23'2167","reported_seq":2189,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.184056+0000","last_change":"2024-08-31T12:15:57.301167+0000","last_active":"2024-08-31T12:28:19.184056+0000","last_peered":"2024-08-31T12:28:19.184056+0000","last_clean":"2024-08-31T12:28:19.184056+0000","last_became_active":"2024-08-31T12:15:57.300646+0000","last_became_peered":"2024-08-31T12:15:57.300646+0000","last_unstale":"2024-08-31T12:28:19.184056+0000","last_undegraded":"2024-08-31T12:28:19.184056+0000","last_fullsized":"2024-08-31T12:28:19.184056+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2167,"log_dups_size":0,"ondisk_log_size":2167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:05:53.039520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150996671,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10769,"num_write_kb":190633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"23'9","reported_seq":33,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:26:00.199906+0000","last_change":"2024-08-31T12:15:56.295139+0000","last_active":"2024-08-31T12:26:00.199906+0000","last_peered":"2024-08-31T12:26:00.199906+0000","last_clean":"2024-08-31T12:26:00.199906+0000","last_became_active":"2024-08-31T12:15:56.294560+0000","last_became_peered":"2024-08-31T12:15:56.294560+0000","last_unstale":"2024-08-31T12:26:00.199906+0000","last_undegraded":"2024-08-31T12:26:00.199906+0000","last_fullsized":"2024-08-31T12:26:00.199906+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:52:16.999507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"23'3348","reported_seq":3371,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.187804+0000","last_change":"2024-08-31T12:15:57.311074+0000","last_active":"2024-08-31T12:28:19.187804+0000","last_peered":"2024-08-31T12:28:19.187804+0000","last_clean":"2024-08-31T12:28:19.187804+0000","last_became_active":"2024-08-31T12:15:57.310262+0000","last_became_peered":"2024-08-31T12:15:57.310262+0000","last_unstale":"2024-08-31T12:28:19.187804+0000","last_undegraded":"2024-08-31T12:28:19.187804+0000","last_fullsized":"2024-08-31T12:28:19.187804+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3348,"log_dups_size":0,"ondisk_log_size":3348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:47:51.267409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251719163,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16721,"num_write_kb":312844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"23'3","reported_seq":28,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:56.212749+0000","last_change":"2024-08-31T12:15:56.285428+0000","last_active":"2024-08-31T12:25:56.212749+0000","last_peered":"2024-08-31T12:25:56.212749+0000","last_clean":"2024-08-31T12:25:56.212749+0000","last_became_active":"2024-08-31T12:15:56.284884+0000","last_became_peered":"2024-08-31T12:15:56.284884+0000","last_unstale":"2024-08-31T12:25:56.212749+0000","last_undegraded":"2024-08-31T12:25:56.212749+0000","last_fullsized":"2024-08-31T12:25:56.212749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:49:25.181432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"23'3167","reported_seq":3190,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.175794+0000","last_change":"2024-08-31T12:15:57.305772+0000","last_active":"2024-08-31T12:28:19.175794+0000","last_peered":"2024-08-31T12:28:19.175794+0000","last_clean":"2024-08-31T12:28:19.175794+0000","last_became_active":"2024-08-31T12:15:57.305409+0000","last_became_peered":"2024-08-31T12:15:57.305409+0000","last_unstale":"2024-08-31T12:28:19.175794+0000","last_undegraded":"2024-08-31T12:28:19.175794+0000","last_fullsized":"2024-08-31T12:28:19.175794+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3167,"log_dups_size":0,"ondisk_log_size":3167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:53:19.523811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222298112,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15945,"num_write_kb":281070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"23'229","reported_seq":254,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:37.337439+0000","last_change":"2024-08-31T12:15:56.294197+0000","last_active":"2024-08-31T12:28:37.337439+0000","last_peered":"2024-08-31T12:28:37.337439+0000","last_clean":"2024-08-31T12:28:37.337439+0000","last_became_active":"2024-08-31T12:15:56.293864+0000","last_became_peered":"2024-08-31T12:15:56.293864+0000","last_unstale":"2024-08-31T12:28:37.337439+0000","last_undegraded":"2024-08-31T12:28:37.337439+0000","last_fullsized":"2024-08-31T12:28:37.337439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":229,"log_dups_size":0,"ondisk_log_size":229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:43:17.443433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1325973,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":228,"num_write_kb":1426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":13424230175,"num_objects":3391,"num_object_clones":0,"num_object_copies":6782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":918154,"num_write_kb":16787942,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26849361920,"data_stored":26848441814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":183697,"ondisk_log_size":183697,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":13960395,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":30,"num_read_kb":64,"num_write":3307,"num_write_kb":14790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28073984,"data_stored":27920790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2759,"ondisk_log_size":2759,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509600,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2618576,"kb_used_data":2384832,"kb_used_omap":1,"kb_used_meta":233726,"kb_avail":91753264,"statfs":{"total":96636764160,"available":93955342336,"internally_reserved":0,"allocated":2442067968,"data_stored":2441803901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":239335878},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509599,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4206900,"kb_used_data":3851044,"kb_used_omap":1,"kb_used_meta":355838,"kb_avail":90164940,"statfs":{"total":96636764160,"available":92328898560,"internally_reserved":0,"allocated":3943469056,"data_stored":3943171530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":364378567},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509600,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2544520,"kb_used_data":2306616,"kb_used_omap":1,"kb_used_meta":237886,"kb_avail":91827320,"statfs":{"total":96636764160,"available":94031175680,"internally_reserved":0,"allocated":2361974784,"data_stored":2361727195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":243595717},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509600,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4141744,"kb_used_data":3795616,"kb_used_omap":1,"kb_used_meta":346110,"kb_avail":90230096,"statfs":{"total":96636764160,"available":92395618304,"internally_reserved":0,"allocated":3886710784,"data_stored":3886395634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354417093},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542303,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4422856,"kb_used_data":4066936,"kb_used_omap":1,"kb_used_meta":355902,"kb_avail":89948984,"statfs":{"total":96636764160,"available":92107759616,"internally_reserved":0,"allocated":4164542464,"data_stored":4164215175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":364444106},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542305,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4162248,"kb_used_data":3793912,"kb_used_omap":1,"kb_used_meta":368318,"kb_avail":90209592,"statfs":{"total":96636764160,"available":92374622208,"internally_reserved":0,"allocated":3884965888,"data_stored":3884692848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":377158085},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542303,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4136736,"kb_used_data":3839056,"kb_used_omap":1,"kb_used_meta":297662,"kb_avail":90235104,"statfs":{"total":96636764160,"available":92400746496,"internally_reserved":0,"allocated":3931193344,"data_stored":3930920211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":304806342},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575008,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2442052,"kb_used_data":2212468,"kb_used_omap":1,"kb_used_meta":229566,"kb_avail":91929788,"statfs":{"total":96636764160,"available":94136102912,"internally_reserved":0,"allocated":2265567232,"data_stored":2265299110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":235076037},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1343488,"data_stored":1326051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":48187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8458240,"data_stored":8437802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4210688,"data_stored":4195018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4210688,"data_stored":4195654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":9736192,"data_stored":9715756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3884539904,"data_stored":3884393932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2361716736,"data_stored":2361640968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2265264128,"data_stored":2265165640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3934777344,"data_stored":3934648445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3926749184,"data_stored":3926639910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3884711936,"data_stored":3884606187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2437623808,"data_stored":2437522964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4153978880,"data_stored":4153823768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-31T12:28:48.937 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-08-31T12:28:48.938 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-31T12:28:49.087 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.085+0000 7fe874f58640 1 -- 172.21.15.19:0/3622517655 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe8700c5370 msgr2=0x7fe87006ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:49.088 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.085+0000 7fe874f58640 1 --2- 172.21.15.19:0/3622517655 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe8700c5370 0x7fe87006ddd0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe860009830 tx=0x7fe86002a5a0 comp rx=0 tx=0).stop 2024-08-31T12:28:49.088 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.085+0000 7fe874f58640 1 -- 172.21.15.19:0/3622517655 shutdown_connections 2024-08-31T12:28:49.088 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.085+0000 7fe874f58640 1 --2- 172.21.15.19:0/3622517655 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe8700cc400 0x7fe8700cc810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:49.088 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.085+0000 7fe874f58640 1 --2- 172.21.15.19:0/3622517655 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe8700c5370 0x7fe87006ddd0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:49.088 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.085+0000 7fe874f58640 1 -- 172.21.15.19:0/3622517655 >> 172.21.15.19:0/3622517655 conn(0x7fe8700ca670 msgr2=0x7fe8700cd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:49.088 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.085+0000 7fe874f58640 1 -- 172.21.15.19:0/3622517655 shutdown_connections 2024-08-31T12:28:49.089 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.085+0000 7fe874f58640 1 -- 172.21.15.19:0/3622517655 wait complete. 2024-08-31T12:28:49.089 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.087+0000 7fe874f58640 1 Processor -- start 2024-08-31T12:28:49.089 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.087+0000 7fe874f58640 1 -- start start 2024-08-31T12:28:49.092 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.088+0000 7fe874f58640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8700c5370 0x7fe870140210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:49.092 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.088+0000 7fe874f58640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe8700cb2b0 0x7fe8701542b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:49.092 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.088+0000 7fe874f58640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe8700cc400 0x7fe870153450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:49.092 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.088+0000 7fe874f58640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fe8700c72e0 con 0x7fe8700cc400 2024-08-31T12:28:49.092 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.088+0000 7fe874f58640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe8700ccd50 con 0x7fe8700c5370 2024-08-31T12:28:49.093 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.088+0000 7fe874f58640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fe8700cb8c0 con 0x7fe8700cb2b0 2024-08-31T12:28:49.093 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.088+0000 7fe86ed9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe8700cc400 0x7fe870153450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:49.093 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.089+0000 7fe86ed9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe8700cc400 0x7fe870153450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:43858/0 (socket says 172.21.15.19:43858) 2024-08-31T12:28:49.093 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.089+0000 7fe86ed9d640 1 -- 172.21.15.19:0/2855808121 learned_addr learned my addr 172.21.15.19:0/2855808121 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:28:49.093 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.089+0000 7fe86dd9b640 1 --2- 172.21.15.19:0/2855808121 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe8700cb2b0 0x7fe8701542b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:49.093 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.089+0000 7fe86e59c640 1 --2- 172.21.15.19:0/2855808121 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8700c5370 0x7fe870140210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:49.093 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.089+0000 7fe86ed9d640 1 -- 172.21.15.19:0/2855808121 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe8700cb2b0 msgr2=0x7fe8701542b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:49.094 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.089+0000 7fe86ed9d640 1 --2- 172.21.15.19:0/2855808121 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe8700cb2b0 0x7fe8701542b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:49.094 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.089+0000 7fe86ed9d640 1 -- 172.21.15.19:0/2855808121 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8700c5370 msgr2=0x7fe870140210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:49.094 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.089+0000 7fe86ed9d640 1 --2- 172.21.15.19:0/2855808121 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8700c5370 0x7fe870140210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:49.094 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.089+0000 7fe86ed9d640 1 -- 172.21.15.19:0/2855808121 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe870147c10 con 0x7fe8700cc400 2024-08-31T12:28:49.094 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.089+0000 7fe86ed9d640 1 --2- 172.21.15.19:0/2855808121 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe8700cc400 0x7fe870153450 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe86800cad0 tx=0x7fe86800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:49.094 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.089+0000 7fe85effd640 1 -- 172.21.15.19:0/2855808121 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86800ae20 con 0x7fe8700cc400 2024-08-31T12:28:49.094 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.089+0000 7fe85effd640 1 -- 172.21.15.19:0/2855808121 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe86800f540 con 0x7fe8700cc400 2024-08-31T12:28:49.095 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.089+0000 7fe874f58640 1 -- 172.21.15.19:0/2855808121 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe870152950 con 0x7fe8700cc400 2024-08-31T12:28:49.095 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.089+0000 7fe85effd640 1 -- 172.21.15.19:0/2855808121 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86800f990 con 0x7fe8700cc400 2024-08-31T12:28:49.095 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.089+0000 7fe874f58640 1 -- 172.21.15.19:0/2855808121 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8701f2bf0 con 0x7fe8700cc400 2024-08-31T12:28:49.096 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.090+0000 7fe874f58640 1 -- 172.21.15.19:0/2855808121 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8700c63f0 con 0x7fe8700cc400 2024-08-31T12:28:49.096 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.091+0000 7fe85effd640 1 -- 172.21.15.19:0/2855808121 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fe86800fb30 con 0x7fe8700cc400 2024-08-31T12:28:49.096 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.091+0000 7fe85effd640 1 --2- 172.21.15.19:0/2855808121 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe850077a50 0x7fe850079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:49.096 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.092+0000 7fe85effd640 1 -- 172.21.15.19:0/2855808121 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fe86809a090 con 0x7fe8700cc400 2024-08-31T12:28:49.096 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.092+0000 7fe86e59c640 1 --2- 172.21.15.19:0/2855808121 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe850077a50 0x7fe850079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:49.096 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.092+0000 7fe86e59c640 1 --2- 172.21.15.19:0/2855808121 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe850077a50 0x7fe850079f60 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe860034000 tx=0x7fe860004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:49.098 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.096+0000 7fe85effd640 1 -- 172.21.15.19:0/2855808121 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fe8700c63f0 con 0x7fe8700cc400 2024-08-31T12:28:49.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.279+0000 7fe874f58640 1 -- 172.21.15.19:0/2855808121 --> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe8701f2e50 con 0x7fe850077a50 2024-08-31T12:28:49.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.284+0000 7fe85effd640 1 -- 172.21.15.19:0/2855808121 <== mgr.4100 v2:172.21.15.43:6800/2824336655 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+254513 (secure 0 0 0) 0x7fe8701f2e50 con 0x7fe850077a50 2024-08-31T12:28:49.288 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:28:49.290 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-08-31T12:28:49.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.287+0000 7fe874f58640 1 -- 172.21.15.19:0/2855808121 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe850077a50 msgr2=0x7fe850079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:49.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.287+0000 7fe874f58640 1 --2- 172.21.15.19:0/2855808121 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe850077a50 0x7fe850079f60 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe860034000 tx=0x7fe860004730 comp rx=0 tx=0).stop 2024-08-31T12:28:49.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.287+0000 7fe874f58640 1 -- 172.21.15.19:0/2855808121 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe8700cc400 msgr2=0x7fe870153450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:49.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.287+0000 7fe874f58640 1 --2- 172.21.15.19:0/2855808121 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe8700cc400 0x7fe870153450 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe86800cad0 tx=0x7fe86800cf90 comp rx=0 tx=0).stop 2024-08-31T12:28:49.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.287+0000 7fe874f58640 1 -- 172.21.15.19:0/2855808121 shutdown_connections 2024-08-31T12:28:49.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.287+0000 7fe874f58640 1 --2- 172.21.15.19:0/2855808121 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe850077a50 0x7fe850079f60 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:49.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.287+0000 7fe874f58640 1 --2- 172.21.15.19:0/2855808121 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe8700cc400 0x7fe870153450 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:49.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.287+0000 7fe874f58640 1 --2- 172.21.15.19:0/2855808121 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe8700cb2b0 0x7fe8701542b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:49.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.287+0000 7fe874f58640 1 --2- 172.21.15.19:0/2855808121 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8700c5370 0x7fe870140210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:49.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.287+0000 7fe874f58640 1 -- 172.21.15.19:0/2855808121 >> 172.21.15.19:0/2855808121 conn(0x7fe8700ca670 msgr2=0x7fe8701443a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:49.293 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.287+0000 7fe874f58640 1 -- 172.21.15.19:0/2855808121 shutdown_connections 2024-08-31T12:28:49.293 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.287+0000 7fe874f58640 1 -- 172.21.15.19:0/2855808121 wait complete. 2024-08-31T12:28:49.300 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":409,"stamp":"2024-08-31T12:28:48.707058+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":13438780938,"num_objects":3466,"num_object_clones":0,"num_object_copies":6932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3466,"num_whiteouts":0,"num_read":96,"num_read_kb":119,"num_write":921562,"num_write_kb":16804360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":186528,"ondisk_log_size":186528,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":28675632,"kb_used_data":26250480,"kb_used_omap":12,"kb_used_meta":2425011,"kb_avail":726299088,"statfs":{"total":773094113280,"available":743730266112,"internally_reserved":0,"allocated":26880491520,"data_stored":26878225604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":2483211825},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5476,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003706"},"pg_stats":[{"pgid":"3.3f","version":"23'2724","reported_seq":2747,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.201520+0000","last_change":"2024-08-31T12:15:57.306560+0000","last_active":"2024-08-31T12:28:19.201520+0000","last_peered":"2024-08-31T12:28:19.201520+0000","last_clean":"2024-08-31T12:28:19.201520+0000","last_became_active":"2024-08-31T12:15:57.306148+0000","last_became_peered":"2024-08-31T12:15:57.306148+0000","last_unstale":"2024-08-31T12:28:19.201520+0000","last_undegraded":"2024-08-31T12:28:19.201520+0000","last_fullsized":"2024-08-31T12:28:19.201520+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2724,"log_dups_size":0,"ondisk_log_size":2724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:35:53.831742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180377297,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13788,"num_write_kb":231441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"23'3448","reported_seq":3471,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.169946+0000","last_change":"2024-08-31T12:15:57.307314+0000","last_active":"2024-08-31T12:28:19.169946+0000","last_peered":"2024-08-31T12:28:19.169946+0000","last_clean":"2024-08-31T12:28:19.169946+0000","last_became_active":"2024-08-31T12:15:57.306940+0000","last_became_peered":"2024-08-31T12:15:57.306940+0000","last_unstale":"2024-08-31T12:28:19.169946+0000","last_undegraded":"2024-08-31T12:28:19.169946+0000","last_fullsized":"2024-08-31T12:28:19.169946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3448,"log_dups_size":0,"ondisk_log_size":3448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:38:30.321214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264252554,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17611,"num_write_kb":328696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"23'3134","reported_seq":3157,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.114777+0000","last_change":"2024-08-31T12:15:57.309372+0000","last_active":"2024-08-31T12:28:19.114777+0000","last_peered":"2024-08-31T12:28:19.114777+0000","last_clean":"2024-08-31T12:28:19.114777+0000","last_became_active":"2024-08-31T12:15:57.305795+0000","last_became_peered":"2024-08-31T12:15:57.305795+0000","last_unstale":"2024-08-31T12:28:19.114777+0000","last_undegraded":"2024-08-31T12:28:19.114777+0000","last_fullsized":"2024-08-31T12:28:19.114777+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3134,"log_dups_size":0,"ondisk_log_size":3134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:37:45.891394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213942725,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15777,"num_write_kb":272211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"23'2785","reported_seq":2808,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.158289+0000","last_change":"2024-08-31T12:15:57.307339+0000","last_active":"2024-08-31T12:28:19.158289+0000","last_peered":"2024-08-31T12:28:19.158289+0000","last_clean":"2024-08-31T12:28:19.158289+0000","last_became_active":"2024-08-31T12:15:57.306957+0000","last_became_peered":"2024-08-31T12:15:57.306957+0000","last_unstale":"2024-08-31T12:28:19.158289+0000","last_undegraded":"2024-08-31T12:28:19.158289+0000","last_fullsized":"2024-08-31T12:28:19.158289+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2785,"log_dups_size":0,"ondisk_log_size":2785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:43:33.705099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14240,"num_write_kb":241389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"23'3277","reported_seq":3299,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.203476+0000","last_change":"2024-08-31T12:15:57.317492+0000","last_active":"2024-08-31T12:28:19.203476+0000","last_peered":"2024-08-31T12:28:19.203476+0000","last_clean":"2024-08-31T12:28:19.203476+0000","last_became_active":"2024-08-31T12:15:57.317079+0000","last_became_peered":"2024-08-31T12:15:57.317079+0000","last_unstale":"2024-08-31T12:28:19.203476+0000","last_undegraded":"2024-08-31T12:28:19.203476+0000","last_fullsized":"2024-08-31T12:28:19.203476+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3277,"log_dups_size":0,"ondisk_log_size":3277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:44:40.712699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247474530,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15722,"num_write_kb":304743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"23'2512","reported_seq":2534,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.179506+0000","last_change":"2024-08-31T12:15:57.307505+0000","last_active":"2024-08-31T12:28:19.179506+0000","last_peered":"2024-08-31T12:28:19.179506+0000","last_clean":"2024-08-31T12:28:19.179506+0000","last_became_active":"2024-08-31T12:15:57.307010+0000","last_became_peered":"2024-08-31T12:15:57.307010+0000","last_unstale":"2024-08-31T12:28:19.179506+0000","last_undegraded":"2024-08-31T12:28:19.179506+0000","last_fullsized":"2024-08-31T12:28:19.179506+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2512,"log_dups_size":0,"ondisk_log_size":2512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:39:31.100565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180356168,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12564,"num_write_kb":226449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"23'2626","reported_seq":2648,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.143842+0000","last_change":"2024-08-31T12:15:57.316892+0000","last_active":"2024-08-31T12:28:19.143842+0000","last_peered":"2024-08-31T12:28:19.143842+0000","last_clean":"2024-08-31T12:28:19.143842+0000","last_became_active":"2024-08-31T12:15:57.316504+0000","last_became_peered":"2024-08-31T12:15:57.316504+0000","last_unstale":"2024-08-31T12:28:19.143842+0000","last_undegraded":"2024-08-31T12:28:19.143842+0000","last_fullsized":"2024-08-31T12:28:19.143842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2626,"log_dups_size":0,"ondisk_log_size":2626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:05:48.614918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201398355,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13306,"num_write_kb":250051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"23'3158","reported_seq":3173,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.134179+0000","last_change":"2024-08-31T12:16:40.694493+0000","last_active":"2024-08-31T12:28:19.134179+0000","last_peered":"2024-08-31T12:28:19.134179+0000","last_clean":"2024-08-31T12:28:19.134179+0000","last_became_active":"2024-08-31T12:16:40.694165+0000","last_became_peered":"2024-08-31T12:16:40.694165+0000","last_unstale":"2024-08-31T12:28:19.134179+0000","last_undegraded":"2024-08-31T12:28:19.134179+0000","last_fullsized":"2024-08-31T12:28:19.134179+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3158,"log_dups_size":0,"ondisk_log_size":3158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:55:57.961306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226497620,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15619,"num_write_kb":283880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.37","version":"23'3491","reported_seq":3513,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.201405+0000","last_change":"2024-08-31T12:15:57.316723+0000","last_active":"2024-08-31T12:28:19.201405+0000","last_peered":"2024-08-31T12:28:19.201405+0000","last_clean":"2024-08-31T12:28:19.201405+0000","last_became_active":"2024-08-31T12:15:57.316354+0000","last_became_peered":"2024-08-31T12:15:57.316354+0000","last_unstale":"2024-08-31T12:28:19.201405+0000","last_undegraded":"2024-08-31T12:28:19.201405+0000","last_fullsized":"2024-08-31T12:28:19.201405+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3491,"log_dups_size":0,"ondisk_log_size":3491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:31:19.860950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251662082,"num_objects":61,"num_object_clones":0,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18100,"num_write_kb":318223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"23'2954","reported_seq":2976,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.206584+0000","last_change":"2024-08-31T12:15:57.311161+0000","last_active":"2024-08-31T12:28:19.206584+0000","last_peered":"2024-08-31T12:28:19.206584+0000","last_clean":"2024-08-31T12:28:19.206584+0000","last_became_active":"2024-08-31T12:15:57.310807+0000","last_became_peered":"2024-08-31T12:15:57.310807+0000","last_unstale":"2024-08-31T12:28:19.206584+0000","last_undegraded":"2024-08-31T12:28:19.206584+0000","last_fullsized":"2024-08-31T12:28:19.206584+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2954,"log_dups_size":0,"ondisk_log_size":2954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:56:03.280339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234894029,"num_objects":59,"num_object_clones":0,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15073,"num_write_kb":289757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"23'2880","reported_seq":2902,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.165528+0000","last_change":"2024-08-31T12:15:57.300930+0000","last_active":"2024-08-31T12:28:19.165528+0000","last_peered":"2024-08-31T12:28:19.165528+0000","last_clean":"2024-08-31T12:28:19.165528+0000","last_became_active":"2024-08-31T12:15:57.300610+0000","last_became_peered":"2024-08-31T12:15:57.300610+0000","last_unstale":"2024-08-31T12:28:19.165528+0000","last_undegraded":"2024-08-31T12:28:19.165528+0000","last_fullsized":"2024-08-31T12:28:19.165528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2880,"log_dups_size":0,"ondisk_log_size":2880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:42:39.731742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201326592,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14139,"num_write_kb":253227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"23'2246","reported_seq":2268,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.123192+0000","last_change":"2024-08-31T12:15:57.303151+0000","last_active":"2024-08-31T12:28:19.123192+0000","last_peered":"2024-08-31T12:28:19.123192+0000","last_clean":"2024-08-31T12:28:19.123192+0000","last_became_active":"2024-08-31T12:15:57.302622+0000","last_became_peered":"2024-08-31T12:15:57.302622+0000","last_unstale":"2024-08-31T12:28:19.123192+0000","last_undegraded":"2024-08-31T12:28:19.123192+0000","last_fullsized":"2024-08-31T12:28:19.123192+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2246,"log_dups_size":0,"ondisk_log_size":2246,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:49:18.493629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176192645,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11403,"num_write_kb":217791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"23'2948","reported_seq":2970,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.215841+0000","last_change":"2024-08-31T12:15:57.305676+0000","last_active":"2024-08-31T12:28:19.215841+0000","last_peered":"2024-08-31T12:28:19.215841+0000","last_clean":"2024-08-31T12:28:19.215841+0000","last_became_active":"2024-08-31T12:15:57.305321+0000","last_became_peered":"2024-08-31T12:15:57.305321+0000","last_unstale":"2024-08-31T12:28:19.215841+0000","last_undegraded":"2024-08-31T12:28:19.215841+0000","last_fullsized":"2024-08-31T12:28:19.215841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2948,"log_dups_size":0,"ondisk_log_size":2948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:42:39.731742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188840677,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15214,"num_write_kb":245235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"23'3122","reported_seq":3144,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.194400+0000","last_change":"2024-08-31T12:15:57.309671+0000","last_active":"2024-08-31T12:28:19.194400+0000","last_peered":"2024-08-31T12:28:19.194400+0000","last_clean":"2024-08-31T12:28:19.194400+0000","last_became_active":"2024-08-31T12:15:57.307013+0000","last_became_peered":"2024-08-31T12:15:57.307013+0000","last_unstale":"2024-08-31T12:28:19.194400+0000","last_undegraded":"2024-08-31T12:28:19.194400+0000","last_fullsized":"2024-08-31T12:28:19.194400+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3122,"log_dups_size":0,"ondisk_log_size":3122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:18:55.451164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213986201,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15743,"num_write_kb":271989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"23'2952","reported_seq":2974,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.181372+0000","last_change":"2024-08-31T12:15:57.298150+0000","last_active":"2024-08-31T12:28:19.181372+0000","last_peered":"2024-08-31T12:28:19.181372+0000","last_clean":"2024-08-31T12:28:19.181372+0000","last_became_active":"2024-08-31T12:15:57.297182+0000","last_became_peered":"2024-08-31T12:15:57.297182+0000","last_unstale":"2024-08-31T12:28:19.181372+0000","last_undegraded":"2024-08-31T12:28:19.181372+0000","last_fullsized":"2024-08-31T12:28:19.181372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2952,"log_dups_size":0,"ondisk_log_size":2952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:16:42.625396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213913551,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14445,"num_write_kb":266758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"23'2546","reported_seq":2562,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.462905+0000","last_change":"2024-08-31T12:16:41.075080+0000","last_active":"2024-08-31T12:28:47.462905+0000","last_peered":"2024-08-31T12:28:47.462905+0000","last_clean":"2024-08-31T12:28:47.462905+0000","last_became_active":"2024-08-31T12:16:41.074773+0000","last_became_peered":"2024-08-31T12:16:41.074773+0000","last_unstale":"2024-08-31T12:28:47.462905+0000","last_undegraded":"2024-08-31T12:28:47.462905+0000","last_fullsized":"2024-08-31T12:28:47.462905+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2546,"log_dups_size":0,"ondisk_log_size":2546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:10:45.435137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184599153,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12917,"num_write_kb":231981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"23'3715","reported_seq":3737,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.196629+0000","last_change":"2024-08-31T12:15:57.308899+0000","last_active":"2024-08-31T12:28:19.196629+0000","last_peered":"2024-08-31T12:28:19.196629+0000","last_clean":"2024-08-31T12:28:19.196629+0000","last_became_active":"2024-08-31T12:15:57.308388+0000","last_became_peered":"2024-08-31T12:15:57.308388+0000","last_unstale":"2024-08-31T12:28:19.196629+0000","last_undegraded":"2024-08-31T12:28:19.196629+0000","last_fullsized":"2024-08-31T12:28:19.196629+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3715,"log_dups_size":0,"ondisk_log_size":3715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:54:03.871840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251747648,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18708,"num_write_kb":320814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"23'2148","reported_seq":2170,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.446094+0000","last_change":"2024-08-31T12:15:57.300201+0000","last_active":"2024-08-31T12:28:47.446094+0000","last_peered":"2024-08-31T12:28:47.446094+0000","last_clean":"2024-08-31T12:28:47.446094+0000","last_became_active":"2024-08-31T12:15:57.299826+0000","last_became_peered":"2024-08-31T12:15:57.299826+0000","last_unstale":"2024-08-31T12:28:47.446094+0000","last_undegraded":"2024-08-31T12:28:47.446094+0000","last_fullsized":"2024-08-31T12:28:47.446094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2148,"log_dups_size":0,"ondisk_log_size":2148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:14:34.488102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167791563,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10514,"num_write_kb":205964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"23'2668","reported_seq":2690,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.054487+0000","last_change":"2024-08-31T12:15:57.317606+0000","last_active":"2024-08-31T12:28:19.054487+0000","last_peered":"2024-08-31T12:28:19.054487+0000","last_clean":"2024-08-31T12:28:19.054487+0000","last_became_active":"2024-08-31T12:15:57.317185+0000","last_became_peered":"2024-08-31T12:15:57.317185+0000","last_unstale":"2024-08-31T12:28:19.054487+0000","last_undegraded":"2024-08-31T12:28:19.054487+0000","last_fullsized":"2024-08-31T12:28:19.054487+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2668,"log_dups_size":0,"ondisk_log_size":2668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:25:46.346490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209762270,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13163,"num_write_kb":257524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"23'2441","reported_seq":2463,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.104779+0000","last_change":"2024-08-31T12:15:57.298778+0000","last_active":"2024-08-31T12:28:19.104779+0000","last_peered":"2024-08-31T12:28:19.104779+0000","last_clean":"2024-08-31T12:28:19.104779+0000","last_became_active":"2024-08-31T12:15:57.298430+0000","last_became_peered":"2024-08-31T12:15:57.298430+0000","last_unstale":"2024-08-31T12:28:19.104779+0000","last_undegraded":"2024-08-31T12:28:19.104779+0000","last_fullsized":"2024-08-31T12:28:19.104779+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2441,"log_dups_size":0,"ondisk_log_size":2441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:22:13.435300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205530518,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12230,"num_write_kb":249721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"23'2939","reported_seq":2961,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.228814+0000","last_change":"2024-08-31T12:15:57.317176+0000","last_active":"2024-08-31T12:28:19.228814+0000","last_peered":"2024-08-31T12:28:19.228814+0000","last_clean":"2024-08-31T12:28:19.228814+0000","last_became_active":"2024-08-31T12:15:57.316870+0000","last_became_peered":"2024-08-31T12:15:57.316870+0000","last_unstale":"2024-08-31T12:28:19.228814+0000","last_undegraded":"2024-08-31T12:28:19.228814+0000","last_fullsized":"2024-08-31T12:28:19.228814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2939,"log_dups_size":0,"ondisk_log_size":2939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:43:12.703971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213923005,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14700,"num_write_kb":267872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"23'3066","reported_seq":3082,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.485438+0000","last_change":"2024-08-31T12:16:40.784525+0000","last_active":"2024-08-31T12:28:47.485438+0000","last_peered":"2024-08-31T12:28:47.485438+0000","last_clean":"2024-08-31T12:28:47.485438+0000","last_became_active":"2024-08-31T12:16:40.784204+0000","last_became_peered":"2024-08-31T12:16:40.784204+0000","last_unstale":"2024-08-31T12:28:47.485438+0000","last_undegraded":"2024-08-31T12:28:47.485438+0000","last_fullsized":"2024-08-31T12:28:47.485438+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3066,"log_dups_size":0,"ondisk_log_size":3066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:51:38.173279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226493660,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15308,"num_write_kb":282531,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"23'2827","reported_seq":2849,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.207366+0000","last_change":"2024-08-31T12:15:57.304032+0000","last_active":"2024-08-31T12:28:19.207366+0000","last_peered":"2024-08-31T12:28:19.207366+0000","last_clean":"2024-08-31T12:28:19.207366+0000","last_became_active":"2024-08-31T12:15:57.303401+0000","last_became_peered":"2024-08-31T12:15:57.303401+0000","last_unstale":"2024-08-31T12:28:19.207366+0000","last_undegraded":"2024-08-31T12:28:19.207366+0000","last_fullsized":"2024-08-31T12:28:19.207366+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2827,"log_dups_size":0,"ondisk_log_size":2827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:56:03.280339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205588184,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14070,"num_write_kb":257066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"23'2468","reported_seq":2490,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.209009+0000","last_change":"2024-08-31T12:15:57.305412+0000","last_active":"2024-08-31T12:28:19.209009+0000","last_peered":"2024-08-31T12:28:19.209009+0000","last_clean":"2024-08-31T12:28:19.209009+0000","last_became_active":"2024-08-31T12:15:57.305075+0000","last_became_peered":"2024-08-31T12:15:57.305075+0000","last_unstale":"2024-08-31T12:28:19.209009+0000","last_undegraded":"2024-08-31T12:28:19.209009+0000","last_fullsized":"2024-08-31T12:28:19.209009+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:30:13.715505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184555785,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12290,"num_write_kb":229419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"23'3164","reported_seq":3186,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.223770+0000","last_change":"2024-08-31T12:15:57.300179+0000","last_active":"2024-08-31T12:28:19.223770+0000","last_peered":"2024-08-31T12:28:19.223770+0000","last_clean":"2024-08-31T12:28:19.223770+0000","last_became_active":"2024-08-31T12:15:57.299861+0000","last_became_peered":"2024-08-31T12:15:57.299861+0000","last_unstale":"2024-08-31T12:28:19.223770+0000","last_undegraded":"2024-08-31T12:28:19.223770+0000","last_fullsized":"2024-08-31T12:28:19.223770+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3164,"log_dups_size":0,"ondisk_log_size":3164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:47:51.267409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243322604,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15709,"num_write_kb":300530,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"23'2729","reported_seq":2751,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.205829+0000","last_change":"2024-08-31T12:15:57.308519+0000","last_active":"2024-08-31T12:28:19.205829+0000","last_peered":"2024-08-31T12:28:19.205829+0000","last_clean":"2024-08-31T12:28:19.205829+0000","last_became_active":"2024-08-31T12:15:57.308189+0000","last_became_peered":"2024-08-31T12:15:57.308189+0000","last_unstale":"2024-08-31T12:28:19.205829+0000","last_undegraded":"2024-08-31T12:28:19.205829+0000","last_fullsized":"2024-08-31T12:28:19.205829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2729,"log_dups_size":0,"ondisk_log_size":2729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:37:02.435678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189169648,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13404,"num_write_kb":238420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"23'3082","reported_seq":3104,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.211281+0000","last_change":"2024-08-31T12:15:57.308321+0000","last_active":"2024-08-31T12:28:19.211281+0000","last_peered":"2024-08-31T12:28:19.211281+0000","last_clean":"2024-08-31T12:28:19.211281+0000","last_became_active":"2024-08-31T12:15:57.304514+0000","last_became_peered":"2024-08-31T12:15:57.304514+0000","last_unstale":"2024-08-31T12:28:19.211281+0000","last_undegraded":"2024-08-31T12:28:19.211281+0000","last_fullsized":"2024-08-31T12:28:19.211281+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3082,"log_dups_size":0,"ondisk_log_size":3082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:09:49.617826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222298838,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15326,"num_write_kb":278485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"23'2570","reported_seq":2592,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.125425+0000","last_change":"2024-08-31T12:15:57.299268+0000","last_active":"2024-08-31T12:28:19.125425+0000","last_peered":"2024-08-31T12:28:19.125425+0000","last_clean":"2024-08-31T12:28:19.125425+0000","last_became_active":"2024-08-31T12:15:57.298766+0000","last_became_peered":"2024-08-31T12:15:57.298766+0000","last_unstale":"2024-08-31T12:28:19.125425+0000","last_undegraded":"2024-08-31T12:28:19.125425+0000","last_fullsized":"2024-08-31T12:28:19.125425+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2570,"log_dups_size":0,"ondisk_log_size":2570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:37:45.891394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188773348,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12878,"num_write_kb":235916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"23'2321","reported_seq":2344,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.234241+0000","last_change":"2024-08-31T12:15:57.310121+0000","last_active":"2024-08-31T12:28:19.234241+0000","last_peered":"2024-08-31T12:28:19.234241+0000","last_clean":"2024-08-31T12:28:19.234241+0000","last_became_active":"2024-08-31T12:15:57.309440+0000","last_became_peered":"2024-08-31T12:15:57.309440+0000","last_unstale":"2024-08-31T12:28:19.234241+0000","last_undegraded":"2024-08-31T12:28:19.234241+0000","last_fullsized":"2024-08-31T12:28:19.234241+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2321,"log_dups_size":0,"ondisk_log_size":2321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:53:19.523811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180357122,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11238,"num_write_kb":221098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"23'3439","reported_seq":3462,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.198926+0000","last_change":"2024-08-31T12:15:57.306214+0000","last_active":"2024-08-31T12:28:19.198926+0000","last_peered":"2024-08-31T12:28:19.198926+0000","last_clean":"2024-08-31T12:28:19.198926+0000","last_became_active":"2024-08-31T12:15:57.305878+0000","last_became_peered":"2024-08-31T12:15:57.305878+0000","last_unstale":"2024-08-31T12:28:19.198926+0000","last_undegraded":"2024-08-31T12:28:19.198926+0000","last_fullsized":"2024-08-31T12:28:19.198926+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3439,"log_dups_size":0,"ondisk_log_size":3439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:05:48.614918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251668879,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17281,"num_write_kb":314954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"23'2880","reported_seq":2903,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.439514+0000","last_change":"2024-08-31T12:15:57.307376+0000","last_active":"2024-08-31T12:28:47.439514+0000","last_peered":"2024-08-31T12:28:47.439514+0000","last_clean":"2024-08-31T12:28:47.439514+0000","last_became_active":"2024-08-31T12:15:57.307056+0000","last_became_peered":"2024-08-31T12:15:57.307056+0000","last_unstale":"2024-08-31T12:28:47.439514+0000","last_undegraded":"2024-08-31T12:28:47.439514+0000","last_fullsized":"2024-08-31T12:28:47.439514+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2880,"log_dups_size":0,"ondisk_log_size":2880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:52:18.004608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222298112,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14386,"num_write_kb":274682,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"23'3113","reported_seq":3136,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.193855+0000","last_change":"2024-08-31T12:15:57.313798+0000","last_active":"2024-08-31T12:28:19.193855+0000","last_peered":"2024-08-31T12:28:19.193855+0000","last_clean":"2024-08-31T12:28:19.193855+0000","last_became_active":"2024-08-31T12:15:57.313463+0000","last_became_peered":"2024-08-31T12:15:57.313463+0000","last_unstale":"2024-08-31T12:28:19.193855+0000","last_undegraded":"2024-08-31T12:28:19.193855+0000","last_fullsized":"2024-08-31T12:28:19.193855+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3113,"log_dups_size":0,"ondisk_log_size":3113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:47:51.267409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260049613,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15263,"num_write_kb":315030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"23'2280","reported_seq":2302,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.088996+0000","last_change":"2024-08-31T12:15:57.293350+0000","last_active":"2024-08-31T12:28:19.088996+0000","last_peered":"2024-08-31T12:28:19.088996+0000","last_clean":"2024-08-31T12:28:19.088996+0000","last_became_active":"2024-08-31T12:15:57.292981+0000","last_became_peered":"2024-08-31T12:15:57.292981+0000","last_unstale":"2024-08-31T12:28:19.088996+0000","last_undegraded":"2024-08-31T12:28:19.088996+0000","last_fullsized":"2024-08-31T12:28:19.088996+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2280,"log_dups_size":0,"ondisk_log_size":2280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:52:00.312013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180377946,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11142,"num_write_kb":220690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"23'569","reported_seq":593,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:56.212722+0000","last_change":"2024-08-31T12:15:56.293497+0000","last_active":"2024-08-31T12:25:56.212722+0000","last_peered":"2024-08-31T12:25:56.212722+0000","last_clean":"2024-08-31T12:25:56.212722+0000","last_became_active":"2024-08-31T12:15:56.293164+0000","last_became_peered":"2024-08-31T12:15:56.293164+0000","last_unstale":"2024-08-31T12:25:56.212722+0000","last_undegraded":"2024-08-31T12:25:56.212722+0000","last_fullsized":"2024-08-31T12:25:56.212722+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":569,"log_dups_size":0,"ondisk_log_size":569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:43:17.443433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4195367,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":566,"num_write_kb":4210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"23'2388","reported_seq":2410,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.228535+0000","last_change":"2024-08-31T12:15:57.312738+0000","last_active":"2024-08-31T12:28:19.228535+0000","last_peered":"2024-08-31T12:28:19.228535+0000","last_clean":"2024-08-31T12:28:19.228535+0000","last_became_active":"2024-08-31T12:15:57.312360+0000","last_became_peered":"2024-08-31T12:15:57.312360+0000","last_unstale":"2024-08-31T12:28:19.228535+0000","last_undegraded":"2024-08-31T12:28:19.228535+0000","last_fullsized":"2024-08-31T12:28:19.228535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2388,"log_dups_size":0,"ondisk_log_size":2388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:22:13.435300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163584295,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11807,"num_write_kb":207035,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"23'5","reported_seq":29,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:56.196855+0000","last_change":"2024-08-31T12:15:56.283526+0000","last_active":"2024-08-31T12:25:56.196855+0000","last_peered":"2024-08-31T12:25:56.196855+0000","last_clean":"2024-08-31T12:25:56.196855+0000","last_became_active":"2024-08-31T12:15:56.283147+0000","last_became_peered":"2024-08-31T12:15:56.283147+0000","last_unstale":"2024-08-31T12:25:56.196855+0000","last_undegraded":"2024-08-31T12:25:56.196855+0000","last_fullsized":"2024-08-31T12:25:56.196855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:51:59.306912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"23'4","reported_seq":28,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:36.210659+0000","last_change":"2024-08-31T12:15:56.291170+0000","last_active":"2024-08-31T12:25:36.210659+0000","last_peered":"2024-08-31T12:25:36.210659+0000","last_clean":"2024-08-31T12:25:36.210659+0000","last_became_active":"2024-08-31T12:15:56.290835+0000","last_became_peered":"2024-08-31T12:15:56.290835+0000","last_unstale":"2024-08-31T12:25:36.210659+0000","last_undegraded":"2024-08-31T12:25:36.210659+0000","last_fullsized":"2024-08-31T12:25:36.210659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:15:51.561831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"23'2535","reported_seq":2557,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.224638+0000","last_change":"2024-08-31T12:15:57.308183+0000","last_active":"2024-08-31T12:28:19.224638+0000","last_peered":"2024-08-31T12:28:19.224638+0000","last_clean":"2024-08-31T12:28:19.224638+0000","last_became_active":"2024-08-31T12:15:57.306962+0000","last_became_peered":"2024-08-31T12:15:57.306962+0000","last_unstale":"2024-08-31T12:28:19.224638+0000","last_undegraded":"2024-08-31T12:28:19.224638+0000","last_fullsized":"2024-08-31T12:28:19.224638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2535,"log_dups_size":0,"ondisk_log_size":2535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:01:18.188264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205541461,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":251825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"23'2","reported_seq":26,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:40.196751+0000","last_change":"2024-08-31T12:15:56.288126+0000","last_active":"2024-08-31T12:25:40.196751+0000","last_peered":"2024-08-31T12:25:40.196751+0000","last_clean":"2024-08-31T12:25:40.196751+0000","last_became_active":"2024-08-31T12:15:56.287552+0000","last_became_peered":"2024-08-31T12:15:56.287552+0000","last_unstale":"2024-08-31T12:25:40.196751+0000","last_undegraded":"2024-08-31T12:25:40.196751+0000","last_fullsized":"2024-08-31T12:25:40.196751+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:54:29.988079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"23'3439","reported_seq":3461,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.208231+0000","last_change":"2024-08-31T12:15:57.305029+0000","last_active":"2024-08-31T12:28:19.208231+0000","last_peered":"2024-08-31T12:28:19.208231+0000","last_clean":"2024-08-31T12:28:19.208231+0000","last_became_active":"2024-08-31T12:15:57.304700+0000","last_became_peered":"2024-08-31T12:15:57.304700+0000","last_unstale":"2024-08-31T12:28:19.208231+0000","last_undegraded":"2024-08-31T12:28:19.208231+0000","last_fullsized":"2024-08-31T12:28:19.208231+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3439,"log_dups_size":0,"ondisk_log_size":3439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:58:17.482204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243316666,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17415,"num_write_kb":307366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"23'4","reported_seq":28,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:36.194403+0000","last_change":"2024-08-31T12:15:56.284907+0000","last_active":"2024-08-31T12:25:36.194403+0000","last_peered":"2024-08-31T12:25:36.194403+0000","last_clean":"2024-08-31T12:25:36.194403+0000","last_became_active":"2024-08-31T12:15:56.284558+0000","last_became_peered":"2024-08-31T12:15:56.284558+0000","last_unstale":"2024-08-31T12:25:36.194403+0000","last_undegraded":"2024-08-31T12:25:36.194403+0000","last_fullsized":"2024-08-31T12:25:36.194403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:29:53.117724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"23'2894","reported_seq":2916,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.203880+0000","last_change":"2024-08-31T12:15:57.308679+0000","last_active":"2024-08-31T12:28:19.203880+0000","last_peered":"2024-08-31T12:28:19.203880+0000","last_clean":"2024-08-31T12:28:19.203880+0000","last_became_active":"2024-08-31T12:15:57.308260+0000","last_became_peered":"2024-08-31T12:15:57.308260+0000","last_unstale":"2024-08-31T12:28:19.203880+0000","last_undegraded":"2024-08-31T12:28:19.203880+0000","last_fullsized":"2024-08-31T12:28:19.203880+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2894,"log_dups_size":0,"ondisk_log_size":2894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:31:19.860950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209955924,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14127,"num_write_kb":261546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"21'2","reported_seq":26,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:40.196825+0000","last_change":"2024-08-31T12:15:56.290763+0000","last_active":"2024-08-31T12:25:40.196825+0000","last_peered":"2024-08-31T12:25:40.196825+0000","last_clean":"2024-08-31T12:25:40.196825+0000","last_became_active":"2024-08-31T12:15:56.290400+0000","last_became_peered":"2024-08-31T12:15:56.290400+0000","last_unstale":"2024-08-31T12:25:40.196825+0000","last_undegraded":"2024-08-31T12:25:40.196825+0000","last_fullsized":"2024-08-31T12:25:40.196825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:53:08.929860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"23'2656","reported_seq":2678,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.161047+0000","last_change":"2024-08-31T12:15:57.311657+0000","last_active":"2024-08-31T12:28:19.161047+0000","last_peered":"2024-08-31T12:28:19.161047+0000","last_clean":"2024-08-31T12:28:19.161047+0000","last_became_active":"2024-08-31T12:15:57.310564+0000","last_became_peered":"2024-08-31T12:15:57.310564+0000","last_unstale":"2024-08-31T12:28:19.161047+0000","last_undegraded":"2024-08-31T12:28:19.161047+0000","last_fullsized":"2024-08-31T12:28:19.161047+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2656,"log_dups_size":0,"ondisk_log_size":2656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:37:02.435678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180410852,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13728,"num_write_kb":231123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"23'19","reported_seq":43,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:36.210981+0000","last_change":"2024-08-31T12:15:56.286792+0000","last_active":"2024-08-31T12:25:36.210981+0000","last_peered":"2024-08-31T12:25:36.210981+0000","last_clean":"2024-08-31T12:25:36.210981+0000","last_became_active":"2024-08-31T12:15:56.286458+0000","last_became_peered":"2024-08-31T12:15:56.286458+0000","last_unstale":"2024-08-31T12:25:36.210981+0000","last_undegraded":"2024-08-31T12:25:36.210981+0000","last_fullsized":"2024-08-31T12:25:36.210981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:49:17.488528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"23'3658","reported_seq":3680,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.159920+0000","last_change":"2024-08-31T12:15:57.309485+0000","last_active":"2024-08-31T12:28:19.159920+0000","last_peered":"2024-08-31T12:28:19.159920+0000","last_clean":"2024-08-31T12:28:19.159920+0000","last_became_active":"2024-08-31T12:15:57.309102+0000","last_became_peered":"2024-08-31T12:15:57.309102+0000","last_unstale":"2024-08-31T12:28:19.159920+0000","last_undegraded":"2024-08-31T12:28:19.159920+0000","last_fullsized":"2024-08-31T12:28:19.159920+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3658,"log_dups_size":0,"ondisk_log_size":3658,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:17:34.692296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":277024326,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17723,"num_write_kb":341816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"23'317","reported_seq":341,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:22.335219+0000","last_change":"2024-08-31T12:15:56.289460+0000","last_active":"2024-08-31T12:28:22.335219+0000","last_peered":"2024-08-31T12:28:22.335219+0000","last_clean":"2024-08-31T12:28:22.335219+0000","last_became_active":"2024-08-31T12:15:56.289096+0000","last_became_peered":"2024-08-31T12:15:56.289096+0000","last_unstale":"2024-08-31T12:28:22.335219+0000","last_undegraded":"2024-08-31T12:28:22.335219+0000","last_fullsized":"2024-08-31T12:28:22.335219+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:06:34.313772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":317,"num_write_kb":4236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"23'2741","reported_seq":2763,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.210058+0000","last_change":"2024-08-31T12:15:57.317968+0000","last_active":"2024-08-31T12:28:19.210058+0000","last_peered":"2024-08-31T12:28:19.210058+0000","last_clean":"2024-08-31T12:28:19.210058+0000","last_became_active":"2024-08-31T12:15:57.317632+0000","last_became_peered":"2024-08-31T12:15:57.317632+0000","last_unstale":"2024-08-31T12:28:19.210058+0000","last_undegraded":"2024-08-31T12:28:19.210058+0000","last_fullsized":"2024-08-31T12:28:19.210058+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2741,"log_dups_size":0,"ondisk_log_size":2741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:22:02.569106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197132288,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13667,"num_write_kb":247166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'6","reported_seq":30,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:16.208499+0000","last_change":"2024-08-31T12:15:56.284014+0000","last_active":"2024-08-31T12:25:16.208499+0000","last_peered":"2024-08-31T12:25:16.208499+0000","last_clean":"2024-08-31T12:25:16.208499+0000","last_became_active":"2024-08-31T12:15:56.283675+0000","last_became_peered":"2024-08-31T12:15:56.283675+0000","last_unstale":"2024-08-31T12:25:16.208499+0000","last_undegraded":"2024-08-31T12:25:16.208499+0000","last_fullsized":"2024-08-31T12:25:16.208499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:51:59.306912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"23'3052","reported_seq":3074,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.204958+0000","last_change":"2024-08-31T12:15:57.312440+0000","last_active":"2024-08-31T12:28:19.204958+0000","last_peered":"2024-08-31T12:28:19.204958+0000","last_clean":"2024-08-31T12:28:19.204958+0000","last_became_active":"2024-08-31T12:15:57.312007+0000","last_became_peered":"2024-08-31T12:15:57.312007+0000","last_unstale":"2024-08-31T12:28:19.204958+0000","last_undegraded":"2024-08-31T12:28:19.204958+0000","last_fullsized":"2024-08-31T12:28:19.204958+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3052,"log_dups_size":0,"ondisk_log_size":3052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:23:43.553625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197138377,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15100,"num_write_kb":253041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"23'4","reported_seq":28,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:20.194216+0000","last_change":"2024-08-31T12:15:56.290968+0000","last_active":"2024-08-31T12:25:20.194216+0000","last_peered":"2024-08-31T12:25:20.194216+0000","last_clean":"2024-08-31T12:25:20.194216+0000","last_became_active":"2024-08-31T12:15:56.290603+0000","last_became_peered":"2024-08-31T12:15:56.290603+0000","last_unstale":"2024-08-31T12:25:20.194216+0000","last_undegraded":"2024-08-31T12:25:20.194216+0000","last_fullsized":"2024-08-31T12:25:20.194216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:50:23.215239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"23'3279","reported_seq":3301,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.194687+0000","last_change":"2024-08-31T12:15:57.312790+0000","last_active":"2024-08-31T12:28:19.194687+0000","last_peered":"2024-08-31T12:28:19.194687+0000","last_clean":"2024-08-31T12:28:19.194687+0000","last_became_active":"2024-08-31T12:15:57.312383+0000","last_became_peered":"2024-08-31T12:15:57.312383+0000","last_unstale":"2024-08-31T12:28:19.194687+0000","last_undegraded":"2024-08-31T12:28:19.194687+0000","last_fullsized":"2024-08-31T12:28:19.194687+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3279,"log_dups_size":0,"ondisk_log_size":3279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:20:41.042375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234891518,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16277,"num_write_kb":294634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"23'5","reported_seq":29,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:16.191885+0000","last_change":"2024-08-31T12:15:56.289795+0000","last_active":"2024-08-31T12:25:16.191885+0000","last_peered":"2024-08-31T12:25:16.191885+0000","last_clean":"2024-08-31T12:25:16.191885+0000","last_became_active":"2024-08-31T12:15:56.289477+0000","last_became_peered":"2024-08-31T12:15:56.289477+0000","last_unstale":"2024-08-31T12:25:16.191885+0000","last_undegraded":"2024-08-31T12:25:16.191885+0000","last_fullsized":"2024-08-31T12:25:16.191885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:15:51.561831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":54,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"23'2828","reported_seq":2850,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.192699+0000","last_change":"2024-08-31T12:15:57.312865+0000","last_active":"2024-08-31T12:28:19.192699+0000","last_peered":"2024-08-31T12:28:19.192699+0000","last_clean":"2024-08-31T12:28:19.192699+0000","last_became_active":"2024-08-31T12:15:57.312462+0000","last_became_peered":"2024-08-31T12:15:57.312462+0000","last_unstale":"2024-08-31T12:28:19.192699+0000","last_undegraded":"2024-08-31T12:28:19.192699+0000","last_fullsized":"2024-08-31T12:28:19.192699+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2828,"log_dups_size":0,"ondisk_log_size":2828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:16:42.625396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222316815,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13883,"num_write_kb":272702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'15","reported_seq":39,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:18.179855+0000","last_change":"2024-08-31T12:15:56.285412+0000","last_active":"2024-08-31T12:25:18.179855+0000","last_peered":"2024-08-31T12:25:18.179855+0000","last_clean":"2024-08-31T12:25:18.179855+0000","last_became_active":"2024-08-31T12:15:56.285108+0000","last_became_peered":"2024-08-31T12:15:56.285108+0000","last_unstale":"2024-08-31T12:25:18.179855+0000","last_undegraded":"2024-08-31T12:25:18.179855+0000","last_fullsized":"2024-08-31T12:25:18.179855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:29:45.580957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"23'2843","reported_seq":2865,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.145892+0000","last_change":"2024-08-31T12:15:57.311701+0000","last_active":"2024-08-31T12:28:19.145892+0000","last_peered":"2024-08-31T12:28:19.145892+0000","last_clean":"2024-08-31T12:28:19.145892+0000","last_became_active":"2024-08-31T12:15:57.311368+0000","last_became_peered":"2024-08-31T12:15:57.311368+0000","last_unstale":"2024-08-31T12:28:19.145892+0000","last_undegraded":"2024-08-31T12:28:19.145892+0000","last_fullsized":"2024-08-31T12:28:19.145892+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2843,"log_dups_size":0,"ondisk_log_size":2843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:30:13.715505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205547202,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14167,"num_write_kb":257592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"23'3","reported_seq":27,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:00.192148+0000","last_change":"2024-08-31T12:15:56.289979+0000","last_active":"2024-08-31T12:25:00.192148+0000","last_peered":"2024-08-31T12:25:00.192148+0000","last_clean":"2024-08-31T12:25:00.192148+0000","last_became_active":"2024-08-31T12:15:56.288289+0000","last_became_peered":"2024-08-31T12:15:56.288289+0000","last_unstale":"2024-08-31T12:25:00.192148+0000","last_undegraded":"2024-08-31T12:25:00.192148+0000","last_fullsized":"2024-08-31T12:25:00.192148+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:46:55.057087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"23'2808","reported_seq":2830,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.206168+0000","last_change":"2024-08-31T12:15:57.311580+0000","last_active":"2024-08-31T12:28:19.206168+0000","last_peered":"2024-08-31T12:28:19.206168+0000","last_clean":"2024-08-31T12:28:19.206168+0000","last_became_active":"2024-08-31T12:15:57.311323+0000","last_became_peered":"2024-08-31T12:15:57.311323+0000","last_unstale":"2024-08-31T12:28:19.206168+0000","last_undegraded":"2024-08-31T12:28:19.206168+0000","last_fullsized":"2024-08-31T12:28:19.206168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2808,"log_dups_size":0,"ondisk_log_size":2808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:54:03.871840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201522906,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13808,"num_write_kb":252081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"23'3","reported_seq":27,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:24:58.177773+0000","last_change":"2024-08-31T12:15:56.290496+0000","last_active":"2024-08-31T12:24:58.177773+0000","last_peered":"2024-08-31T12:24:58.177773+0000","last_clean":"2024-08-31T12:24:58.177773+0000","last_became_active":"2024-08-31T12:15:56.290196+0000","last_became_peered":"2024-08-31T12:15:56.290196+0000","last_unstale":"2024-08-31T12:24:58.177773+0000","last_undegraded":"2024-08-31T12:24:58.177773+0000","last_fullsized":"2024-08-31T12:24:58.177773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:52:20.055757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"23'3282","reported_seq":3304,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.175690+0000","last_change":"2024-08-31T12:15:57.313317+0000","last_active":"2024-08-31T12:28:19.175690+0000","last_peered":"2024-08-31T12:28:19.175690+0000","last_clean":"2024-08-31T12:28:19.175690+0000","last_became_active":"2024-08-31T12:15:57.312145+0000","last_became_peered":"2024-08-31T12:15:57.312145+0000","last_unstale":"2024-08-31T12:28:19.175690+0000","last_undegraded":"2024-08-31T12:28:19.175690+0000","last_fullsized":"2024-08-31T12:28:19.175690+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3282,"log_dups_size":0,"ondisk_log_size":3282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:09:31.904512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243277619,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16586,"num_write_kb":304033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":25,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:24:56.207028+0000","last_change":"2024-08-31T12:15:56.281809+0000","last_active":"2024-08-31T12:24:56.207028+0000","last_peered":"2024-08-31T12:24:56.207028+0000","last_clean":"2024-08-31T12:24:56.207028+0000","last_became_active":"2024-08-31T12:15:56.281368+0000","last_became_peered":"2024-08-31T12:15:56.281368+0000","last_unstale":"2024-08-31T12:24:56.207028+0000","last_undegraded":"2024-08-31T12:24:56.207028+0000","last_fullsized":"2024-08-31T12:24:56.207028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:17:33.687196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"23'3289","reported_seq":3311,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.183134+0000","last_change":"2024-08-31T12:15:57.305874+0000","last_active":"2024-08-31T12:28:19.183134+0000","last_peered":"2024-08-31T12:28:19.183134+0000","last_clean":"2024-08-31T12:28:19.183134+0000","last_became_active":"2024-08-31T12:15:57.305507+0000","last_became_peered":"2024-08-31T12:15:57.305507+0000","last_unstale":"2024-08-31T12:28:19.183134+0000","last_undegraded":"2024-08-31T12:28:19.183134+0000","last_fullsized":"2024-08-31T12:28:19.183134+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3289,"log_dups_size":0,"ondisk_log_size":3289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:22:02.569106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218117314,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16432,"num_write_kb":278837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"23'4","reported_seq":28,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:24:58.177745+0000","last_change":"2024-08-31T12:15:56.291851+0000","last_active":"2024-08-31T12:24:58.177745+0000","last_peered":"2024-08-31T12:24:58.177745+0000","last_clean":"2024-08-31T12:24:58.177745+0000","last_became_active":"2024-08-31T12:15:56.291446+0000","last_became_peered":"2024-08-31T12:15:56.291446+0000","last_unstale":"2024-08-31T12:24:58.177745+0000","last_undegraded":"2024-08-31T12:24:58.177745+0000","last_fullsized":"2024-08-31T12:24:58.177745+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:54:29.988079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"23'2737","reported_seq":2759,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.167644+0000","last_change":"2024-08-31T12:15:57.316154+0000","last_active":"2024-08-31T12:28:19.167644+0000","last_peered":"2024-08-31T12:28:19.167644+0000","last_clean":"2024-08-31T12:28:19.167644+0000","last_became_active":"2024-08-31T12:15:57.315830+0000","last_became_peered":"2024-08-31T12:15:57.315830+0000","last_unstale":"2024-08-31T12:28:19.167644+0000","last_undegraded":"2024-08-31T12:28:19.167644+0000","last_fullsized":"2024-08-31T12:28:19.167644+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2737,"log_dups_size":0,"ondisk_log_size":2737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:05:53.039520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209813064,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13700,"num_write_kb":259755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":121,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:50.198157+0000","last_change":"2024-08-31T12:15:47.084912+0000","last_active":"2024-08-31T12:25:50.198157+0000","last_peered":"2024-08-31T12:25:50.198157+0000","last_clean":"2024-08-31T12:25:50.198157+0000","last_became_active":"2024-08-31T12:15:47.083771+0000","last_became_peered":"2024-08-31T12:15:47.083771+0000","last_unstale":"2024-08-31T12:25:50.198157+0000","last_undegraded":"2024-08-31T12:25:50.198157+0000","last_fullsized":"2024-08-31T12:25:50.198157+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:45.213688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:45.213688+0000","last_clean_scrub_stamp":"2024-08-31T12:15:45.213688+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:25:24.827530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"23'4","reported_seq":29,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:26:25.205795+0000","last_change":"2024-08-31T12:15:56.285355+0000","last_active":"2024-08-31T12:26:25.205795+0000","last_peered":"2024-08-31T12:26:25.205795+0000","last_clean":"2024-08-31T12:26:25.205795+0000","last_became_active":"2024-08-31T12:15:56.284470+0000","last_became_peered":"2024-08-31T12:15:56.284470+0000","last_unstale":"2024-08-31T12:26:25.205795+0000","last_undegraded":"2024-08-31T12:26:25.205795+0000","last_fullsized":"2024-08-31T12:26:25.205795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:52:20.055757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"23'2767","reported_seq":2790,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.201878+0000","last_change":"2024-08-31T12:15:57.309039+0000","last_active":"2024-08-31T12:28:19.201878+0000","last_peered":"2024-08-31T12:28:19.201878+0000","last_clean":"2024-08-31T12:28:19.201878+0000","last_became_active":"2024-08-31T12:15:57.308721+0000","last_became_peered":"2024-08-31T12:15:57.308721+0000","last_unstale":"2024-08-31T12:28:19.201878+0000","last_undegraded":"2024-08-31T12:28:19.201878+0000","last_fullsized":"2024-08-31T12:28:19.201878+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2767,"log_dups_size":0,"ondisk_log_size":2767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:43:12.703971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218112028,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13723,"num_write_kb":267984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"23'4","reported_seq":29,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:27:20.870746+0000","last_change":"2024-08-31T12:15:56.293221+0000","last_active":"2024-08-31T12:27:20.870746+0000","last_peered":"2024-08-31T12:27:20.870746+0000","last_clean":"2024-08-31T12:27:20.870746+0000","last_became_active":"2024-08-31T12:15:56.292910+0000","last_became_peered":"2024-08-31T12:15:56.292910+0000","last_unstale":"2024-08-31T12:27:20.870746+0000","last_undegraded":"2024-08-31T12:27:20.870746+0000","last_fullsized":"2024-08-31T12:27:20.870746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:14:33.483002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"23'2757","reported_seq":2780,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.182199+0000","last_change":"2024-08-31T12:15:57.307266+0000","last_active":"2024-08-31T12:28:19.182199+0000","last_peered":"2024-08-31T12:28:19.182199+0000","last_clean":"2024-08-31T12:28:19.182199+0000","last_became_active":"2024-08-31T12:15:57.306529+0000","last_became_peered":"2024-08-31T12:15:57.306529+0000","last_unstale":"2024-08-31T12:28:19.182199+0000","last_undegraded":"2024-08-31T12:28:19.182199+0000","last_fullsized":"2024-08-31T12:28:19.182199+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2757,"log_dups_size":0,"ondisk_log_size":2757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:42:39.731742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201337916,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14174,"num_write_kb":253444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"23'35","reported_seq":90,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:26:23.185903+0000","last_change":"2024-08-31T12:15:56.291822+0000","last_active":"2024-08-31T12:26:23.185903+0000","last_peered":"2024-08-31T12:26:23.185903+0000","last_clean":"2024-08-31T12:26:23.185903+0000","last_became_active":"2024-08-31T12:15:56.291064+0000","last_became_peered":"2024-08-31T12:15:56.291064+0000","last_unstale":"2024-08-31T12:26:23.185903+0000","last_undegraded":"2024-08-31T12:26:23.185903+0000","last_fullsized":"2024-08-31T12:26:23.185903+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:53:08.929860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47459,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":30,"num_read_kb":64,"num_write":34,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"23'2938","reported_seq":2961,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.167628+0000","last_change":"2024-08-31T12:15:57.299212+0000","last_active":"2024-08-31T12:28:19.167628+0000","last_peered":"2024-08-31T12:28:19.167628+0000","last_clean":"2024-08-31T12:28:19.167628+0000","last_became_active":"2024-08-31T12:15:57.298892+0000","last_became_peered":"2024-08-31T12:15:57.298892+0000","last_unstale":"2024-08-31T12:28:19.167628+0000","last_undegraded":"2024-08-31T12:28:19.167628+0000","last_fullsized":"2024-08-31T12:28:19.167628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2938,"log_dups_size":0,"ondisk_log_size":2938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:53:19.523811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197144226,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14947,"num_write_kb":252544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"23'2","reported_seq":27,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:26:25.205779+0000","last_change":"2024-08-31T12:15:56.286720+0000","last_active":"2024-08-31T12:26:25.205779+0000","last_peered":"2024-08-31T12:26:25.205779+0000","last_clean":"2024-08-31T12:26:25.205779+0000","last_became_active":"2024-08-31T12:15:56.285648+0000","last_became_peered":"2024-08-31T12:15:56.285648+0000","last_unstale":"2024-08-31T12:26:25.205779+0000","last_undegraded":"2024-08-31T12:26:25.205779+0000","last_fullsized":"2024-08-31T12:26:25.205779+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:39:31.946792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"23'2762","reported_seq":2785,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.152132+0000","last_change":"2024-08-31T12:15:57.311785+0000","last_active":"2024-08-31T12:28:19.152132+0000","last_peered":"2024-08-31T12:28:19.152132+0000","last_clean":"2024-08-31T12:28:19.152132+0000","last_became_active":"2024-08-31T12:15:57.311469+0000","last_became_peered":"2024-08-31T12:15:57.311469+0000","last_unstale":"2024-08-31T12:28:19.152132+0000","last_undegraded":"2024-08-31T12:28:19.152132+0000","last_fullsized":"2024-08-31T12:28:19.152132+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2762,"log_dups_size":0,"ondisk_log_size":2762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:29:54.122825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197139313,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14081,"num_write_kb":248914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"23'1","reported_seq":26,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:26:21.215756+0000","last_change":"2024-08-31T12:15:56.284302+0000","last_active":"2024-08-31T12:26:21.215756+0000","last_peered":"2024-08-31T12:26:21.215756+0000","last_clean":"2024-08-31T12:26:21.215756+0000","last_became_active":"2024-08-31T12:15:56.283979+0000","last_became_peered":"2024-08-31T12:15:56.283979+0000","last_unstale":"2024-08-31T12:26:21.215756+0000","last_undegraded":"2024-08-31T12:26:21.215756+0000","last_fullsized":"2024-08-31T12:26:21.215756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:51:59.306912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"23'2527","reported_seq":2543,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.473715+0000","last_change":"2024-08-31T12:16:41.195070+0000","last_active":"2024-08-31T12:28:47.473715+0000","last_peered":"2024-08-31T12:28:47.473715+0000","last_clean":"2024-08-31T12:28:47.473715+0000","last_became_active":"2024-08-31T12:16:41.194724+0000","last_became_peered":"2024-08-31T12:16:41.194724+0000","last_unstale":"2024-08-31T12:28:47.473715+0000","last_undegraded":"2024-08-31T12:28:47.473715+0000","last_fullsized":"2024-08-31T12:28:47.473715+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2527,"log_dups_size":0,"ondisk_log_size":2527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:24:46.945850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184558382,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13011,"num_write_kb":232344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"21'1","reported_seq":26,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:26:25.202953+0000","last_change":"2024-08-31T12:15:56.297648+0000","last_active":"2024-08-31T12:26:25.202953+0000","last_peered":"2024-08-31T12:26:25.202953+0000","last_clean":"2024-08-31T12:26:25.202953+0000","last_became_active":"2024-08-31T12:15:56.297306+0000","last_became_peered":"2024-08-31T12:15:56.297306+0000","last_unstale":"2024-08-31T12:26:25.202953+0000","last_undegraded":"2024-08-31T12:26:25.202953+0000","last_fullsized":"2024-08-31T12:26:25.202953+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:29:54.693819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"23'3164","reported_seq":3187,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.097415+0000","last_change":"2024-08-31T12:15:57.297230+0000","last_active":"2024-08-31T12:28:19.097415+0000","last_peered":"2024-08-31T12:28:19.097415+0000","last_clean":"2024-08-31T12:28:19.097415+0000","last_became_active":"2024-08-31T12:15:57.296892+0000","last_became_peered":"2024-08-31T12:15:57.296892+0000","last_unstale":"2024-08-31T12:28:19.097415+0000","last_undegraded":"2024-08-31T12:28:19.097415+0000","last_fullsized":"2024-08-31T12:28:19.097415+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3164,"log_dups_size":0,"ondisk_log_size":3164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:42:36.033586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213956769,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16152,"num_write_kb":273638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"23'427","reported_seq":451,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:23.458871+0000","last_change":"2024-08-31T12:15:56.286504+0000","last_active":"2024-08-31T12:28:23.458871+0000","last_peered":"2024-08-31T12:28:23.458871+0000","last_clean":"2024-08-31T12:28:23.458871+0000","last_became_active":"2024-08-31T12:15:56.286173+0000","last_became_peered":"2024-08-31T12:15:56.286173+0000","last_unstale":"2024-08-31T12:28:23.458871+0000","last_undegraded":"2024-08-31T12:28:23.458871+0000","last_fullsized":"2024-08-31T12:28:23.458871+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":427,"log_dups_size":0,"ondisk_log_size":427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:06:34.313772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":975,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"23'2916","reported_seq":2938,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.229676+0000","last_change":"2024-08-31T12:15:57.308013+0000","last_active":"2024-08-31T12:28:19.229676+0000","last_peered":"2024-08-31T12:28:19.229676+0000","last_clean":"2024-08-31T12:28:19.229676+0000","last_became_active":"2024-08-31T12:15:57.307701+0000","last_became_peered":"2024-08-31T12:15:57.307701+0000","last_unstale":"2024-08-31T12:28:19.229676+0000","last_undegraded":"2024-08-31T12:28:19.229676+0000","last_fullsized":"2024-08-31T12:28:19.229676+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2916,"log_dups_size":0,"ondisk_log_size":2916,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:06:15.161589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205521115,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14225,"num_write_kb":257747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'4","reported_seq":28,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:24:36.187735+0000","last_change":"2024-08-31T12:15:56.294852+0000","last_active":"2024-08-31T12:24:36.187735+0000","last_peered":"2024-08-31T12:24:36.187735+0000","last_clean":"2024-08-31T12:24:36.187735+0000","last_became_active":"2024-08-31T12:15:56.294496+0000","last_became_peered":"2024-08-31T12:15:56.294496+0000","last_unstale":"2024-08-31T12:24:36.187735+0000","last_undegraded":"2024-08-31T12:24:36.187735+0000","last_fullsized":"2024-08-31T12:24:36.187735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:49:17.488528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"23'2537","reported_seq":2559,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.168378+0000","last_change":"2024-08-31T12:15:57.305202+0000","last_active":"2024-08-31T12:28:19.168378+0000","last_peered":"2024-08-31T12:28:19.168378+0000","last_clean":"2024-08-31T12:28:19.168378+0000","last_became_active":"2024-08-31T12:15:57.304602+0000","last_became_peered":"2024-08-31T12:15:57.304602+0000","last_unstale":"2024-08-31T12:28:19.168378+0000","last_undegraded":"2024-08-31T12:28:19.168378+0000","last_fullsized":"2024-08-31T12:28:19.168378+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2537,"log_dups_size":0,"ondisk_log_size":2537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:56:03.280339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176167762,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12536,"num_write_kb":222149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"23'3","reported_seq":27,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:24:40.189661+0000","last_change":"2024-08-31T12:15:56.297542+0000","last_active":"2024-08-31T12:24:40.189661+0000","last_peered":"2024-08-31T12:24:40.189661+0000","last_clean":"2024-08-31T12:24:40.189661+0000","last_became_active":"2024-08-31T12:15:56.297168+0000","last_became_peered":"2024-08-31T12:15:56.297168+0000","last_unstale":"2024-08-31T12:24:40.189661+0000","last_undegraded":"2024-08-31T12:24:40.189661+0000","last_fullsized":"2024-08-31T12:24:40.189661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:56:41.924772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"23'2320","reported_seq":2342,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.187968+0000","last_change":"2024-08-31T12:15:57.298845+0000","last_active":"2024-08-31T12:28:19.187968+0000","last_peered":"2024-08-31T12:28:19.187968+0000","last_clean":"2024-08-31T12:28:19.187968+0000","last_became_active":"2024-08-31T12:15:57.298520+0000","last_became_peered":"2024-08-31T12:15:57.298520+0000","last_unstale":"2024-08-31T12:28:19.187968+0000","last_undegraded":"2024-08-31T12:28:19.187968+0000","last_fullsized":"2024-08-31T12:28:19.187968+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2320,"log_dups_size":0,"ondisk_log_size":2320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:06:35.318873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197144309,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11287,"num_write_kb":237691,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"21'3","reported_seq":27,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:24:36.205095+0000","last_change":"2024-08-31T12:15:56.290475+0000","last_active":"2024-08-31T12:24:36.205095+0000","last_peered":"2024-08-31T12:24:36.205095+0000","last_clean":"2024-08-31T12:24:36.205095+0000","last_became_active":"2024-08-31T12:15:56.289960+0000","last_became_peered":"2024-08-31T12:15:56.289960+0000","last_unstale":"2024-08-31T12:24:36.205095+0000","last_undegraded":"2024-08-31T12:24:36.205095+0000","last_fullsized":"2024-08-31T12:24:36.205095+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:29:53.117724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"23'2970","reported_seq":2992,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.225999+0000","last_change":"2024-08-31T12:15:57.301460+0000","last_active":"2024-08-31T12:28:19.225999+0000","last_peered":"2024-08-31T12:28:19.225999+0000","last_clean":"2024-08-31T12:28:19.225999+0000","last_became_active":"2024-08-31T12:15:57.301157+0000","last_became_peered":"2024-08-31T12:15:57.301157+0000","last_unstale":"2024-08-31T12:28:19.225999+0000","last_undegraded":"2024-08-31T12:28:19.225999+0000","last_fullsized":"2024-08-31T12:28:19.225999+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2970,"log_dups_size":0,"ondisk_log_size":2970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:22:52.173578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209728881,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15223,"num_write_kb":265883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"21'1","reported_seq":25,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:24:36.205125+0000","last_change":"2024-08-31T12:15:56.289720+0000","last_active":"2024-08-31T12:24:36.205125+0000","last_peered":"2024-08-31T12:24:36.205125+0000","last_clean":"2024-08-31T12:24:36.205125+0000","last_became_active":"2024-08-31T12:15:56.289332+0000","last_became_peered":"2024-08-31T12:15:56.289332+0000","last_unstale":"2024-08-31T12:24:36.205125+0000","last_undegraded":"2024-08-31T12:24:36.205125+0000","last_fullsized":"2024-08-31T12:24:36.205125+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:15:51.561831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"23'2415","reported_seq":2437,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.230233+0000","last_change":"2024-08-31T12:15:57.313802+0000","last_active":"2024-08-31T12:28:19.230233+0000","last_peered":"2024-08-31T12:28:19.230233+0000","last_clean":"2024-08-31T12:28:19.230233+0000","last_became_active":"2024-08-31T12:15:57.313428+0000","last_became_peered":"2024-08-31T12:15:57.313428+0000","last_unstale":"2024-08-31T12:28:19.230233+0000","last_undegraded":"2024-08-31T12:28:19.230233+0000","last_fullsized":"2024-08-31T12:28:19.230233+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2415,"log_dups_size":0,"ondisk_log_size":2415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:22:13.435300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176173175,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11447,"num_write_kb":217885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"23'1068","reported_seq":1092,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:27:20.763645+0000","last_change":"2024-08-31T12:15:56.296105+0000","last_active":"2024-08-31T12:27:20.763645+0000","last_peered":"2024-08-31T12:27:20.763645+0000","last_clean":"2024-08-31T12:27:20.763645+0000","last_became_active":"2024-08-31T12:15:56.295524+0000","last_became_peered":"2024-08-31T12:15:56.295524+0000","last_unstale":"2024-08-31T12:27:20.763645+0000","last_undegraded":"2024-08-31T12:27:20.763645+0000","last_fullsized":"2024-08-31T12:27:20.763645+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":1068,"log_dups_size":0,"ondisk_log_size":1068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:41:55.482044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1064,"num_write_kb":4401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"23'2502","reported_seq":2524,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.061863+0000","last_change":"2024-08-31T12:15:57.300155+0000","last_active":"2024-08-31T12:28:19.061863+0000","last_peered":"2024-08-31T12:28:19.061863+0000","last_clean":"2024-08-31T12:28:19.061863+0000","last_became_active":"2024-08-31T12:15:57.299436+0000","last_became_peered":"2024-08-31T12:15:57.299436+0000","last_unstale":"2024-08-31T12:28:19.061863+0000","last_undegraded":"2024-08-31T12:28:19.061863+0000","last_fullsized":"2024-08-31T12:28:19.061863+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2502,"log_dups_size":0,"ondisk_log_size":2502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:37:45.891394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188744397,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12696,"num_write_kb":235198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"23'3358","reported_seq":3380,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.189014+0000","last_change":"2024-08-31T12:15:57.300529+0000","last_active":"2024-08-31T12:28:19.189014+0000","last_peered":"2024-08-31T12:28:19.189014+0000","last_clean":"2024-08-31T12:28:19.189014+0000","last_became_active":"2024-08-31T12:15:57.300159+0000","last_became_peered":"2024-08-31T12:15:57.300159+0000","last_unstale":"2024-08-31T12:28:19.189014+0000","last_undegraded":"2024-08-31T12:28:19.189014+0000","last_fullsized":"2024-08-31T12:28:19.189014+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3358,"log_dups_size":0,"ondisk_log_size":3358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:35:53.831742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251730727,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16759,"num_write_kb":312885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":26,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:26:00.199883+0000","last_change":"2024-08-31T12:15:56.297791+0000","last_active":"2024-08-31T12:26:00.199883+0000","last_peered":"2024-08-31T12:26:00.199883+0000","last_clean":"2024-08-31T12:26:00.199883+0000","last_became_active":"2024-08-31T12:15:56.296976+0000","last_became_peered":"2024-08-31T12:15:56.296976+0000","last_unstale":"2024-08-31T12:26:00.199883+0000","last_undegraded":"2024-08-31T12:26:00.199883+0000","last_fullsized":"2024-08-31T12:26:00.199883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:07:54.963050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"23'2167","reported_seq":2189,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.184056+0000","last_change":"2024-08-31T12:15:57.301167+0000","last_active":"2024-08-31T12:28:19.184056+0000","last_peered":"2024-08-31T12:28:19.184056+0000","last_clean":"2024-08-31T12:28:19.184056+0000","last_became_active":"2024-08-31T12:15:57.300646+0000","last_became_peered":"2024-08-31T12:15:57.300646+0000","last_unstale":"2024-08-31T12:28:19.184056+0000","last_undegraded":"2024-08-31T12:28:19.184056+0000","last_fullsized":"2024-08-31T12:28:19.184056+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2167,"log_dups_size":0,"ondisk_log_size":2167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:05:53.039520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150996671,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10769,"num_write_kb":190633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"23'9","reported_seq":33,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:26:00.199906+0000","last_change":"2024-08-31T12:15:56.295139+0000","last_active":"2024-08-31T12:26:00.199906+0000","last_peered":"2024-08-31T12:26:00.199906+0000","last_clean":"2024-08-31T12:26:00.199906+0000","last_became_active":"2024-08-31T12:15:56.294560+0000","last_became_peered":"2024-08-31T12:15:56.294560+0000","last_unstale":"2024-08-31T12:26:00.199906+0000","last_undegraded":"2024-08-31T12:26:00.199906+0000","last_fullsized":"2024-08-31T12:26:00.199906+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:52:16.999507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"23'3348","reported_seq":3371,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.187804+0000","last_change":"2024-08-31T12:15:57.311074+0000","last_active":"2024-08-31T12:28:19.187804+0000","last_peered":"2024-08-31T12:28:19.187804+0000","last_clean":"2024-08-31T12:28:19.187804+0000","last_became_active":"2024-08-31T12:15:57.310262+0000","last_became_peered":"2024-08-31T12:15:57.310262+0000","last_unstale":"2024-08-31T12:28:19.187804+0000","last_undegraded":"2024-08-31T12:28:19.187804+0000","last_fullsized":"2024-08-31T12:28:19.187804+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3348,"log_dups_size":0,"ondisk_log_size":3348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:47:51.267409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251719163,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16721,"num_write_kb":312844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"23'3","reported_seq":28,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:56.212749+0000","last_change":"2024-08-31T12:15:56.285428+0000","last_active":"2024-08-31T12:25:56.212749+0000","last_peered":"2024-08-31T12:25:56.212749+0000","last_clean":"2024-08-31T12:25:56.212749+0000","last_became_active":"2024-08-31T12:15:56.284884+0000","last_became_peered":"2024-08-31T12:15:56.284884+0000","last_unstale":"2024-08-31T12:25:56.212749+0000","last_undegraded":"2024-08-31T12:25:56.212749+0000","last_fullsized":"2024-08-31T12:25:56.212749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:49:25.181432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"23'3167","reported_seq":3190,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:19.175794+0000","last_change":"2024-08-31T12:15:57.305772+0000","last_active":"2024-08-31T12:28:19.175794+0000","last_peered":"2024-08-31T12:28:19.175794+0000","last_clean":"2024-08-31T12:28:19.175794+0000","last_became_active":"2024-08-31T12:15:57.305409+0000","last_became_peered":"2024-08-31T12:15:57.305409+0000","last_unstale":"2024-08-31T12:28:19.175794+0000","last_undegraded":"2024-08-31T12:28:19.175794+0000","last_fullsized":"2024-08-31T12:28:19.175794+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3167,"log_dups_size":0,"ondisk_log_size":3167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:53:19.523811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222298112,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15945,"num_write_kb":281070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"23'229","reported_seq":254,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:37.337439+0000","last_change":"2024-08-31T12:15:56.294197+0000","last_active":"2024-08-31T12:28:37.337439+0000","last_peered":"2024-08-31T12:28:37.337439+0000","last_clean":"2024-08-31T12:28:37.337439+0000","last_became_active":"2024-08-31T12:15:56.293864+0000","last_became_peered":"2024-08-31T12:15:56.293864+0000","last_unstale":"2024-08-31T12:28:37.337439+0000","last_undegraded":"2024-08-31T12:28:37.337439+0000","last_fullsized":"2024-08-31T12:28:37.337439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":229,"log_dups_size":0,"ondisk_log_size":229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:43:17.443433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1325973,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":228,"num_write_kb":1426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":13424230175,"num_objects":3391,"num_object_clones":0,"num_object_copies":6782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":918154,"num_write_kb":16787942,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26849361920,"data_stored":26848441814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":183697,"ondisk_log_size":183697,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":13960395,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":30,"num_read_kb":64,"num_write":3307,"num_write_kb":14790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28073984,"data_stored":27920790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2759,"ondisk_log_size":2759,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509600,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2618576,"kb_used_data":2384832,"kb_used_omap":1,"kb_used_meta":233726,"kb_avail":91753264,"statfs":{"total":96636764160,"available":93955342336,"internally_reserved":0,"allocated":2442067968,"data_stored":2441803901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":239335878},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509599,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4206900,"kb_used_data":3851044,"kb_used_omap":1,"kb_used_meta":355838,"kb_avail":90164940,"statfs":{"total":96636764160,"available":92328898560,"internally_reserved":0,"allocated":3943469056,"data_stored":3943171530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":364378567},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509600,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2544520,"kb_used_data":2306616,"kb_used_omap":1,"kb_used_meta":237886,"kb_avail":91827320,"statfs":{"total":96636764160,"available":94031175680,"internally_reserved":0,"allocated":2361974784,"data_stored":2361727195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":243595717},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509600,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4141744,"kb_used_data":3795616,"kb_used_omap":1,"kb_used_meta":346110,"kb_avail":90230096,"statfs":{"total":96636764160,"available":92395618304,"internally_reserved":0,"allocated":3886710784,"data_stored":3886395634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354417093},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542303,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4422856,"kb_used_data":4066936,"kb_used_omap":1,"kb_used_meta":355902,"kb_avail":89948984,"statfs":{"total":96636764160,"available":92107759616,"internally_reserved":0,"allocated":4164542464,"data_stored":4164215175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":364444106},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542305,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4162248,"kb_used_data":3793912,"kb_used_omap":1,"kb_used_meta":368318,"kb_avail":90209592,"statfs":{"total":96636764160,"available":92374622208,"internally_reserved":0,"allocated":3884965888,"data_stored":3884692848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":377158085},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542303,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4136736,"kb_used_data":3839056,"kb_used_omap":1,"kb_used_meta":297662,"kb_avail":90235104,"statfs":{"total":96636764160,"available":92400746496,"internally_reserved":0,"allocated":3931193344,"data_stored":3930920211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":304806342},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575008,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2442052,"kb_used_data":2212468,"kb_used_omap":1,"kb_used_meta":229566,"kb_avail":91929788,"statfs":{"total":96636764160,"available":94136102912,"internally_reserved":0,"allocated":2265567232,"data_stored":2265299110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":235076037},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1343488,"data_stored":1326051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":48187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8458240,"data_stored":8437802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4210688,"data_stored":4195018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4210688,"data_stored":4195654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":9736192,"data_stored":9715756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3884539904,"data_stored":3884393932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2361716736,"data_stored":2361640968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2265264128,"data_stored":2265165640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3934777344,"data_stored":3934648445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3926749184,"data_stored":3926639910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3884711936,"data_stored":3884606187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2437623808,"data_stored":2437522964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4153978880,"data_stored":4153823768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-31T12:28:49.304 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-31T12:28:49.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.452+0000 7f9c512ab640 1 -- 172.21.15.19:0/3183591795 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9c4c0c9850 msgr2=0x7f9c4c155300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:49.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.452+0000 7f9c512ab640 1 --2- 172.21.15.19:0/3183591795 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9c4c0c9850 0x7f9c4c155300 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c005fd0 tx=0x7f9c3c02a940 comp rx=0 tx=0).stop 2024-08-31T12:28:49.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.452+0000 7f9c512ab640 1 -- 172.21.15.19:0/3183591795 shutdown_connections 2024-08-31T12:28:49.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.452+0000 7f9c512ab640 1 --2- 172.21.15.19:0/3183591795 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9c4c0c9850 0x7f9c4c155300 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:49.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.452+0000 7f9c512ab640 1 -- 172.21.15.19:0/3183591795 >> 172.21.15.19:0/3183591795 conn(0x7f9c4c0690d0 msgr2=0x7f9c4c0d0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:49.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.453+0000 7f9c512ab640 1 -- 172.21.15.19:0/3183591795 shutdown_connections 2024-08-31T12:28:49.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.453+0000 7f9c512ab640 1 -- 172.21.15.19:0/3183591795 wait complete. 2024-08-31T12:28:49.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.454+0000 7f9c512ab640 1 Processor -- start 2024-08-31T12:28:49.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.454+0000 7f9c512ab640 1 -- start start 2024-08-31T12:28:49.459 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.456+0000 7f9c512ab640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9c4c06e0e0 0x7f9c4c0cea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:49.459 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.456+0000 7f9c512ab640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9c4c0c8a40 0x7f9c4c154390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:49.459 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.456+0000 7f9c512ab640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c4c0c9850 0x7f9c4c0c2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:49.459 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.456+0000 7f9c512ab640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f9c4c15b250 con 0x7f9c4c06e0e0 2024-08-31T12:28:49.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.456+0000 7f9c512ab640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9c4c15af70 con 0x7f9c4c0c9850 2024-08-31T12:28:49.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.456+0000 7f9c512ab640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f9c4c0c8610 con 0x7f9c4c0c8a40 2024-08-31T12:28:49.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.456+0000 7f9c4affd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9c4c06e0e0 0x7f9c4c0cea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:49.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.456+0000 7f9c4affd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9c4c06e0e0 0x7f9c4c0cea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:43872/0 (socket says 172.21.15.19:43872) 2024-08-31T12:28:49.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.456+0000 7f9c4affd640 1 -- 172.21.15.19:0/1457476547 learned_addr learned my addr 172.21.15.19:0/1457476547 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:28:49.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.456+0000 7f9c4b7fe640 1 --2- 172.21.15.19:0/1457476547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c4c0c9850 0x7f9c4c0c2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:49.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.456+0000 7f9c4affd640 1 -- 172.21.15.19:0/1457476547 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9c4c0c8a40 msgr2=0x7f9c4c154390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:49.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.456+0000 7f9c4a7fc640 1 --2- 172.21.15.19:0/1457476547 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9c4c0c8a40 0x7f9c4c154390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:49.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.456+0000 7f9c4affd640 1 --2- 172.21.15.19:0/1457476547 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9c4c0c8a40 0x7f9c4c154390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:49.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.456+0000 7f9c4affd640 1 -- 172.21.15.19:0/1457476547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c4c0c9850 msgr2=0x7f9c4c0c2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:49.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.456+0000 7f9c4affd640 1 --2- 172.21.15.19:0/1457476547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c4c0c9850 0x7f9c4c0c2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:49.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.456+0000 7f9c4affd640 1 -- 172.21.15.19:0/1457476547 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c4c0cef60 con 0x7f9c4c06e0e0 2024-08-31T12:28:49.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.456+0000 7f9c4b7fe640 1 --2- 172.21.15.19:0/1457476547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c4c0c9850 0x7f9c4c0c2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-31T12:28:49.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.456+0000 7f9c4affd640 1 --2- 172.21.15.19:0/1457476547 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9c4c06e0e0 0x7f9c4c0cea20 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f9c4400b3f0 tx=0x7f9c4400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:49.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.456+0000 7f9c437fe640 1 -- 172.21.15.19:0/1457476547 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c44015020 con 0x7f9c4c06e0e0 2024-08-31T12:28:49.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.456+0000 7f9c437fe640 1 -- 172.21.15.19:0/1457476547 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9c44011920 con 0x7f9c4c06e0e0 2024-08-31T12:28:49.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.456+0000 7f9c512ab640 1 -- 172.21.15.19:0/1457476547 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c4c1f13e0 con 0x7f9c4c06e0e0 2024-08-31T12:28:49.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.456+0000 7f9c437fe640 1 -- 172.21.15.19:0/1457476547 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c44011d70 con 0x7f9c4c06e0e0 2024-08-31T12:28:49.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.456+0000 7f9c512ab640 1 -- 172.21.15.19:0/1457476547 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c4c1f1c60 con 0x7f9c4c06e0e0 2024-08-31T12:28:49.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.458+0000 7f9c512ab640 1 -- 172.21.15.19:0/1457476547 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c4c155300 con 0x7f9c4c06e0e0 2024-08-31T12:28:49.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.458+0000 7f9c437fe640 1 -- 172.21.15.19:0/1457476547 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f9c44018020 con 0x7f9c4c06e0e0 2024-08-31T12:28:49.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.459+0000 7f9c437fe640 1 --2- 172.21.15.19:0/1457476547 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9c2c077a50 0x7f9c2c079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:49.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.459+0000 7f9c437fe640 1 -- 172.21.15.19:0/1457476547 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f9c440996a0 con 0x7f9c4c06e0e0 2024-08-31T12:28:49.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.459+0000 7f9c4a7fc640 1 --2- 172.21.15.19:0/1457476547 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9c2c077a50 0x7f9c2c079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:49.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.460+0000 7f9c4a7fc640 1 --2- 172.21.15.19:0/1457476547 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9c2c077a50 0x7f9c2c079f60 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c158910 tx=0x7f9c34009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:49.465 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.463+0000 7f9c437fe640 1 -- 172.21.15.19:0/1457476547 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f9c4401f020 con 0x7f9c4c06e0e0 2024-08-31T12:28:49.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.649+0000 7f9c512ab640 1 -- 172.21.15.19:0/1457476547 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9c4c0ccf20 con 0x7f9c4c06e0e0 2024-08-31T12:28:49.665 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.663+0000 7f9c437fe640 1 -- 172.21.15.19:0/1457476547 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) ==== 74+0+16866 (secure 0 0 0) 0x7f9c44062390 con 0x7f9c4c06e0e0 2024-08-31T12:28:49.665 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:28:49.666 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":23,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","created":"2024-08-31T12:15:32.084309+0000","modified":"2024-08-31T12:16:40.637326+0000","last_up_change":"2024-08-31T12:15:45.213688+0000","last_in_change":"2024-08-31T12:15:35.545202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-31T12:15:44.508197+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-31T12:15:54.512584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-31T12:15:55.618251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b80d6a3f-7e4e-4d4d-b1b4-0df71c9176ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6800","nonce":680447146},{"type":"v1","addr":"172.21.15.19:6801","nonce":680447146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":680447146},{"type":"v1","addr":"172.21.15.19:6803","nonce":680447146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":680447146},{"type":"v1","addr":"172.21.15.19:6807","nonce":680447146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":680447146},{"type":"v1","addr":"172.21.15.19:6805","nonce":680447146}]},"public_addr":"172.21.15.19:6801/680447146","cluster_addr":"172.21.15.19:6803/680447146","heartbeat_back_addr":"172.21.15.19:6807/680447146","heartbeat_front_addr":"172.21.15.19:6805/680447146","state":["exists","up"]},{"osd":1,"uuid":"6f986366-82a6-4133-9f2e-43f84c7b03bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":1522610034},{"type":"v1","addr":"172.21.15.19:6817","nonce":1522610034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":1522610034},{"type":"v1","addr":"172.21.15.19:6819","nonce":1522610034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":1522610034},{"type":"v1","addr":"172.21.15.19:6823","nonce":1522610034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":1522610034},{"type":"v1","addr":"172.21.15.19:6821","nonce":1522610034}]},"public_addr":"172.21.15.19:6817/1522610034","cluster_addr":"172.21.15.19:6819/1522610034","heartbeat_back_addr":"172.21.15.19:6823/1522610034","heartbeat_front_addr":"172.21.15.19:6821/1522610034","state":["exists","up"]},{"osd":2,"uuid":"ce8c2068-845c-4cbb-a4d1-077375d605ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6809","nonce":2332368824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6811","nonce":2332368824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6815","nonce":2332368824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6813","nonce":2332368824}]},"public_addr":"172.21.15.19:6809/2332368824","cluster_addr":"172.21.15.19:6811/2332368824","heartbeat_back_addr":"172.21.15.19:6815/2332368824","heartbeat_front_addr":"172.21.15.19:6813/2332368824","state":["exists","up"]},{"osd":3,"uuid":"b1cb0ef4-aa9b-421b-a94e-82de0c46e5cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":3785550598},{"type":"v1","addr":"172.21.15.19:6825","nonce":3785550598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":3785550598},{"type":"v1","addr":"172.21.15.19:6827","nonce":3785550598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":3785550598},{"type":"v1","addr":"172.21.15.19:6831","nonce":3785550598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":3785550598},{"type":"v1","addr":"172.21.15.19:6829","nonce":3785550598}]},"public_addr":"172.21.15.19:6825/3785550598","cluster_addr":"172.21.15.19:6827/3785550598","heartbeat_back_addr":"172.21.15.19:6831/3785550598","heartbeat_front_addr":"172.21.15.19:6829/3785550598","state":["exists","up"]},{"osd":4,"uuid":"825b58ec-a6da-4515-b6a5-81debe327011","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":1985519349},{"type":"v1","addr":"172.21.15.43:6819","nonce":1985519349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":1985519349},{"type":"v1","addr":"172.21.15.43:6821","nonce":1985519349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":1985519349},{"type":"v1","addr":"172.21.15.43:6825","nonce":1985519349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":1985519349},{"type":"v1","addr":"172.21.15.43:6823","nonce":1985519349}]},"public_addr":"172.21.15.43:6819/1985519349","cluster_addr":"172.21.15.43:6821/1985519349","heartbeat_back_addr":"172.21.15.43:6825/1985519349","heartbeat_front_addr":"172.21.15.43:6823/1985519349","state":["exists","up"]},{"osd":5,"uuid":"7f34e537-5997-40e4-92e7-2570507160c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":4271971521},{"type":"v1","addr":"172.21.15.43:6803","nonce":4271971521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":4271971521},{"type":"v1","addr":"172.21.15.43:6805","nonce":4271971521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":4271971521},{"type":"v1","addr":"172.21.15.43:6809","nonce":4271971521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":4271971521},{"type":"v1","addr":"172.21.15.43:6807","nonce":4271971521}]},"public_addr":"172.21.15.43:6803/4271971521","cluster_addr":"172.21.15.43:6805/4271971521","heartbeat_back_addr":"172.21.15.43:6809/4271971521","heartbeat_front_addr":"172.21.15.43:6807/4271971521","state":["exists","up"]},{"osd":6,"uuid":"195dd8ad-640a-4fa3-9ee4-06e4172e0464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2863384694},{"type":"v1","addr":"172.21.15.43:6827","nonce":2863384694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2863384694},{"type":"v1","addr":"172.21.15.43:6829","nonce":2863384694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":2863384694},{"type":"v1","addr":"172.21.15.43:6833","nonce":2863384694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2863384694},{"type":"v1","addr":"172.21.15.43:6831","nonce":2863384694}]},"public_addr":"172.21.15.43:6827/2863384694","cluster_addr":"172.21.15.43:6829/2863384694","heartbeat_back_addr":"172.21.15.43:6833/2863384694","heartbeat_front_addr":"172.21.15.43:6831/2863384694","state":["exists","up"]},{"osd":7,"uuid":"4645661f-f430-4f05-bdd1-69138dcd1bbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":792572234},{"type":"v1","addr":"172.21.15.43:6811","nonce":792572234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":792572234},{"type":"v1","addr":"172.21.15.43:6813","nonce":792572234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":792572234},{"type":"v1","addr":"172.21.15.43:6817","nonce":792572234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":792572234},{"type":"v1","addr":"172.21.15.43:6815","nonce":792572234}]},"public_addr":"172.21.15.43:6811/792572234","cluster_addr":"172.21.15.43:6813/792572234","heartbeat_back_addr":"172.21.15.43:6817/792572234","heartbeat_front_addr":"172.21.15.43:6815/792572234","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.156485+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.578126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.075440+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.639867+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.543602+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:41.919592+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.615557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:41.813337+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-31T12:28:49.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.665+0000 7f9c512ab640 1 -- 172.21.15.19:0/1457476547 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9c2c077a50 msgr2=0x7f9c2c079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:49.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.665+0000 7f9c512ab640 1 --2- 172.21.15.19:0/1457476547 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9c2c077a50 0x7f9c2c079f60 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c158910 tx=0x7f9c34009040 comp rx=0 tx=0).stop 2024-08-31T12:28:49.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.665+0000 7f9c512ab640 1 -- 172.21.15.19:0/1457476547 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9c4c06e0e0 msgr2=0x7f9c4c0cea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:49.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.665+0000 7f9c512ab640 1 --2- 172.21.15.19:0/1457476547 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9c4c06e0e0 0x7f9c4c0cea20 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f9c4400b3f0 tx=0x7f9c4400b8b0 comp rx=0 tx=0).stop 2024-08-31T12:28:49.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.665+0000 7f9c512ab640 1 -- 172.21.15.19:0/1457476547 shutdown_connections 2024-08-31T12:28:49.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.665+0000 7f9c512ab640 1 --2- 172.21.15.19:0/1457476547 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f9c2c077a50 0x7f9c2c079f60 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:49.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.665+0000 7f9c512ab640 1 --2- 172.21.15.19:0/1457476547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c4c0c9850 0x7f9c4c0c2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:49.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.665+0000 7f9c512ab640 1 --2- 172.21.15.19:0/1457476547 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9c4c0c8a40 0x7f9c4c154390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:49.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.665+0000 7f9c512ab640 1 --2- 172.21.15.19:0/1457476547 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9c4c06e0e0 0x7f9c4c0cea20 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:49.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.665+0000 7f9c512ab640 1 -- 172.21.15.19:0/1457476547 >> 172.21.15.19:0/1457476547 conn(0x7f9c4c0690d0 msgr2=0x7f9c4c15a740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:49.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.666+0000 7f9c512ab640 1 -- 172.21.15.19:0/1457476547 shutdown_connections 2024-08-31T12:28:49.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:49.666+0000 7f9c512ab640 1 -- 172.21.15.19:0/1457476547 wait complete. 2024-08-31T12:28:50.680 INFO:tasks.ceph:Scrubbing osd.0 2024-08-31T12:28:50.681 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-08-31T12:28:50.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.812+0000 7fd01f9e4640 1 -- 172.21.15.19:0/1917823975 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd0180ccd30 msgr2=0x7fd0180cd140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:50.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.812+0000 7fd01f9e4640 1 --2- 172.21.15.19:0/1917823975 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd0180ccd30 0x7fd0180cd140 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd01806b190 tx=0x7fd00c02a5f0 comp rx=0 tx=0).stop 2024-08-31T12:28:50.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.812+0000 7fd01f9e4640 1 -- 172.21.15.19:0/1917823975 shutdown_connections 2024-08-31T12:28:50.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.812+0000 7fd01f9e4640 1 --2- 172.21.15.19:0/1917823975 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd0180ccd30 0x7fd0180cd140 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:50.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.812+0000 7fd01f9e4640 1 --2- 172.21.15.19:0/1917823975 >> v2:172.21.15.43:3300/0 conn(0x7fd0180c6a90 0x7fd018151e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:50.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.812+0000 7fd01f9e4640 1 -- 172.21.15.19:0/1917823975 >> 172.21.15.19:0/1917823975 conn(0x7fd01806cf00 msgr2=0x7fd01806d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:50.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.812+0000 7fd01f9e4640 1 -- 172.21.15.19:0/1917823975 shutdown_connections 2024-08-31T12:28:50.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.812+0000 7fd01f9e4640 1 -- 172.21.15.19:0/1917823975 wait complete. 2024-08-31T12:28:50.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.814+0000 7fd01f9e4640 1 Processor -- start 2024-08-31T12:28:50.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.814+0000 7fd01f9e4640 1 -- start start 2024-08-31T12:28:50.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.815+0000 7fd01f9e4640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd0180c4020 0x7fd0181ef900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:50.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.815+0000 7fd01f9e4640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd0180c6a90 0x7fd0181e3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:50.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.815+0000 7fd01f9e4640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd0180ccd30 0x7fd0181401e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:50.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.815+0000 7fd01f9e4640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fd018072f00 con 0x7fd0180c6a90 2024-08-31T12:28:50.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.815+0000 7fd01f9e4640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd0181523d0 con 0x7fd0180ccd30 2024-08-31T12:28:50.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.815+0000 7fd01f9e4640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fd0180c9040 con 0x7fd0180c4020 2024-08-31T12:28:50.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.816+0000 7fd01cf7f640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd0180c6a90 0x7fd0181e3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:50.820 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.816+0000 7fd01cf7f640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd0180c6a90 0x7fd0181e3420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:43902/0 (socket says 172.21.15.19:43902) 2024-08-31T12:28:50.820 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.816+0000 7fd01cf7f640 1 -- 172.21.15.19:0/2173396318 learned_addr learned my addr 172.21.15.19:0/2173396318 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:28:50.820 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.816+0000 7fd01d780640 1 --2- 172.21.15.19:0/2173396318 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd0180c4020 0x7fd0181ef900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:50.820 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.816+0000 7fd01df81640 1 --2- 172.21.15.19:0/2173396318 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd0180ccd30 0x7fd0181401e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:50.820 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.816+0000 7fd01cf7f640 1 -- 172.21.15.19:0/2173396318 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd0180c4020 msgr2=0x7fd0181ef900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:50.820 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.816+0000 7fd01cf7f640 1 --2- 172.21.15.19:0/2173396318 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd0180c4020 0x7fd0181ef900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:50.820 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.816+0000 7fd01cf7f640 1 -- 172.21.15.19:0/2173396318 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd0180ccd30 msgr2=0x7fd0181401e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:50.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.816+0000 7fd01cf7f640 1 --2- 172.21.15.19:0/2173396318 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd0180ccd30 0x7fd0181401e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:50.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.816+0000 7fd01cf7f640 1 -- 172.21.15.19:0/2173396318 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0181f3660 con 0x7fd0180c6a90 2024-08-31T12:28:50.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.816+0000 7fd01cf7f640 1 --2- 172.21.15.19:0/2173396318 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd0180c6a90 0x7fd0181e3420 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd00c000c00 tx=0x7fd00c003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:50.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.816+0000 7fd015ffb640 1 -- 172.21.15.19:0/2173396318 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd00c03e030 con 0x7fd0180c6a90 2024-08-31T12:28:50.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.816+0000 7fd015ffb640 1 -- 172.21.15.19:0/2173396318 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd00c003e00 con 0x7fd0180c6a90 2024-08-31T12:28:50.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.816+0000 7fd01f9e4640 1 -- 172.21.15.19:0/2173396318 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0181ef170 con 0x7fd0180c6a90 2024-08-31T12:28:50.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.816+0000 7fd015ffb640 1 -- 172.21.15.19:0/2173396318 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd00c035040 con 0x7fd0180c6a90 2024-08-31T12:28:50.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.817+0000 7fd01f9e4640 1 -- 172.21.15.19:0/2173396318 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0181e0970 con 0x7fd0180c6a90 2024-08-31T12:28:50.823 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.817+0000 7fd01f9e4640 1 -- 172.21.15.19:0/2173396318 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd0180c3720 con 0x7fd0180c6a90 2024-08-31T12:28:50.823 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.818+0000 7fd015ffb640 1 -- 172.21.15.19:0/2173396318 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fd00c041030 con 0x7fd0180c6a90 2024-08-31T12:28:50.823 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.819+0000 7fd015ffb640 1 --2- 172.21.15.19:0/2173396318 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd004077a80 0x7fd004079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:50.823 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.819+0000 7fd015ffb640 1 -- 172.21.15.19:0/2173396318 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fd00c0ba4f0 con 0x7fd0180c6a90 2024-08-31T12:28:50.823 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.819+0000 7fd015ffb640 1 --2- 172.21.15.19:0/2173396318 >> [v2:172.21.15.19:6800/680447146,v1:172.21.15.19:6801/680447146] conn(0x7fd00407fe50 0x7fd004082320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:50.823 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.819+0000 7fd01d780640 1 --2- 172.21.15.19:0/2173396318 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd004077a80 0x7fd004079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:50.823 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.819+0000 7fd015ffb640 1 -- 172.21.15.19:0/2173396318 --> [v2:172.21.15.19:6800/680447146,v1:172.21.15.19:6801/680447146] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd00c005310 con 0x7fd00407fe50 2024-08-31T12:28:50.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.819+0000 7fd015ffb640 1 -- 172.21.15.19:0/2173396318 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fd00c0ba8a0 con 0x7fd0180c6a90 2024-08-31T12:28:50.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.819+0000 7fd01df81640 1 --2- 172.21.15.19:0/2173396318 >> [v2:172.21.15.19:6800/680447146,v1:172.21.15.19:6801/680447146] conn(0x7fd00407fe50 0x7fd004082320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:50.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.820+0000 7fd01df81640 1 --2- 172.21.15.19:0/2173396318 >> [v2:172.21.15.19:6800/680447146,v1:172.21.15.19:6801/680447146] conn(0x7fd00407fe50 0x7fd004082320 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:50.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.820+0000 7fd01d780640 1 --2- 172.21.15.19:0/2173396318 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd004077a80 0x7fd004079f90 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fd008005fd0 tx=0x7fd0080095e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:50.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.820+0000 7fd015ffb640 1 -- 172.21.15.19:0/2173396318 <== osd.0 v2:172.21.15.19:6800/680447146 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd00c005310 con 0x7fd00407fe50 2024-08-31T12:28:50.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.833+0000 7fd01f9e4640 1 -- 172.21.15.19:0/2173396318 --> [v2:172.21.15.19:6800/680447146,v1:172.21.15.19:6801/680447146] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fd0181f3360 con 0x7fd00407fe50 2024-08-31T12:28:50.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.834+0000 7fd015ffb640 1 -- 172.21.15.19:0/2173396318 <== osd.0 v2:172.21.15.19:6800/680447146 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fd0181f3360 con 0x7fd00407fe50 2024-08-31T12:28:50.836 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:28:50.836 INFO:teuthology.orchestra.run.smithi019.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-08-31T12:28:50.836 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:28:50.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.834+0000 7fd01f9e4640 1 -- 172.21.15.19:0/2173396318 >> [v2:172.21.15.19:6800/680447146,v1:172.21.15.19:6801/680447146] conn(0x7fd00407fe50 msgr2=0x7fd004082320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:50.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.834+0000 7fd01f9e4640 1 --2- 172.21.15.19:0/2173396318 >> [v2:172.21.15.19:6800/680447146,v1:172.21.15.19:6801/680447146] conn(0x7fd00407fe50 0x7fd004082320 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:50.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.834+0000 7fd01f9e4640 1 -- 172.21.15.19:0/2173396318 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd004077a80 msgr2=0x7fd004079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:50.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.834+0000 7fd01f9e4640 1 --2- 172.21.15.19:0/2173396318 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fd004077a80 0x7fd004079f90 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fd008005fd0 tx=0x7fd0080095e0 comp rx=0 tx=0).stop 2024-08-31T12:28:50.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.834+0000 7fd01f9e4640 1 -- 172.21.15.19:0/2173396318 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd0180c6a90 msgr2=0x7fd0181e3420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:50.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.834+0000 7fd01f9e4640 1 --2- 172.21.15.19:0/2173396318 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd0180c6a90 0x7fd0181e3420 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd00c000c00 tx=0x7fd00c003910 comp rx=0 tx=0).stop 2024-08-31T12:28:50.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.834+0000 7fd01df81640 1 -- 172.21.15.19:0/2173396318 reap_dead start 2024-08-31T12:28:50.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.834+0000 7fd01f9e4640 1 -- 172.21.15.19:0/2173396318 shutdown_connections 2024-08-31T12:28:50.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.834+0000 7fd01f9e4640 1 -- 172.21.15.19:0/2173396318 >> 172.21.15.19:0/2173396318 conn(0x7fd01806cf00 msgr2=0x7fd01806f2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:50.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.834+0000 7fd01f9e4640 1 -- 172.21.15.19:0/2173396318 shutdown_connections 2024-08-31T12:28:50.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.834+0000 7fd01f9e4640 1 -- 172.21.15.19:0/2173396318 wait complete. 2024-08-31T12:28:50.847 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-08-31T12:28:50.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.960+0000 7f0bf961d640 1 -- 172.21.15.19:0/1565900003 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0bf40c8a40 msgr2=0x7f0bf40c9850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:50.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.960+0000 7f0bf961d640 1 --2- 172.21.15.19:0/1565900003 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0bf40c8a40 0x7f0bf40c9850 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0be4005fd0 tx=0x7f0be402ab20 comp rx=0 tx=0).stop 2024-08-31T12:28:50.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.960+0000 7f0bf961d640 1 -- 172.21.15.19:0/1565900003 shutdown_connections 2024-08-31T12:28:50.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.960+0000 7f0bf961d640 1 --2- 172.21.15.19:0/1565900003 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0bf4155300 0x7f0bf406d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:50.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.960+0000 7f0bf961d640 1 --2- 172.21.15.19:0/1565900003 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0bf40c8a40 0x7f0bf40c9850 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:50.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.960+0000 7f0bf961d640 1 -- 172.21.15.19:0/1565900003 >> 172.21.15.19:0/1565900003 conn(0x7f0bf40690d0 msgr2=0x7f0bf40d0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:50.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.960+0000 7f0bf961d640 1 -- 172.21.15.19:0/1565900003 shutdown_connections 2024-08-31T12:28:50.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.960+0000 7f0bf961d640 1 -- 172.21.15.19:0/1565900003 wait complete. 2024-08-31T12:28:50.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.961+0000 7f0bf961d640 1 Processor -- start 2024-08-31T12:28:50.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.962+0000 7f0bf961d640 1 -- start start 2024-08-31T12:28:50.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.963+0000 7f0bf961d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0bf406e0e0 0x7f0bf40c7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:50.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.963+0000 7f0bf961d640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0bf40c8a40 0x7f0bf41f1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:50.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.963+0000 7f0bf961d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0bf4155300 0x7f0bf40ce640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:50.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.963+0000 7f0bf961d640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf40c8330 con 0x7f0bf4155300 2024-08-31T12:28:50.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.963+0000 7f0bf961d640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf415b250 con 0x7f0bf40c8a40 2024-08-31T12:28:50.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.963+0000 7f0bf961d640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f0bf40c8610 con 0x7f0bf406e0e0 2024-08-31T12:28:50.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.963+0000 7f0bf37fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0bf4155300 0x7f0bf40ce640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:50.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.963+0000 7f0bf37fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0bf4155300 0x7f0bf40ce640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:43912/0 (socket says 172.21.15.19:43912) 2024-08-31T12:28:50.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.963+0000 7f0bf37fe640 1 -- 172.21.15.19:0/2201605684 learned_addr learned my addr 172.21.15.19:0/2201605684 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:28:50.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.963+0000 7f0bf27fc640 1 --2- 172.21.15.19:0/2201605684 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0bf40c8a40 0x7f0bf41f1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:50.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.963+0000 7f0bf2ffd640 1 --2- 172.21.15.19:0/2201605684 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0bf406e0e0 0x7f0bf40c7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:50.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.963+0000 7f0bf37fe640 1 -- 172.21.15.19:0/2201605684 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0bf406e0e0 msgr2=0x7f0bf40c7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:50.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.963+0000 7f0bf37fe640 1 --2- 172.21.15.19:0/2201605684 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0bf406e0e0 0x7f0bf40c7030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:50.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.963+0000 7f0bf37fe640 1 -- 172.21.15.19:0/2201605684 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0bf40c8a40 msgr2=0x7f0bf41f1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:50.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.963+0000 7f0bf37fe640 1 --2- 172.21.15.19:0/2201605684 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0bf40c8a40 0x7f0bf41f1f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:50.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.963+0000 7f0bf37fe640 1 -- 172.21.15.19:0/2201605684 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bf40cf5b0 con 0x7f0bf4155300 2024-08-31T12:28:50.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.963+0000 7f0bf37fe640 1 --2- 172.21.15.19:0/2201605684 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0bf4155300 0x7f0bf40ce640 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0bec00e6a0 tx=0x7f0bec00eb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:50.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.964+0000 7f0bdb7fe640 1 -- 172.21.15.19:0/2201605684 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bec00cab0 con 0x7f0bf4155300 2024-08-31T12:28:50.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.964+0000 7f0bdb7fe640 1 -- 172.21.15.19:0/2201605684 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0bec00cc50 con 0x7f0bf4155300 2024-08-31T12:28:50.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.964+0000 7f0bf961d640 1 -- 172.21.15.19:0/2201605684 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bf40cfb90 con 0x7f0bf4155300 2024-08-31T12:28:50.969 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.964+0000 7f0bdb7fe640 1 -- 172.21.15.19:0/2201605684 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bec005280 con 0x7f0bf4155300 2024-08-31T12:28:50.969 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.964+0000 7f0bf961d640 1 -- 172.21.15.19:0/2201605684 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bf4154180 con 0x7f0bf4155300 2024-08-31T12:28:50.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.965+0000 7f0bf961d640 1 -- 172.21.15.19:0/2201605684 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bf40c9850 con 0x7f0bf4155300 2024-08-31T12:28:50.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.966+0000 7f0bdb7fe640 1 -- 172.21.15.19:0/2201605684 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f0bec005420 con 0x7f0bf4155300 2024-08-31T12:28:50.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.966+0000 7f0bdb7fe640 1 --2- 172.21.15.19:0/2201605684 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0be0077a50 0x7f0be0079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:50.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.966+0000 7f0bdb7fe640 1 -- 172.21.15.19:0/2201605684 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f0bec099f60 con 0x7f0bf4155300 2024-08-31T12:28:50.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.967+0000 7f0bf2ffd640 1 --2- 172.21.15.19:0/2201605684 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0be0077a50 0x7f0be0079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:50.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.967+0000 7f0bf2ffd640 1 --2- 172.21.15.19:0/2201605684 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0be0077a50 0x7f0be0079f60 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f0be4035000 tx=0x7f0be4038040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:50.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:50.971+0000 7f0bdb7fe640 1 -- 172.21.15.19:0/2201605684 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f0bec00ce20 con 0x7f0bf4155300 2024-08-31T12:28:51.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.154+0000 7f0bf961d640 1 -- 172.21.15.19:0/2201605684 --> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f0bf40ccf20 con 0x7f0be0077a50 2024-08-31T12:28:51.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.155+0000 7f0bdb7fe640 1 -- 172.21.15.19:0/2201605684 <== mgr.4100 v2:172.21.15.43:6800/2824336655 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f0bf40ccf20 con 0x7f0be0077a50 2024-08-31T12:28:51.157 INFO:teuthology.orchestra.run.smithi019.stderr:instructed osd(s) 0 to deep-scrub 2024-08-31T12:28:51.159 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.157+0000 7f0bf961d640 1 -- 172.21.15.19:0/2201605684 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0be0077a50 msgr2=0x7f0be0079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:51.159 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.157+0000 7f0bf961d640 1 --2- 172.21.15.19:0/2201605684 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0be0077a50 0x7f0be0079f60 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f0be4035000 tx=0x7f0be4038040 comp rx=0 tx=0).stop 2024-08-31T12:28:51.159 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.157+0000 7f0bf961d640 1 -- 172.21.15.19:0/2201605684 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0bf4155300 msgr2=0x7f0bf40ce640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:51.159 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.157+0000 7f0bf961d640 1 --2- 172.21.15.19:0/2201605684 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0bf4155300 0x7f0bf40ce640 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0bec00e6a0 tx=0x7f0bec00eb60 comp rx=0 tx=0).stop 2024-08-31T12:28:51.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.157+0000 7f0bf961d640 1 -- 172.21.15.19:0/2201605684 shutdown_connections 2024-08-31T12:28:51.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.157+0000 7f0bf961d640 1 --2- 172.21.15.19:0/2201605684 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0be0077a50 0x7f0be0079f60 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:51.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.157+0000 7f0bf961d640 1 --2- 172.21.15.19:0/2201605684 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0bf4155300 0x7f0bf40ce640 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:51.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.157+0000 7f0bf961d640 1 --2- 172.21.15.19:0/2201605684 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0bf40c8a40 0x7f0bf41f1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:51.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.157+0000 7f0bf961d640 1 --2- 172.21.15.19:0/2201605684 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0bf406e0e0 0x7f0bf40c7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:51.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.157+0000 7f0bf961d640 1 -- 172.21.15.19:0/2201605684 >> 172.21.15.19:0/2201605684 conn(0x7f0bf40690d0 msgr2=0x7f0bf4150c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:51.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.157+0000 7f0bf961d640 1 -- 172.21.15.19:0/2201605684 shutdown_connections 2024-08-31T12:28:51.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.157+0000 7f0bf961d640 1 -- 172.21.15.19:0/2201605684 wait complete. 2024-08-31T12:28:51.171 INFO:tasks.ceph:Scrubbing osd.1 2024-08-31T12:28:51.171 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-08-31T12:28:51.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.288+0000 7f419da8b640 1 -- 172.21.15.19:0/2727379246 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f41980cbcf0 msgr2=0x7f41980cc150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:51.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.288+0000 7f419da8b640 1 --2- 172.21.15.19:0/2727379246 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f41980cbcf0 0x7f41980cc150 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4188009820 tx=0x7f418802a5a0 comp rx=0 tx=0).stop 2024-08-31T12:28:51.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.288+0000 7f419da8b640 1 -- 172.21.15.19:0/2727379246 shutdown_connections 2024-08-31T12:28:51.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.288+0000 7f419da8b640 1 --2- 172.21.15.19:0/2727379246 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f419815e830 0x7f419815ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:51.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.288+0000 7f419da8b640 1 --2- 172.21.15.19:0/2727379246 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f41980cbcf0 0x7f41980cc150 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:51.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.288+0000 7f419da8b640 1 -- 172.21.15.19:0/2727379246 >> 172.21.15.19:0/2727379246 conn(0x7f4198074210 msgr2=0x7f4198074620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:51.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.288+0000 7f419da8b640 1 -- 172.21.15.19:0/2727379246 shutdown_connections 2024-08-31T12:28:51.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.288+0000 7f419da8b640 1 -- 172.21.15.19:0/2727379246 wait complete. 2024-08-31T12:28:51.293 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.290+0000 7f419da8b640 1 Processor -- start 2024-08-31T12:28:51.293 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.290+0000 7f419da8b640 1 -- start start 2024-08-31T12:28:51.295 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.291+0000 7f419da8b640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41980ca6c0 0x7f41980b3110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:51.295 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.291+0000 7f419da8b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f41980cbcf0 0x7f41981e5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:51.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.291+0000 7f419da8b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f419815e830 0x7f41981f1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:51.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.291+0000 7f419da8b640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f41980b4680 con 0x7f41980cbcf0 2024-08-31T12:28:51.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.291+0000 7f419da8b640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f419815f180 con 0x7f41980ca6c0 2024-08-31T12:28:51.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.291+0000 7f419da8b640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f41980b3880 con 0x7f419815e830 2024-08-31T12:28:51.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.292+0000 7f4196ffd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f41980cbcf0 0x7f41981e5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:51.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.292+0000 7f4196ffd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f41980cbcf0 0x7f41981e5db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:43926/0 (socket says 172.21.15.19:43926) 2024-08-31T12:28:51.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.292+0000 7f4196ffd640 1 -- 172.21.15.19:0/2884934425 learned_addr learned my addr 172.21.15.19:0/2884934425 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:28:51.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.292+0000 7f41977fe640 1 --2- 172.21.15.19:0/2884934425 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41980ca6c0 0x7f41980b3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:51.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.292+0000 7f4197fff640 1 --2- 172.21.15.19:0/2884934425 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f419815e830 0x7f41981f1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:51.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.292+0000 7f4196ffd640 1 -- 172.21.15.19:0/2884934425 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f419815e830 msgr2=0x7f41981f1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:51.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.292+0000 7f4196ffd640 1 --2- 172.21.15.19:0/2884934425 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f419815e830 0x7f41981f1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:51.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.292+0000 7f4196ffd640 1 -- 172.21.15.19:0/2884934425 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41980ca6c0 msgr2=0x7f41980b3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:51.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.292+0000 7f4196ffd640 1 --2- 172.21.15.19:0/2884934425 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41980ca6c0 0x7f41980b3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:51.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.292+0000 7f4196ffd640 1 -- 172.21.15.19:0/2884934425 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41981f1b40 con 0x7f41980cbcf0 2024-08-31T12:28:51.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.292+0000 7f4196ffd640 1 --2- 172.21.15.19:0/2884934425 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f41980cbcf0 0x7f41981e5db0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f418c00c3c0 tx=0x7f418c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:51.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.292+0000 7f417ffff640 1 -- 172.21.15.19:0/2884934425 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f418c017070 con 0x7f41980cbcf0 2024-08-31T12:28:51.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.292+0000 7f417ffff640 1 -- 172.21.15.19:0/2884934425 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f418c0035f0 con 0x7f41980cbcf0 2024-08-31T12:28:51.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.292+0000 7f417ffff640 1 -- 172.21.15.19:0/2884934425 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f418c003a20 con 0x7f41980cbcf0 2024-08-31T12:28:51.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.292+0000 7f419da8b640 1 -- 172.21.15.19:0/2884934425 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4198154de0 con 0x7f41980cbcf0 2024-08-31T12:28:51.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.292+0000 7f419da8b640 1 -- 172.21.15.19:0/2884934425 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4198155080 con 0x7f41980cbcf0 2024-08-31T12:28:51.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.293+0000 7f419da8b640 1 -- 172.21.15.19:0/2884934425 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f41980caad0 con 0x7f41980cbcf0 2024-08-31T12:28:51.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.294+0000 7f417ffff640 1 -- 172.21.15.19:0/2884934425 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f418c01a070 con 0x7f41980cbcf0 2024-08-31T12:28:51.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.294+0000 7f417ffff640 1 --2- 172.21.15.19:0/2884934425 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4184077a80 0x7f4184079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:51.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.295+0000 7f417ffff640 1 -- 172.21.15.19:0/2884934425 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f418c098990 con 0x7f41980cbcf0 2024-08-31T12:28:51.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.295+0000 7f417ffff640 1 --2- 172.21.15.19:0/2884934425 >> [v2:172.21.15.19:6816/1522610034,v1:172.21.15.19:6817/1522610034] conn(0x7f418407fe50 0x7f4184082320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:51.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.295+0000 7f417ffff640 1 -- 172.21.15.19:0/2884934425 --> [v2:172.21.15.19:6816/1522610034,v1:172.21.15.19:6817/1522610034] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f41840829f0 con 0x7f418407fe50 2024-08-31T12:28:51.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.295+0000 7f417ffff640 1 -- 172.21.15.19:0/2884934425 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f418c09d2c0 con 0x7f41980cbcf0 2024-08-31T12:28:51.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.295+0000 7f41977fe640 1 --2- 172.21.15.19:0/2884934425 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4184077a80 0x7f4184079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:51.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.295+0000 7f4197fff640 1 --2- 172.21.15.19:0/2884934425 >> [v2:172.21.15.19:6816/1522610034,v1:172.21.15.19:6817/1522610034] conn(0x7f418407fe50 0x7f4184082320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:51.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.295+0000 7f4197fff640 1 --2- 172.21.15.19:0/2884934425 >> [v2:172.21.15.19:6816/1522610034,v1:172.21.15.19:6817/1522610034] conn(0x7f418407fe50 0x7f4184082320 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:51.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.295+0000 7f41977fe640 1 --2- 172.21.15.19:0/2884934425 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4184077a80 0x7f4184079f90 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f4188034000 tx=0x7f4188004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:51.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.296+0000 7f417ffff640 1 -- 172.21.15.19:0/2884934425 <== osd.1 v2:172.21.15.19:6816/1522610034 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f41840829f0 con 0x7f418407fe50 2024-08-31T12:28:51.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.308+0000 7f419da8b640 1 -- 172.21.15.19:0/2884934425 --> [v2:172.21.15.19:6816/1522610034,v1:172.21.15.19:6817/1522610034] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f419815c530 con 0x7f418407fe50 2024-08-31T12:28:51.311 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.309+0000 7f417ffff640 1 -- 172.21.15.19:0/2884934425 <== osd.1 v2:172.21.15.19:6816/1522610034 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f419815c530 con 0x7f418407fe50 2024-08-31T12:28:51.311 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:28:51.311 INFO:teuthology.orchestra.run.smithi019.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-08-31T12:28:51.311 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:28:51.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.309+0000 7f419da8b640 1 -- 172.21.15.19:0/2884934425 >> [v2:172.21.15.19:6816/1522610034,v1:172.21.15.19:6817/1522610034] conn(0x7f418407fe50 msgr2=0x7f4184082320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:51.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.309+0000 7f419da8b640 1 --2- 172.21.15.19:0/2884934425 >> [v2:172.21.15.19:6816/1522610034,v1:172.21.15.19:6817/1522610034] conn(0x7f418407fe50 0x7f4184082320 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:51.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.309+0000 7f419da8b640 1 -- 172.21.15.19:0/2884934425 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4184077a80 msgr2=0x7f4184079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:51.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.309+0000 7f419da8b640 1 --2- 172.21.15.19:0/2884934425 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f4184077a80 0x7f4184079f90 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f4188034000 tx=0x7f4188004730 comp rx=0 tx=0).stop 2024-08-31T12:28:51.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.309+0000 7f419da8b640 1 -- 172.21.15.19:0/2884934425 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f41980cbcf0 msgr2=0x7f41981e5db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:51.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.309+0000 7f419da8b640 1 --2- 172.21.15.19:0/2884934425 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f41980cbcf0 0x7f41981e5db0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f418c00c3c0 tx=0x7f418c00c880 comp rx=0 tx=0).stop 2024-08-31T12:28:51.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.309+0000 7f4197fff640 1 -- 172.21.15.19:0/2884934425 reap_dead start 2024-08-31T12:28:51.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.309+0000 7f419da8b640 1 -- 172.21.15.19:0/2884934425 shutdown_connections 2024-08-31T12:28:51.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.309+0000 7f419da8b640 1 -- 172.21.15.19:0/2884934425 >> 172.21.15.19:0/2884934425 conn(0x7f4198074210 msgr2=0x7f41981590b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:51.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.309+0000 7f419da8b640 1 -- 172.21.15.19:0/2884934425 shutdown_connections 2024-08-31T12:28:51.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.309+0000 7f419da8b640 1 -- 172.21.15.19:0/2884934425 wait complete. 2024-08-31T12:28:51.322 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-08-31T12:28:51.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.434+0000 7fe3a10bd640 1 -- 172.21.15.19:0/200083724 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe39c0c4650 msgr2=0x7fe39c0ccfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:51.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.434+0000 7fe3a10bd640 1 --2- 172.21.15.19:0/200083724 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe39c0c4650 0x7fe39c0ccfe0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fe394005fd0 tx=0x7fe39402a5d0 comp rx=0 tx=0).stop 2024-08-31T12:28:51.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.434+0000 7fe3a10bd640 1 -- 172.21.15.19:0/200083724 shutdown_connections 2024-08-31T12:28:51.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.434+0000 7fe3a10bd640 1 --2- 172.21.15.19:0/200083724 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe39c159af0 0x7fe39c070510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:51.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.434+0000 7fe3a10bd640 1 --2- 172.21.15.19:0/200083724 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe39c0c4650 0x7fe39c0ccfe0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:51.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.434+0000 7fe3a10bd640 1 -- 172.21.15.19:0/200083724 >> 172.21.15.19:0/200083724 conn(0x7fe39c06d200 msgr2=0x7fe39c0cc740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:51.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.434+0000 7fe3a10bd640 1 -- 172.21.15.19:0/200083724 shutdown_connections 2024-08-31T12:28:51.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.434+0000 7fe3a10bd640 1 -- 172.21.15.19:0/200083724 wait complete. 2024-08-31T12:28:51.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.436+0000 7fe3a10bd640 1 Processor -- start 2024-08-31T12:28:51.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.436+0000 7fe3a10bd640 1 -- start start 2024-08-31T12:28:51.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.437+0000 7fe3a10bd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe39c0c4650 0x7fe39c1f1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:51.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.437+0000 7fe3a10bd640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe39c152fc0 0x7fe39c074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:51.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.437+0000 7fe3a10bd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe39c1494b0 0x7fe39c1498c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:51.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.437+0000 7fe3a10bd640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fe39c0cd520 con 0x7fe39c0c4650 2024-08-31T12:28:51.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.437+0000 7fe3a10bd640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe39c0c5830 con 0x7fe39c152fc0 2024-08-31T12:28:51.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.437+0000 7fe3a10bd640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fe39c1533d0 con 0x7fe39c1494b0 2024-08-31T12:28:51.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.437+0000 7fe39ad9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe39c0c4650 0x7fe39c1f1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:51.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.437+0000 7fe39ad9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe39c0c4650 0x7fe39c1f1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:43940/0 (socket says 172.21.15.19:43940) 2024-08-31T12:28:51.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.437+0000 7fe39ad9d640 1 -- 172.21.15.19:0/489178437 learned_addr learned my addr 172.21.15.19:0/489178437 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:28:51.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.437+0000 7fe39a59c640 1 --2- 172.21.15.19:0/489178437 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe39c152fc0 0x7fe39c074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:51.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.437+0000 7fe39b59e640 1 --2- 172.21.15.19:0/489178437 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe39c1494b0 0x7fe39c1498c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:51.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.438+0000 7fe39ad9d640 1 -- 172.21.15.19:0/489178437 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe39c1494b0 msgr2=0x7fe39c1498c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:51.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.438+0000 7fe39ad9d640 1 --2- 172.21.15.19:0/489178437 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe39c1494b0 0x7fe39c1498c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:51.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.438+0000 7fe39ad9d640 1 -- 172.21.15.19:0/489178437 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe39c152fc0 msgr2=0x7fe39c074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:51.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.438+0000 7fe39ad9d640 1 --2- 172.21.15.19:0/489178437 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe39c152fc0 0x7fe39c074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:51.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.438+0000 7fe39ad9d640 1 -- 172.21.15.19:0/489178437 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe39c154b60 con 0x7fe39c0c4650 2024-08-31T12:28:51.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.438+0000 7fe39ad9d640 1 --2- 172.21.15.19:0/489178437 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe39c0c4650 0x7fe39c1f1da0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fe394000c00 tx=0x7fe3940038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:51.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.438+0000 7fe3937fe640 1 -- 172.21.15.19:0/489178437 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe39403e030 con 0x7fe39c0c4650 2024-08-31T12:28:51.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.438+0000 7fe3937fe640 1 -- 172.21.15.19:0/489178437 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe394003ca0 con 0x7fe39c0c4650 2024-08-31T12:28:51.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.438+0000 7fe3a10bd640 1 -- 172.21.15.19:0/489178437 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe39c1f5370 con 0x7fe39c0c4650 2024-08-31T12:28:51.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.438+0000 7fe3937fe640 1 -- 172.21.15.19:0/489178437 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe394035040 con 0x7fe39c0c4650 2024-08-31T12:28:51.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.438+0000 7fe3a10bd640 1 -- 172.21.15.19:0/489178437 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe39c1f15a0 con 0x7fe39c0c4650 2024-08-31T12:28:51.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.439+0000 7fe3a10bd640 1 -- 172.21.15.19:0/489178437 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe39c0ccfe0 con 0x7fe39c0c4650 2024-08-31T12:28:51.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.440+0000 7fe3937fe640 1 -- 172.21.15.19:0/489178437 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fe394041030 con 0x7fe39c0c4650 2024-08-31T12:28:51.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.440+0000 7fe3937fe640 1 --2- 172.21.15.19:0/489178437 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe37c077a50 0x7fe37c079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:51.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.440+0000 7fe3937fe640 1 -- 172.21.15.19:0/489178437 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fe3940b9ca0 con 0x7fe39c0c4650 2024-08-31T12:28:51.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.441+0000 7fe39a59c640 1 --2- 172.21.15.19:0/489178437 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe37c077a50 0x7fe37c079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:51.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.441+0000 7fe39a59c640 1 --2- 172.21.15.19:0/489178437 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe37c077a50 0x7fe37c079f60 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fe39c157580 tx=0x7fe384009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:51.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.445+0000 7fe3937fe640 1 -- 172.21.15.19:0/489178437 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fe394082230 con 0x7fe39c0c4650 2024-08-31T12:28:51.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.629+0000 7fe3a10bd640 1 -- 172.21.15.19:0/489178437 --> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7fe39c1f5650 con 0x7fe37c077a50 2024-08-31T12:28:51.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.630+0000 7fe3937fe640 1 -- 172.21.15.19:0/489178437 <== mgr.4100 v2:172.21.15.43:6800/2824336655 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fe39c1f5650 con 0x7fe37c077a50 2024-08-31T12:28:51.632 INFO:teuthology.orchestra.run.smithi019.stderr:instructed osd(s) 1 to deep-scrub 2024-08-31T12:28:51.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.632+0000 7fe3a10bd640 1 -- 172.21.15.19:0/489178437 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe37c077a50 msgr2=0x7fe37c079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:51.635 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.632+0000 7fe3a10bd640 1 --2- 172.21.15.19:0/489178437 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe37c077a50 0x7fe37c079f60 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fe39c157580 tx=0x7fe384009040 comp rx=0 tx=0).stop 2024-08-31T12:28:51.635 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.632+0000 7fe3a10bd640 1 -- 172.21.15.19:0/489178437 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe39c0c4650 msgr2=0x7fe39c1f1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:51.635 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.632+0000 7fe3a10bd640 1 --2- 172.21.15.19:0/489178437 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe39c0c4650 0x7fe39c1f1da0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fe394000c00 tx=0x7fe3940038a0 comp rx=0 tx=0).stop 2024-08-31T12:28:51.635 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.632+0000 7fe3a10bd640 1 -- 172.21.15.19:0/489178437 shutdown_connections 2024-08-31T12:28:51.635 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.632+0000 7fe3a10bd640 1 --2- 172.21.15.19:0/489178437 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fe37c077a50 0x7fe37c079f60 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:51.635 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.632+0000 7fe3a10bd640 1 --2- 172.21.15.19:0/489178437 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe39c1494b0 0x7fe39c1498c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:51.635 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.632+0000 7fe3a10bd640 1 --2- 172.21.15.19:0/489178437 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe39c152fc0 0x7fe39c074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:51.635 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.632+0000 7fe3a10bd640 1 --2- 172.21.15.19:0/489178437 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe39c0c4650 0x7fe39c1f1da0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:51.635 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.632+0000 7fe3a10bd640 1 -- 172.21.15.19:0/489178437 >> 172.21.15.19:0/489178437 conn(0x7fe39c06d200 msgr2=0x7fe39c1522e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:51.636 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.632+0000 7fe3a10bd640 1 -- 172.21.15.19:0/489178437 shutdown_connections 2024-08-31T12:28:51.636 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.632+0000 7fe3a10bd640 1 -- 172.21.15.19:0/489178437 wait complete. 2024-08-31T12:28:51.646 INFO:tasks.ceph:Scrubbing osd.2 2024-08-31T12:28:51.646 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-08-31T12:28:51.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.762+0000 7f511652b640 1 -- 172.21.15.19:0/2905136655 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f51100cc990 msgr2=0x7f511015bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:51.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.762+0000 7f511652b640 1 --2- 172.21.15.19:0/2905136655 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f51100cc990 0x7f511015bcc0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f51100ccda0 tx=0x7f510002a5f0 comp rx=0 tx=0).stop 2024-08-31T12:28:51.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.763+0000 7f511652b640 1 -- 172.21.15.19:0/2905136655 shutdown_connections 2024-08-31T12:28:51.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.763+0000 7f511652b640 1 --2- 172.21.15.19:0/2905136655 >> v2:172.21.15.43:3300/0 conn(0x7f511015c730 0x7f51100d0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:51.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.763+0000 7f511652b640 1 --2- 172.21.15.19:0/2905136655 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f51100cc990 0x7f511015bcc0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:51.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.763+0000 7f511652b640 1 -- 172.21.15.19:0/2905136655 >> 172.21.15.19:0/2905136655 conn(0x7f511006da20 msgr2=0x7f511006de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:51.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.763+0000 7f511652b640 1 -- 172.21.15.19:0/2905136655 shutdown_connections 2024-08-31T12:28:51.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.763+0000 7f511652b640 1 -- 172.21.15.19:0/2905136655 wait complete. 2024-08-31T12:28:51.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.764+0000 7f511652b640 1 Processor -- start 2024-08-31T12:28:51.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.764+0000 7f511652b640 1 -- start start 2024-08-31T12:28:51.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.766+0000 7f511652b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f51100cc990 0x7f51100b4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:51.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.766+0000 7f511652b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f511015c730 0x7f51100c4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:51.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.766+0000 7f511652b640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f511015cb40 0x7f51100c6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:51.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.766+0000 7f511652b640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f51100cc6a0 con 0x7f511015c730 2024-08-31T12:28:51.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.766+0000 7f511652b640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f511015f130 con 0x7f511015cb40 2024-08-31T12:28:51.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.766+0000 7f511652b640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f51100cc380 con 0x7f51100cc990 2024-08-31T12:28:51.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.766+0000 7f510f7fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f511015c730 0x7f51100c4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:51.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.766+0000 7f510f7fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f511015c730 0x7f51100c4f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:43962/0 (socket says 172.21.15.19:43962) 2024-08-31T12:28:51.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.766+0000 7f510f7fe640 1 -- 172.21.15.19:0/2986888705 learned_addr learned my addr 172.21.15.19:0/2986888705 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:28:51.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.766+0000 7f5114ac8640 1 --2- 172.21.15.19:0/2986888705 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f511015cb40 0x7f51100c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:51.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.766+0000 7f510ffff640 1 --2- 172.21.15.19:0/2986888705 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f51100cc990 0x7f51100b4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:51.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.766+0000 7f510f7fe640 1 -- 172.21.15.19:0/2986888705 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f51100cc990 msgr2=0x7f51100b4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:51.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.766+0000 7f510f7fe640 1 --2- 172.21.15.19:0/2986888705 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f51100cc990 0x7f51100b4210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:51.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.766+0000 7f510f7fe640 1 -- 172.21.15.19:0/2986888705 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f511015cb40 msgr2=0x7f51100c6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:51.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.766+0000 7f510f7fe640 1 --2- 172.21.15.19:0/2986888705 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f511015cb40 0x7f51100c6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:51.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.766+0000 7f510f7fe640 1 -- 172.21.15.19:0/2986888705 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51100c5490 con 0x7f511015c730 2024-08-31T12:28:51.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.766+0000 7f5114ac8640 1 --2- 172.21.15.19:0/2986888705 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f511015cb40 0x7f51100c6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-31T12:28:51.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.766+0000 7f510f7fe640 1 --2- 172.21.15.19:0/2986888705 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f511015c730 0x7f51100c4f50 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f510400d6d0 tx=0x7f510400db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:51.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.766+0000 7f510cff9640 1 -- 172.21.15.19:0/2986888705 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5104017070 con 0x7f511015c730 2024-08-31T12:28:51.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.766+0000 7f510cff9640 1 -- 172.21.15.19:0/2986888705 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5104004310 con 0x7f511015c730 2024-08-31T12:28:51.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.766+0000 7f510cff9640 1 -- 172.21.15.19:0/2986888705 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5104009a80 con 0x7f511015c730 2024-08-31T12:28:51.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.766+0000 7f511652b640 1 -- 172.21.15.19:0/2986888705 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51100b3c50 con 0x7f511015c730 2024-08-31T12:28:51.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.766+0000 7f511652b640 1 -- 172.21.15.19:0/2986888705 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51100b5110 con 0x7f511015c730 2024-08-31T12:28:51.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.767+0000 7f511652b640 1 -- 172.21.15.19:0/2986888705 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f511015f650 con 0x7f511015c730 2024-08-31T12:28:51.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.768+0000 7f510cff9640 1 -- 172.21.15.19:0/2986888705 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f5104009c20 con 0x7f511015c730 2024-08-31T12:28:51.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.769+0000 7f510cff9640 1 --2- 172.21.15.19:0/2986888705 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f50fc077a80 0x7f50fc079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:51.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.769+0000 7f510cff9640 1 -- 172.21.15.19:0/2986888705 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f5104099cc0 con 0x7f511015c730 2024-08-31T12:28:51.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.769+0000 7f510ffff640 1 --2- 172.21.15.19:0/2986888705 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f50fc077a80 0x7f50fc079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:51.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.769+0000 7f510cff9640 1 --2- 172.21.15.19:0/2986888705 >> [v2:172.21.15.19:6808/2332368824,v1:172.21.15.19:6809/2332368824] conn(0x7f50fc07fe50 0x7f50fc082320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:51.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.769+0000 7f510cff9640 1 -- 172.21.15.19:0/2986888705 --> [v2:172.21.15.19:6808/2332368824,v1:172.21.15.19:6809/2332368824] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5104021e50 con 0x7f50fc07fe50 2024-08-31T12:28:51.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.769+0000 7f510cff9640 1 -- 172.21.15.19:0/2986888705 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f510409a070 con 0x7f511015c730 2024-08-31T12:28:51.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.769+0000 7f5114ac8640 1 --2- 172.21.15.19:0/2986888705 >> [v2:172.21.15.19:6808/2332368824,v1:172.21.15.19:6809/2332368824] conn(0x7f50fc07fe50 0x7f50fc082320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:51.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.770+0000 7f510ffff640 1 --2- 172.21.15.19:0/2986888705 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f50fc077a80 0x7f50fc079f90 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f5100033000 tx=0x7f5100004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:51.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.770+0000 7f5114ac8640 1 --2- 172.21.15.19:0/2986888705 >> [v2:172.21.15.19:6808/2332368824,v1:172.21.15.19:6809/2332368824] conn(0x7f50fc07fe50 0x7f50fc082320 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:51.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.770+0000 7f510cff9640 1 -- 172.21.15.19:0/2986888705 <== osd.2 v2:172.21.15.19:6808/2332368824 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5104021e50 con 0x7f50fc07fe50 2024-08-31T12:28:51.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.783+0000 7f511652b640 1 -- 172.21.15.19:0/2986888705 --> [v2:172.21.15.19:6808/2332368824,v1:172.21.15.19:6809/2332368824] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f51101f1fd0 con 0x7f50fc07fe50 2024-08-31T12:28:51.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.783+0000 7f510cff9640 1 -- 172.21.15.19:0/2986888705 <== osd.2 v2:172.21.15.19:6808/2332368824 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f51101f1fd0 con 0x7f50fc07fe50 2024-08-31T12:28:51.786 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:28:51.786 INFO:teuthology.orchestra.run.smithi019.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-08-31T12:28:51.786 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:28:51.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.783+0000 7f511652b640 1 -- 172.21.15.19:0/2986888705 >> [v2:172.21.15.19:6808/2332368824,v1:172.21.15.19:6809/2332368824] conn(0x7f50fc07fe50 msgr2=0x7f50fc082320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:51.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.783+0000 7f511652b640 1 --2- 172.21.15.19:0/2986888705 >> [v2:172.21.15.19:6808/2332368824,v1:172.21.15.19:6809/2332368824] conn(0x7f50fc07fe50 0x7f50fc082320 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:51.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.783+0000 7f511652b640 1 -- 172.21.15.19:0/2986888705 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f50fc077a80 msgr2=0x7f50fc079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:51.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.783+0000 7f511652b640 1 --2- 172.21.15.19:0/2986888705 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f50fc077a80 0x7f50fc079f90 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f5100033000 tx=0x7f5100004730 comp rx=0 tx=0).stop 2024-08-31T12:28:51.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.783+0000 7f511652b640 1 -- 172.21.15.19:0/2986888705 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f511015c730 msgr2=0x7f51100c4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:51.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.783+0000 7f511652b640 1 --2- 172.21.15.19:0/2986888705 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f511015c730 0x7f51100c4f50 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f510400d6d0 tx=0x7f510400db90 comp rx=0 tx=0).stop 2024-08-31T12:28:51.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.783+0000 7f5114ac8640 1 -- 172.21.15.19:0/2986888705 reap_dead start 2024-08-31T12:28:51.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.783+0000 7f511652b640 1 -- 172.21.15.19:0/2986888705 shutdown_connections 2024-08-31T12:28:51.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.783+0000 7f511652b640 1 -- 172.21.15.19:0/2986888705 >> 172.21.15.19:0/2986888705 conn(0x7f511006da20 msgr2=0x7f51100ca270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:51.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.784+0000 7f511652b640 1 -- 172.21.15.19:0/2986888705 shutdown_connections 2024-08-31T12:28:51.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.784+0000 7f511652b640 1 -- 172.21.15.19:0/2986888705 wait complete. 2024-08-31T12:28:51.797 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-08-31T12:28:51.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.909+0000 7f2a2427e640 1 -- 172.21.15.19:0/4095427952 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a1c161860 msgr2=0x7f2a1c163c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:51.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.909+0000 7f2a2427e640 1 --2- 172.21.15.19:0/4095427952 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a1c161860 0x7f2a1c163c80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2a14005fd0 tx=0x7f2a1402a9d0 comp rx=0 tx=0).stop 2024-08-31T12:28:51.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.910+0000 7f2a2427e640 1 -- 172.21.15.19:0/4095427952 shutdown_connections 2024-08-31T12:28:51.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.910+0000 7f2a2427e640 1 --2- 172.21.15.19:0/4095427952 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a1c161860 0x7f2a1c163c80 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:51.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.910+0000 7f2a2427e640 1 -- 172.21.15.19:0/4095427952 >> 172.21.15.19:0/4095427952 conn(0x7f2a1c0c9660 msgr2=0x7f2a1c0c9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:51.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.910+0000 7f2a2427e640 1 -- 172.21.15.19:0/4095427952 shutdown_connections 2024-08-31T12:28:51.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.910+0000 7f2a2427e640 1 -- 172.21.15.19:0/4095427952 wait complete. 2024-08-31T12:28:51.914 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.911+0000 7f2a2427e640 1 Processor -- start 2024-08-31T12:28:51.914 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.911+0000 7f2a2427e640 1 -- start start 2024-08-31T12:28:51.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.913+0000 7f2a2427e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2a1c14eb10 0x7f2a1c14fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:51.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.913+0000 7f2a2427e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a1c158960 0x7f2a1c146d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:51.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.913+0000 7f2a2427e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2a1c161860 0x7f2a1c141410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:51.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.913+0000 7f2a2427e640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f2a1c14ef90 con 0x7f2a1c14eb10 2024-08-31T12:28:51.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.913+0000 7f2a2427e640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2a1c068d80 con 0x7f2a1c158960 2024-08-31T12:28:51.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.913+0000 7f2a2427e640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f2a1c0ca180 con 0x7f2a1c161860 2024-08-31T12:28:51.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.913+0000 7f2a2201a640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2a1c14eb10 0x7f2a1c14fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:51.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.913+0000 7f2a2201a640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2a1c14eb10 0x7f2a1c14fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:43970/0 (socket says 172.21.15.19:43970) 2024-08-31T12:28:51.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.913+0000 7f2a2201a640 1 -- 172.21.15.19:0/1998639649 learned_addr learned my addr 172.21.15.19:0/1998639649 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:28:51.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.913+0000 7f2a21819640 1 --2- 172.21.15.19:0/1998639649 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a1c158960 0x7f2a1c146d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:51.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.913+0000 7f2a2281b640 1 --2- 172.21.15.19:0/1998639649 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2a1c161860 0x7f2a1c141410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:51.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.913+0000 7f2a2201a640 1 -- 172.21.15.19:0/1998639649 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2a1c161860 msgr2=0x7f2a1c141410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:51.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.913+0000 7f2a2201a640 1 --2- 172.21.15.19:0/1998639649 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2a1c161860 0x7f2a1c141410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:51.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.913+0000 7f2a2201a640 1 -- 172.21.15.19:0/1998639649 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a1c158960 msgr2=0x7f2a1c146d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:51.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.913+0000 7f2a2201a640 1 --2- 172.21.15.19:0/1998639649 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a1c158960 0x7f2a1c146d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:51.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.913+0000 7f2a2201a640 1 -- 172.21.15.19:0/1998639649 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a1c157160 con 0x7f2a1c14eb10 2024-08-31T12:28:51.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.913+0000 7f2a2281b640 1 --2- 172.21.15.19:0/1998639649 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2a1c161860 0x7f2a1c141410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-08-31T12:28:51.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.913+0000 7f2a2201a640 1 --2- 172.21.15.19:0/1998639649 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2a1c14eb10 0x7f2a1c14fbb0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f2a1800c430 tx=0x7f2a1800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:51.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.913+0000 7f2a12ffd640 1 -- 172.21.15.19:0/1998639649 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a18017070 con 0x7f2a1c14eb10 2024-08-31T12:28:51.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.913+0000 7f2a12ffd640 1 -- 172.21.15.19:0/1998639649 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2a180053c0 con 0x7f2a1c14eb10 2024-08-31T12:28:51.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.913+0000 7f2a12ffd640 1 -- 172.21.15.19:0/1998639649 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a18003850 con 0x7f2a1c14eb10 2024-08-31T12:28:51.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.913+0000 7f2a2427e640 1 -- 172.21.15.19:0/1998639649 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a1c1500f0 con 0x7f2a1c14eb10 2024-08-31T12:28:51.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.914+0000 7f2a2427e640 1 -- 172.21.15.19:0/1998639649 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a1c14f3f0 con 0x7f2a1c14eb10 2024-08-31T12:28:51.920 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.914+0000 7f2a2427e640 1 -- 172.21.15.19:0/1998639649 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a1c14d140 con 0x7f2a1c14eb10 2024-08-31T12:28:51.920 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.915+0000 7f2a12ffd640 1 -- 172.21.15.19:0/1998639649 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f2a1801a070 con 0x7f2a1c14eb10 2024-08-31T12:28:51.920 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.916+0000 7f2a12ffd640 1 --2- 172.21.15.19:0/1998639649 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2a00077a50 0x7f2a00079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:51.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.916+0000 7f2a12ffd640 1 -- 172.21.15.19:0/1998639649 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f2a18098bd0 con 0x7f2a1c14eb10 2024-08-31T12:28:51.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.916+0000 7f2a21819640 1 --2- 172.21.15.19:0/1998639649 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2a00077a50 0x7f2a00079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:51.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.917+0000 7f2a21819640 1 --2- 172.21.15.19:0/1998639649 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2a00077a50 0x7f2a00079f60 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c0c8690 tx=0x7f2a0c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:51.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:51.920+0000 7f2a12ffd640 1 -- 172.21.15.19:0/1998639649 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f2a1c14d140 con 0x7f2a1c14eb10 2024-08-31T12:28:52.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.104+0000 7f2a2427e640 1 -- 172.21.15.19:0/1998639649 --> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f2a1c14d420 con 0x7f2a00077a50 2024-08-31T12:28:52.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.106+0000 7f2a12ffd640 1 -- 172.21.15.19:0/1998639649 <== mgr.4100 v2:172.21.15.43:6800/2824336655 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f2a1c14d420 con 0x7f2a00077a50 2024-08-31T12:28:52.107 INFO:teuthology.orchestra.run.smithi019.stderr:instructed osd(s) 2 to deep-scrub 2024-08-31T12:28:52.110 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.107+0000 7f2a2427e640 1 -- 172.21.15.19:0/1998639649 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2a00077a50 msgr2=0x7f2a00079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:52.110 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.107+0000 7f2a2427e640 1 --2- 172.21.15.19:0/1998639649 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2a00077a50 0x7f2a00079f60 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c0c8690 tx=0x7f2a0c009040 comp rx=0 tx=0).stop 2024-08-31T12:28:52.110 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.107+0000 7f2a2427e640 1 -- 172.21.15.19:0/1998639649 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2a1c14eb10 msgr2=0x7f2a1c14fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:52.110 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.107+0000 7f2a2427e640 1 --2- 172.21.15.19:0/1998639649 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2a1c14eb10 0x7f2a1c14fbb0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f2a1800c430 tx=0x7f2a1800c8f0 comp rx=0 tx=0).stop 2024-08-31T12:28:52.110 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.107+0000 7f2a2427e640 1 -- 172.21.15.19:0/1998639649 shutdown_connections 2024-08-31T12:28:52.111 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.107+0000 7f2a2427e640 1 --2- 172.21.15.19:0/1998639649 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2a00077a50 0x7f2a00079f60 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:52.111 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.107+0000 7f2a2427e640 1 --2- 172.21.15.19:0/1998639649 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2a1c161860 0x7f2a1c141410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:52.111 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.107+0000 7f2a2427e640 1 --2- 172.21.15.19:0/1998639649 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a1c158960 0x7f2a1c146d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:52.111 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.107+0000 7f2a2427e640 1 --2- 172.21.15.19:0/1998639649 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2a1c14eb10 0x7f2a1c14fbb0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:52.111 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.107+0000 7f2a2427e640 1 -- 172.21.15.19:0/1998639649 >> 172.21.15.19:0/1998639649 conn(0x7f2a1c0c9660 msgr2=0x7f2a1c14a650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:52.111 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.108+0000 7f2a2427e640 1 -- 172.21.15.19:0/1998639649 shutdown_connections 2024-08-31T12:28:52.112 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.108+0000 7f2a2427e640 1 -- 172.21.15.19:0/1998639649 wait complete. 2024-08-31T12:28:52.121 INFO:tasks.ceph:Scrubbing osd.3 2024-08-31T12:28:52.121 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-08-31T12:28:52.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.238+0000 7f8fadc8d640 1 -- 172.21.15.19:0/1049441785 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8fa815cee0 msgr2=0x7f8fa80c6ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:52.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.238+0000 7f8fadc8d640 1 --2- 172.21.15.19:0/1049441785 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8fa815cee0 0x7f8fa80c6ba0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f8fa0005fd0 tx=0x7f8fa002aa90 comp rx=0 tx=0).stop 2024-08-31T12:28:52.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.238+0000 7f8fadc8d640 1 -- 172.21.15.19:0/1049441785 shutdown_connections 2024-08-31T12:28:52.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.238+0000 7f8fadc8d640 1 --2- 172.21.15.19:0/1049441785 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8fa815cee0 0x7f8fa80c6ba0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:52.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.238+0000 7f8fadc8d640 1 --2- 172.21.15.19:0/1049441785 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8fa80c7e90 0x7f8fa8159a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:52.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.238+0000 7f8fadc8d640 1 -- 172.21.15.19:0/1049441785 >> 172.21.15.19:0/1049441785 conn(0x7f8fa80ca5f0 msgr2=0x7f8fa80cd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:52.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.238+0000 7f8fadc8d640 1 -- 172.21.15.19:0/1049441785 shutdown_connections 2024-08-31T12:28:52.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.238+0000 7f8fadc8d640 1 -- 172.21.15.19:0/1049441785 wait complete. 2024-08-31T12:28:52.242 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.240+0000 7f8fadc8d640 1 Processor -- start 2024-08-31T12:28:52.242 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.240+0000 7f8fadc8d640 1 -- start start 2024-08-31T12:28:52.244 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.241+0000 7f8fadc8d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8fa80c7e90 0x7f8fa81f4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:52.244 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.241+0000 7f8fadc8d640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8fa80c90a0 0x7f8fa80c2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:52.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.241+0000 7f8fadc8d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8fa815cee0 0x7f8fa81f17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:52.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.241+0000 7f8fadc8d640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f8fa8159500 con 0x7f8fa815cee0 2024-08-31T12:28:52.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.241+0000 7f8fadc8d640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8fa8159220 con 0x7f8fa80c90a0 2024-08-31T12:28:52.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.241+0000 7f8fadc8d640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f8fa815d2f0 con 0x7f8fa80c7e90 2024-08-31T12:28:52.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.241+0000 7f8fa7fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8fa815cee0 0x7f8fa81f17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:52.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.241+0000 7f8fa7fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8fa815cee0 0x7f8fa81f17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:43980/0 (socket says 172.21.15.19:43980) 2024-08-31T12:28:52.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.241+0000 7f8fa7fff640 1 -- 172.21.15.19:0/2492378817 learned_addr learned my addr 172.21.15.19:0/2492378817 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:28:52.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.241+0000 7f8fa77fe640 1 --2- 172.21.15.19:0/2492378817 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8fa80c7e90 0x7f8fa81f4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:52.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.241+0000 7f8fa6ffd640 1 --2- 172.21.15.19:0/2492378817 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8fa80c90a0 0x7f8fa80c2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:52.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.242+0000 7f8fa7fff640 1 -- 172.21.15.19:0/2492378817 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8fa80c7e90 msgr2=0x7f8fa81f4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:52.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.242+0000 7f8fa7fff640 1 --2- 172.21.15.19:0/2492378817 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8fa80c7e90 0x7f8fa81f4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:52.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.242+0000 7f8fa7fff640 1 -- 172.21.15.19:0/2492378817 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8fa80c90a0 msgr2=0x7f8fa80c2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:52.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.242+0000 7f8fa7fff640 1 --2- 172.21.15.19:0/2492378817 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8fa80c90a0 0x7f8fa80c2c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:52.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.242+0000 7f8fa7fff640 1 -- 172.21.15.19:0/2492378817 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fa81f1ce0 con 0x7f8fa815cee0 2024-08-31T12:28:52.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.242+0000 7f8fa7fff640 1 --2- 172.21.15.19:0/2492378817 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8fa815cee0 0x7f8fa81f17a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8fa0009d90 tx=0x7f8fa0003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:52.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.242+0000 7f8f8ffff640 1 -- 172.21.15.19:0/2492378817 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa0044070 con 0x7f8fa815cee0 2024-08-31T12:28:52.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.242+0000 7f8f8ffff640 1 -- 172.21.15.19:0/2492378817 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8fa0003e00 con 0x7f8fa815cee0 2024-08-31T12:28:52.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.242+0000 7f8f8ffff640 1 -- 172.21.15.19:0/2492378817 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa00044b0 con 0x7f8fa815cee0 2024-08-31T12:28:52.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.242+0000 7f8fadc8d640 1 -- 172.21.15.19:0/2492378817 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fa81f43b0 con 0x7f8fa815cee0 2024-08-31T12:28:52.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.242+0000 7f8fadc8d640 1 -- 172.21.15.19:0/2492378817 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fa81f4650 con 0x7f8fa815cee0 2024-08-31T12:28:52.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.243+0000 7f8fadc8d640 1 -- 172.21.15.19:0/2492378817 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8fa8074fc0 con 0x7f8fa815cee0 2024-08-31T12:28:52.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.244+0000 7f8f8ffff640 1 -- 172.21.15.19:0/2492378817 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f8fa0047070 con 0x7f8fa815cee0 2024-08-31T12:28:52.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.245+0000 7f8f8ffff640 1 --2- 172.21.15.19:0/2492378817 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8f94077a80 0x7f8f94079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:52.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.245+0000 7f8f8ffff640 1 -- 172.21.15.19:0/2492378817 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f8fa00ba580 con 0x7f8fa815cee0 2024-08-31T12:28:52.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.245+0000 7f8fa77fe640 1 --2- 172.21.15.19:0/2492378817 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8f94077a80 0x7f8f94079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:52.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.245+0000 7f8f8ffff640 1 --2- 172.21.15.19:0/2492378817 >> [v2:172.21.15.19:6824/3785550598,v1:172.21.15.19:6825/3785550598] conn(0x7f8f9407fe50 0x7f8f94082320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:52.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.245+0000 7f8f8ffff640 1 -- 172.21.15.19:0/2492378817 --> [v2:172.21.15.19:6824/3785550598,v1:172.21.15.19:6825/3785550598] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8fa0044e70 con 0x7f8f9407fe50 2024-08-31T12:28:52.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.245+0000 7f8f8ffff640 1 -- 172.21.15.19:0/2492378817 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f8fa00c1050 con 0x7f8fa815cee0 2024-08-31T12:28:52.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.245+0000 7f8fa6ffd640 1 --2- 172.21.15.19:0/2492378817 >> [v2:172.21.15.19:6824/3785550598,v1:172.21.15.19:6825/3785550598] conn(0x7f8f9407fe50 0x7f8f94082320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:52.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.245+0000 7f8fa77fe640 1 --2- 172.21.15.19:0/2492378817 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8f94077a80 0x7f8f94079f90 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f8f98001720 tx=0x7f8f98009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:52.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.246+0000 7f8fa6ffd640 1 --2- 172.21.15.19:0/2492378817 >> [v2:172.21.15.19:6824/3785550598,v1:172.21.15.19:6825/3785550598] conn(0x7f8f9407fe50 0x7f8f94082320 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:52.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.246+0000 7f8f8ffff640 1 -- 172.21.15.19:0/2492378817 <== osd.3 v2:172.21.15.19:6824/3785550598 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8fa0044e70 con 0x7f8f9407fe50 2024-08-31T12:28:52.264 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.262+0000 7f8fadc8d640 1 -- 172.21.15.19:0/2492378817 --> [v2:172.21.15.19:6824/3785550598,v1:172.21.15.19:6825/3785550598] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f8fa80c6980 con 0x7f8f9407fe50 2024-08-31T12:28:52.264 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.262+0000 7f8f8ffff640 1 -- 172.21.15.19:0/2492378817 <== osd.3 v2:172.21.15.19:6824/3785550598 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f8fa80c6980 con 0x7f8f9407fe50 2024-08-31T12:28:52.264 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:28:52.264 INFO:teuthology.orchestra.run.smithi019.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-08-31T12:28:52.265 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:28:52.266 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.262+0000 7f8fadc8d640 1 -- 172.21.15.19:0/2492378817 >> [v2:172.21.15.19:6824/3785550598,v1:172.21.15.19:6825/3785550598] conn(0x7f8f9407fe50 msgr2=0x7f8f94082320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:52.266 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.262+0000 7f8fadc8d640 1 --2- 172.21.15.19:0/2492378817 >> [v2:172.21.15.19:6824/3785550598,v1:172.21.15.19:6825/3785550598] conn(0x7f8f9407fe50 0x7f8f94082320 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:52.266 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.262+0000 7f8fadc8d640 1 -- 172.21.15.19:0/2492378817 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8f94077a80 msgr2=0x7f8f94079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:52.266 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.262+0000 7f8fadc8d640 1 --2- 172.21.15.19:0/2492378817 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8f94077a80 0x7f8f94079f90 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f8f98001720 tx=0x7f8f98009040 comp rx=0 tx=0).stop 2024-08-31T12:28:52.266 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.262+0000 7f8fadc8d640 1 -- 172.21.15.19:0/2492378817 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8fa815cee0 msgr2=0x7f8fa81f17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:52.266 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.262+0000 7f8fadc8d640 1 --2- 172.21.15.19:0/2492378817 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8fa815cee0 0x7f8fa81f17a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8fa0009d90 tx=0x7f8fa0003c10 comp rx=0 tx=0).stop 2024-08-31T12:28:52.266 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.262+0000 7f8fa7fff640 1 -- 172.21.15.19:0/2492378817 reap_dead start 2024-08-31T12:28:52.266 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.262+0000 7f8fadc8d640 1 -- 172.21.15.19:0/2492378817 shutdown_connections 2024-08-31T12:28:52.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.262+0000 7f8fadc8d640 1 -- 172.21.15.19:0/2492378817 >> 172.21.15.19:0/2492378817 conn(0x7f8fa80ca5f0 msgr2=0x7f8fa814f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:52.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.263+0000 7f8fadc8d640 1 -- 172.21.15.19:0/2492378817 shutdown_connections 2024-08-31T12:28:52.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.263+0000 7f8fadc8d640 1 -- 172.21.15.19:0/2492378817 wait complete. 2024-08-31T12:28:52.276 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-08-31T12:28:52.408 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.405+0000 7f8de5b19640 1 -- 172.21.15.19:0/3784103415 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8de00cc400 msgr2=0x7f8de00cc810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:52.408 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.405+0000 7f8de5b19640 1 --2- 172.21.15.19:0/3784103415 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8de00cc400 0x7f8de00cc810 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8de0073680 tx=0x7f8dd802ab40 comp rx=0 tx=0).stop 2024-08-31T12:28:52.408 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.405+0000 7f8de5b19640 1 -- 172.21.15.19:0/3784103415 shutdown_connections 2024-08-31T12:28:52.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.405+0000 7f8de5b19640 1 --2- 172.21.15.19:0/3784103415 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8de00cc400 0x7f8de00cc810 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:52.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.405+0000 7f8de5b19640 1 --2- 172.21.15.19:0/3784103415 >> v2:172.21.15.43:3300/0 conn(0x7f8de00c5370 0x7f8de006ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:52.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.405+0000 7f8de5b19640 1 -- 172.21.15.19:0/3784103415 >> 172.21.15.19:0/3784103415 conn(0x7f8de00ca670 msgr2=0x7f8de00cd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:52.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.405+0000 7f8de5b19640 1 -- 172.21.15.19:0/3784103415 shutdown_connections 2024-08-31T12:28:52.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.405+0000 7f8de5b19640 1 -- 172.21.15.19:0/3784103415 wait complete. 2024-08-31T12:28:52.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.407+0000 7f8de5b19640 1 Processor -- start 2024-08-31T12:28:52.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.407+0000 7f8de5b19640 1 -- start start 2024-08-31T12:28:52.412 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.409+0000 7f8de5b19640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8de00c5370 0x7f8de01f45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:52.412 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.409+0000 7f8de5b19640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8de00cb2b0 0x7f8de01e38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:52.412 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.409+0000 7f8de5b19640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8de00cc400 0x7f8de006cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:52.412 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.409+0000 7f8de5b19640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f8de00c72e0 con 0x7f8de00cb2b0 2024-08-31T12:28:52.412 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.409+0000 7f8de5b19640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8de00ccd50 con 0x7f8de00cc400 2024-08-31T12:28:52.412 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.409+0000 7f8de5b19640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f8de00cb8c0 con 0x7f8de00c5370 2024-08-31T12:28:52.412 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.409+0000 7f8dd6ffd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8de00cb2b0 0x7f8de01e38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:52.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.409+0000 7f8dd6ffd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8de00cb2b0 0x7f8de01e38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:43996/0 (socket says 172.21.15.19:43996) 2024-08-31T12:28:52.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.409+0000 7f8ddffff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8de00cc400 0x7f8de006cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:52.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.409+0000 7f8ddf7fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8de00c5370 0x7f8de01f45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:52.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.409+0000 7f8dd6ffd640 1 -- 172.21.15.19:0/3814619654 learned_addr learned my addr 172.21.15.19:0/3814619654 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:28:52.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.409+0000 7f8dd6ffd640 1 -- 172.21.15.19:0/3814619654 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8de00c5370 msgr2=0x7f8de01f45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:52.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.409+0000 7f8dd6ffd640 1 --2- 172.21.15.19:0/3814619654 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8de00c5370 0x7f8de01f45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:52.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.409+0000 7f8dd6ffd640 1 -- 172.21.15.19:0/3814619654 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8de00cc400 msgr2=0x7f8de006cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:52.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.409+0000 7f8dd6ffd640 1 --2- 172.21.15.19:0/3814619654 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8de00cc400 0x7f8de006cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:52.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.409+0000 7f8dd6ffd640 1 -- 172.21.15.19:0/3814619654 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8de01e3df0 con 0x7f8de00cb2b0 2024-08-31T12:28:52.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.409+0000 7f8dd6ffd640 1 --2- 172.21.15.19:0/3814619654 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8de00cb2b0 0x7f8de01e38b0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8dc800c3d0 tx=0x7f8dc800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:52.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.409+0000 7f8ddcff9640 1 -- 172.21.15.19:0/3814619654 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dc8017070 con 0x7f8de00cb2b0 2024-08-31T12:28:52.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.410+0000 7f8ddcff9640 1 -- 172.21.15.19:0/3814619654 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8dc80053c0 con 0x7f8de00cb2b0 2024-08-31T12:28:52.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.410+0000 7f8de5b19640 1 -- 172.21.15.19:0/3814619654 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8de01e4770 con 0x7f8de00cb2b0 2024-08-31T12:28:52.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.410+0000 7f8ddcff9640 1 -- 172.21.15.19:0/3814619654 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dc8003850 con 0x7f8de00cb2b0 2024-08-31T12:28:52.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.410+0000 7f8de5b19640 1 -- 172.21.15.19:0/3814619654 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8de0147ee0 con 0x7f8de00cb2b0 2024-08-31T12:28:52.416 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.411+0000 7f8de5b19640 1 -- 172.21.15.19:0/3814619654 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8de006e760 con 0x7f8de00cb2b0 2024-08-31T12:28:52.416 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.411+0000 7f8ddcff9640 1 -- 172.21.15.19:0/3814619654 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f8dc801a070 con 0x7f8de00cb2b0 2024-08-31T12:28:52.416 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.412+0000 7f8ddcff9640 1 --2- 172.21.15.19:0/3814619654 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8dc0077a50 0x7f8dc0079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:52.416 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.412+0000 7f8ddcff9640 1 -- 172.21.15.19:0/3814619654 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f8dc8098990 con 0x7f8de00cb2b0 2024-08-31T12:28:52.416 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.412+0000 7f8ddf7fe640 1 --2- 172.21.15.19:0/3814619654 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8dc0077a50 0x7f8dc0079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:52.416 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.412+0000 7f8ddf7fe640 1 --2- 172.21.15.19:0/3814619654 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8dc0077a50 0x7f8dc0079f60 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f8dd0009b30 tx=0x7f8dd0005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:52.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.417+0000 7f8ddcff9640 1 -- 172.21.15.19:0/3814619654 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f8dc8061030 con 0x7f8de00cb2b0 2024-08-31T12:28:52.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.638+0000 7f8de5b19640 1 -- 172.21.15.19:0/3814619654 --> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f8de01e4520 con 0x7f8dc0077a50 2024-08-31T12:28:52.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.639+0000 7f8ddcff9640 1 -- 172.21.15.19:0/3814619654 <== mgr.4100 v2:172.21.15.43:6800/2824336655 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f8de01e4520 con 0x7f8dc0077a50 2024-08-31T12:28:52.641 INFO:teuthology.orchestra.run.smithi019.stderr:instructed osd(s) 3 to deep-scrub 2024-08-31T12:28:52.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.641+0000 7f8de5b19640 1 -- 172.21.15.19:0/3814619654 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8dc0077a50 msgr2=0x7f8dc0079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:52.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.641+0000 7f8de5b19640 1 --2- 172.21.15.19:0/3814619654 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8dc0077a50 0x7f8dc0079f60 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f8dd0009b30 tx=0x7f8dd0005f90 comp rx=0 tx=0).stop 2024-08-31T12:28:52.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.641+0000 7f8de5b19640 1 -- 172.21.15.19:0/3814619654 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8de00cb2b0 msgr2=0x7f8de01e38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:52.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.641+0000 7f8de5b19640 1 --2- 172.21.15.19:0/3814619654 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8de00cb2b0 0x7f8de01e38b0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8dc800c3d0 tx=0x7f8dc800c890 comp rx=0 tx=0).stop 2024-08-31T12:28:52.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.641+0000 7f8de5b19640 1 -- 172.21.15.19:0/3814619654 shutdown_connections 2024-08-31T12:28:52.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.641+0000 7f8de5b19640 1 --2- 172.21.15.19:0/3814619654 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f8dc0077a50 0x7f8dc0079f60 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:52.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.641+0000 7f8de5b19640 1 --2- 172.21.15.19:0/3814619654 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8de00cc400 0x7f8de006cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:52.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.641+0000 7f8de5b19640 1 --2- 172.21.15.19:0/3814619654 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8de00cb2b0 0x7f8de01e38b0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:52.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.641+0000 7f8de5b19640 1 --2- 172.21.15.19:0/3814619654 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8de00c5370 0x7f8de01f45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:52.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.641+0000 7f8de5b19640 1 -- 172.21.15.19:0/3814619654 >> 172.21.15.19:0/3814619654 conn(0x7f8de00ca670 msgr2=0x7f8de014a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:52.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.641+0000 7f8de5b19640 1 -- 172.21.15.19:0/3814619654 shutdown_connections 2024-08-31T12:28:52.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.641+0000 7f8de5b19640 1 -- 172.21.15.19:0/3814619654 wait complete. 2024-08-31T12:28:52.655 INFO:tasks.ceph:Scrubbing osd.4 2024-08-31T12:28:52.656 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-08-31T12:28:52.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.772+0000 7f64ee76e640 1 -- 172.21.15.19:0/1748524981 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64e814fa00 msgr2=0x7f64e8164950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:52.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.772+0000 7f64ee76e640 1 --2- 172.21.15.19:0/1748524981 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64e814fa00 0x7f64e8164950 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f64e0005fd0 tx=0x7f64e002a800 comp rx=0 tx=0).stop 2024-08-31T12:28:52.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.772+0000 7f64ee76e640 1 -- 172.21.15.19:0/1748524981 shutdown_connections 2024-08-31T12:28:52.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.772+0000 7f64ee76e640 1 --2- 172.21.15.19:0/1748524981 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64e814fa00 0x7f64e8164950 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:52.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.772+0000 7f64ee76e640 1 -- 172.21.15.19:0/1748524981 >> 172.21.15.19:0/1748524981 conn(0x7f64e8068aa0 msgr2=0x7f64e8068eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:52.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.772+0000 7f64ee76e640 1 -- 172.21.15.19:0/1748524981 shutdown_connections 2024-08-31T12:28:52.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.772+0000 7f64ee76e640 1 -- 172.21.15.19:0/1748524981 wait complete. 2024-08-31T12:28:52.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.774+0000 7f64ee76e640 1 Processor -- start 2024-08-31T12:28:52.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.774+0000 7f64ee76e640 1 -- start start 2024-08-31T12:28:52.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.775+0000 7f64ee76e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f64e80c43a0 0x7f64e81e40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:52.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.775+0000 7f64ee76e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64e80c9140 0x7f64e814b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:52.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.775+0000 7f64ee76e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64e814fa00 0x7f64e8147020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:52.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.775+0000 7f64ee76e640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f64e8159b00 con 0x7f64e80c43a0 2024-08-31T12:28:52.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.775+0000 7f64ee76e640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f64e80c47b0 con 0x7f64e80c9140 2024-08-31T12:28:52.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.775+0000 7f64ee76e640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f64e814fe10 con 0x7f64e814fa00 2024-08-31T12:28:52.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.775+0000 7f64e7fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f64e80c43a0 0x7f64e81e40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:52.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.775+0000 7f64e7fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f64e80c43a0 0x7f64e81e40c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:44004/0 (socket says 172.21.15.19:44004) 2024-08-31T12:28:52.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.775+0000 7f64e7fff640 1 -- 172.21.15.19:0/2412394457 learned_addr learned my addr 172.21.15.19:0/2412394457 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:28:52.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.776+0000 7f64e77fe640 1 --2- 172.21.15.19:0/2412394457 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64e80c9140 0x7f64e814b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:52.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.776+0000 7f64ecd0b640 1 --2- 172.21.15.19:0/2412394457 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64e814fa00 0x7f64e8147020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:52.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.776+0000 7f64e7fff640 1 -- 172.21.15.19:0/2412394457 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64e814fa00 msgr2=0x7f64e8147020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:52.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.776+0000 7f64e7fff640 1 --2- 172.21.15.19:0/2412394457 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64e814fa00 0x7f64e8147020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:52.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.776+0000 7f64e7fff640 1 -- 172.21.15.19:0/2412394457 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64e80c9140 msgr2=0x7f64e814b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:52.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.776+0000 7f64e7fff640 1 --2- 172.21.15.19:0/2412394457 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64e80c9140 0x7f64e814b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:52.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.776+0000 7f64e7fff640 1 -- 172.21.15.19:0/2412394457 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64e81e0060 con 0x7f64e80c43a0 2024-08-31T12:28:52.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.776+0000 7f64e77fe640 1 --2- 172.21.15.19:0/2412394457 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64e80c9140 0x7f64e814b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:28:52.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.776+0000 7f64ecd0b640 1 --2- 172.21.15.19:0/2412394457 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64e814fa00 0x7f64e8147020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:28:52.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.776+0000 7f64e7fff640 1 --2- 172.21.15.19:0/2412394457 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f64e80c43a0 0x7f64e81e40c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f64d800c480 tx=0x7f64d800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:52.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.776+0000 7f64e4ff9640 1 -- 172.21.15.19:0/2412394457 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64d8017070 con 0x7f64e80c43a0 2024-08-31T12:28:52.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.776+0000 7f64e4ff9640 1 -- 172.21.15.19:0/2412394457 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f64d80035f0 con 0x7f64e80c43a0 2024-08-31T12:28:52.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.776+0000 7f64ee76e640 1 -- 172.21.15.19:0/2412394457 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64e81e0340 con 0x7f64e80c43a0 2024-08-31T12:28:52.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.776+0000 7f64e4ff9640 1 -- 172.21.15.19:0/2412394457 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64d8003a40 con 0x7f64e80c43a0 2024-08-31T12:28:52.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.776+0000 7f64ee76e640 1 -- 172.21.15.19:0/2412394457 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64e81e0900 con 0x7f64e80c43a0 2024-08-31T12:28:52.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.777+0000 7f64ee76e640 1 -- 172.21.15.19:0/2412394457 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f64e81dfe10 con 0x7f64e80c43a0 2024-08-31T12:28:52.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.778+0000 7f64e4ff9640 1 -- 172.21.15.19:0/2412394457 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f64d801a070 con 0x7f64e80c43a0 2024-08-31T12:28:52.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.778+0000 7f64e4ff9640 1 --2- 172.21.15.19:0/2412394457 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f64d4077a50 0x7f64d4079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:52.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.778+0000 7f64e4ff9640 1 -- 172.21.15.19:0/2412394457 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f64d80991a0 con 0x7f64e80c43a0 2024-08-31T12:28:52.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.779+0000 7f64e4ff9640 1 --2- 172.21.15.19:0/2412394457 >> [v2:172.21.15.43:6818/1985519349,v1:172.21.15.43:6819/1985519349] conn(0x7f64d407fdc0 0x7f64d4082290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:52.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.779+0000 7f64e4ff9640 1 -- 172.21.15.19:0/2412394457 --> [v2:172.21.15.43:6818/1985519349,v1:172.21.15.43:6819/1985519349] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f64d4082960 con 0x7f64d407fdc0 2024-08-31T12:28:52.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.779+0000 7f64e4ff9640 1 -- 172.21.15.19:0/2412394457 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f64d8099630 con 0x7f64e80c43a0 2024-08-31T12:28:52.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.779+0000 7f64e77fe640 1 --2- 172.21.15.19:0/2412394457 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f64d4077a50 0x7f64d4079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:52.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.779+0000 7f64ecd0b640 1 --2- 172.21.15.19:0/2412394457 >> [v2:172.21.15.43:6818/1985519349,v1:172.21.15.43:6819/1985519349] conn(0x7f64d407fdc0 0x7f64d4082290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:52.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.779+0000 7f64e77fe640 1 --2- 172.21.15.19:0/2412394457 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f64d4077a50 0x7f64d4079f60 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f64dc007fd0 tx=0x7f64dc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:52.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.779+0000 7f64ecd0b640 1 --2- 172.21.15.19:0/2412394457 >> [v2:172.21.15.43:6818/1985519349,v1:172.21.15.43:6819/1985519349] conn(0x7f64d407fdc0 0x7f64d4082290 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:52.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.780+0000 7f64e4ff9640 1 -- 172.21.15.19:0/2412394457 <== osd.4 v2:172.21.15.43:6818/1985519349 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f64d4082960 con 0x7f64d407fdc0 2024-08-31T12:28:52.795 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.793+0000 7f64ee76e640 1 -- 172.21.15.19:0/2412394457 --> [v2:172.21.15.43:6818/1985519349,v1:172.21.15.43:6819/1985519349] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f64e80c4c30 con 0x7f64d407fdc0 2024-08-31T12:28:52.795 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.793+0000 7f64e4ff9640 1 -- 172.21.15.19:0/2412394457 <== osd.4 v2:172.21.15.43:6818/1985519349 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f64e80c4c30 con 0x7f64d407fdc0 2024-08-31T12:28:52.796 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:28:52.796 INFO:teuthology.orchestra.run.smithi019.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-08-31T12:28:52.796 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:28:52.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.794+0000 7f64ee76e640 1 -- 172.21.15.19:0/2412394457 >> [v2:172.21.15.43:6818/1985519349,v1:172.21.15.43:6819/1985519349] conn(0x7f64d407fdc0 msgr2=0x7f64d4082290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:52.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.794+0000 7f64ee76e640 1 --2- 172.21.15.19:0/2412394457 >> [v2:172.21.15.43:6818/1985519349,v1:172.21.15.43:6819/1985519349] conn(0x7f64d407fdc0 0x7f64d4082290 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:52.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.794+0000 7f64ee76e640 1 -- 172.21.15.19:0/2412394457 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f64d4077a50 msgr2=0x7f64d4079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:52.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.794+0000 7f64ee76e640 1 --2- 172.21.15.19:0/2412394457 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f64d4077a50 0x7f64d4079f60 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f64dc007fd0 tx=0x7f64dc009040 comp rx=0 tx=0).stop 2024-08-31T12:28:52.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.794+0000 7f64ee76e640 1 -- 172.21.15.19:0/2412394457 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f64e80c43a0 msgr2=0x7f64e81e40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:52.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.794+0000 7f64ee76e640 1 --2- 172.21.15.19:0/2412394457 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f64e80c43a0 0x7f64e81e40c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f64d800c480 tx=0x7f64d800c940 comp rx=0 tx=0).stop 2024-08-31T12:28:52.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.794+0000 7f64ecd0b640 1 -- 172.21.15.19:0/2412394457 reap_dead start 2024-08-31T12:28:52.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.794+0000 7f64ee76e640 1 -- 172.21.15.19:0/2412394457 shutdown_connections 2024-08-31T12:28:52.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.794+0000 7f64ee76e640 1 -- 172.21.15.19:0/2412394457 >> 172.21.15.19:0/2412394457 conn(0x7f64e8068aa0 msgr2=0x7f64e8149270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:52.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.794+0000 7f64ee76e640 1 -- 172.21.15.19:0/2412394457 shutdown_connections 2024-08-31T12:28:52.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.794+0000 7f64ee76e640 1 -- 172.21.15.19:0/2412394457 wait complete. 2024-08-31T12:28:52.807 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-08-31T12:28:52.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.923+0000 7fba44cbd640 1 -- 172.21.15.19:0/2155662125 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fba2c004d90 msgr2=0x7fba2c005220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:52.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.923+0000 7fba44cbd640 1 --2- 172.21.15.19:0/2155662125 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fba2c004d90 0x7fba2c005220 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fba38009b20 tx=0x7fba2c005830 comp rx=0 tx=0).stop 2024-08-31T12:28:52.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.923+0000 7fba3ed9d640 1 -- 172.21.15.19:0/2155662125 reap_dead start 2024-08-31T12:28:52.926 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.923+0000 7fba44cbd640 1 -- 172.21.15.19:0/2155662125 shutdown_connections 2024-08-31T12:28:52.926 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.923+0000 7fba44cbd640 1 -- 172.21.15.19:0/2155662125 >> 172.21.15.19:0/2155662125 conn(0x7fba400690d0 msgr2=0x7fba400d0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:52.926 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.923+0000 7fba44cbd640 1 -- 172.21.15.19:0/2155662125 shutdown_connections 2024-08-31T12:28:52.926 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.923+0000 7fba44cbd640 1 -- 172.21.15.19:0/2155662125 wait complete. 2024-08-31T12:28:52.926 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.925+0000 7fba44cbd640 1 Processor -- start 2024-08-31T12:28:52.927 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.925+0000 7fba44cbd640 1 -- start start 2024-08-31T12:28:52.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.926+0000 7fba44cbd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fba2c004d90 0x7fba400be9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:52.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.926+0000 7fba44cbd640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba400c0420 0x7fba4014cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:52.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.926+0000 7fba44cbd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fba400c0830 0x7fba4014d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:52.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.926+0000 7fba44cbd640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fba4015b250 con 0x7fba400c0830 2024-08-31T12:28:52.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.926+0000 7fba44cbd640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fba4015af70 con 0x7fba400c0420 2024-08-31T12:28:52.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.926+0000 7fba44cbd640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fba400c8610 con 0x7fba2c004d90 2024-08-31T12:28:52.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.926+0000 7fba3ed9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fba400c0830 0x7fba4014d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:52.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.926+0000 7fba3dd9b640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba400c0420 0x7fba4014cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:52.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.926+0000 7fba3ed9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fba400c0830 0x7fba4014d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:44018/0 (socket says 172.21.15.19:44018) 2024-08-31T12:28:52.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.926+0000 7fba3ed9d640 1 -- 172.21.15.19:0/2561081375 learned_addr learned my addr 172.21.15.19:0/2561081375 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:28:52.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.926+0000 7fba3e59c640 1 --2- 172.21.15.19:0/2561081375 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fba2c004d90 0x7fba400be9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:52.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.926+0000 7fba3ed9d640 1 -- 172.21.15.19:0/2561081375 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fba2c004d90 msgr2=0x7fba400be9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:52.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.926+0000 7fba3ed9d640 1 --2- 172.21.15.19:0/2561081375 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fba2c004d90 0x7fba400be9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:52.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.926+0000 7fba3ed9d640 1 -- 172.21.15.19:0/2561081375 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba400c0420 msgr2=0x7fba4014cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:52.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.926+0000 7fba3ed9d640 1 --2- 172.21.15.19:0/2561081375 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba400c0420 0x7fba4014cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:52.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.926+0000 7fba3ed9d640 1 -- 172.21.15.19:0/2561081375 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba400c0040 con 0x7fba400c0830 2024-08-31T12:28:52.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.926+0000 7fba3dd9b640 1 --2- 172.21.15.19:0/2561081375 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba400c0420 0x7fba4014cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:28:52.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.927+0000 7fba3ed9d640 1 --2- 172.21.15.19:0/2561081375 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fba400c0830 0x7fba4014d440 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fba38036040 tx=0x7fba38034680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:52.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.927+0000 7fba36ffd640 1 -- 172.21.15.19:0/2561081375 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba38038040 con 0x7fba400c0830 2024-08-31T12:28:52.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.927+0000 7fba36ffd640 1 -- 172.21.15.19:0/2561081375 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fba38042070 con 0x7fba400c0830 2024-08-31T12:28:52.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.927+0000 7fba44cbd640 1 -- 172.21.15.19:0/2561081375 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba4014fdb0 con 0x7fba400c0830 2024-08-31T12:28:52.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.927+0000 7fba36ffd640 1 -- 172.21.15.19:0/2561081375 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba38034b80 con 0x7fba400c0830 2024-08-31T12:28:52.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.927+0000 7fba44cbd640 1 -- 172.21.15.19:0/2561081375 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba4014fad0 con 0x7fba400c0830 2024-08-31T12:28:52.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.928+0000 7fba44cbd640 1 -- 172.21.15.19:0/2561081375 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba400d0200 con 0x7fba400c0830 2024-08-31T12:28:52.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.929+0000 7fba36ffd640 1 -- 172.21.15.19:0/2561081375 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fba38047460 con 0x7fba400c0830 2024-08-31T12:28:52.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.929+0000 7fba36ffd640 1 --2- 172.21.15.19:0/2561081375 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fba20077a50 0x7fba20079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:52.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.929+0000 7fba36ffd640 1 -- 172.21.15.19:0/2561081375 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fba380b9f70 con 0x7fba400c0830 2024-08-31T12:28:52.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.929+0000 7fba3e59c640 1 --2- 172.21.15.19:0/2561081375 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fba20077a50 0x7fba20079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:52.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.930+0000 7fba3e59c640 1 --2- 172.21.15.19:0/2561081375 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fba20077a50 0x7fba20079f60 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fba30001ab0 tx=0x7fba30006e90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:52.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:52.934+0000 7fba36ffd640 1 -- 172.21.15.19:0/2561081375 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fba38082590 con 0x7fba400c0830 2024-08-31T12:28:53.120 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.118+0000 7fba44cbd640 1 -- 172.21.15.19:0/2561081375 --> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7fba40155300 con 0x7fba20077a50 2024-08-31T12:28:53.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.119+0000 7fba36ffd640 1 -- 172.21.15.19:0/2561081375 <== mgr.4100 v2:172.21.15.43:6800/2824336655 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fba40155300 con 0x7fba20077a50 2024-08-31T12:28:53.121 INFO:teuthology.orchestra.run.smithi019.stderr:instructed osd(s) 4 to deep-scrub 2024-08-31T12:28:53.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.121+0000 7fba44cbd640 1 -- 172.21.15.19:0/2561081375 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fba20077a50 msgr2=0x7fba20079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:53.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.121+0000 7fba44cbd640 1 --2- 172.21.15.19:0/2561081375 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fba20077a50 0x7fba20079f60 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fba30001ab0 tx=0x7fba30006e90 comp rx=0 tx=0).stop 2024-08-31T12:28:53.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.121+0000 7fba44cbd640 1 -- 172.21.15.19:0/2561081375 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fba400c0830 msgr2=0x7fba4014d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:53.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.121+0000 7fba44cbd640 1 --2- 172.21.15.19:0/2561081375 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fba400c0830 0x7fba4014d440 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fba38036040 tx=0x7fba38034680 comp rx=0 tx=0).stop 2024-08-31T12:28:53.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.121+0000 7fba44cbd640 1 -- 172.21.15.19:0/2561081375 shutdown_connections 2024-08-31T12:28:53.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.121+0000 7fba44cbd640 1 --2- 172.21.15.19:0/2561081375 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fba20077a50 0x7fba20079f60 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:53.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.121+0000 7fba44cbd640 1 --2- 172.21.15.19:0/2561081375 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fba400c0830 0x7fba4014d440 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:53.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.121+0000 7fba44cbd640 1 --2- 172.21.15.19:0/2561081375 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba400c0420 0x7fba4014cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:53.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.121+0000 7fba44cbd640 1 --2- 172.21.15.19:0/2561081375 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fba2c004d90 0x7fba400be9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:53.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.121+0000 7fba44cbd640 1 -- 172.21.15.19:0/2561081375 >> 172.21.15.19:0/2561081375 conn(0x7fba400690d0 msgr2=0x7fba4015a460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:53.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.121+0000 7fba44cbd640 1 -- 172.21.15.19:0/2561081375 shutdown_connections 2024-08-31T12:28:53.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.121+0000 7fba44cbd640 1 -- 172.21.15.19:0/2561081375 wait complete. 2024-08-31T12:28:53.135 INFO:tasks.ceph:Scrubbing osd.5 2024-08-31T12:28:53.136 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-08-31T12:28:53.262 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.260+0000 7fc9dc9c3640 1 -- 172.21.15.19:0/4001707679 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc9d8152020 msgr2=0x7fc9d80cdce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:53.263 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.260+0000 7fc9dc9c3640 1 --2- 172.21.15.19:0/4001707679 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc9d8152020 0x7fc9d80cdce0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc009830 tx=0x7fc9cc02a650 comp rx=0 tx=0).stop 2024-08-31T12:28:53.263 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.260+0000 7fc9dc9c3640 1 -- 172.21.15.19:0/4001707679 shutdown_connections 2024-08-31T12:28:53.263 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.260+0000 7fc9dc9c3640 1 --2- 172.21.15.19:0/4001707679 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc9d8152020 0x7fc9d80cdce0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:53.263 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.260+0000 7fc9dc9c3640 1 -- 172.21.15.19:0/4001707679 >> 172.21.15.19:0/4001707679 conn(0x7fc9d80ca5f0 msgr2=0x7fc9d80cd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:53.263 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.260+0000 7fc9dc9c3640 1 -- 172.21.15.19:0/4001707679 shutdown_connections 2024-08-31T12:28:53.263 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.260+0000 7fc9dc9c3640 1 -- 172.21.15.19:0/4001707679 wait complete. 2024-08-31T12:28:53.264 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.262+0000 7fc9dc9c3640 1 Processor -- start 2024-08-31T12:28:53.264 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.262+0000 7fc9dc9c3640 1 -- start start 2024-08-31T12:28:53.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.264+0000 7fc9dc9c3640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc9d806dbf0 0x7fc9d80cb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:53.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.264+0000 7fc9dc9c3640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc9d80c7e90 0x7fc9d81e4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:53.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.264+0000 7fc9dc9c3640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc9d8152020 0x7fc9d813bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:53.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.264+0000 7fc9dc9c3640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d80c6b90 con 0x7fc9d8152020 2024-08-31T12:28:53.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.264+0000 7fc9dc9c3640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d80c6e90 con 0x7fc9d806dbf0 2024-08-31T12:28:53.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.264+0000 7fc9dc9c3640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fc9d8152430 con 0x7fc9d80c7e90 2024-08-31T12:28:53.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.264+0000 7fc9d6d9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc9d8152020 0x7fc9d813bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:53.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.264+0000 7fc9d6d9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc9d8152020 0x7fc9d813bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:44030/0 (socket says 172.21.15.19:44030) 2024-08-31T12:28:53.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.264+0000 7fc9d6d9d640 1 -- 172.21.15.19:0/3885818110 learned_addr learned my addr 172.21.15.19:0/3885818110 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:28:53.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.264+0000 7fc9d659c640 1 --2- 172.21.15.19:0/3885818110 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc9d806dbf0 0x7fc9d80cb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:53.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.264+0000 7fc9d5d9b640 1 --2- 172.21.15.19:0/3885818110 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc9d80c7e90 0x7fc9d81e4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:53.269 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.264+0000 7fc9d6d9d640 1 -- 172.21.15.19:0/3885818110 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc9d80c7e90 msgr2=0x7fc9d81e4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:53.269 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.264+0000 7fc9d6d9d640 1 --2- 172.21.15.19:0/3885818110 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc9d80c7e90 0x7fc9d81e4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:53.269 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.265+0000 7fc9d6d9d640 1 -- 172.21.15.19:0/3885818110 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc9d806dbf0 msgr2=0x7fc9d80cb5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:53.269 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.265+0000 7fc9d6d9d640 1 --2- 172.21.15.19:0/3885818110 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc9d806dbf0 0x7fc9d80cb5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:53.269 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.265+0000 7fc9d6d9d640 1 -- 172.21.15.19:0/3885818110 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9d81ee0d0 con 0x7fc9d8152020 2024-08-31T12:28:53.270 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.265+0000 7fc9d6d9d640 1 --2- 172.21.15.19:0/3885818110 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc9d8152020 0x7fc9d813bd30 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fc9d000c760 tx=0x7fc9d000cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:53.270 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.265+0000 7fc9c6ffd640 1 -- 172.21.15.19:0/3885818110 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d0003b70 con 0x7fc9d8152020 2024-08-31T12:28:53.270 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.265+0000 7fc9c6ffd640 1 -- 172.21.15.19:0/3885818110 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc9d0003d10 con 0x7fc9d8152020 2024-08-31T12:28:53.270 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.265+0000 7fc9dc9c3640 1 -- 172.21.15.19:0/3885818110 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9d80c4590 con 0x7fc9d8152020 2024-08-31T12:28:53.270 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.265+0000 7fc9c6ffd640 1 -- 172.21.15.19:0/3885818110 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d00052d0 con 0x7fc9d8152020 2024-08-31T12:28:53.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.265+0000 7fc9dc9c3640 1 -- 172.21.15.19:0/3885818110 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9d81f16c0 con 0x7fc9d8152020 2024-08-31T12:28:53.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.266+0000 7fc9dc9c3640 1 -- 172.21.15.19:0/3885818110 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc9d81ede70 con 0x7fc9d8152020 2024-08-31T12:28:53.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.267+0000 7fc9c6ffd640 1 -- 172.21.15.19:0/3885818110 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fc9d0005470 con 0x7fc9d8152020 2024-08-31T12:28:53.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.268+0000 7fc9c6ffd640 1 --2- 172.21.15.19:0/3885818110 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc9b8077a80 0x7fc9b8079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:53.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.268+0000 7fc9c6ffd640 1 -- 172.21.15.19:0/3885818110 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fc9d00993e0 con 0x7fc9d8152020 2024-08-31T12:28:53.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.268+0000 7fc9d659c640 1 --2- 172.21.15.19:0/3885818110 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc9b8077a80 0x7fc9b8079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:53.273 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.268+0000 7fc9c6ffd640 1 --2- 172.21.15.19:0/3885818110 >> [v2:172.21.15.43:6802/4271971521,v1:172.21.15.43:6803/4271971521] conn(0x7fc9b807fe50 0x7fc9b8082320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:53.273 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.268+0000 7fc9c6ffd640 1 -- 172.21.15.19:0/3885818110 --> [v2:172.21.15.43:6802/4271971521,v1:172.21.15.43:6803/4271971521] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc9b80829f0 con 0x7fc9b807fe50 2024-08-31T12:28:53.273 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.268+0000 7fc9c6ffd640 1 -- 172.21.15.19:0/3885818110 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fc9d00a0050 con 0x7fc9d8152020 2024-08-31T12:28:53.273 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.268+0000 7fc9d5d9b640 1 --2- 172.21.15.19:0/3885818110 >> [v2:172.21.15.43:6802/4271971521,v1:172.21.15.43:6803/4271971521] conn(0x7fc9b807fe50 0x7fc9b8082320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:53.273 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.269+0000 7fc9d659c640 1 --2- 172.21.15.19:0/3885818110 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc9b8077a80 0x7fc9b8079f90 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fc9c8007fd0 tx=0x7fc9c8001370 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:53.273 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.269+0000 7fc9d5d9b640 1 --2- 172.21.15.19:0/3885818110 >> [v2:172.21.15.43:6802/4271971521,v1:172.21.15.43:6803/4271971521] conn(0x7fc9b807fe50 0x7fc9b8082320 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:53.273 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.270+0000 7fc9c6ffd640 1 -- 172.21.15.19:0/3885818110 <== osd.5 v2:172.21.15.43:6802/4271971521 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc9b80829f0 con 0x7fc9b807fe50 2024-08-31T12:28:53.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.286+0000 7fc9dc9c3640 1 -- 172.21.15.19:0/3885818110 --> [v2:172.21.15.43:6802/4271971521,v1:172.21.15.43:6803/4271971521] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fc9d8156030 con 0x7fc9b807fe50 2024-08-31T12:28:53.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.286+0000 7fc9c6ffd640 1 -- 172.21.15.19:0/3885818110 <== osd.5 v2:172.21.15.43:6802/4271971521 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fc9d8156030 con 0x7fc9b807fe50 2024-08-31T12:28:53.289 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:28:53.289 INFO:teuthology.orchestra.run.smithi019.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-08-31T12:28:53.289 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:28:53.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.287+0000 7fc9dc9c3640 1 -- 172.21.15.19:0/3885818110 >> [v2:172.21.15.43:6802/4271971521,v1:172.21.15.43:6803/4271971521] conn(0x7fc9b807fe50 msgr2=0x7fc9b8082320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:53.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.287+0000 7fc9dc9c3640 1 --2- 172.21.15.19:0/3885818110 >> [v2:172.21.15.43:6802/4271971521,v1:172.21.15.43:6803/4271971521] conn(0x7fc9b807fe50 0x7fc9b8082320 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:53.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.287+0000 7fc9dc9c3640 1 -- 172.21.15.19:0/3885818110 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc9b8077a80 msgr2=0x7fc9b8079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:53.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.287+0000 7fc9dc9c3640 1 --2- 172.21.15.19:0/3885818110 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fc9b8077a80 0x7fc9b8079f90 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fc9c8007fd0 tx=0x7fc9c8001370 comp rx=0 tx=0).stop 2024-08-31T12:28:53.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.287+0000 7fc9dc9c3640 1 -- 172.21.15.19:0/3885818110 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc9d8152020 msgr2=0x7fc9d813bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:53.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.287+0000 7fc9dc9c3640 1 --2- 172.21.15.19:0/3885818110 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc9d8152020 0x7fc9d813bd30 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fc9d000c760 tx=0x7fc9d000cb60 comp rx=0 tx=0).stop 2024-08-31T12:28:53.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.287+0000 7fc9d6d9d640 1 -- 172.21.15.19:0/3885818110 reap_dead start 2024-08-31T12:28:53.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.287+0000 7fc9dc9c3640 1 -- 172.21.15.19:0/3885818110 shutdown_connections 2024-08-31T12:28:53.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.287+0000 7fc9dc9c3640 1 -- 172.21.15.19:0/3885818110 >> 172.21.15.19:0/3885818110 conn(0x7fc9d80ca5f0 msgr2=0x7fc9d80c8b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:53.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.287+0000 7fc9dc9c3640 1 -- 172.21.15.19:0/3885818110 shutdown_connections 2024-08-31T12:28:53.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.287+0000 7fc9dc9c3640 1 -- 172.21.15.19:0/3885818110 wait complete. 2024-08-31T12:28:53.302 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-08-31T12:28:53.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.438+0000 7efe8e721640 1 -- 172.21.15.19:0/2138276449 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efe8815cfc0 msgr2=0x7efe88159880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:53.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.438+0000 7efe8e721640 1 --2- 172.21.15.19:0/2138276449 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efe8815cfc0 0x7efe88159880 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7efe880c5860 tx=0x7efe7802a5c0 comp rx=0 tx=0).stop 2024-08-31T12:28:53.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.439+0000 7efe8e721640 1 -- 172.21.15.19:0/2138276449 shutdown_connections 2024-08-31T12:28:53.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.439+0000 7efe8e721640 1 --2- 172.21.15.19:0/2138276449 >> v2:172.21.15.43:3300/0 conn(0x7efe880c9e20 0x7efe8814f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:53.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.439+0000 7efe8e721640 1 --2- 172.21.15.19:0/2138276449 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efe8815cfc0 0x7efe88159880 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:53.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.439+0000 7efe8e721640 1 -- 172.21.15.19:0/2138276449 >> 172.21.15.19:0/2138276449 conn(0x7efe880690d0 msgr2=0x7efe88072800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:53.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.439+0000 7efe8e721640 1 -- 172.21.15.19:0/2138276449 shutdown_connections 2024-08-31T12:28:53.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.439+0000 7efe8e721640 1 -- 172.21.15.19:0/2138276449 wait complete. 2024-08-31T12:28:53.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.441+0000 7efe8e721640 1 Processor -- start 2024-08-31T12:28:53.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.441+0000 7efe8e721640 1 -- start start 2024-08-31T12:28:53.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.443+0000 7efe8e721640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efe88070df0 0x7efe88157160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:53.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.443+0000 7efe8e721640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe880c9e20 0x7efe8806ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:53.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.443+0000 7efe8e721640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efe8815cfc0 0x7efe88148e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:53.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.443+0000 7efe8e721640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7efe880ca230 con 0x7efe8815cfc0 2024-08-31T12:28:53.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.443+0000 7efe8e721640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7efe881534f0 con 0x7efe880c9e20 2024-08-31T12:28:53.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.443+0000 7efe8e721640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7efe880c8f40 con 0x7efe88070df0 2024-08-31T12:28:53.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.443+0000 7efe8ccbe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efe8815cfc0 0x7efe88148e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:53.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.443+0000 7efe8ccbe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efe8815cfc0 0x7efe88148e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:44056/0 (socket says 172.21.15.19:44056) 2024-08-31T12:28:53.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.443+0000 7efe87fff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efe88070df0 0x7efe88157160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:53.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.443+0000 7efe877fe640 1 --2- 172.21.15.19:0/3602373520 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe880c9e20 0x7efe8806ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:53.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.443+0000 7efe8ccbe640 1 -- 172.21.15.19:0/3602373520 learned_addr learned my addr 172.21.15.19:0/3602373520 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:28:53.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.443+0000 7efe8ccbe640 1 -- 172.21.15.19:0/3602373520 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efe88070df0 msgr2=0x7efe88157160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:53.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.443+0000 7efe8ccbe640 1 --2- 172.21.15.19:0/3602373520 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efe88070df0 0x7efe88157160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:53.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.443+0000 7efe8ccbe640 1 -- 172.21.15.19:0/3602373520 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe880c9e20 msgr2=0x7efe8806ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:53.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.443+0000 7efe8ccbe640 1 --2- 172.21.15.19:0/3602373520 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe880c9e20 0x7efe8806ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:53.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.443+0000 7efe8ccbe640 1 -- 172.21.15.19:0/3602373520 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe8815b310 con 0x7efe8815cfc0 2024-08-31T12:28:53.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.443+0000 7efe877fe640 1 --2- 172.21.15.19:0/3602373520 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe880c9e20 0x7efe8806ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:28:53.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.443+0000 7efe8ccbe640 1 --2- 172.21.15.19:0/3602373520 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efe8815cfc0 0x7efe88148e90 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7efe8000cb10 tx=0x7efe8000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:53.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.444+0000 7efe84ff9640 1 -- 172.21.15.19:0/3602373520 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe80016020 con 0x7efe8815cfc0 2024-08-31T12:28:53.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.444+0000 7efe84ff9640 1 -- 172.21.15.19:0/3602373520 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efe8000ad60 con 0x7efe8815cfc0 2024-08-31T12:28:53.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.444+0000 7efe84ff9640 1 -- 172.21.15.19:0/3602373520 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe80012680 con 0x7efe8815cfc0 2024-08-31T12:28:53.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.444+0000 7efe8e721640 1 -- 172.21.15.19:0/3602373520 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe88157760 con 0x7efe8815cfc0 2024-08-31T12:28:53.449 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.444+0000 7efe8e721640 1 -- 172.21.15.19:0/3602373520 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe8815a320 con 0x7efe8815cfc0 2024-08-31T12:28:53.449 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.445+0000 7efe8e721640 1 -- 172.21.15.19:0/3602373520 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe8815a580 con 0x7efe8815cfc0 2024-08-31T12:28:53.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.446+0000 7efe84ff9640 1 -- 172.21.15.19:0/3602373520 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7efe80019020 con 0x7efe8815cfc0 2024-08-31T12:28:53.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.446+0000 7efe84ff9640 1 --2- 172.21.15.19:0/3602373520 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7efe74077a50 0x7efe74079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:53.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.446+0000 7efe84ff9640 1 -- 172.21.15.19:0/3602373520 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7efe800998b0 con 0x7efe8815cfc0 2024-08-31T12:28:53.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.446+0000 7efe87fff640 1 --2- 172.21.15.19:0/3602373520 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7efe74077a50 0x7efe74079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:53.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.447+0000 7efe87fff640 1 --2- 172.21.15.19:0/3602373520 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7efe74077a50 0x7efe74079f60 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7efe78034000 tx=0x7efe78004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:53.454 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.452+0000 7efe84ff9640 1 -- 172.21.15.19:0/3602373520 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7efe80020030 con 0x7efe8815cfc0 2024-08-31T12:28:53.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.666+0000 7efe8e721640 1 -- 172.21.15.19:0/3602373520 --> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7efe88149e30 con 0x7efe74077a50 2024-08-31T12:28:53.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.667+0000 7efe84ff9640 1 -- 172.21.15.19:0/3602373520 <== mgr.4100 v2:172.21.15.43:6800/2824336655 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7efe88149e30 con 0x7efe74077a50 2024-08-31T12:28:53.670 INFO:teuthology.orchestra.run.smithi019.stderr:instructed osd(s) 5 to deep-scrub 2024-08-31T12:28:53.672 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.670+0000 7efe8e721640 1 -- 172.21.15.19:0/3602373520 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7efe74077a50 msgr2=0x7efe74079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:53.672 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.670+0000 7efe8e721640 1 --2- 172.21.15.19:0/3602373520 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7efe74077a50 0x7efe74079f60 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7efe78034000 tx=0x7efe78004730 comp rx=0 tx=0).stop 2024-08-31T12:28:53.672 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.670+0000 7efe8e721640 1 -- 172.21.15.19:0/3602373520 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efe8815cfc0 msgr2=0x7efe88148e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:53.672 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.670+0000 7efe8e721640 1 --2- 172.21.15.19:0/3602373520 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efe8815cfc0 0x7efe88148e90 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7efe8000cb10 tx=0x7efe8000a060 comp rx=0 tx=0).stop 2024-08-31T12:28:53.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.670+0000 7efe8e721640 1 -- 172.21.15.19:0/3602373520 shutdown_connections 2024-08-31T12:28:53.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.670+0000 7efe8e721640 1 --2- 172.21.15.19:0/3602373520 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7efe74077a50 0x7efe74079f60 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:53.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.670+0000 7efe8e721640 1 --2- 172.21.15.19:0/3602373520 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efe8815cfc0 0x7efe88148e90 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:53.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.670+0000 7efe8e721640 1 --2- 172.21.15.19:0/3602373520 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe880c9e20 0x7efe8806ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:53.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.670+0000 7efe8e721640 1 --2- 172.21.15.19:0/3602373520 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efe88070df0 0x7efe88157160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:53.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.670+0000 7efe8e721640 1 -- 172.21.15.19:0/3602373520 >> 172.21.15.19:0/3602373520 conn(0x7efe880690d0 msgr2=0x7efe880cb950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:53.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.670+0000 7efe8e721640 1 -- 172.21.15.19:0/3602373520 shutdown_connections 2024-08-31T12:28:53.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.670+0000 7efe8e721640 1 -- 172.21.15.19:0/3602373520 wait complete. 2024-08-31T12:28:53.683 INFO:tasks.ceph:Scrubbing osd.6 2024-08-31T12:28:53.683 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-08-31T12:28:53.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.800+0000 7f2aa2d1a640 1 -- 172.21.15.19:0/3347691138 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2a9c0cc150 msgr2=0x7f2a9c0d0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:53.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.800+0000 7f2aa2d1a640 1 --2- 172.21.15.19:0/3347691138 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2a9c0cc150 0x7f2a9c0d0bf0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f2a88009830 tx=0x7f2a8802a640 comp rx=0 tx=0).stop 2024-08-31T12:28:53.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.801+0000 7f2aa2d1a640 1 -- 172.21.15.19:0/3347691138 shutdown_connections 2024-08-31T12:28:53.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.801+0000 7f2aa2d1a640 1 --2- 172.21.15.19:0/3347691138 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2a9c0cc150 0x7f2a9c0d0bf0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:53.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.801+0000 7f2aa2d1a640 1 -- 172.21.15.19:0/3347691138 >> 172.21.15.19:0/3347691138 conn(0x7f2a9c074210 msgr2=0x7f2a9c074620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:53.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.801+0000 7f2aa2d1a640 1 -- 172.21.15.19:0/3347691138 shutdown_connections 2024-08-31T12:28:53.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.801+0000 7f2aa2d1a640 1 -- 172.21.15.19:0/3347691138 wait complete. 2024-08-31T12:28:53.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.802+0000 7f2aa2d1a640 1 Processor -- start 2024-08-31T12:28:53.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.802+0000 7f2aa2d1a640 1 -- start start 2024-08-31T12:28:53.807 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.804+0000 7f2aa2d1a640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2a9c0cbcf0 0x7f2a9c1e4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:53.807 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.804+0000 7f2aa2d1a640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2a9c0cc150 0x7f2a9c0c94e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:53.807 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.804+0000 7f2aa2d1a640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a9c0d1130 0x7f2a9c152a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:53.807 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.804+0000 7f2aa2d1a640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f2a9c0cc560 con 0x7f2a9c0cbcf0 2024-08-31T12:28:53.807 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.804+0000 7f2aa2d1a640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2a9c0b4680 con 0x7f2a9c0d1130 2024-08-31T12:28:53.807 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.804+0000 7f2aa2d1a640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f2a9c0b3880 con 0x7f2a9c0cc150 2024-08-31T12:28:53.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.804+0000 7f2aa0ab6640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2a9c0cbcf0 0x7f2a9c1e4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:53.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.804+0000 7f2aa0ab6640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2a9c0cbcf0 0x7f2a9c1e4d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:44078/0 (socket says 172.21.15.19:44078) 2024-08-31T12:28:53.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.804+0000 7f2aa0ab6640 1 -- 172.21.15.19:0/947086133 learned_addr learned my addr 172.21.15.19:0/947086133 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:28:53.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.804+0000 7f2aa12b7640 1 --2- 172.21.15.19:0/947086133 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a9c0d1130 0x7f2a9c152a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:53.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.804+0000 7f2a9bfff640 1 --2- 172.21.15.19:0/947086133 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2a9c0cc150 0x7f2a9c0c94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:53.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.804+0000 7f2aa0ab6640 1 -- 172.21.15.19:0/947086133 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2a9c0cc150 msgr2=0x7f2a9c0c94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:53.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.804+0000 7f2aa0ab6640 1 --2- 172.21.15.19:0/947086133 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2a9c0cc150 0x7f2a9c0c94e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:53.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.804+0000 7f2aa0ab6640 1 -- 172.21.15.19:0/947086133 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a9c0d1130 msgr2=0x7f2a9c152a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:53.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.804+0000 7f2aa0ab6640 1 --2- 172.21.15.19:0/947086133 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a9c0d1130 0x7f2a9c152a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:53.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.804+0000 7f2aa0ab6640 1 -- 172.21.15.19:0/947086133 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a9c1e65d0 con 0x7f2a9c0cbcf0 2024-08-31T12:28:53.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.804+0000 7f2aa0ab6640 1 --2- 172.21.15.19:0/947086133 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2a9c0cbcf0 0x7f2a9c1e4d80 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f2a9000c390 tx=0x7f2a9000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:53.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.804+0000 7f2a997fa640 1 -- 172.21.15.19:0/947086133 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a90017070 con 0x7f2a9c0cbcf0 2024-08-31T12:28:53.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.804+0000 7f2a997fa640 1 -- 172.21.15.19:0/947086133 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2a900053c0 con 0x7f2a9c0cbcf0 2024-08-31T12:28:53.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.804+0000 7f2aa2d1a640 1 -- 172.21.15.19:0/947086133 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a9c1e1690 con 0x7f2a9c0cbcf0 2024-08-31T12:28:53.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.805+0000 7f2a997fa640 1 -- 172.21.15.19:0/947086133 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a90003880 con 0x7f2a9c0cbcf0 2024-08-31T12:28:53.810 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.805+0000 7f2aa2d1a640 1 -- 172.21.15.19:0/947086133 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a9c1e3770 con 0x7f2a9c0cbcf0 2024-08-31T12:28:53.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.805+0000 7f2aa2d1a640 1 -- 172.21.15.19:0/947086133 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2a9c0d0bf0 con 0x7f2a9c0cbcf0 2024-08-31T12:28:53.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.806+0000 7f2a997fa640 1 -- 172.21.15.19:0/947086133 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f2a9001a070 con 0x7f2a9c0cbcf0 2024-08-31T12:28:53.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.807+0000 7f2a997fa640 1 --2- 172.21.15.19:0/947086133 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2a80077a50 0x7f2a80079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:53.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.807+0000 7f2a997fa640 1 -- 172.21.15.19:0/947086133 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f2a90099110 con 0x7f2a9c0cbcf0 2024-08-31T12:28:53.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.807+0000 7f2a997fa640 1 --2- 172.21.15.19:0/947086133 >> [v2:172.21.15.43:6826/2863384694,v1:172.21.15.43:6827/2863384694] conn(0x7f2a8007fe20 0x7f2a800822f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:53.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.807+0000 7f2a997fa640 1 -- 172.21.15.19:0/947086133 --> [v2:172.21.15.43:6826/2863384694,v1:172.21.15.43:6827/2863384694] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2a90009e50 con 0x7f2a8007fe20 2024-08-31T12:28:53.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.807+0000 7f2a997fa640 1 -- 172.21.15.19:0/947086133 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f2a900995a0 con 0x7f2a9c0cbcf0 2024-08-31T12:28:53.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.807+0000 7f2a9bfff640 1 --2- 172.21.15.19:0/947086133 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2a80077a50 0x7f2a80079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:53.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.807+0000 7f2aa12b7640 1 --2- 172.21.15.19:0/947086133 >> [v2:172.21.15.43:6826/2863384694,v1:172.21.15.43:6827/2863384694] conn(0x7f2a8007fe20 0x7f2a800822f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:53.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.808+0000 7f2a9bfff640 1 --2- 172.21.15.19:0/947086133 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2a80077a50 0x7f2a80079f60 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f2a88034000 tx=0x7f2a88004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:53.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.808+0000 7f2aa12b7640 1 --2- 172.21.15.19:0/947086133 >> [v2:172.21.15.43:6826/2863384694,v1:172.21.15.43:6827/2863384694] conn(0x7f2a8007fe20 0x7f2a800822f0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:53.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.809+0000 7f2a997fa640 1 -- 172.21.15.19:0/947086133 <== osd.6 v2:172.21.15.43:6826/2863384694 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2a90009e50 con 0x7f2a8007fe20 2024-08-31T12:28:53.823 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.821+0000 7f2aa2d1a640 1 -- 172.21.15.19:0/947086133 --> [v2:172.21.15.43:6826/2863384694,v1:172.21.15.43:6827/2863384694] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f2a9c1e72e0 con 0x7f2a8007fe20 2024-08-31T12:28:53.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.822+0000 7f2a997fa640 1 -- 172.21.15.19:0/947086133 <== osd.6 v2:172.21.15.43:6826/2863384694 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f2a9c1e72e0 con 0x7f2a8007fe20 2024-08-31T12:28:53.824 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:28:53.824 INFO:teuthology.orchestra.run.smithi019.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-08-31T12:28:53.824 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:28:53.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.822+0000 7f2aa2d1a640 1 -- 172.21.15.19:0/947086133 >> [v2:172.21.15.43:6826/2863384694,v1:172.21.15.43:6827/2863384694] conn(0x7f2a8007fe20 msgr2=0x7f2a800822f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:53.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.822+0000 7f2aa2d1a640 1 --2- 172.21.15.19:0/947086133 >> [v2:172.21.15.43:6826/2863384694,v1:172.21.15.43:6827/2863384694] conn(0x7f2a8007fe20 0x7f2a800822f0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:53.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.822+0000 7f2aa2d1a640 1 -- 172.21.15.19:0/947086133 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2a80077a50 msgr2=0x7f2a80079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:53.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.822+0000 7f2aa2d1a640 1 --2- 172.21.15.19:0/947086133 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2a80077a50 0x7f2a80079f60 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f2a88034000 tx=0x7f2a88004730 comp rx=0 tx=0).stop 2024-08-31T12:28:53.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.822+0000 7f2aa2d1a640 1 -- 172.21.15.19:0/947086133 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2a9c0cbcf0 msgr2=0x7f2a9c1e4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:53.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.822+0000 7f2aa2d1a640 1 --2- 172.21.15.19:0/947086133 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2a9c0cbcf0 0x7f2a9c1e4d80 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f2a9000c390 tx=0x7f2a9000c850 comp rx=0 tx=0).stop 2024-08-31T12:28:53.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.822+0000 7f2aa12b7640 1 -- 172.21.15.19:0/947086133 reap_dead start 2024-08-31T12:28:53.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.822+0000 7f2aa2d1a640 1 -- 172.21.15.19:0/947086133 shutdown_connections 2024-08-31T12:28:53.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.822+0000 7f2aa2d1a640 1 -- 172.21.15.19:0/947086133 >> 172.21.15.19:0/947086133 conn(0x7f2a9c074210 msgr2=0x7f2a9c15aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:53.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.822+0000 7f2aa2d1a640 1 -- 172.21.15.19:0/947086133 shutdown_connections 2024-08-31T12:28:53.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.822+0000 7f2aa2d1a640 1 -- 172.21.15.19:0/947086133 wait complete. 2024-08-31T12:28:53.836 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-08-31T12:28:53.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.949+0000 7f62381ad640 1 -- 172.21.15.19:0/3905533204 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f623015d890 msgr2=0x7f623015dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:53.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.949+0000 7f62381ad640 1 --2- 172.21.15.19:0/3905533204 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f623015d890 0x7f623015dca0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f623008e3f0 tx=0x7f622802a5d0 comp rx=0 tx=0).stop 2024-08-31T12:28:53.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.949+0000 7f62381ad640 1 -- 172.21.15.19:0/3905533204 shutdown_connections 2024-08-31T12:28:53.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.949+0000 7f62381ad640 1 --2- 172.21.15.19:0/3905533204 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f623015d890 0x7f623015dca0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:53.952 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.949+0000 7f62381ad640 1 --2- 172.21.15.19:0/3905533204 >> v2:172.21.15.43:3300/0 conn(0x7f62300cc620 0x7f62300cdab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:53.952 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.949+0000 7f62381ad640 1 -- 172.21.15.19:0/3905533204 >> 172.21.15.19:0/3905533204 conn(0x7f623006e6e0 msgr2=0x7f623006eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:53.952 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.949+0000 7f62381ad640 1 -- 172.21.15.19:0/3905533204 shutdown_connections 2024-08-31T12:28:53.952 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.949+0000 7f62381ad640 1 -- 172.21.15.19:0/3905533204 wait complete. 2024-08-31T12:28:53.953 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.950+0000 7f62381ad640 1 Processor -- start 2024-08-31T12:28:53.953 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.950+0000 7f62381ad640 1 -- start start 2024-08-31T12:28:53.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.952+0000 7f62381ad640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62300ca770 0x7f6230153110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:53.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.952+0000 7f62381ad640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62300cc620 0x7f6230155df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:53.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.952+0000 7f62381ad640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f623015d890 0x7f62300ad790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:53.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.952+0000 7f62381ad640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f623015d090 con 0x7f62300ca770 2024-08-31T12:28:53.956 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.952+0000 7f62381ad640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f623015cdb0 con 0x7f62300cc620 2024-08-31T12:28:53.956 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.952+0000 7f62381ad640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f623015e1e0 con 0x7f623015d890 2024-08-31T12:28:53.956 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.952+0000 7f6235f49640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62300ca770 0x7f6230153110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:53.956 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.952+0000 7f6235f49640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62300ca770 0x7f6230153110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:44096/0 (socket says 172.21.15.19:44096) 2024-08-31T12:28:53.956 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.952+0000 7f6235f49640 1 -- 172.21.15.19:0/1374737198 learned_addr learned my addr 172.21.15.19:0/1374737198 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:28:53.956 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.952+0000 7f6235748640 1 --2- 172.21.15.19:0/1374737198 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62300cc620 0x7f6230155df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:53.956 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.952+0000 7f623674a640 1 --2- 172.21.15.19:0/1374737198 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f623015d890 0x7f62300ad790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:53.956 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.952+0000 7f6235f49640 1 -- 172.21.15.19:0/1374737198 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f623015d890 msgr2=0x7f62300ad790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:53.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.952+0000 7f6235f49640 1 --2- 172.21.15.19:0/1374737198 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f623015d890 0x7f62300ad790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:53.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.952+0000 7f6235f49640 1 -- 172.21.15.19:0/1374737198 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62300cc620 msgr2=0x7f6230155df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:53.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.952+0000 7f6235f49640 1 --2- 172.21.15.19:0/1374737198 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62300cc620 0x7f6230155df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:53.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.952+0000 7f6235f49640 1 -- 172.21.15.19:0/1374737198 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6230153650 con 0x7f62300ca770 2024-08-31T12:28:53.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.952+0000 7f6235f49640 1 --2- 172.21.15.19:0/1374737198 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62300ca770 0x7f6230153110 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f622c00ea50 tx=0x7f622c00ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:53.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.952+0000 7f62267fc640 1 -- 172.21.15.19:0/1374737198 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f622c00cdc0 con 0x7f62300ca770 2024-08-31T12:28:53.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.952+0000 7f62267fc640 1 -- 172.21.15.19:0/1374737198 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f622c011560 con 0x7f62300ca770 2024-08-31T12:28:53.958 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.952+0000 7f62381ad640 1 -- 172.21.15.19:0/1374737198 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6230156330 con 0x7f62300ca770 2024-08-31T12:28:53.958 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.952+0000 7f62267fc640 1 -- 172.21.15.19:0/1374737198 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f622c011990 con 0x7f62300ca770 2024-08-31T12:28:53.958 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.953+0000 7f62381ad640 1 -- 172.21.15.19:0/1374737198 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62301f1790 con 0x7f62300ca770 2024-08-31T12:28:53.959 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.953+0000 7f62381ad640 1 -- 172.21.15.19:0/1374737198 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62300cf920 con 0x7f62300ca770 2024-08-31T12:28:53.959 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.954+0000 7f62267fc640 1 -- 172.21.15.19:0/1374737198 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f622c011b30 con 0x7f62300ca770 2024-08-31T12:28:53.959 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.955+0000 7f62267fc640 1 --2- 172.21.15.19:0/1374737198 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6214077a80 0x7f6214079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:53.959 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.955+0000 7f62267fc640 1 -- 172.21.15.19:0/1374737198 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f622c09a2d0 con 0x7f62300ca770 2024-08-31T12:28:53.959 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.955+0000 7f6235748640 1 --2- 172.21.15.19:0/1374737198 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6214077a80 0x7f6214079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:53.959 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.955+0000 7f6235748640 1 --2- 172.21.15.19:0/1374737198 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6214077a80 0x7f6214079f90 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f62300cb900 tx=0x7f6220009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:53.961 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:53.959+0000 7f62267fc640 1 -- 172.21.15.19:0/1374737198 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f62300cf920 con 0x7f62300ca770 2024-08-31T12:28:54.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.143+0000 7f62381ad640 1 -- 172.21.15.19:0/1374737198 --> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7f6230159270 con 0x7f6214077a80 2024-08-31T12:28:54.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.144+0000 7f62267fc640 1 -- 172.21.15.19:0/1374737198 <== mgr.4100 v2:172.21.15.43:6800/2824336655 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f6230159270 con 0x7f6214077a80 2024-08-31T12:28:54.146 INFO:teuthology.orchestra.run.smithi019.stderr:instructed osd(s) 6 to deep-scrub 2024-08-31T12:28:54.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.146+0000 7f62381ad640 1 -- 172.21.15.19:0/1374737198 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6214077a80 msgr2=0x7f6214079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:54.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.146+0000 7f62381ad640 1 --2- 172.21.15.19:0/1374737198 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6214077a80 0x7f6214079f90 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f62300cb900 tx=0x7f6220009040 comp rx=0 tx=0).stop 2024-08-31T12:28:54.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.146+0000 7f62381ad640 1 -- 172.21.15.19:0/1374737198 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62300ca770 msgr2=0x7f6230153110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:54.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.146+0000 7f62381ad640 1 --2- 172.21.15.19:0/1374737198 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62300ca770 0x7f6230153110 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f622c00ea50 tx=0x7f622c00ef10 comp rx=0 tx=0).stop 2024-08-31T12:28:54.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.146+0000 7f62381ad640 1 -- 172.21.15.19:0/1374737198 shutdown_connections 2024-08-31T12:28:54.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.146+0000 7f62381ad640 1 --2- 172.21.15.19:0/1374737198 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f6214077a80 0x7f6214079f90 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:54.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.146+0000 7f62381ad640 1 --2- 172.21.15.19:0/1374737198 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f623015d890 0x7f62300ad790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:54.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.146+0000 7f62381ad640 1 --2- 172.21.15.19:0/1374737198 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f62300cc620 0x7f6230155df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:54.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.146+0000 7f62381ad640 1 --2- 172.21.15.19:0/1374737198 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62300ca770 0x7f6230153110 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:54.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.146+0000 7f62381ad640 1 -- 172.21.15.19:0/1374737198 >> 172.21.15.19:0/1374737198 conn(0x7f623006e6e0 msgr2=0x7f62300cd0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:54.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.146+0000 7f62381ad640 1 -- 172.21.15.19:0/1374737198 shutdown_connections 2024-08-31T12:28:54.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.146+0000 7f62381ad640 1 -- 172.21.15.19:0/1374737198 wait complete. 2024-08-31T12:28:54.159 INFO:tasks.ceph:Scrubbing osd.7 2024-08-31T12:28:54.159 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-08-31T12:28:54.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.279+0000 7f2b9f6ec640 1 -- 172.21.15.19:0/1553911576 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2b9806ea50 msgr2=0x7f2b980b3450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:54.282 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.279+0000 7f2b9f6ec640 1 --2- 172.21.15.19:0/1553911576 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2b9806ea50 0x7f2b980b3450 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2b9808e880 tx=0x7f2b9402a530 comp rx=0 tx=0).stop 2024-08-31T12:28:54.282 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.279+0000 7f2b9f6ec640 1 -- 172.21.15.19:0/1553911576 shutdown_connections 2024-08-31T12:28:54.282 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.279+0000 7f2b9f6ec640 1 --2- 172.21.15.19:0/1553911576 >> v2:172.21.15.43:3300/0 conn(0x7f2b980b3d10 0x7f2b980b5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:54.282 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.279+0000 7f2b9f6ec640 1 --2- 172.21.15.19:0/1553911576 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2b9806ea50 0x7f2b980b3450 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:54.282 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.279+0000 7f2b9f6ec640 1 -- 172.21.15.19:0/1553911576 >> 172.21.15.19:0/1553911576 conn(0x7f2b98069290 msgr2=0x7f2b980d0770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:54.282 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.279+0000 7f2b9f6ec640 1 -- 172.21.15.19:0/1553911576 shutdown_connections 2024-08-31T12:28:54.282 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.279+0000 7f2b9f6ec640 1 -- 172.21.15.19:0/1553911576 wait complete. 2024-08-31T12:28:54.283 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.281+0000 7f2b9f6ec640 1 Processor -- start 2024-08-31T12:28:54.283 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.281+0000 7f2b9f6ec640 1 -- start start 2024-08-31T12:28:54.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.283+0000 7f2b9f6ec640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2b9806ea50 0x7f2b980cee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:54.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.283+0000 7f2b9f6ec640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2b980b3d10 0x7f2b98152a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:54.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.283+0000 7f2b9f6ec640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2b980b5590 0x7f2b9814b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:54.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.283+0000 7f2b9f6ec640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f2b980b3990 con 0x7f2b980b3d10 2024-08-31T12:28:54.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.283+0000 7f2b9f6ec640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2b980b4b30 con 0x7f2b980b5590 2024-08-31T12:28:54.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.283+0000 7f2b9f6ec640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f2b980b3b10 con 0x7f2b9806ea50 2024-08-31T12:28:54.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.283+0000 7f2b9cc87640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2b980b3d10 0x7f2b98152a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:54.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.283+0000 7f2b9cc87640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2b980b3d10 0x7f2b98152a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:44118/0 (socket says 172.21.15.19:44118) 2024-08-31T12:28:54.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.283+0000 7f2b9cc87640 1 -- 172.21.15.19:0/3296089370 learned_addr learned my addr 172.21.15.19:0/3296089370 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:28:54.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.283+0000 7f2b9dc89640 1 --2- 172.21.15.19:0/3296089370 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2b980b5590 0x7f2b9814b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:54.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.283+0000 7f2b9d488640 1 --2- 172.21.15.19:0/3296089370 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2b9806ea50 0x7f2b980cee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:54.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.283+0000 7f2b9cc87640 1 -- 172.21.15.19:0/3296089370 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2b9806ea50 msgr2=0x7f2b980cee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:54.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.283+0000 7f2b9cc87640 1 --2- 172.21.15.19:0/3296089370 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2b9806ea50 0x7f2b980cee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:54.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.283+0000 7f2b9cc87640 1 -- 172.21.15.19:0/3296089370 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2b980b5590 msgr2=0x7f2b9814b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:54.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.283+0000 7f2b9cc87640 1 --2- 172.21.15.19:0/3296089370 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2b980b5590 0x7f2b9814b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:54.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.283+0000 7f2b9cc87640 1 -- 172.21.15.19:0/3296089370 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b980cf3c0 con 0x7f2b980b3d10 2024-08-31T12:28:54.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.283+0000 7f2b9cc87640 1 --2- 172.21.15.19:0/3296089370 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2b980b3d10 0x7f2b98152a50 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2b88009fd0 tx=0x7f2b8800ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:54.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.284+0000 7f2b8dffb640 1 -- 172.21.15.19:0/3296089370 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b8800ccc0 con 0x7f2b980b3d10 2024-08-31T12:28:54.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.284+0000 7f2b8dffb640 1 -- 172.21.15.19:0/3296089370 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2b8800ee70 con 0x7f2b980b3d10 2024-08-31T12:28:54.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.284+0000 7f2b9f6ec640 1 -- 172.21.15.19:0/3296089370 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b980d0100 con 0x7f2b980b3d10 2024-08-31T12:28:54.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.284+0000 7f2b8dffb640 1 -- 172.21.15.19:0/3296089370 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b8800ce60 con 0x7f2b980b3d10 2024-08-31T12:28:54.289 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.284+0000 7f2b9f6ec640 1 -- 172.21.15.19:0/3296089370 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b981568d0 con 0x7f2b980b3d10 2024-08-31T12:28:54.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.285+0000 7f2b9f6ec640 1 -- 172.21.15.19:0/3296089370 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2b980cfeb0 con 0x7f2b980b3d10 2024-08-31T12:28:54.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.286+0000 7f2b8dffb640 1 -- 172.21.15.19:0/3296089370 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f2b88005260 con 0x7f2b980b3d10 2024-08-31T12:28:54.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.286+0000 7f2b8dffb640 1 --2- 172.21.15.19:0/3296089370 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2b7c077a50 0x7f2b7c079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:54.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.287+0000 7f2b8dffb640 1 -- 172.21.15.19:0/3296089370 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f2b88099f30 con 0x7f2b980b3d10 2024-08-31T12:28:54.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.287+0000 7f2b9d488640 1 --2- 172.21.15.19:0/3296089370 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2b7c077a50 0x7f2b7c079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:54.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.287+0000 7f2b8dffb640 1 --2- 172.21.15.19:0/3296089370 >> [v2:172.21.15.43:6810/792572234,v1:172.21.15.43:6811/792572234] conn(0x7f2b7c07fe20 0x7f2b7c0822f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:54.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.287+0000 7f2b8dffb640 1 -- 172.21.15.19:0/3296089370 --> [v2:172.21.15.43:6810/792572234,v1:172.21.15.43:6811/792572234] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2b7c0829c0 con 0x7f2b7c07fe20 2024-08-31T12:28:54.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.287+0000 7f2b8dffb640 1 -- 172.21.15.19:0/3296089370 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f2b8809c630 con 0x7f2b980b3d10 2024-08-31T12:28:54.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.287+0000 7f2b9dc89640 1 --2- 172.21.15.19:0/3296089370 >> [v2:172.21.15.43:6810/792572234,v1:172.21.15.43:6811/792572234] conn(0x7f2b7c07fe20 0x7f2b7c0822f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:54.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.287+0000 7f2b9d488640 1 --2- 172.21.15.19:0/3296089370 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2b7c077a50 0x7f2b7c079f60 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f2b94034000 tx=0x7f2b94004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:54.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.287+0000 7f2b9dc89640 1 --2- 172.21.15.19:0/3296089370 >> [v2:172.21.15.43:6810/792572234,v1:172.21.15.43:6811/792572234] conn(0x7f2b7c07fe20 0x7f2b7c0822f0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:54.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.288+0000 7f2b8dffb640 1 -- 172.21.15.19:0/3296089370 <== osd.7 v2:172.21.15.43:6810/792572234 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2b7c0829c0 con 0x7f2b7c07fe20 2024-08-31T12:28:54.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.305+0000 7f2b9f6ec640 1 -- 172.21.15.19:0/3296089370 --> [v2:172.21.15.43:6810/792572234,v1:172.21.15.43:6811/792572234] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f2b980c8b00 con 0x7f2b7c07fe20 2024-08-31T12:28:54.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.305+0000 7f2b8dffb640 1 -- 172.21.15.19:0/3296089370 <== osd.7 v2:172.21.15.43:6810/792572234 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f2b980c8b00 con 0x7f2b7c07fe20 2024-08-31T12:28:54.307 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-08-31T12:28:54.308 INFO:teuthology.orchestra.run.smithi019.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-08-31T12:28:54.308 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-08-31T12:28:54.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.305+0000 7f2b9f6ec640 1 -- 172.21.15.19:0/3296089370 >> [v2:172.21.15.43:6810/792572234,v1:172.21.15.43:6811/792572234] conn(0x7f2b7c07fe20 msgr2=0x7f2b7c0822f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:54.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.305+0000 7f2b9f6ec640 1 --2- 172.21.15.19:0/3296089370 >> [v2:172.21.15.43:6810/792572234,v1:172.21.15.43:6811/792572234] conn(0x7f2b7c07fe20 0x7f2b7c0822f0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:54.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.306+0000 7f2b9f6ec640 1 -- 172.21.15.19:0/3296089370 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2b7c077a50 msgr2=0x7f2b7c079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:54.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.306+0000 7f2b9f6ec640 1 --2- 172.21.15.19:0/3296089370 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f2b7c077a50 0x7f2b7c079f60 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f2b94034000 tx=0x7f2b94004730 comp rx=0 tx=0).stop 2024-08-31T12:28:54.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.306+0000 7f2b9f6ec640 1 -- 172.21.15.19:0/3296089370 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2b980b3d10 msgr2=0x7f2b98152a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:54.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.306+0000 7f2b9f6ec640 1 --2- 172.21.15.19:0/3296089370 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2b980b3d10 0x7f2b98152a50 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2b88009fd0 tx=0x7f2b8800ec20 comp rx=0 tx=0).stop 2024-08-31T12:28:54.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.306+0000 7f2b9dc89640 1 -- 172.21.15.19:0/3296089370 reap_dead start 2024-08-31T12:28:54.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.306+0000 7f2b9f6ec640 1 -- 172.21.15.19:0/3296089370 shutdown_connections 2024-08-31T12:28:54.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.306+0000 7f2b9f6ec640 1 -- 172.21.15.19:0/3296089370 >> 172.21.15.19:0/3296089370 conn(0x7f2b98069290 msgr2=0x7f2b980cc930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:54.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.306+0000 7f2b9f6ec640 1 -- 172.21.15.19:0/3296089370 shutdown_connections 2024-08-31T12:28:54.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.306+0000 7f2b9f6ec640 1 -- 172.21.15.19:0/3296089370 wait complete. 2024-08-31T12:28:54.321 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-08-31T12:28:54.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.454+0000 7fdf1822e640 1 -- 172.21.15.19:0/3790976658 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdf100c5370 msgr2=0x7fdf1006ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:54.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.454+0000 7fdf1822e640 1 --2- 172.21.15.19:0/3790976658 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdf100c5370 0x7fdf1006ddd0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c005fd0 tx=0x7fdf0c02a940 comp rx=0 tx=0).stop 2024-08-31T12:28:54.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.454+0000 7fdf1822e640 1 -- 172.21.15.19:0/3790976658 shutdown_connections 2024-08-31T12:28:54.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.454+0000 7fdf1822e640 1 --2- 172.21.15.19:0/3790976658 >> v2:172.21.15.19:3300/0 conn(0x7fdf100cb2b0 0x7fdf1014e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:54.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.454+0000 7fdf1822e640 1 --2- 172.21.15.19:0/3790976658 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdf100c5370 0x7fdf1006ddd0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:54.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.454+0000 7fdf1822e640 1 -- 172.21.15.19:0/3790976658 >> 172.21.15.19:0/3790976658 conn(0x7fdf100ca670 msgr2=0x7fdf100cd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:54.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.454+0000 7fdf1822e640 1 -- 172.21.15.19:0/3790976658 shutdown_connections 2024-08-31T12:28:54.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.454+0000 7fdf1822e640 1 -- 172.21.15.19:0/3790976658 wait complete. 2024-08-31T12:28:54.459 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.456+0000 7fdf1822e640 1 Processor -- start 2024-08-31T12:28:54.459 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.457+0000 7fdf1822e640 1 -- start start 2024-08-31T12:28:54.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.458+0000 7fdf1822e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdf100c5370 0x7fdf1015a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:54.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.458+0000 7fdf1822e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdf100cb2b0 0x7fdf1006ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:54.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.458+0000 7fdf1822e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf1014e9a0 0x7fdf10154a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:54.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.458+0000 7fdf1822e640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fdf100c72e0 con 0x7fdf100c5370 2024-08-31T12:28:54.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.458+0000 7fdf1822e640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fdf100c7650 con 0x7fdf1014e9a0 2024-08-31T12:28:54.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.458+0000 7fdf1822e640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fdf100cb8c0 con 0x7fdf100cb2b0 2024-08-31T12:28:54.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.458+0000 7fdf15fca640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdf100c5370 0x7fdf1015a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:54.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.458+0000 7fdf15fca640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdf100c5370 0x7fdf1015a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:44136/0 (socket says 172.21.15.19:44136) 2024-08-31T12:28:54.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.458+0000 7fdf157c9640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdf100cb2b0 0x7fdf1006ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:54.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.458+0000 7fdf15fca640 1 -- 172.21.15.19:0/3367916557 learned_addr learned my addr 172.21.15.19:0/3367916557 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:28:54.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.458+0000 7fdf167cb640 1 --2- 172.21.15.19:0/3367916557 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf1014e9a0 0x7fdf10154a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:54.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.459+0000 7fdf15fca640 1 -- 172.21.15.19:0/3367916557 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdf100cb2b0 msgr2=0x7fdf1006ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:54.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.459+0000 7fdf15fca640 1 --2- 172.21.15.19:0/3367916557 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdf100cb2b0 0x7fdf1006ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:54.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.459+0000 7fdf15fca640 1 -- 172.21.15.19:0/3367916557 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf1014e9a0 msgr2=0x7fdf10154a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:54.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.459+0000 7fdf15fca640 1 --2- 172.21.15.19:0/3367916557 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf1014e9a0 0x7fdf10154a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:54.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.459+0000 7fdf15fca640 1 -- 172.21.15.19:0/3367916557 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf100c8c30 con 0x7fdf100c5370 2024-08-31T12:28:54.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.459+0000 7fdf157c9640 1 --2- 172.21.15.19:0/3367916557 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdf100cb2b0 0x7fdf1006ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:28:54.464 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.459+0000 7fdf167cb640 1 --2- 172.21.15.19:0/3367916557 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf1014e9a0 0x7fdf10154a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-31T12:28:54.464 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.459+0000 7fdf15fca640 1 --2- 172.21.15.19:0/3367916557 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdf100c5370 0x7fdf1015a120 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c009ca0 tx=0x7fdf0c031ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:54.464 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.459+0000 7fdf067fc640 1 -- 172.21.15.19:0/3367916557 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf0c03e070 con 0x7fdf100c5370 2024-08-31T12:28:54.464 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.459+0000 7fdf1822e640 1 -- 172.21.15.19:0/3367916557 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf1014d7f0 con 0x7fdf100c5370 2024-08-31T12:28:54.464 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.459+0000 7fdf067fc640 1 -- 172.21.15.19:0/3367916557 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdf0c004540 con 0x7fdf100c5370 2024-08-31T12:28:54.464 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.459+0000 7fdf067fc640 1 -- 172.21.15.19:0/3367916557 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf0c003e60 con 0x7fdf100c5370 2024-08-31T12:28:54.464 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.459+0000 7fdf1822e640 1 -- 172.21.15.19:0/3367916557 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf100c8820 con 0x7fdf100c5370 2024-08-31T12:28:54.465 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.461+0000 7fdf1822e640 1 -- 172.21.15.19:0/3367916557 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf1014da30 con 0x7fdf100c5370 2024-08-31T12:28:54.465 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.462+0000 7fdf067fc640 1 -- 172.21.15.19:0/3367916557 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fdf0c03a440 con 0x7fdf100c5370 2024-08-31T12:28:54.465 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.463+0000 7fdf067fc640 1 --2- 172.21.15.19:0/3367916557 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fdef4077ad0 0x7fdef4079fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:54.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.463+0000 7fdf067fc640 1 -- 172.21.15.19:0/3367916557 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fdf0c0baef0 con 0x7fdf100c5370 2024-08-31T12:28:54.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.463+0000 7fdf157c9640 1 --2- 172.21.15.19:0/3367916557 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fdef4077ad0 0x7fdef4079fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:54.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.463+0000 7fdf157c9640 1 --2- 172.21.15.19:0/3367916557 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fdef4077ad0 0x7fdef4079fe0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fdf000097a0 tx=0x7fdf00009260 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:54.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.468+0000 7fdf067fc640 1 -- 172.21.15.19:0/3367916557 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fdf0c083700 con 0x7fdf100c5370 2024-08-31T12:28:54.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.705+0000 7fdf1822e640 1 -- 172.21.15.19:0/3367916557 --> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7fdf1014adc0 con 0x7fdef4077ad0 2024-08-31T12:28:54.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.705+0000 7fdf067fc640 1 -- 172.21.15.19:0/3367916557 <== mgr.4100 v2:172.21.15.43:6800/2824336655 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fdf1014adc0 con 0x7fdef4077ad0 2024-08-31T12:28:54.707 INFO:teuthology.orchestra.run.smithi019.stderr:instructed osd(s) 7 to deep-scrub 2024-08-31T12:28:54.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.708+0000 7fdf1822e640 1 -- 172.21.15.19:0/3367916557 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fdef4077ad0 msgr2=0x7fdef4079fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:54.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.708+0000 7fdf1822e640 1 --2- 172.21.15.19:0/3367916557 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fdef4077ad0 0x7fdef4079fe0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fdf000097a0 tx=0x7fdf00009260 comp rx=0 tx=0).stop 2024-08-31T12:28:54.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.708+0000 7fdf1822e640 1 -- 172.21.15.19:0/3367916557 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdf100c5370 msgr2=0x7fdf1015a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:54.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.708+0000 7fdf1822e640 1 --2- 172.21.15.19:0/3367916557 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdf100c5370 0x7fdf1015a120 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c009ca0 tx=0x7fdf0c031ea0 comp rx=0 tx=0).stop 2024-08-31T12:28:54.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.708+0000 7fdf1822e640 1 -- 172.21.15.19:0/3367916557 shutdown_connections 2024-08-31T12:28:54.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.708+0000 7fdf1822e640 1 --2- 172.21.15.19:0/3367916557 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fdef4077ad0 0x7fdef4079fe0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:54.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.708+0000 7fdf1822e640 1 --2- 172.21.15.19:0/3367916557 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf1014e9a0 0x7fdf10154a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:54.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.708+0000 7fdf1822e640 1 --2- 172.21.15.19:0/3367916557 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdf100cb2b0 0x7fdf1006ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:54.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.708+0000 7fdf1822e640 1 --2- 172.21.15.19:0/3367916557 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdf100c5370 0x7fdf1015a120 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:54.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.708+0000 7fdf1822e640 1 -- 172.21.15.19:0/3367916557 >> 172.21.15.19:0/3367916557 conn(0x7fdf100ca670 msgr2=0x7fdf10149f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:54.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.708+0000 7fdf1822e640 1 -- 172.21.15.19:0/3367916557 shutdown_connections 2024-08-31T12:28:54.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.708+0000 7fdf1822e640 1 -- 172.21.15.19:0/3367916557 wait complete. 2024-08-31T12:28:54.726 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-31T12:28:54.858 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.856+0000 7f290f044640 1 -- 172.21.15.19:0/3189752386 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f29080c85a0 msgr2=0x7f29080c89b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:54.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.856+0000 7f290f044640 1 --2- 172.21.15.19:0/3189752386 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f29080c85a0 0x7f29080c89b0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f2900005fd0 tx=0x7f290002aa90 comp rx=0 tx=0).stop 2024-08-31T12:28:54.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.856+0000 7f290f044640 1 -- 172.21.15.19:0/3189752386 shutdown_connections 2024-08-31T12:28:54.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.856+0000 7f290f044640 1 --2- 172.21.15.19:0/3189752386 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f29080c85a0 0x7f29080c89b0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:54.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.856+0000 7f290f044640 1 --2- 172.21.15.19:0/3189752386 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f29081587f0 0x7f290814e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:54.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.856+0000 7f290f044640 1 -- 172.21.15.19:0/3189752386 >> 172.21.15.19:0/3189752386 conn(0x7f290806cae0 msgr2=0x7f290806cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:54.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.856+0000 7f290f044640 1 -- 172.21.15.19:0/3189752386 shutdown_connections 2024-08-31T12:28:54.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.856+0000 7f290f044640 1 -- 172.21.15.19:0/3189752386 wait complete. 2024-08-31T12:28:54.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.858+0000 7f290f044640 1 Processor -- start 2024-08-31T12:28:54.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.858+0000 7f290f044640 1 -- start start 2024-08-31T12:28:54.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.860+0000 7f290f044640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f29080c7ac0 0x7f2908156e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:54.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.860+0000 7f290f044640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f29080c85a0 0x7f29081364e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:54.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.860+0000 7f290f044640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29081587f0 0x7f2908136a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:54.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.860+0000 7f290f044640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f2908152380 con 0x7f29080c85a0 2024-08-31T12:28:54.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.860+0000 7f290f044640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f29080c8ef0 con 0x7f29081587f0 2024-08-31T12:28:54.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.860+0000 7f290f044640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f29080c3e90 con 0x7f29080c7ac0 2024-08-31T12:28:54.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.860+0000 7f2907fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f29080c85a0 0x7f29081364e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:54.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.860+0000 7f2907fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f29080c85a0 0x7f29081364e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:44148/0 (socket says 172.21.15.19:44148) 2024-08-31T12:28:54.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.860+0000 7f2907fff640 1 -- 172.21.15.19:0/3843679566 learned_addr learned my addr 172.21.15.19:0/3843679566 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:28:54.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.860+0000 7f290d5e1640 1 --2- 172.21.15.19:0/3843679566 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29081587f0 0x7f2908136a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:54.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.860+0000 7f290cde0640 1 --2- 172.21.15.19:0/3843679566 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f29080c7ac0 0x7f2908156e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:54.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.860+0000 7f2907fff640 1 -- 172.21.15.19:0/3843679566 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f29080c7ac0 msgr2=0x7f2908156e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:54.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.860+0000 7f2907fff640 1 --2- 172.21.15.19:0/3843679566 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f29080c7ac0 0x7f2908156e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:54.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.860+0000 7f2907fff640 1 -- 172.21.15.19:0/3843679566 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29081587f0 msgr2=0x7f2908136a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:54.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.860+0000 7f2907fff640 1 --2- 172.21.15.19:0/3843679566 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29081587f0 0x7f2908136a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:54.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.860+0000 7f2907fff640 1 -- 172.21.15.19:0/3843679566 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f290813fc00 con 0x7f29080c85a0 2024-08-31T12:28:54.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.861+0000 7f2907fff640 1 --2- 172.21.15.19:0/3843679566 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f29080c85a0 0x7f29081364e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f28f400ea20 tx=0x7f28f400eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:54.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.861+0000 7f29057fa640 1 -- 172.21.15.19:0/3843679566 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f400ce40 con 0x7f29080c85a0 2024-08-31T12:28:54.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.861+0000 7f29057fa640 1 -- 172.21.15.19:0/3843679566 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f28f4004020 con 0x7f29080c85a0 2024-08-31T12:28:54.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.861+0000 7f290f044640 1 -- 172.21.15.19:0/3843679566 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f290813fea0 con 0x7f29080c85a0 2024-08-31T12:28:54.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.861+0000 7f29057fa640 1 -- 172.21.15.19:0/3843679566 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f4004470 con 0x7f29080c85a0 2024-08-31T12:28:54.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.861+0000 7f290f044640 1 -- 172.21.15.19:0/3843679566 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2908136f60 con 0x7f29080c85a0 2024-08-31T12:28:54.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.862+0000 7f290f044640 1 -- 172.21.15.19:0/3843679566 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f290814d960 con 0x7f29080c85a0 2024-08-31T12:28:54.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.863+0000 7f29057fa640 1 -- 172.21.15.19:0/3843679566 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f28f40035f0 con 0x7f29080c85a0 2024-08-31T12:28:54.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.864+0000 7f29057fa640 1 --2- 172.21.15.19:0/3843679566 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f28ec077a80 0x7f28ec079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:28:54.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.864+0000 7f29057fa640 1 -- 172.21.15.19:0/3843679566 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f28f409a040 con 0x7f29080c85a0 2024-08-31T12:28:54.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.864+0000 7f290cde0640 1 --2- 172.21.15.19:0/3843679566 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f28ec077a80 0x7f28ec079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:28:54.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.865+0000 7f290cde0640 1 --2- 172.21.15.19:0/3843679566 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f28ec077a80 0x7f28ec079f90 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f28fc007fd0 tx=0x7f28fc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:28:54.871 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:54.869+0000 7f29057fa640 1 -- 172.21.15.19:0/3843679566 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f28f40626e0 con 0x7f29080c85a0 2024-08-31T12:28:55.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:55.117+0000 7f290f044640 1 -- 172.21.15.19:0/3843679566 --> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f29080c89b0 con 0x7f28ec077a80 2024-08-31T12:28:55.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:55.122+0000 7f29057fa640 1 -- 172.21.15.19:0/3843679566 <== mgr.4100 v2:172.21.15.43:6800/2824336655 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+254619 (secure 0 0 0) 0x7f29080c89b0 con 0x7f28ec077a80 2024-08-31T12:28:55.127 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:28:55.128 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-08-31T12:28:55.129 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:55.126+0000 7f290f044640 1 -- 172.21.15.19:0/3843679566 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f28ec077a80 msgr2=0x7f28ec079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:55.129 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:55.126+0000 7f290f044640 1 --2- 172.21.15.19:0/3843679566 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f28ec077a80 0x7f28ec079f90 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f28fc007fd0 tx=0x7f28fc009040 comp rx=0 tx=0).stop 2024-08-31T12:28:55.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:55.126+0000 7f290f044640 1 -- 172.21.15.19:0/3843679566 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f29080c85a0 msgr2=0x7f29081364e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:28:55.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:55.126+0000 7f290f044640 1 --2- 172.21.15.19:0/3843679566 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f29080c85a0 0x7f29081364e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f28f400ea20 tx=0x7f28f400eee0 comp rx=0 tx=0).stop 2024-08-31T12:28:55.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:55.126+0000 7f290f044640 1 -- 172.21.15.19:0/3843679566 shutdown_connections 2024-08-31T12:28:55.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:55.126+0000 7f290f044640 1 --2- 172.21.15.19:0/3843679566 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f28ec077a80 0x7f28ec079f90 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:55.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:55.126+0000 7f290f044640 1 --2- 172.21.15.19:0/3843679566 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29081587f0 0x7f2908136a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:55.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:55.126+0000 7f290f044640 1 --2- 172.21.15.19:0/3843679566 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f29080c85a0 0x7f29081364e0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:55.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:55.126+0000 7f290f044640 1 --2- 172.21.15.19:0/3843679566 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f29080c7ac0 0x7f2908156e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:28:55.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:55.126+0000 7f290f044640 1 -- 172.21.15.19:0/3843679566 >> 172.21.15.19:0/3843679566 conn(0x7f290806cae0 msgr2=0x7f290814a3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:28:55.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:55.126+0000 7f290f044640 1 -- 172.21.15.19:0/3843679566 shutdown_connections 2024-08-31T12:28:55.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:28:55.126+0000 7f290f044640 1 -- 172.21.15.19:0/3843679566 wait complete. 2024-08-31T12:28:55.144 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":412,"stamp":"2024-08-31T12:28:54.708736+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":13438838229,"num_objects":3466,"num_object_clones":0,"num_object_copies":6932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3466,"num_whiteouts":0,"num_read":96,"num_read_kb":119,"num_write":921570,"num_write_kb":16804420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":186997,"ondisk_log_size":186997,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":28675744,"kb_used_data":26250592,"kb_used_omap":12,"kb_used_meta":2425011,"kb_avail":726298976,"statfs":{"total":773094113280,"available":743730151424,"internally_reserved":0,"allocated":26880606208,"data_stored":26878340186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":2483211825},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":57291,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003570"},"pg_stats":[{"pgid":"3.3f","version":"23'2732","reported_seq":2755,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.515247+0000","last_change":"2024-08-31T12:15:57.306560+0000","last_active":"2024-08-31T12:28:47.515247+0000","last_peered":"2024-08-31T12:28:47.515247+0000","last_clean":"2024-08-31T12:28:47.515247+0000","last_became_active":"2024-08-31T12:15:57.306148+0000","last_became_peered":"2024-08-31T12:15:57.306148+0000","last_unstale":"2024-08-31T12:28:47.515247+0000","last_undegraded":"2024-08-31T12:28:47.515247+0000","last_fullsized":"2024-08-31T12:28:47.515247+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2732,"log_dups_size":0,"ondisk_log_size":2732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:35:53.831742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180377297,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13788,"num_write_kb":231441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"23'3452","reported_seq":3475,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.406775+0000","last_change":"2024-08-31T12:15:57.307314+0000","last_active":"2024-08-31T12:28:47.406775+0000","last_peered":"2024-08-31T12:28:47.406775+0000","last_clean":"2024-08-31T12:28:47.406775+0000","last_became_active":"2024-08-31T12:15:57.306940+0000","last_became_peered":"2024-08-31T12:15:57.306940+0000","last_unstale":"2024-08-31T12:28:47.406775+0000","last_undegraded":"2024-08-31T12:28:47.406775+0000","last_fullsized":"2024-08-31T12:28:47.406775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3452,"log_dups_size":0,"ondisk_log_size":3452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:38:30.321214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264252554,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17611,"num_write_kb":328696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"23'3141","reported_seq":3164,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.457018+0000","last_change":"2024-08-31T12:15:57.309372+0000","last_active":"2024-08-31T12:28:47.457018+0000","last_peered":"2024-08-31T12:28:47.457018+0000","last_clean":"2024-08-31T12:28:47.457018+0000","last_became_active":"2024-08-31T12:15:57.305795+0000","last_became_peered":"2024-08-31T12:15:57.305795+0000","last_unstale":"2024-08-31T12:28:47.457018+0000","last_undegraded":"2024-08-31T12:28:47.457018+0000","last_fullsized":"2024-08-31T12:28:47.457018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3141,"log_dups_size":0,"ondisk_log_size":3141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:37:45.891394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213942725,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15777,"num_write_kb":272211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"23'2793","reported_seq":2816,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.486426+0000","last_change":"2024-08-31T12:15:57.307339+0000","last_active":"2024-08-31T12:28:47.486426+0000","last_peered":"2024-08-31T12:28:47.486426+0000","last_clean":"2024-08-31T12:28:47.486426+0000","last_became_active":"2024-08-31T12:15:57.306957+0000","last_became_peered":"2024-08-31T12:15:57.306957+0000","last_unstale":"2024-08-31T12:28:47.486426+0000","last_undegraded":"2024-08-31T12:28:47.486426+0000","last_fullsized":"2024-08-31T12:28:47.486426+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2793,"log_dups_size":0,"ondisk_log_size":2793,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:43:33.705099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14240,"num_write_kb":241389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"23'3281","reported_seq":3303,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.514017+0000","last_change":"2024-08-31T12:15:57.317492+0000","last_active":"2024-08-31T12:28:47.514017+0000","last_peered":"2024-08-31T12:28:47.514017+0000","last_clean":"2024-08-31T12:28:47.514017+0000","last_became_active":"2024-08-31T12:15:57.317079+0000","last_became_peered":"2024-08-31T12:15:57.317079+0000","last_unstale":"2024-08-31T12:28:47.514017+0000","last_undegraded":"2024-08-31T12:28:47.514017+0000","last_fullsized":"2024-08-31T12:28:47.514017+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3281,"log_dups_size":0,"ondisk_log_size":3281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:44:40.712699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247474530,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15722,"num_write_kb":304743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"23'2518","reported_seq":2540,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.484034+0000","last_change":"2024-08-31T12:15:57.307505+0000","last_active":"2024-08-31T12:28:47.484034+0000","last_peered":"2024-08-31T12:28:47.484034+0000","last_clean":"2024-08-31T12:28:47.484034+0000","last_became_active":"2024-08-31T12:15:57.307010+0000","last_became_peered":"2024-08-31T12:15:57.307010+0000","last_unstale":"2024-08-31T12:28:47.484034+0000","last_undegraded":"2024-08-31T12:28:47.484034+0000","last_fullsized":"2024-08-31T12:28:47.484034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2518,"log_dups_size":0,"ondisk_log_size":2518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:39:31.100565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180356168,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12564,"num_write_kb":226449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"23'2631","reported_seq":2653,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.457738+0000","last_change":"2024-08-31T12:15:57.316892+0000","last_active":"2024-08-31T12:28:47.457738+0000","last_peered":"2024-08-31T12:28:47.457738+0000","last_clean":"2024-08-31T12:28:47.457738+0000","last_became_active":"2024-08-31T12:15:57.316504+0000","last_became_peered":"2024-08-31T12:15:57.316504+0000","last_unstale":"2024-08-31T12:28:47.457738+0000","last_undegraded":"2024-08-31T12:28:47.457738+0000","last_fullsized":"2024-08-31T12:28:47.457738+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2631,"log_dups_size":0,"ondisk_log_size":2631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:05:48.614918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201398355,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13306,"num_write_kb":250051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"23'3167","reported_seq":3182,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.457071+0000","last_change":"2024-08-31T12:16:40.694493+0000","last_active":"2024-08-31T12:28:47.457071+0000","last_peered":"2024-08-31T12:28:47.457071+0000","last_clean":"2024-08-31T12:28:47.457071+0000","last_became_active":"2024-08-31T12:16:40.694165+0000","last_became_peered":"2024-08-31T12:16:40.694165+0000","last_unstale":"2024-08-31T12:28:47.457071+0000","last_undegraded":"2024-08-31T12:28:47.457071+0000","last_fullsized":"2024-08-31T12:28:47.457071+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3167,"log_dups_size":0,"ondisk_log_size":3167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:55:57.961306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226497620,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15619,"num_write_kb":283880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.37","version":"23'3502","reported_seq":3524,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.527864+0000","last_change":"2024-08-31T12:15:57.316723+0000","last_active":"2024-08-31T12:28:47.527864+0000","last_peered":"2024-08-31T12:28:47.527864+0000","last_clean":"2024-08-31T12:28:47.527864+0000","last_became_active":"2024-08-31T12:15:57.316354+0000","last_became_peered":"2024-08-31T12:15:57.316354+0000","last_unstale":"2024-08-31T12:28:47.527864+0000","last_undegraded":"2024-08-31T12:28:47.527864+0000","last_fullsized":"2024-08-31T12:28:47.527864+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3502,"log_dups_size":0,"ondisk_log_size":3502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:31:19.860950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251662082,"num_objects":61,"num_object_clones":0,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18100,"num_write_kb":318223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"23'2970","reported_seq":2992,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.500679+0000","last_change":"2024-08-31T12:15:57.311161+0000","last_active":"2024-08-31T12:28:47.500679+0000","last_peered":"2024-08-31T12:28:47.500679+0000","last_clean":"2024-08-31T12:28:47.500679+0000","last_became_active":"2024-08-31T12:15:57.310807+0000","last_became_peered":"2024-08-31T12:15:57.310807+0000","last_unstale":"2024-08-31T12:28:47.500679+0000","last_undegraded":"2024-08-31T12:28:47.500679+0000","last_fullsized":"2024-08-31T12:28:47.500679+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2970,"log_dups_size":0,"ondisk_log_size":2970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:56:03.280339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234894029,"num_objects":59,"num_object_clones":0,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15073,"num_write_kb":289757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"23'2886","reported_seq":2908,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.482462+0000","last_change":"2024-08-31T12:15:57.300930+0000","last_active":"2024-08-31T12:28:47.482462+0000","last_peered":"2024-08-31T12:28:47.482462+0000","last_clean":"2024-08-31T12:28:47.482462+0000","last_became_active":"2024-08-31T12:15:57.300610+0000","last_became_peered":"2024-08-31T12:15:57.300610+0000","last_unstale":"2024-08-31T12:28:47.482462+0000","last_undegraded":"2024-08-31T12:28:47.482462+0000","last_fullsized":"2024-08-31T12:28:47.482462+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2886,"log_dups_size":0,"ondisk_log_size":2886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:42:39.731742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201326592,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14139,"num_write_kb":253227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"23'2253","reported_seq":2275,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.441081+0000","last_change":"2024-08-31T12:15:57.303151+0000","last_active":"2024-08-31T12:28:47.441081+0000","last_peered":"2024-08-31T12:28:47.441081+0000","last_clean":"2024-08-31T12:28:47.441081+0000","last_became_active":"2024-08-31T12:15:57.302622+0000","last_became_peered":"2024-08-31T12:15:57.302622+0000","last_unstale":"2024-08-31T12:28:47.441081+0000","last_undegraded":"2024-08-31T12:28:47.441081+0000","last_fullsized":"2024-08-31T12:28:47.441081+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2253,"log_dups_size":0,"ondisk_log_size":2253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:49:18.493629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176192645,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11403,"num_write_kb":217791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"23'2958","reported_seq":2980,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.471452+0000","last_change":"2024-08-31T12:15:57.305676+0000","last_active":"2024-08-31T12:28:47.471452+0000","last_peered":"2024-08-31T12:28:47.471452+0000","last_clean":"2024-08-31T12:28:47.471452+0000","last_became_active":"2024-08-31T12:15:57.305321+0000","last_became_peered":"2024-08-31T12:15:57.305321+0000","last_unstale":"2024-08-31T12:28:47.471452+0000","last_undegraded":"2024-08-31T12:28:47.471452+0000","last_fullsized":"2024-08-31T12:28:47.471452+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2958,"log_dups_size":0,"ondisk_log_size":2958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:42:39.731742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188840677,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15214,"num_write_kb":245235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"23'3129","reported_seq":3151,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.462358+0000","last_change":"2024-08-31T12:15:57.309671+0000","last_active":"2024-08-31T12:28:47.462358+0000","last_peered":"2024-08-31T12:28:47.462358+0000","last_clean":"2024-08-31T12:28:47.462358+0000","last_became_active":"2024-08-31T12:15:57.307013+0000","last_became_peered":"2024-08-31T12:15:57.307013+0000","last_unstale":"2024-08-31T12:28:47.462358+0000","last_undegraded":"2024-08-31T12:28:47.462358+0000","last_fullsized":"2024-08-31T12:28:47.462358+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3129,"log_dups_size":0,"ondisk_log_size":3129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:18:55.451164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213986201,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15743,"num_write_kb":271989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"23'2959","reported_seq":2981,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.427712+0000","last_change":"2024-08-31T12:15:57.298150+0000","last_active":"2024-08-31T12:28:47.427712+0000","last_peered":"2024-08-31T12:28:47.427712+0000","last_clean":"2024-08-31T12:28:47.427712+0000","last_became_active":"2024-08-31T12:15:57.297182+0000","last_became_peered":"2024-08-31T12:15:57.297182+0000","last_unstale":"2024-08-31T12:28:47.427712+0000","last_undegraded":"2024-08-31T12:28:47.427712+0000","last_fullsized":"2024-08-31T12:28:47.427712+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2959,"log_dups_size":0,"ondisk_log_size":2959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:16:42.625396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213913551,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14445,"num_write_kb":266758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"23'2546","reported_seq":2562,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.462905+0000","last_change":"2024-08-31T12:16:41.075080+0000","last_active":"2024-08-31T12:28:47.462905+0000","last_peered":"2024-08-31T12:28:47.462905+0000","last_clean":"2024-08-31T12:28:47.462905+0000","last_became_active":"2024-08-31T12:16:41.074773+0000","last_became_peered":"2024-08-31T12:16:41.074773+0000","last_unstale":"2024-08-31T12:28:47.462905+0000","last_undegraded":"2024-08-31T12:28:47.462905+0000","last_fullsized":"2024-08-31T12:28:47.462905+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2546,"log_dups_size":0,"ondisk_log_size":2546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:10:45.435137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184599153,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12917,"num_write_kb":231981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"23'3723","reported_seq":3745,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.527964+0000","last_change":"2024-08-31T12:15:57.308899+0000","last_active":"2024-08-31T12:28:47.527964+0000","last_peered":"2024-08-31T12:28:47.527964+0000","last_clean":"2024-08-31T12:28:47.527964+0000","last_became_active":"2024-08-31T12:15:57.308388+0000","last_became_peered":"2024-08-31T12:15:57.308388+0000","last_unstale":"2024-08-31T12:28:47.527964+0000","last_undegraded":"2024-08-31T12:28:47.527964+0000","last_fullsized":"2024-08-31T12:28:47.527964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3723,"log_dups_size":0,"ondisk_log_size":3723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:54:03.871840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251747648,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18708,"num_write_kb":320814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"23'2148","reported_seq":2170,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.446094+0000","last_change":"2024-08-31T12:15:57.300201+0000","last_active":"2024-08-31T12:28:47.446094+0000","last_peered":"2024-08-31T12:28:47.446094+0000","last_clean":"2024-08-31T12:28:47.446094+0000","last_became_active":"2024-08-31T12:15:57.299826+0000","last_became_peered":"2024-08-31T12:15:57.299826+0000","last_unstale":"2024-08-31T12:28:47.446094+0000","last_undegraded":"2024-08-31T12:28:47.446094+0000","last_fullsized":"2024-08-31T12:28:47.446094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2148,"log_dups_size":0,"ondisk_log_size":2148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:14:34.488102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167791563,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10514,"num_write_kb":205964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"23'2675","reported_seq":2697,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.491633+0000","last_change":"2024-08-31T12:15:57.317606+0000","last_active":"2024-08-31T12:28:47.491633+0000","last_peered":"2024-08-31T12:28:47.491633+0000","last_clean":"2024-08-31T12:28:47.491633+0000","last_became_active":"2024-08-31T12:15:57.317185+0000","last_became_peered":"2024-08-31T12:15:57.317185+0000","last_unstale":"2024-08-31T12:28:47.491633+0000","last_undegraded":"2024-08-31T12:28:47.491633+0000","last_fullsized":"2024-08-31T12:28:47.491633+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2675,"log_dups_size":0,"ondisk_log_size":2675,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:25:46.346490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209762270,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13163,"num_write_kb":257524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"23'2449","reported_seq":2471,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.458310+0000","last_change":"2024-08-31T12:15:57.298778+0000","last_active":"2024-08-31T12:28:47.458310+0000","last_peered":"2024-08-31T12:28:47.458310+0000","last_clean":"2024-08-31T12:28:47.458310+0000","last_became_active":"2024-08-31T12:15:57.298430+0000","last_became_peered":"2024-08-31T12:15:57.298430+0000","last_unstale":"2024-08-31T12:28:47.458310+0000","last_undegraded":"2024-08-31T12:28:47.458310+0000","last_fullsized":"2024-08-31T12:28:47.458310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2449,"log_dups_size":0,"ondisk_log_size":2449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:22:13.435300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205530518,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12230,"num_write_kb":249721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"23'2950","reported_seq":2972,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.512435+0000","last_change":"2024-08-31T12:15:57.317176+0000","last_active":"2024-08-31T12:28:47.512435+0000","last_peered":"2024-08-31T12:28:47.512435+0000","last_clean":"2024-08-31T12:28:47.512435+0000","last_became_active":"2024-08-31T12:15:57.316870+0000","last_became_peered":"2024-08-31T12:15:57.316870+0000","last_unstale":"2024-08-31T12:28:47.512435+0000","last_undegraded":"2024-08-31T12:28:47.512435+0000","last_fullsized":"2024-08-31T12:28:47.512435+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2950,"log_dups_size":0,"ondisk_log_size":2950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:43:12.703971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213923005,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14700,"num_write_kb":267872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"23'3066","reported_seq":3082,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.485438+0000","last_change":"2024-08-31T12:16:40.784525+0000","last_active":"2024-08-31T12:28:47.485438+0000","last_peered":"2024-08-31T12:28:47.485438+0000","last_clean":"2024-08-31T12:28:47.485438+0000","last_became_active":"2024-08-31T12:16:40.784204+0000","last_became_peered":"2024-08-31T12:16:40.784204+0000","last_unstale":"2024-08-31T12:28:47.485438+0000","last_undegraded":"2024-08-31T12:28:47.485438+0000","last_fullsized":"2024-08-31T12:28:47.485438+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3066,"log_dups_size":0,"ondisk_log_size":3066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:51:38.173279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226493660,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15308,"num_write_kb":282531,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"23'2835","reported_seq":2857,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.468867+0000","last_change":"2024-08-31T12:15:57.304032+0000","last_active":"2024-08-31T12:28:47.468867+0000","last_peered":"2024-08-31T12:28:47.468867+0000","last_clean":"2024-08-31T12:28:47.468867+0000","last_became_active":"2024-08-31T12:15:57.303401+0000","last_became_peered":"2024-08-31T12:15:57.303401+0000","last_unstale":"2024-08-31T12:28:47.468867+0000","last_undegraded":"2024-08-31T12:28:47.468867+0000","last_fullsized":"2024-08-31T12:28:47.468867+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:56:03.280339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205588184,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14070,"num_write_kb":257066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"23'2474","reported_seq":2496,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.476373+0000","last_change":"2024-08-31T12:15:57.305412+0000","last_active":"2024-08-31T12:28:47.476373+0000","last_peered":"2024-08-31T12:28:47.476373+0000","last_clean":"2024-08-31T12:28:47.476373+0000","last_became_active":"2024-08-31T12:15:57.305075+0000","last_became_peered":"2024-08-31T12:15:57.305075+0000","last_unstale":"2024-08-31T12:28:47.476373+0000","last_undegraded":"2024-08-31T12:28:47.476373+0000","last_fullsized":"2024-08-31T12:28:47.476373+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2474,"log_dups_size":0,"ondisk_log_size":2474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:30:13.715505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184555785,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12290,"num_write_kb":229419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"23'3166","reported_seq":3188,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.421868+0000","last_change":"2024-08-31T12:15:57.300179+0000","last_active":"2024-08-31T12:28:47.421868+0000","last_peered":"2024-08-31T12:28:47.421868+0000","last_clean":"2024-08-31T12:28:47.421868+0000","last_became_active":"2024-08-31T12:15:57.299861+0000","last_became_peered":"2024-08-31T12:15:57.299861+0000","last_unstale":"2024-08-31T12:28:47.421868+0000","last_undegraded":"2024-08-31T12:28:47.421868+0000","last_fullsized":"2024-08-31T12:28:47.421868+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3166,"log_dups_size":0,"ondisk_log_size":3166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:47:51.267409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243322604,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15709,"num_write_kb":300530,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"23'2744","reported_seq":2766,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.489208+0000","last_change":"2024-08-31T12:15:57.308519+0000","last_active":"2024-08-31T12:28:47.489208+0000","last_peered":"2024-08-31T12:28:47.489208+0000","last_clean":"2024-08-31T12:28:47.489208+0000","last_became_active":"2024-08-31T12:15:57.308189+0000","last_became_peered":"2024-08-31T12:15:57.308189+0000","last_unstale":"2024-08-31T12:28:47.489208+0000","last_undegraded":"2024-08-31T12:28:47.489208+0000","last_fullsized":"2024-08-31T12:28:47.489208+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2744,"log_dups_size":0,"ondisk_log_size":2744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:37:02.435678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189169648,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13404,"num_write_kb":238420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"23'3088","reported_seq":3110,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.429424+0000","last_change":"2024-08-31T12:15:57.308321+0000","last_active":"2024-08-31T12:28:47.429424+0000","last_peered":"2024-08-31T12:28:47.429424+0000","last_clean":"2024-08-31T12:28:47.429424+0000","last_became_active":"2024-08-31T12:15:57.304514+0000","last_became_peered":"2024-08-31T12:15:57.304514+0000","last_unstale":"2024-08-31T12:28:47.429424+0000","last_undegraded":"2024-08-31T12:28:47.429424+0000","last_fullsized":"2024-08-31T12:28:47.429424+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3088,"log_dups_size":0,"ondisk_log_size":3088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:09:49.617826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222298838,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15326,"num_write_kb":278485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"23'2574","reported_seq":2596,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.475910+0000","last_change":"2024-08-31T12:15:57.299268+0000","last_active":"2024-08-31T12:28:47.475910+0000","last_peered":"2024-08-31T12:28:47.475910+0000","last_clean":"2024-08-31T12:28:47.475910+0000","last_became_active":"2024-08-31T12:15:57.298766+0000","last_became_peered":"2024-08-31T12:15:57.298766+0000","last_unstale":"2024-08-31T12:28:47.475910+0000","last_undegraded":"2024-08-31T12:28:47.475910+0000","last_fullsized":"2024-08-31T12:28:47.475910+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2574,"log_dups_size":0,"ondisk_log_size":2574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:37:45.891394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188773348,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12878,"num_write_kb":235916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"23'2328","reported_seq":2351,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.475414+0000","last_change":"2024-08-31T12:15:57.310121+0000","last_active":"2024-08-31T12:28:47.475414+0000","last_peered":"2024-08-31T12:28:47.475414+0000","last_clean":"2024-08-31T12:28:47.475414+0000","last_became_active":"2024-08-31T12:15:57.309440+0000","last_became_peered":"2024-08-31T12:15:57.309440+0000","last_unstale":"2024-08-31T12:28:47.475414+0000","last_undegraded":"2024-08-31T12:28:47.475414+0000","last_fullsized":"2024-08-31T12:28:47.475414+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2328,"log_dups_size":0,"ondisk_log_size":2328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:53:19.523811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180357122,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11238,"num_write_kb":221098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"23'3447","reported_seq":3470,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.484164+0000","last_change":"2024-08-31T12:15:57.306214+0000","last_active":"2024-08-31T12:28:47.484164+0000","last_peered":"2024-08-31T12:28:47.484164+0000","last_clean":"2024-08-31T12:28:47.484164+0000","last_became_active":"2024-08-31T12:15:57.305878+0000","last_became_peered":"2024-08-31T12:15:57.305878+0000","last_unstale":"2024-08-31T12:28:47.484164+0000","last_undegraded":"2024-08-31T12:28:47.484164+0000","last_fullsized":"2024-08-31T12:28:47.484164+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3447,"log_dups_size":0,"ondisk_log_size":3447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:05:48.614918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251668879,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17281,"num_write_kb":314954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"23'2880","reported_seq":2903,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.439514+0000","last_change":"2024-08-31T12:15:57.307376+0000","last_active":"2024-08-31T12:28:47.439514+0000","last_peered":"2024-08-31T12:28:47.439514+0000","last_clean":"2024-08-31T12:28:47.439514+0000","last_became_active":"2024-08-31T12:15:57.307056+0000","last_became_peered":"2024-08-31T12:15:57.307056+0000","last_unstale":"2024-08-31T12:28:47.439514+0000","last_undegraded":"2024-08-31T12:28:47.439514+0000","last_fullsized":"2024-08-31T12:28:47.439514+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2880,"log_dups_size":0,"ondisk_log_size":2880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:52:18.004608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222298112,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14386,"num_write_kb":274682,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"23'3122","reported_seq":3145,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.445220+0000","last_change":"2024-08-31T12:15:57.313798+0000","last_active":"2024-08-31T12:28:47.445220+0000","last_peered":"2024-08-31T12:28:47.445220+0000","last_clean":"2024-08-31T12:28:47.445220+0000","last_became_active":"2024-08-31T12:15:57.313463+0000","last_became_peered":"2024-08-31T12:15:57.313463+0000","last_unstale":"2024-08-31T12:28:47.445220+0000","last_undegraded":"2024-08-31T12:28:47.445220+0000","last_fullsized":"2024-08-31T12:28:47.445220+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3122,"log_dups_size":0,"ondisk_log_size":3122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:47:51.267409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260049613,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15263,"num_write_kb":315030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"23'2287","reported_seq":2309,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.416957+0000","last_change":"2024-08-31T12:15:57.293350+0000","last_active":"2024-08-31T12:28:47.416957+0000","last_peered":"2024-08-31T12:28:47.416957+0000","last_clean":"2024-08-31T12:28:47.416957+0000","last_became_active":"2024-08-31T12:15:57.292981+0000","last_became_peered":"2024-08-31T12:15:57.292981+0000","last_unstale":"2024-08-31T12:28:47.416957+0000","last_undegraded":"2024-08-31T12:28:47.416957+0000","last_fullsized":"2024-08-31T12:28:47.416957+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2287,"log_dups_size":0,"ondisk_log_size":2287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:52:00.312013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180377946,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11142,"num_write_kb":220690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"23'570","reported_seq":594,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:26.622146+0000","last_change":"2024-08-31T12:15:56.293497+0000","last_active":"2024-08-31T12:28:26.622146+0000","last_peered":"2024-08-31T12:28:26.622146+0000","last_clean":"2024-08-31T12:28:26.622146+0000","last_became_active":"2024-08-31T12:15:56.293164+0000","last_became_peered":"2024-08-31T12:15:56.293164+0000","last_unstale":"2024-08-31T12:28:26.622146+0000","last_undegraded":"2024-08-31T12:28:26.622146+0000","last_fullsized":"2024-08-31T12:28:26.622146+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":570,"log_dups_size":0,"ondisk_log_size":570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:43:17.443433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4195508,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":567,"num_write_kb":4211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"23'2396","reported_seq":2418,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.477656+0000","last_change":"2024-08-31T12:15:57.312738+0000","last_active":"2024-08-31T12:28:47.477656+0000","last_peered":"2024-08-31T12:28:47.477656+0000","last_clean":"2024-08-31T12:28:47.477656+0000","last_became_active":"2024-08-31T12:15:57.312360+0000","last_became_peered":"2024-08-31T12:15:57.312360+0000","last_unstale":"2024-08-31T12:28:47.477656+0000","last_undegraded":"2024-08-31T12:28:47.477656+0000","last_fullsized":"2024-08-31T12:28:47.477656+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2396,"log_dups_size":0,"ondisk_log_size":2396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:22:13.435300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163584295,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11807,"num_write_kb":207035,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"23'5","reported_seq":29,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:56.196855+0000","last_change":"2024-08-31T12:15:56.283526+0000","last_active":"2024-08-31T12:25:56.196855+0000","last_peered":"2024-08-31T12:25:56.196855+0000","last_clean":"2024-08-31T12:25:56.196855+0000","last_became_active":"2024-08-31T12:15:56.283147+0000","last_became_peered":"2024-08-31T12:15:56.283147+0000","last_unstale":"2024-08-31T12:25:56.196855+0000","last_undegraded":"2024-08-31T12:25:56.196855+0000","last_fullsized":"2024-08-31T12:25:56.196855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:51:59.306912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"23'4","reported_seq":28,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:36.210659+0000","last_change":"2024-08-31T12:15:56.291170+0000","last_active":"2024-08-31T12:25:36.210659+0000","last_peered":"2024-08-31T12:25:36.210659+0000","last_clean":"2024-08-31T12:25:36.210659+0000","last_became_active":"2024-08-31T12:15:56.290835+0000","last_became_peered":"2024-08-31T12:15:56.290835+0000","last_unstale":"2024-08-31T12:25:36.210659+0000","last_undegraded":"2024-08-31T12:25:36.210659+0000","last_fullsized":"2024-08-31T12:25:36.210659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:15:51.561831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"23'2543","reported_seq":2565,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.441933+0000","last_change":"2024-08-31T12:15:57.308183+0000","last_active":"2024-08-31T12:28:47.441933+0000","last_peered":"2024-08-31T12:28:47.441933+0000","last_clean":"2024-08-31T12:28:47.441933+0000","last_became_active":"2024-08-31T12:15:57.306962+0000","last_became_peered":"2024-08-31T12:15:57.306962+0000","last_unstale":"2024-08-31T12:28:47.441933+0000","last_undegraded":"2024-08-31T12:28:47.441933+0000","last_fullsized":"2024-08-31T12:28:47.441933+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2543,"log_dups_size":0,"ondisk_log_size":2543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:01:18.188264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205541461,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":251825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"23'2","reported_seq":26,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:40.196751+0000","last_change":"2024-08-31T12:15:56.288126+0000","last_active":"2024-08-31T12:25:40.196751+0000","last_peered":"2024-08-31T12:25:40.196751+0000","last_clean":"2024-08-31T12:25:40.196751+0000","last_became_active":"2024-08-31T12:15:56.287552+0000","last_became_peered":"2024-08-31T12:15:56.287552+0000","last_unstale":"2024-08-31T12:25:40.196751+0000","last_undegraded":"2024-08-31T12:25:40.196751+0000","last_fullsized":"2024-08-31T12:25:40.196751+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:54:29.988079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"23'3450","reported_seq":3472,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.472510+0000","last_change":"2024-08-31T12:15:57.305029+0000","last_active":"2024-08-31T12:28:47.472510+0000","last_peered":"2024-08-31T12:28:47.472510+0000","last_clean":"2024-08-31T12:28:47.472510+0000","last_became_active":"2024-08-31T12:15:57.304700+0000","last_became_peered":"2024-08-31T12:15:57.304700+0000","last_unstale":"2024-08-31T12:28:47.472510+0000","last_undegraded":"2024-08-31T12:28:47.472510+0000","last_fullsized":"2024-08-31T12:28:47.472510+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3450,"log_dups_size":0,"ondisk_log_size":3450,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:58:17.482204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243316666,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17415,"num_write_kb":307366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"23'5","reported_seq":29,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:01.224710+0000","last_change":"2024-08-31T12:15:56.284907+0000","last_active":"2024-08-31T12:28:01.224710+0000","last_peered":"2024-08-31T12:28:01.224710+0000","last_clean":"2024-08-31T12:28:01.224710+0000","last_became_active":"2024-08-31T12:15:56.284558+0000","last_became_peered":"2024-08-31T12:15:56.284558+0000","last_unstale":"2024-08-31T12:28:01.224710+0000","last_undegraded":"2024-08-31T12:28:01.224710+0000","last_fullsized":"2024-08-31T12:28:01.224710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:29:53.117724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"23'2904","reported_seq":2926,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.509044+0000","last_change":"2024-08-31T12:15:57.308679+0000","last_active":"2024-08-31T12:28:47.509044+0000","last_peered":"2024-08-31T12:28:47.509044+0000","last_clean":"2024-08-31T12:28:47.509044+0000","last_became_active":"2024-08-31T12:15:57.308260+0000","last_became_peered":"2024-08-31T12:15:57.308260+0000","last_unstale":"2024-08-31T12:28:47.509044+0000","last_undegraded":"2024-08-31T12:28:47.509044+0000","last_fullsized":"2024-08-31T12:28:47.509044+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2904,"log_dups_size":0,"ondisk_log_size":2904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:31:19.860950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209955924,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14127,"num_write_kb":261546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"21'2","reported_seq":26,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:40.196825+0000","last_change":"2024-08-31T12:15:56.290763+0000","last_active":"2024-08-31T12:25:40.196825+0000","last_peered":"2024-08-31T12:25:40.196825+0000","last_clean":"2024-08-31T12:25:40.196825+0000","last_became_active":"2024-08-31T12:15:56.290400+0000","last_became_peered":"2024-08-31T12:15:56.290400+0000","last_unstale":"2024-08-31T12:25:40.196825+0000","last_undegraded":"2024-08-31T12:25:40.196825+0000","last_fullsized":"2024-08-31T12:25:40.196825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:53:08.929860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"23'2665","reported_seq":2687,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.511808+0000","last_change":"2024-08-31T12:15:57.311657+0000","last_active":"2024-08-31T12:28:47.511808+0000","last_peered":"2024-08-31T12:28:47.511808+0000","last_clean":"2024-08-31T12:28:47.511808+0000","last_became_active":"2024-08-31T12:15:57.310564+0000","last_became_peered":"2024-08-31T12:15:57.310564+0000","last_unstale":"2024-08-31T12:28:47.511808+0000","last_undegraded":"2024-08-31T12:28:47.511808+0000","last_fullsized":"2024-08-31T12:28:47.511808+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2665,"log_dups_size":0,"ondisk_log_size":2665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:37:02.435678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180410852,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13728,"num_write_kb":231123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"23'19","reported_seq":43,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:36.210981+0000","last_change":"2024-08-31T12:15:56.286792+0000","last_active":"2024-08-31T12:25:36.210981+0000","last_peered":"2024-08-31T12:25:36.210981+0000","last_clean":"2024-08-31T12:25:36.210981+0000","last_became_active":"2024-08-31T12:15:56.286458+0000","last_became_peered":"2024-08-31T12:15:56.286458+0000","last_unstale":"2024-08-31T12:25:36.210981+0000","last_undegraded":"2024-08-31T12:25:36.210981+0000","last_fullsized":"2024-08-31T12:25:36.210981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:49:17.488528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"23'3669","reported_seq":3691,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.475304+0000","last_change":"2024-08-31T12:15:57.309485+0000","last_active":"2024-08-31T12:28:47.475304+0000","last_peered":"2024-08-31T12:28:47.475304+0000","last_clean":"2024-08-31T12:28:47.475304+0000","last_became_active":"2024-08-31T12:15:57.309102+0000","last_became_peered":"2024-08-31T12:15:57.309102+0000","last_unstale":"2024-08-31T12:28:47.475304+0000","last_undegraded":"2024-08-31T12:28:47.475304+0000","last_fullsized":"2024-08-31T12:28:47.475304+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3669,"log_dups_size":0,"ondisk_log_size":3669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:17:34.692296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":277024326,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17723,"num_write_kb":341816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"23'317","reported_seq":341,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:22.335219+0000","last_change":"2024-08-31T12:15:56.289460+0000","last_active":"2024-08-31T12:28:22.335219+0000","last_peered":"2024-08-31T12:28:22.335219+0000","last_clean":"2024-08-31T12:28:22.335219+0000","last_became_active":"2024-08-31T12:15:56.289096+0000","last_became_peered":"2024-08-31T12:15:56.289096+0000","last_unstale":"2024-08-31T12:28:22.335219+0000","last_undegraded":"2024-08-31T12:28:22.335219+0000","last_fullsized":"2024-08-31T12:28:22.335219+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:06:34.313772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":317,"num_write_kb":4236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"23'2747","reported_seq":2769,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.493162+0000","last_change":"2024-08-31T12:15:57.317968+0000","last_active":"2024-08-31T12:28:47.493162+0000","last_peered":"2024-08-31T12:28:47.493162+0000","last_clean":"2024-08-31T12:28:47.493162+0000","last_became_active":"2024-08-31T12:15:57.317632+0000","last_became_peered":"2024-08-31T12:15:57.317632+0000","last_unstale":"2024-08-31T12:28:47.493162+0000","last_undegraded":"2024-08-31T12:28:47.493162+0000","last_fullsized":"2024-08-31T12:28:47.493162+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2747,"log_dups_size":0,"ondisk_log_size":2747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:22:02.569106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197132288,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13667,"num_write_kb":247166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"23'10","reported_seq":42,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:51.225974+0000","last_change":"2024-08-31T12:28:51.225974+0000","last_active":"2024-08-31T12:28:51.225974+0000","last_peered":"2024-08-31T12:28:51.225974+0000","last_clean":"2024-08-31T12:28:51.225974+0000","last_became_active":"2024-08-31T12:15:56.283675+0000","last_became_peered":"2024-08-31T12:15:56.283675+0000","last_unstale":"2024-08-31T12:28:51.225974+0000","last_undegraded":"2024-08-31T12:28:51.225974+0000","last_fullsized":"2024-08-31T12:28:51.225974+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'10","last_scrub_stamp":"2024-08-31T12:28:51.225832+0000","last_deep_scrub":"23'10","last_deep_scrub_stamp":"2024-08-31T12:28:51.225832+0000","last_clean_scrub_stamp":"2024-08-31T12:28:51.225832+0000","objects_scrubbed":4,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:46:50.223589+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"23'3061","reported_seq":3083,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.459667+0000","last_change":"2024-08-31T12:15:57.312440+0000","last_active":"2024-08-31T12:28:47.459667+0000","last_peered":"2024-08-31T12:28:47.459667+0000","last_clean":"2024-08-31T12:28:47.459667+0000","last_became_active":"2024-08-31T12:15:57.312007+0000","last_became_peered":"2024-08-31T12:15:57.312007+0000","last_unstale":"2024-08-31T12:28:47.459667+0000","last_undegraded":"2024-08-31T12:28:47.459667+0000","last_fullsized":"2024-08-31T12:28:47.459667+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3061,"log_dups_size":0,"ondisk_log_size":3061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:23:43.553625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197138377,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15100,"num_write_kb":253041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"23'4","reported_seq":28,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:20.194216+0000","last_change":"2024-08-31T12:15:56.290968+0000","last_active":"2024-08-31T12:25:20.194216+0000","last_peered":"2024-08-31T12:25:20.194216+0000","last_clean":"2024-08-31T12:25:20.194216+0000","last_became_active":"2024-08-31T12:15:56.290603+0000","last_became_peered":"2024-08-31T12:15:56.290603+0000","last_unstale":"2024-08-31T12:25:20.194216+0000","last_undegraded":"2024-08-31T12:25:20.194216+0000","last_fullsized":"2024-08-31T12:25:20.194216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:50:23.215239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"23'3294","reported_seq":3316,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.520094+0000","last_change":"2024-08-31T12:15:57.312790+0000","last_active":"2024-08-31T12:28:47.520094+0000","last_peered":"2024-08-31T12:28:47.520094+0000","last_clean":"2024-08-31T12:28:47.520094+0000","last_became_active":"2024-08-31T12:15:57.312383+0000","last_became_peered":"2024-08-31T12:15:57.312383+0000","last_unstale":"2024-08-31T12:28:47.520094+0000","last_undegraded":"2024-08-31T12:28:47.520094+0000","last_fullsized":"2024-08-31T12:28:47.520094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3294,"log_dups_size":0,"ondisk_log_size":3294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:20:41.042375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234891518,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16277,"num_write_kb":294634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"23'5","reported_seq":29,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:16.191885+0000","last_change":"2024-08-31T12:15:56.289795+0000","last_active":"2024-08-31T12:25:16.191885+0000","last_peered":"2024-08-31T12:25:16.191885+0000","last_clean":"2024-08-31T12:25:16.191885+0000","last_became_active":"2024-08-31T12:15:56.289477+0000","last_became_peered":"2024-08-31T12:15:56.289477+0000","last_unstale":"2024-08-31T12:25:16.191885+0000","last_undegraded":"2024-08-31T12:25:16.191885+0000","last_fullsized":"2024-08-31T12:25:16.191885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:15:51.561831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":54,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"23'2834","reported_seq":2856,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.422961+0000","last_change":"2024-08-31T12:15:57.312865+0000","last_active":"2024-08-31T12:28:47.422961+0000","last_peered":"2024-08-31T12:28:47.422961+0000","last_clean":"2024-08-31T12:28:47.422961+0000","last_became_active":"2024-08-31T12:15:57.312462+0000","last_became_peered":"2024-08-31T12:15:57.312462+0000","last_unstale":"2024-08-31T12:28:47.422961+0000","last_undegraded":"2024-08-31T12:28:47.422961+0000","last_fullsized":"2024-08-31T12:28:47.422961+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2834,"log_dups_size":0,"ondisk_log_size":2834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:16:42.625396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222316815,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13883,"num_write_kb":272702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'15","reported_seq":39,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:18.179855+0000","last_change":"2024-08-31T12:15:56.285412+0000","last_active":"2024-08-31T12:25:18.179855+0000","last_peered":"2024-08-31T12:25:18.179855+0000","last_clean":"2024-08-31T12:25:18.179855+0000","last_became_active":"2024-08-31T12:15:56.285108+0000","last_became_peered":"2024-08-31T12:15:56.285108+0000","last_unstale":"2024-08-31T12:25:18.179855+0000","last_undegraded":"2024-08-31T12:25:18.179855+0000","last_fullsized":"2024-08-31T12:25:18.179855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:29:45.580957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"23'2849","reported_seq":2871,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.435389+0000","last_change":"2024-08-31T12:15:57.311701+0000","last_active":"2024-08-31T12:28:47.435389+0000","last_peered":"2024-08-31T12:28:47.435389+0000","last_clean":"2024-08-31T12:28:47.435389+0000","last_became_active":"2024-08-31T12:15:57.311368+0000","last_became_peered":"2024-08-31T12:15:57.311368+0000","last_unstale":"2024-08-31T12:28:47.435389+0000","last_undegraded":"2024-08-31T12:28:47.435389+0000","last_fullsized":"2024-08-31T12:28:47.435389+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2849,"log_dups_size":0,"ondisk_log_size":2849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:30:13.715505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205547202,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14167,"num_write_kb":257592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"23'3","reported_seq":27,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:00.192148+0000","last_change":"2024-08-31T12:15:56.289979+0000","last_active":"2024-08-31T12:25:00.192148+0000","last_peered":"2024-08-31T12:25:00.192148+0000","last_clean":"2024-08-31T12:25:00.192148+0000","last_became_active":"2024-08-31T12:15:56.288289+0000","last_became_peered":"2024-08-31T12:15:56.288289+0000","last_unstale":"2024-08-31T12:25:00.192148+0000","last_undegraded":"2024-08-31T12:25:00.192148+0000","last_fullsized":"2024-08-31T12:25:00.192148+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:46:55.057087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"23'2813","reported_seq":2835,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.480495+0000","last_change":"2024-08-31T12:15:57.311580+0000","last_active":"2024-08-31T12:28:47.480495+0000","last_peered":"2024-08-31T12:28:47.480495+0000","last_clean":"2024-08-31T12:28:47.480495+0000","last_became_active":"2024-08-31T12:15:57.311323+0000","last_became_peered":"2024-08-31T12:15:57.311323+0000","last_unstale":"2024-08-31T12:28:47.480495+0000","last_undegraded":"2024-08-31T12:28:47.480495+0000","last_fullsized":"2024-08-31T12:28:47.480495+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2813,"log_dups_size":0,"ondisk_log_size":2813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:54:03.871840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201522906,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13808,"num_write_kb":252081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"23'3","reported_seq":27,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:24:58.177773+0000","last_change":"2024-08-31T12:15:56.290496+0000","last_active":"2024-08-31T12:24:58.177773+0000","last_peered":"2024-08-31T12:24:58.177773+0000","last_clean":"2024-08-31T12:24:58.177773+0000","last_became_active":"2024-08-31T12:15:56.290196+0000","last_became_peered":"2024-08-31T12:15:56.290196+0000","last_unstale":"2024-08-31T12:24:58.177773+0000","last_undegraded":"2024-08-31T12:24:58.177773+0000","last_fullsized":"2024-08-31T12:24:58.177773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:52:20.055757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"23'3291","reported_seq":3313,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.519305+0000","last_change":"2024-08-31T12:15:57.313317+0000","last_active":"2024-08-31T12:28:47.519305+0000","last_peered":"2024-08-31T12:28:47.519305+0000","last_clean":"2024-08-31T12:28:47.519305+0000","last_became_active":"2024-08-31T12:15:57.312145+0000","last_became_peered":"2024-08-31T12:15:57.312145+0000","last_unstale":"2024-08-31T12:28:47.519305+0000","last_undegraded":"2024-08-31T12:28:47.519305+0000","last_fullsized":"2024-08-31T12:28:47.519305+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3291,"log_dups_size":0,"ondisk_log_size":3291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:09:31.904512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243277619,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16586,"num_write_kb":304033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":25,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:24:56.207028+0000","last_change":"2024-08-31T12:15:56.281809+0000","last_active":"2024-08-31T12:24:56.207028+0000","last_peered":"2024-08-31T12:24:56.207028+0000","last_clean":"2024-08-31T12:24:56.207028+0000","last_became_active":"2024-08-31T12:15:56.281368+0000","last_became_peered":"2024-08-31T12:15:56.281368+0000","last_unstale":"2024-08-31T12:24:56.207028+0000","last_undegraded":"2024-08-31T12:24:56.207028+0000","last_fullsized":"2024-08-31T12:24:56.207028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:17:33.687196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"23'3295","reported_seq":3317,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.486094+0000","last_change":"2024-08-31T12:15:57.305874+0000","last_active":"2024-08-31T12:28:47.486094+0000","last_peered":"2024-08-31T12:28:47.486094+0000","last_clean":"2024-08-31T12:28:47.486094+0000","last_became_active":"2024-08-31T12:15:57.305507+0000","last_became_peered":"2024-08-31T12:15:57.305507+0000","last_unstale":"2024-08-31T12:28:47.486094+0000","last_undegraded":"2024-08-31T12:28:47.486094+0000","last_fullsized":"2024-08-31T12:28:47.486094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3295,"log_dups_size":0,"ondisk_log_size":3295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:22:02.569106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218117314,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16432,"num_write_kb":278837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"23'4","reported_seq":28,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:24:58.177745+0000","last_change":"2024-08-31T12:15:56.291851+0000","last_active":"2024-08-31T12:24:58.177745+0000","last_peered":"2024-08-31T12:24:58.177745+0000","last_clean":"2024-08-31T12:24:58.177745+0000","last_became_active":"2024-08-31T12:15:56.291446+0000","last_became_peered":"2024-08-31T12:15:56.291446+0000","last_unstale":"2024-08-31T12:24:58.177745+0000","last_undegraded":"2024-08-31T12:24:58.177745+0000","last_fullsized":"2024-08-31T12:24:58.177745+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:54:29.988079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"23'2744","reported_seq":2766,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.496685+0000","last_change":"2024-08-31T12:15:57.316154+0000","last_active":"2024-08-31T12:28:47.496685+0000","last_peered":"2024-08-31T12:28:47.496685+0000","last_clean":"2024-08-31T12:28:47.496685+0000","last_became_active":"2024-08-31T12:15:57.315830+0000","last_became_peered":"2024-08-31T12:15:57.315830+0000","last_unstale":"2024-08-31T12:28:47.496685+0000","last_undegraded":"2024-08-31T12:28:47.496685+0000","last_fullsized":"2024-08-31T12:28:47.496685+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2744,"log_dups_size":0,"ondisk_log_size":2744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:05:53.039520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209813064,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13700,"num_write_kb":259755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":121,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:50.198157+0000","last_change":"2024-08-31T12:15:47.084912+0000","last_active":"2024-08-31T12:25:50.198157+0000","last_peered":"2024-08-31T12:25:50.198157+0000","last_clean":"2024-08-31T12:25:50.198157+0000","last_became_active":"2024-08-31T12:15:47.083771+0000","last_became_peered":"2024-08-31T12:15:47.083771+0000","last_unstale":"2024-08-31T12:25:50.198157+0000","last_undegraded":"2024-08-31T12:25:50.198157+0000","last_fullsized":"2024-08-31T12:25:50.198157+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:45.213688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:45.213688+0000","last_clean_scrub_stamp":"2024-08-31T12:15:45.213688+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:25:24.827530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"23'4","reported_seq":29,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:26:25.205795+0000","last_change":"2024-08-31T12:15:56.285355+0000","last_active":"2024-08-31T12:26:25.205795+0000","last_peered":"2024-08-31T12:26:25.205795+0000","last_clean":"2024-08-31T12:26:25.205795+0000","last_became_active":"2024-08-31T12:15:56.284470+0000","last_became_peered":"2024-08-31T12:15:56.284470+0000","last_unstale":"2024-08-31T12:26:25.205795+0000","last_undegraded":"2024-08-31T12:26:25.205795+0000","last_fullsized":"2024-08-31T12:26:25.205795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:52:20.055757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"23'2775","reported_seq":2798,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.472877+0000","last_change":"2024-08-31T12:15:57.309039+0000","last_active":"2024-08-31T12:28:47.472877+0000","last_peered":"2024-08-31T12:28:47.472877+0000","last_clean":"2024-08-31T12:28:47.472877+0000","last_became_active":"2024-08-31T12:15:57.308721+0000","last_became_peered":"2024-08-31T12:15:57.308721+0000","last_unstale":"2024-08-31T12:28:47.472877+0000","last_undegraded":"2024-08-31T12:28:47.472877+0000","last_fullsized":"2024-08-31T12:28:47.472877+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2775,"log_dups_size":0,"ondisk_log_size":2775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:43:12.703971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218112028,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13723,"num_write_kb":267984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"23'4","reported_seq":29,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:27:20.870746+0000","last_change":"2024-08-31T12:15:56.293221+0000","last_active":"2024-08-31T12:27:20.870746+0000","last_peered":"2024-08-31T12:27:20.870746+0000","last_clean":"2024-08-31T12:27:20.870746+0000","last_became_active":"2024-08-31T12:15:56.292910+0000","last_became_peered":"2024-08-31T12:15:56.292910+0000","last_unstale":"2024-08-31T12:27:20.870746+0000","last_undegraded":"2024-08-31T12:27:20.870746+0000","last_fullsized":"2024-08-31T12:27:20.870746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:14:33.483002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"23'2762","reported_seq":2785,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.457744+0000","last_change":"2024-08-31T12:15:57.307266+0000","last_active":"2024-08-31T12:28:47.457744+0000","last_peered":"2024-08-31T12:28:47.457744+0000","last_clean":"2024-08-31T12:28:47.457744+0000","last_became_active":"2024-08-31T12:15:57.306529+0000","last_became_peered":"2024-08-31T12:15:57.306529+0000","last_unstale":"2024-08-31T12:28:47.457744+0000","last_undegraded":"2024-08-31T12:28:47.457744+0000","last_fullsized":"2024-08-31T12:28:47.457744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2762,"log_dups_size":0,"ondisk_log_size":2762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:42:39.731742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201337916,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14174,"num_write_kb":253444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"23'38","reported_seq":101,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:52.310657+0000","last_change":"2024-08-31T12:28:52.310657+0000","last_active":"2024-08-31T12:28:52.310657+0000","last_peered":"2024-08-31T12:28:52.310657+0000","last_clean":"2024-08-31T12:28:52.310657+0000","last_became_active":"2024-08-31T12:15:56.291064+0000","last_became_peered":"2024-08-31T12:15:56.291064+0000","last_unstale":"2024-08-31T12:28:52.310657+0000","last_undegraded":"2024-08-31T12:28:52.310657+0000","last_fullsized":"2024-08-31T12:28:52.310657+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'38","last_scrub_stamp":"2024-08-31T12:28:52.310562+0000","last_deep_scrub":"23'38","last_deep_scrub_stamp":"2024-08-31T12:28:52.310562+0000","last_clean_scrub_stamp":"2024-08-31T12:28:52.310562+0000","objects_scrubbed":3,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:52:09.840889+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47459,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":30,"num_read_kb":64,"num_write":34,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"23'2946","reported_seq":2969,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.447731+0000","last_change":"2024-08-31T12:15:57.299212+0000","last_active":"2024-08-31T12:28:47.447731+0000","last_peered":"2024-08-31T12:28:47.447731+0000","last_clean":"2024-08-31T12:28:47.447731+0000","last_became_active":"2024-08-31T12:15:57.298892+0000","last_became_peered":"2024-08-31T12:15:57.298892+0000","last_unstale":"2024-08-31T12:28:47.447731+0000","last_undegraded":"2024-08-31T12:28:47.447731+0000","last_fullsized":"2024-08-31T12:28:47.447731+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2946,"log_dups_size":0,"ondisk_log_size":2946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:53:19.523811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197144226,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14947,"num_write_kb":252544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"23'2","reported_seq":27,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:26:25.205779+0000","last_change":"2024-08-31T12:15:56.286720+0000","last_active":"2024-08-31T12:26:25.205779+0000","last_peered":"2024-08-31T12:26:25.205779+0000","last_clean":"2024-08-31T12:26:25.205779+0000","last_became_active":"2024-08-31T12:15:56.285648+0000","last_became_peered":"2024-08-31T12:15:56.285648+0000","last_unstale":"2024-08-31T12:26:25.205779+0000","last_undegraded":"2024-08-31T12:26:25.205779+0000","last_fullsized":"2024-08-31T12:26:25.205779+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:39:31.946792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"23'2769","reported_seq":2792,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.472313+0000","last_change":"2024-08-31T12:15:57.311785+0000","last_active":"2024-08-31T12:28:47.472313+0000","last_peered":"2024-08-31T12:28:47.472313+0000","last_clean":"2024-08-31T12:28:47.472313+0000","last_became_active":"2024-08-31T12:15:57.311469+0000","last_became_peered":"2024-08-31T12:15:57.311469+0000","last_unstale":"2024-08-31T12:28:47.472313+0000","last_undegraded":"2024-08-31T12:28:47.472313+0000","last_fullsized":"2024-08-31T12:28:47.472313+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2769,"log_dups_size":0,"ondisk_log_size":2769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:29:54.122825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197139313,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14081,"num_write_kb":248914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"23'1","reported_seq":26,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:26:21.215756+0000","last_change":"2024-08-31T12:15:56.284302+0000","last_active":"2024-08-31T12:26:21.215756+0000","last_peered":"2024-08-31T12:26:21.215756+0000","last_clean":"2024-08-31T12:26:21.215756+0000","last_became_active":"2024-08-31T12:15:56.283979+0000","last_became_peered":"2024-08-31T12:15:56.283979+0000","last_unstale":"2024-08-31T12:26:21.215756+0000","last_undegraded":"2024-08-31T12:26:21.215756+0000","last_fullsized":"2024-08-31T12:26:21.215756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:51:59.306912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"23'2527","reported_seq":2543,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.473715+0000","last_change":"2024-08-31T12:16:41.195070+0000","last_active":"2024-08-31T12:28:47.473715+0000","last_peered":"2024-08-31T12:28:47.473715+0000","last_clean":"2024-08-31T12:28:47.473715+0000","last_became_active":"2024-08-31T12:16:41.194724+0000","last_became_peered":"2024-08-31T12:16:41.194724+0000","last_unstale":"2024-08-31T12:28:47.473715+0000","last_undegraded":"2024-08-31T12:28:47.473715+0000","last_fullsized":"2024-08-31T12:28:47.473715+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2527,"log_dups_size":0,"ondisk_log_size":2527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:24:46.945850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184558382,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13011,"num_write_kb":232344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"21'1","reported_seq":26,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:26:25.202953+0000","last_change":"2024-08-31T12:15:56.297648+0000","last_active":"2024-08-31T12:26:25.202953+0000","last_peered":"2024-08-31T12:26:25.202953+0000","last_clean":"2024-08-31T12:26:25.202953+0000","last_became_active":"2024-08-31T12:15:56.297306+0000","last_became_peered":"2024-08-31T12:15:56.297306+0000","last_unstale":"2024-08-31T12:26:25.202953+0000","last_undegraded":"2024-08-31T12:26:25.202953+0000","last_fullsized":"2024-08-31T12:26:25.202953+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:29:54.693819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"23'3173","reported_seq":3196,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.445424+0000","last_change":"2024-08-31T12:15:57.297230+0000","last_active":"2024-08-31T12:28:47.445424+0000","last_peered":"2024-08-31T12:28:47.445424+0000","last_clean":"2024-08-31T12:28:47.445424+0000","last_became_active":"2024-08-31T12:15:57.296892+0000","last_became_peered":"2024-08-31T12:15:57.296892+0000","last_unstale":"2024-08-31T12:28:47.445424+0000","last_undegraded":"2024-08-31T12:28:47.445424+0000","last_fullsized":"2024-08-31T12:28:47.445424+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3173,"log_dups_size":0,"ondisk_log_size":3173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:42:36.033586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213956769,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16152,"num_write_kb":273638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"23'428","reported_seq":452,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:24.459222+0000","last_change":"2024-08-31T12:15:56.286504+0000","last_active":"2024-08-31T12:28:24.459222+0000","last_peered":"2024-08-31T12:28:24.459222+0000","last_clean":"2024-08-31T12:28:24.459222+0000","last_became_active":"2024-08-31T12:15:56.286173+0000","last_became_peered":"2024-08-31T12:15:56.286173+0000","last_unstale":"2024-08-31T12:28:24.459222+0000","last_undegraded":"2024-08-31T12:28:24.459222+0000","last_fullsized":"2024-08-31T12:28:24.459222+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":428,"log_dups_size":0,"ondisk_log_size":428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:06:34.313772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":977,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"23'2921","reported_seq":2943,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.483253+0000","last_change":"2024-08-31T12:15:57.308013+0000","last_active":"2024-08-31T12:28:47.483253+0000","last_peered":"2024-08-31T12:28:47.483253+0000","last_clean":"2024-08-31T12:28:47.483253+0000","last_became_active":"2024-08-31T12:15:57.307701+0000","last_became_peered":"2024-08-31T12:15:57.307701+0000","last_unstale":"2024-08-31T12:28:47.483253+0000","last_undegraded":"2024-08-31T12:28:47.483253+0000","last_fullsized":"2024-08-31T12:28:47.483253+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2921,"log_dups_size":0,"ondisk_log_size":2921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:06:15.161589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205521115,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14225,"num_write_kb":257747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'4","reported_seq":28,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:24:36.187735+0000","last_change":"2024-08-31T12:15:56.294852+0000","last_active":"2024-08-31T12:24:36.187735+0000","last_peered":"2024-08-31T12:24:36.187735+0000","last_clean":"2024-08-31T12:24:36.187735+0000","last_became_active":"2024-08-31T12:15:56.294496+0000","last_became_peered":"2024-08-31T12:15:56.294496+0000","last_unstale":"2024-08-31T12:24:36.187735+0000","last_undegraded":"2024-08-31T12:24:36.187735+0000","last_fullsized":"2024-08-31T12:24:36.187735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:49:17.488528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"23'2544","reported_seq":2566,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.455480+0000","last_change":"2024-08-31T12:15:57.305202+0000","last_active":"2024-08-31T12:28:47.455480+0000","last_peered":"2024-08-31T12:28:47.455480+0000","last_clean":"2024-08-31T12:28:47.455480+0000","last_became_active":"2024-08-31T12:15:57.304602+0000","last_became_peered":"2024-08-31T12:15:57.304602+0000","last_unstale":"2024-08-31T12:28:47.455480+0000","last_undegraded":"2024-08-31T12:28:47.455480+0000","last_fullsized":"2024-08-31T12:28:47.455480+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2544,"log_dups_size":0,"ondisk_log_size":2544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:56:03.280339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176167762,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12536,"num_write_kb":222149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"23'3","reported_seq":27,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:24:40.189661+0000","last_change":"2024-08-31T12:15:56.297542+0000","last_active":"2024-08-31T12:24:40.189661+0000","last_peered":"2024-08-31T12:24:40.189661+0000","last_clean":"2024-08-31T12:24:40.189661+0000","last_became_active":"2024-08-31T12:15:56.297168+0000","last_became_peered":"2024-08-31T12:15:56.297168+0000","last_unstale":"2024-08-31T12:24:40.189661+0000","last_undegraded":"2024-08-31T12:24:40.189661+0000","last_fullsized":"2024-08-31T12:24:40.189661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:56:41.924772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"23'2322","reported_seq":2344,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.414629+0000","last_change":"2024-08-31T12:15:57.298845+0000","last_active":"2024-08-31T12:28:47.414629+0000","last_peered":"2024-08-31T12:28:47.414629+0000","last_clean":"2024-08-31T12:28:47.414629+0000","last_became_active":"2024-08-31T12:15:57.298520+0000","last_became_peered":"2024-08-31T12:15:57.298520+0000","last_unstale":"2024-08-31T12:28:47.414629+0000","last_undegraded":"2024-08-31T12:28:47.414629+0000","last_fullsized":"2024-08-31T12:28:47.414629+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2322,"log_dups_size":0,"ondisk_log_size":2322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:06:35.318873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197144309,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11287,"num_write_kb":237691,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"21'3","reported_seq":27,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:24:36.205095+0000","last_change":"2024-08-31T12:15:56.290475+0000","last_active":"2024-08-31T12:24:36.205095+0000","last_peered":"2024-08-31T12:24:36.205095+0000","last_clean":"2024-08-31T12:24:36.205095+0000","last_became_active":"2024-08-31T12:15:56.289960+0000","last_became_peered":"2024-08-31T12:15:56.289960+0000","last_unstale":"2024-08-31T12:24:36.205095+0000","last_undegraded":"2024-08-31T12:24:36.205095+0000","last_fullsized":"2024-08-31T12:24:36.205095+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:29:53.117724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"23'2979","reported_seq":3001,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.466373+0000","last_change":"2024-08-31T12:15:57.301460+0000","last_active":"2024-08-31T12:28:47.466373+0000","last_peered":"2024-08-31T12:28:47.466373+0000","last_clean":"2024-08-31T12:28:47.466373+0000","last_became_active":"2024-08-31T12:15:57.301157+0000","last_became_peered":"2024-08-31T12:15:57.301157+0000","last_unstale":"2024-08-31T12:28:47.466373+0000","last_undegraded":"2024-08-31T12:28:47.466373+0000","last_fullsized":"2024-08-31T12:28:47.466373+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2979,"log_dups_size":0,"ondisk_log_size":2979,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:22:52.173578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209728881,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15223,"num_write_kb":265883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"21'1","reported_seq":25,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:24:36.205125+0000","last_change":"2024-08-31T12:15:56.289720+0000","last_active":"2024-08-31T12:24:36.205125+0000","last_peered":"2024-08-31T12:24:36.205125+0000","last_clean":"2024-08-31T12:24:36.205125+0000","last_became_active":"2024-08-31T12:15:56.289332+0000","last_became_peered":"2024-08-31T12:15:56.289332+0000","last_unstale":"2024-08-31T12:24:36.205125+0000","last_undegraded":"2024-08-31T12:24:36.205125+0000","last_fullsized":"2024-08-31T12:24:36.205125+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:15:51.561831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"23'2423","reported_seq":2445,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.473180+0000","last_change":"2024-08-31T12:15:57.313802+0000","last_active":"2024-08-31T12:28:47.473180+0000","last_peered":"2024-08-31T12:28:47.473180+0000","last_clean":"2024-08-31T12:28:47.473180+0000","last_became_active":"2024-08-31T12:15:57.313428+0000","last_became_peered":"2024-08-31T12:15:57.313428+0000","last_unstale":"2024-08-31T12:28:47.473180+0000","last_undegraded":"2024-08-31T12:28:47.473180+0000","last_fullsized":"2024-08-31T12:28:47.473180+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2423,"log_dups_size":0,"ondisk_log_size":2423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:22:13.435300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176173175,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11447,"num_write_kb":217885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"23'1068","reported_seq":1092,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:27:20.763645+0000","last_change":"2024-08-31T12:15:56.296105+0000","last_active":"2024-08-31T12:27:20.763645+0000","last_peered":"2024-08-31T12:27:20.763645+0000","last_clean":"2024-08-31T12:27:20.763645+0000","last_became_active":"2024-08-31T12:15:56.295524+0000","last_became_peered":"2024-08-31T12:15:56.295524+0000","last_unstale":"2024-08-31T12:27:20.763645+0000","last_undegraded":"2024-08-31T12:27:20.763645+0000","last_fullsized":"2024-08-31T12:27:20.763645+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":1068,"log_dups_size":0,"ondisk_log_size":1068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:41:55.482044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1064,"num_write_kb":4401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"23'2507","reported_seq":2529,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.435175+0000","last_change":"2024-08-31T12:15:57.300155+0000","last_active":"2024-08-31T12:28:47.435175+0000","last_peered":"2024-08-31T12:28:47.435175+0000","last_clean":"2024-08-31T12:28:47.435175+0000","last_became_active":"2024-08-31T12:15:57.299436+0000","last_became_peered":"2024-08-31T12:15:57.299436+0000","last_unstale":"2024-08-31T12:28:47.435175+0000","last_undegraded":"2024-08-31T12:28:47.435175+0000","last_fullsized":"2024-08-31T12:28:47.435175+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2507,"log_dups_size":0,"ondisk_log_size":2507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:37:45.891394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188744397,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12696,"num_write_kb":235198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"23'3373","reported_seq":3395,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.503787+0000","last_change":"2024-08-31T12:15:57.300529+0000","last_active":"2024-08-31T12:28:47.503787+0000","last_peered":"2024-08-31T12:28:47.503787+0000","last_clean":"2024-08-31T12:28:47.503787+0000","last_became_active":"2024-08-31T12:15:57.300159+0000","last_became_peered":"2024-08-31T12:15:57.300159+0000","last_unstale":"2024-08-31T12:28:47.503787+0000","last_undegraded":"2024-08-31T12:28:47.503787+0000","last_fullsized":"2024-08-31T12:28:47.503787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3373,"log_dups_size":0,"ondisk_log_size":3373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:35:53.831742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251730727,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16759,"num_write_kb":312885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":26,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:26:00.199883+0000","last_change":"2024-08-31T12:15:56.297791+0000","last_active":"2024-08-31T12:26:00.199883+0000","last_peered":"2024-08-31T12:26:00.199883+0000","last_clean":"2024-08-31T12:26:00.199883+0000","last_became_active":"2024-08-31T12:15:56.296976+0000","last_became_peered":"2024-08-31T12:15:56.296976+0000","last_unstale":"2024-08-31T12:26:00.199883+0000","last_undegraded":"2024-08-31T12:26:00.199883+0000","last_fullsized":"2024-08-31T12:26:00.199883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:07:54.963050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"23'2176","reported_seq":2198,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.487154+0000","last_change":"2024-08-31T12:15:57.301167+0000","last_active":"2024-08-31T12:28:47.487154+0000","last_peered":"2024-08-31T12:28:47.487154+0000","last_clean":"2024-08-31T12:28:47.487154+0000","last_became_active":"2024-08-31T12:15:57.300646+0000","last_became_peered":"2024-08-31T12:15:57.300646+0000","last_unstale":"2024-08-31T12:28:47.487154+0000","last_undegraded":"2024-08-31T12:28:47.487154+0000","last_fullsized":"2024-08-31T12:28:47.487154+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2176,"log_dups_size":0,"ondisk_log_size":2176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:05:53.039520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150996671,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10769,"num_write_kb":190633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"23'9","reported_seq":33,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:26:00.199906+0000","last_change":"2024-08-31T12:15:56.295139+0000","last_active":"2024-08-31T12:26:00.199906+0000","last_peered":"2024-08-31T12:26:00.199906+0000","last_clean":"2024-08-31T12:26:00.199906+0000","last_became_active":"2024-08-31T12:15:56.294560+0000","last_became_peered":"2024-08-31T12:15:56.294560+0000","last_unstale":"2024-08-31T12:26:00.199906+0000","last_undegraded":"2024-08-31T12:26:00.199906+0000","last_fullsized":"2024-08-31T12:26:00.199906+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:52:16.999507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"23'3354","reported_seq":3377,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.512326+0000","last_change":"2024-08-31T12:15:57.311074+0000","last_active":"2024-08-31T12:28:47.512326+0000","last_peered":"2024-08-31T12:28:47.512326+0000","last_clean":"2024-08-31T12:28:47.512326+0000","last_became_active":"2024-08-31T12:15:57.310262+0000","last_became_peered":"2024-08-31T12:15:57.310262+0000","last_unstale":"2024-08-31T12:28:47.512326+0000","last_undegraded":"2024-08-31T12:28:47.512326+0000","last_fullsized":"2024-08-31T12:28:47.512326+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3354,"log_dups_size":0,"ondisk_log_size":3354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:47:51.267409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251719163,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16721,"num_write_kb":312844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"23'3","reported_seq":28,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:25:56.212749+0000","last_change":"2024-08-31T12:15:56.285428+0000","last_active":"2024-08-31T12:25:56.212749+0000","last_peered":"2024-08-31T12:25:56.212749+0000","last_clean":"2024-08-31T12:25:56.212749+0000","last_became_active":"2024-08-31T12:15:56.284884+0000","last_became_peered":"2024-08-31T12:15:56.284884+0000","last_unstale":"2024-08-31T12:25:56.212749+0000","last_undegraded":"2024-08-31T12:25:56.212749+0000","last_fullsized":"2024-08-31T12:25:56.212749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:49:25.181432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"23'3172","reported_seq":3195,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.448314+0000","last_change":"2024-08-31T12:15:57.305772+0000","last_active":"2024-08-31T12:28:47.448314+0000","last_peered":"2024-08-31T12:28:47.448314+0000","last_clean":"2024-08-31T12:28:47.448314+0000","last_became_active":"2024-08-31T12:15:57.305409+0000","last_became_peered":"2024-08-31T12:15:57.305409+0000","last_unstale":"2024-08-31T12:28:47.448314+0000","last_undegraded":"2024-08-31T12:28:47.448314+0000","last_fullsized":"2024-08-31T12:28:47.448314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":3172,"log_dups_size":0,"ondisk_log_size":3172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:53:19.523811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222298112,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15945,"num_write_kb":281070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"23'233","reported_seq":258,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.337968+0000","last_change":"2024-08-31T12:15:56.294197+0000","last_active":"2024-08-31T12:28:47.337968+0000","last_peered":"2024-08-31T12:28:47.337968+0000","last_clean":"2024-08-31T12:28:47.337968+0000","last_became_active":"2024-08-31T12:15:56.293864+0000","last_became_peered":"2024-08-31T12:15:56.293864+0000","last_unstale":"2024-08-31T12:28:47.337968+0000","last_undegraded":"2024-08-31T12:28:47.337968+0000","last_fullsized":"2024-08-31T12:28:47.337968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:55.259987+0000","last_clean_scrub_stamp":"2024-08-31T12:15:55.259987+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:43:17.443433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1383123,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":232,"num_write_kb":1484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":13424230175,"num_objects":3391,"num_object_clones":0,"num_object_copies":6782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":918154,"num_write_kb":16787942,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26849361920,"data_stored":26848441814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":184152,"ondisk_log_size":184152,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":14017686,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":30,"num_read_kb":64,"num_write":3315,"num_write_kb":14850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28188672,"data_stored":28035372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2773,"ondisk_log_size":2773,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509601,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2618576,"kb_used_data":2384832,"kb_used_omap":1,"kb_used_meta":233726,"kb_avail":91753264,"statfs":{"total":96636764160,"available":93955342336,"internally_reserved":0,"allocated":2442067968,"data_stored":2441803901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":239335878},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509600,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4206900,"kb_used_data":3851044,"kb_used_omap":1,"kb_used_meta":355838,"kb_avail":90164940,"statfs":{"total":96636764160,"available":92328898560,"internally_reserved":0,"allocated":3943469056,"data_stored":3943171671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":364378567},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509601,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2544520,"kb_used_data":2306616,"kb_used_omap":1,"kb_used_meta":237886,"kb_avail":91827320,"statfs":{"total":96636764160,"available":94031175680,"internally_reserved":0,"allocated":2361974784,"data_stored":2361727195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":243595717},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509601,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4141800,"kb_used_data":3795672,"kb_used_omap":1,"kb_used_meta":346110,"kb_avail":90230040,"statfs":{"total":96636764160,"available":92395560960,"internally_reserved":0,"allocated":3886768128,"data_stored":3886452784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354417093},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542304,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4422912,"kb_used_data":4066992,"kb_used_omap":1,"kb_used_meta":355902,"kb_avail":89948928,"statfs":{"total":96636764160,"available":92107702272,"internally_reserved":0,"allocated":4164599808,"data_stored":4164272466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":364444106},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542306,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4162248,"kb_used_data":3793912,"kb_used_omap":1,"kb_used_meta":368318,"kb_avail":90209592,"statfs":{"total":96636764160,"available":92374622208,"internally_reserved":0,"allocated":3884965888,"data_stored":3884692848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":377158085},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542304,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4136736,"kb_used_data":3839056,"kb_used_omap":1,"kb_used_meta":297662,"kb_avail":90235104,"statfs":{"total":96636764160,"available":92400746496,"internally_reserved":0,"allocated":3931193344,"data_stored":3930920211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":304806342},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575009,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2442052,"kb_used_data":2212468,"kb_used_omap":1,"kb_used_meta":229566,"kb_avail":91929788,"statfs":{"total":96636764160,"available":94136102912,"internally_reserved":0,"allocated":2265567232,"data_stored":2265299110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":235076037},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1400832,"data_stored":1383201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":48187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8458240,"data_stored":8437943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4210688,"data_stored":4195018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4210688,"data_stored":4195654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":9793536,"data_stored":9773047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3884539904,"data_stored":3884393932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2361716736,"data_stored":2361640968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2265264128,"data_stored":2265165640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3934777344,"data_stored":3934648445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3926749184,"data_stored":3926639910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3884711936,"data_stored":3884606187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2437623808,"data_stored":2437522964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4153978880,"data_stored":4153823768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-31T12:28:55.149 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.150 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.150 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.150 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.150 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.150 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.150 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.151 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.151 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.151 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.151 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.151 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.151 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.152 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.152 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.152 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.152 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.152 INFO:tasks.ceph:pgid 3.2e last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.152 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.152 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.153 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.153 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.153 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.153 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.153 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.153 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.154 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.154 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.154 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.154 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.154 INFO:tasks.ceph:pgid 3.21 last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.154 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.155 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.155 INFO:tasks.ceph:pgid 2.1b last_scrub_stamp 2024-08-31T12:15:55.259987+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=55, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.155 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.155 INFO:tasks.ceph:pgid 2.1a last_scrub_stamp 2024-08-31T12:15:55.259987+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=55, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.155 INFO:tasks.ceph:pgid 2.19 last_scrub_stamp 2024-08-31T12:15:55.259987+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=55, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.155 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.156 INFO:tasks.ceph:pgid 2.18 last_scrub_stamp 2024-08-31T12:15:55.259987+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=55, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.156 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.156 INFO:tasks.ceph:pgid 2.17 last_scrub_stamp 2024-08-31T12:15:55.259987+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=55, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.156 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.156 INFO:tasks.ceph:pgid 2.16 last_scrub_stamp 2024-08-31T12:15:55.259987+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=55, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.156 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.156 INFO:tasks.ceph:pgid 2.15 last_scrub_stamp 2024-08-31T12:15:55.259987+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=55, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.157 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.157 INFO:tasks.ceph:pgid 2.14 last_scrub_stamp 2024-08-31T12:15:55.259987+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=55, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.157 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.157 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.157 INFO:tasks.ceph:pgid 2.12 last_scrub_stamp 2024-08-31T12:15:55.259987+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=55, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.157 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.158 INFO:tasks.ceph:pgid 2.11 last_scrub_stamp 2024-08-31T12:15:55.259987+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=55, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.158 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.158 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2024-08-31T12:15:55.259987+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=55, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.158 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.158 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2024-08-31T12:15:55.259987+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=55, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.158 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.159 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2024-08-31T12:15:55.259987+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=55, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.159 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.159 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2024-08-31T12:15:55.259987+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=55, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.159 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.159 INFO:tasks.ceph:pgid 2.c last_scrub_stamp 2024-08-31T12:15:55.259987+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=55, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.159 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.160 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-08-31T12:15:45.213688+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=45, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.160 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-08-31T12:15:55.259987+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=55, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.160 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.160 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-08-31T12:15:55.259987+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=55, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.160 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.160 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.161 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-08-31T12:15:55.259987+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=55, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.161 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.161 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-08-31T12:15:55.259987+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=55, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.161 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.161 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-08-31T12:15:55.259987+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=55, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.161 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.161 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-08-31T12:15:55.259987+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=55, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.162 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.162 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-08-31T12:15:55.259987+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=55, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.162 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.162 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2024-08-31T12:15:55.259987+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=55, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.162 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.162 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2024-08-31T12:15:55.259987+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=55, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.163 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.163 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2024-08-31T12:15:55.259987+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=55, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.163 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.163 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2024-08-31T12:15:55.259987+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=55, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.163 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.163 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.163 INFO:tasks.ceph:pgid 2.1c last_scrub_stamp 2024-08-31T12:15:55.259987+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=55, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.164 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.164 INFO:tasks.ceph:pgid 2.1d last_scrub_stamp 2024-08-31T12:15:55.259987+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=55, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.164 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.164 INFO:tasks.ceph:pgid 2.1e last_scrub_stamp 2024-08-31T12:15:55.259987+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=55, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.164 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.164 INFO:tasks.ceph:pgid 2.1f last_scrub_stamp 2024-08-31T12:15:55.259987+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=55, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:28:55.165 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-08-31T12:28:58.395 DEBUG:teuthology.orchestra.run.smithi019:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:28:58.398 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:29:15.166 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-31T12:29:15.295 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.293+0000 7f80d2dc3640 1 -- 172.21.15.19:0/3240717218 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f80cc06eb00 msgr2=0x7f80cc0b54a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:15.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.293+0000 7f80d2dc3640 1 --2- 172.21.15.19:0/3240717218 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f80cc06eb00 0x7f80cc0b54a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f80c4009ca0 tx=0x7f80c402aa30 comp rx=0 tx=0).stop 2024-08-31T12:29:15.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.293+0000 7f80d2dc3640 1 -- 172.21.15.19:0/3240717218 shutdown_connections 2024-08-31T12:29:15.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.293+0000 7f80d2dc3640 1 --2- 172.21.15.19:0/3240717218 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f80cc06eb00 0x7f80cc0b54a0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:15.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.293+0000 7f80d2dc3640 1 --2- 172.21.15.19:0/3240717218 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f80cc0cc820 0x7f80cc0b4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:15.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.293+0000 7f80d2dc3640 1 -- 172.21.15.19:0/3240717218 >> 172.21.15.19:0/3240717218 conn(0x7f80cc069260 msgr2=0x7f80cc0d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:29:15.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.293+0000 7f80d2dc3640 1 -- 172.21.15.19:0/3240717218 shutdown_connections 2024-08-31T12:29:15.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.293+0000 7f80d2dc3640 1 -- 172.21.15.19:0/3240717218 wait complete. 2024-08-31T12:29:15.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.295+0000 7f80d2dc3640 1 Processor -- start 2024-08-31T12:29:15.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.295+0000 7f80d2dc3640 1 -- start start 2024-08-31T12:29:15.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.296+0000 7f80d2dc3640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f80cc06eb00 0x7f80cc0c4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:29:15.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.296+0000 7f80d2dc3640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f80cc0cc820 0x7f80cc0c6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:29:15.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.296+0000 7f80d2dc3640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80cc15b0a0 0x7f80cc0c6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:29:15.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.296+0000 7f80d2dc3640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f80cc0ccf90 con 0x7f80cc0cc820 2024-08-31T12:29:15.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.296+0000 7f80d2dc3640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f80cc0b4a40 con 0x7f80cc15b0a0 2024-08-31T12:29:15.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.296+0000 7f80d2dc3640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f80cc06dff0 con 0x7f80cc06eb00 2024-08-31T12:29:15.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.296+0000 7f80cbfff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f80cc0cc820 0x7f80cc0c6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:29:15.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.296+0000 7f80cbfff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f80cc0cc820 0x7f80cc0c6800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:55580/0 (socket says 172.21.15.19:55580) 2024-08-31T12:29:15.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.296+0000 7f80cbfff640 1 -- 172.21.15.19:0/1251162356 learned_addr learned my addr 172.21.15.19:0/1251162356 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:29:15.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.296+0000 7f80d1360640 1 --2- 172.21.15.19:0/1251162356 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80cc15b0a0 0x7f80cc0c6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:29:15.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.296+0000 7f80d0b5f640 1 --2- 172.21.15.19:0/1251162356 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f80cc06eb00 0x7f80cc0c4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:29:15.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.297+0000 7f80cbfff640 1 -- 172.21.15.19:0/1251162356 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f80cc06eb00 msgr2=0x7f80cc0c4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:15.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.297+0000 7f80cbfff640 1 --2- 172.21.15.19:0/1251162356 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f80cc06eb00 0x7f80cc0c4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:15.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.297+0000 7f80cbfff640 1 -- 172.21.15.19:0/1251162356 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80cc15b0a0 msgr2=0x7f80cc0c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:15.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.297+0000 7f80cbfff640 1 --2- 172.21.15.19:0/1251162356 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80cc15b0a0 0x7f80cc0c6d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:15.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.297+0000 7f80cbfff640 1 -- 172.21.15.19:0/1251162356 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80cc0c4c80 con 0x7f80cc0cc820 2024-08-31T12:29:15.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.297+0000 7f80cbfff640 1 --2- 172.21.15.19:0/1251162356 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f80cc0cc820 0x7f80cc0c6800 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f80c000b410 tx=0x7f80c0009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:29:15.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.297+0000 7f80c97fa640 1 -- 172.21.15.19:0/1251162356 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c0015030 con 0x7f80cc0cc820 2024-08-31T12:29:15.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.297+0000 7f80c97fa640 1 -- 172.21.15.19:0/1251162356 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f80c0004110 con 0x7f80cc0cc820 2024-08-31T12:29:15.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.297+0000 7f80d2dc3640 1 -- 172.21.15.19:0/1251162356 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80cc1e3040 con 0x7f80cc0cc820 2024-08-31T12:29:15.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.297+0000 7f80c97fa640 1 -- 172.21.15.19:0/1251162356 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c0011630 con 0x7f80cc0cc820 2024-08-31T12:29:15.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.298+0000 7f80d2dc3640 1 -- 172.21.15.19:0/1251162356 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80cc0cf9c0 con 0x7f80cc0cc820 2024-08-31T12:29:15.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.299+0000 7f80d2dc3640 1 -- 172.21.15.19:0/1251162356 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80cc0ca860 con 0x7f80cc0cc820 2024-08-31T12:29:15.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.299+0000 7f80c97fa640 1 -- 172.21.15.19:0/1251162356 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f80c0018070 con 0x7f80cc0cc820 2024-08-31T12:29:15.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.300+0000 7f80c97fa640 1 --2- 172.21.15.19:0/1251162356 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f80b8077a80 0x7f80b8079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:29:15.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.300+0000 7f80c97fa640 1 -- 172.21.15.19:0/1251162356 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f80c0099140 con 0x7f80cc0cc820 2024-08-31T12:29:15.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.300+0000 7f80d0b5f640 1 --2- 172.21.15.19:0/1251162356 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f80b8077a80 0x7f80b8079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:29:15.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.300+0000 7f80d0b5f640 1 --2- 172.21.15.19:0/1251162356 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f80b8077a80 0x7f80b8079f90 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f80bc005fd0 tx=0x7f80bc00a250 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:29:15.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.304+0000 7f80c97fa640 1 -- 172.21.15.19:0/1251162356 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f80c00616d0 con 0x7f80cc0cc820 2024-08-31T12:29:15.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.489+0000 7f80d2dc3640 1 -- 172.21.15.19:0/1251162356 --> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f80cc1f2470 con 0x7f80b8077a80 2024-08-31T12:29:15.496 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.494+0000 7f80c97fa640 1 -- 172.21.15.19:0/1251162356 <== mgr.4100 v2:172.21.15.43:6800/2824336655 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+255326 (secure 0 0 0) 0x7f80cc1f2470 con 0x7f80b8077a80 2024-08-31T12:29:15.499 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:29:15.500 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-08-31T12:29:15.502 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.498+0000 7f80d2dc3640 1 -- 172.21.15.19:0/1251162356 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f80b8077a80 msgr2=0x7f80b8079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:15.502 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.498+0000 7f80d2dc3640 1 --2- 172.21.15.19:0/1251162356 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f80b8077a80 0x7f80b8079f90 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f80bc005fd0 tx=0x7f80bc00a250 comp rx=0 tx=0).stop 2024-08-31T12:29:15.502 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.498+0000 7f80d2dc3640 1 -- 172.21.15.19:0/1251162356 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f80cc0cc820 msgr2=0x7f80cc0c6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:15.502 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.498+0000 7f80d2dc3640 1 --2- 172.21.15.19:0/1251162356 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f80cc0cc820 0x7f80cc0c6800 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f80c000b410 tx=0x7f80c0009f90 comp rx=0 tx=0).stop 2024-08-31T12:29:15.502 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.498+0000 7f80d2dc3640 1 -- 172.21.15.19:0/1251162356 shutdown_connections 2024-08-31T12:29:15.503 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.498+0000 7f80d2dc3640 1 --2- 172.21.15.19:0/1251162356 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f80b8077a80 0x7f80b8079f90 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:15.503 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.498+0000 7f80d2dc3640 1 --2- 172.21.15.19:0/1251162356 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80cc15b0a0 0x7f80cc0c6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:15.503 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.498+0000 7f80d2dc3640 1 --2- 172.21.15.19:0/1251162356 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f80cc0cc820 0x7f80cc0c6800 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:15.503 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.498+0000 7f80d2dc3640 1 --2- 172.21.15.19:0/1251162356 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f80cc06eb00 0x7f80cc0c4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:15.503 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.498+0000 7f80d2dc3640 1 -- 172.21.15.19:0/1251162356 >> 172.21.15.19:0/1251162356 conn(0x7f80cc069260 msgr2=0x7f80cc154a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:29:15.503 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.498+0000 7f80d2dc3640 1 -- 172.21.15.19:0/1251162356 shutdown_connections 2024-08-31T12:29:15.503 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:15.498+0000 7f80d2dc3640 1 -- 172.21.15.19:0/1251162356 wait complete. 2024-08-31T12:29:15.512 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":422,"stamp":"2024-08-31T12:29:14.713967+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":13438846028,"num_objects":3466,"num_object_clones":0,"num_object_copies":6932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3466,"num_whiteouts":0,"num_read":96,"num_read_kb":119,"num_write":921575,"num_write_kb":16804431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":66114,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":190316,"ondisk_log_size":190316,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":28671600,"kb_used_data":26250608,"kb_used_omap":12,"kb_used_meta":2420851,"kb_avail":726303120,"statfs":{"total":773094113280,"available":743734394880,"internally_reserved":0,"allocated":26880622592,"data_stored":26878355784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":2478951985},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1653,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003264"},"pg_stats":[{"pgid":"3.3f","version":"23'2732","reported_seq":2755,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.515247+0000","last_change":"2024-08-31T12:15:57.306560+0000","last_active":"2024-08-31T12:28:47.515247+0000","last_peered":"2024-08-31T12:28:47.515247+0000","last_clean":"2024-08-31T12:28:47.515247+0000","last_became_active":"2024-08-31T12:15:57.306148+0000","last_became_peered":"2024-08-31T12:15:57.306148+0000","last_unstale":"2024-08-31T12:28:47.515247+0000","last_undegraded":"2024-08-31T12:28:47.515247+0000","last_fullsized":"2024-08-31T12:28:47.515247+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2732,"log_dups_size":0,"ondisk_log_size":2732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:35:53.831742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180377297,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13788,"num_write_kb":231441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"23'3518","reported_seq":3551,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:04.479860+0000","last_change":"2024-08-31T12:29:04.479860+0000","last_active":"2024-08-31T12:29:04.479860+0000","last_peered":"2024-08-31T12:29:04.479860+0000","last_clean":"2024-08-31T12:29:04.479860+0000","last_became_active":"2024-08-31T12:15:57.306940+0000","last_became_peered":"2024-08-31T12:15:57.306940+0000","last_unstale":"2024-08-31T12:29:04.479860+0000","last_undegraded":"2024-08-31T12:29:04.479860+0000","last_fullsized":"2024-08-31T12:29:04.479860+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3518","last_scrub_stamp":"2024-08-31T12:29:04.479738+0000","last_deep_scrub":"23'3518","last_deep_scrub_stamp":"2024-08-31T12:29:04.479738+0000","last_clean_scrub_stamp":"2024-08-31T12:29:04.479738+0000","objects_scrubbed":66,"log_size":3518,"log_dups_size":0,"ondisk_log_size":3518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:20:30.975452+0000","scrub_duration":591,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264252554,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17611,"num_write_kb":328696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"23'3196","reported_seq":3229,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:01.068808+0000","last_change":"2024-08-31T12:29:01.068808+0000","last_active":"2024-08-31T12:29:01.068808+0000","last_peered":"2024-08-31T12:29:01.068808+0000","last_clean":"2024-08-31T12:29:01.068808+0000","last_became_active":"2024-08-31T12:15:57.305795+0000","last_became_peered":"2024-08-31T12:15:57.305795+0000","last_unstale":"2024-08-31T12:29:01.068808+0000","last_undegraded":"2024-08-31T12:29:01.068808+0000","last_fullsized":"2024-08-31T12:29:01.068808+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3196","last_scrub_stamp":"2024-08-31T12:29:01.068596+0000","last_deep_scrub":"23'3196","last_deep_scrub_stamp":"2024-08-31T12:29:01.068596+0000","last_clean_scrub_stamp":"2024-08-31T12:29:01.068596+0000","objects_scrubbed":55,"log_size":3196,"log_dups_size":0,"ondisk_log_size":3196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:43:43.076508+0000","scrub_duration":768,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213942725,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15777,"num_write_kb":272211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"23'2838","reported_seq":2870,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:07.638189+0000","last_change":"2024-08-31T12:29:07.638189+0000","last_active":"2024-08-31T12:29:07.638189+0000","last_peered":"2024-08-31T12:29:07.638189+0000","last_clean":"2024-08-31T12:29:07.638189+0000","last_became_active":"2024-08-31T12:15:57.306957+0000","last_became_peered":"2024-08-31T12:15:57.306957+0000","last_unstale":"2024-08-31T12:29:07.638189+0000","last_undegraded":"2024-08-31T12:29:07.638189+0000","last_fullsized":"2024-08-31T12:29:07.638189+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2838","last_scrub_stamp":"2024-08-31T12:29:07.638081+0000","last_deep_scrub":"23'2838","last_deep_scrub_stamp":"2024-08-31T12:29:07.638081+0000","last_clean_scrub_stamp":"2024-08-31T12:29:07.638081+0000","objects_scrubbed":45,"log_size":2838,"log_dups_size":0,"ondisk_log_size":2838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:02:16.520446+0000","scrub_duration":366,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14240,"num_write_kb":241389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"23'3305","reported_seq":3332,"reported_epoch":23,"state":"active+clean+scrubbing+deep","last_fresh":"2024-08-31T12:29:10.189509+0000","last_change":"2024-08-31T12:29:10.034534+0000","last_active":"2024-08-31T12:29:10.189509+0000","last_peered":"2024-08-31T12:29:10.189509+0000","last_clean":"2024-08-31T12:29:10.189509+0000","last_became_active":"2024-08-31T12:15:57.317079+0000","last_became_peered":"2024-08-31T12:15:57.317079+0000","last_unstale":"2024-08-31T12:29:10.189509+0000","last_undegraded":"2024-08-31T12:29:10.189509+0000","last_fullsized":"2024-08-31T12:29:10.189509+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":24,"log_size":3305,"log_dups_size":0,"ondisk_log_size":3305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247474530,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15722,"num_write_kb":304743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"23'2562","reported_seq":2594,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:05.561895+0000","last_change":"2024-08-31T12:29:05.561895+0000","last_active":"2024-08-31T12:29:05.561895+0000","last_peered":"2024-08-31T12:29:05.561895+0000","last_clean":"2024-08-31T12:29:05.561895+0000","last_became_active":"2024-08-31T12:15:57.307010+0000","last_became_peered":"2024-08-31T12:15:57.307010+0000","last_unstale":"2024-08-31T12:29:05.561895+0000","last_undegraded":"2024-08-31T12:29:05.561895+0000","last_fullsized":"2024-08-31T12:29:05.561895+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2562","last_scrub_stamp":"2024-08-31T12:29:05.561780+0000","last_deep_scrub":"23'2562","last_deep_scrub_stamp":"2024-08-31T12:29:05.561780+0000","last_clean_scrub_stamp":"2024-08-31T12:29:05.561780+0000","objects_scrubbed":44,"log_size":2562,"log_dups_size":0,"ondisk_log_size":2562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:43:35.106630+0000","scrub_duration":324,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180356168,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12564,"num_write_kb":226449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"23'2683","reported_seq":2715,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:07.339239+0000","last_change":"2024-08-31T12:29:07.339239+0000","last_active":"2024-08-31T12:29:07.339239+0000","last_peered":"2024-08-31T12:29:07.339239+0000","last_clean":"2024-08-31T12:29:07.339239+0000","last_became_active":"2024-08-31T12:15:57.316504+0000","last_became_peered":"2024-08-31T12:15:57.316504+0000","last_unstale":"2024-08-31T12:29:07.339239+0000","last_undegraded":"2024-08-31T12:29:07.339239+0000","last_fullsized":"2024-08-31T12:29:07.339239+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2683","last_scrub_stamp":"2024-08-31T12:29:07.339100+0000","last_deep_scrub":"23'2683","last_deep_scrub_stamp":"2024-08-31T12:29:07.339100+0000","last_clean_scrub_stamp":"2024-08-31T12:29:07.339100+0000","objects_scrubbed":52,"log_size":2683,"log_dups_size":0,"ondisk_log_size":2683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:16:32.863297+0000","scrub_duration":370,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201398355,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13306,"num_write_kb":250051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"23'3222","reported_seq":3248,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:03.296289+0000","last_change":"2024-08-31T12:29:03.296217+0000","last_active":"2024-08-31T12:29:03.296289+0000","last_peered":"2024-08-31T12:29:03.296289+0000","last_clean":"2024-08-31T12:29:03.296289+0000","last_became_active":"2024-08-31T12:16:40.694165+0000","last_became_peered":"2024-08-31T12:16:40.694165+0000","last_unstale":"2024-08-31T12:29:03.296289+0000","last_undegraded":"2024-08-31T12:29:03.296289+0000","last_fullsized":"2024-08-31T12:29:03.296289+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3222","last_scrub_stamp":"2024-08-31T12:29:03.296130+0000","last_deep_scrub":"23'3222","last_deep_scrub_stamp":"2024-08-31T12:29:03.296130+0000","last_clean_scrub_stamp":"2024-08-31T12:29:03.296130+0000","objects_scrubbed":55,"log_size":3222,"log_dups_size":0,"ondisk_log_size":3222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:18:47.300381+0000","scrub_duration":434,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226497620,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15619,"num_write_kb":283880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.37","version":"23'3563","reported_seq":3595,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:08.348653+0000","last_change":"2024-08-31T12:29:08.348653+0000","last_active":"2024-08-31T12:29:08.348653+0000","last_peered":"2024-08-31T12:29:08.348653+0000","last_clean":"2024-08-31T12:29:08.348653+0000","last_became_active":"2024-08-31T12:15:57.316354+0000","last_became_peered":"2024-08-31T12:15:57.316354+0000","last_unstale":"2024-08-31T12:29:08.348653+0000","last_undegraded":"2024-08-31T12:29:08.348653+0000","last_fullsized":"2024-08-31T12:29:08.348653+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3563","last_scrub_stamp":"2024-08-31T12:29:08.348558+0000","last_deep_scrub":"23'3563","last_deep_scrub_stamp":"2024-08-31T12:29:08.348558+0000","last_clean_scrub_stamp":"2024-08-31T12:29:08.348558+0000","objects_scrubbed":61,"log_size":3563,"log_dups_size":0,"ondisk_log_size":3563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:21:41.849206+0000","scrub_duration":397,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251662082,"num_objects":61,"num_object_clones":0,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18100,"num_write_kb":318223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"23'3029","reported_seq":3061,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:05.887322+0000","last_change":"2024-08-31T12:29:05.887322+0000","last_active":"2024-08-31T12:29:05.887322+0000","last_peered":"2024-08-31T12:29:05.887322+0000","last_clean":"2024-08-31T12:29:05.887322+0000","last_became_active":"2024-08-31T12:15:57.310807+0000","last_became_peered":"2024-08-31T12:15:57.310807+0000","last_unstale":"2024-08-31T12:29:05.887322+0000","last_undegraded":"2024-08-31T12:29:05.887322+0000","last_fullsized":"2024-08-31T12:29:05.887322+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3029","last_scrub_stamp":"2024-08-31T12:29:05.887218+0000","last_deep_scrub":"23'3029","last_deep_scrub_stamp":"2024-08-31T12:29:05.887218+0000","last_clean_scrub_stamp":"2024-08-31T12:29:05.887218+0000","objects_scrubbed":59,"log_size":3029,"log_dups_size":0,"ondisk_log_size":3029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:18:49.891469+0000","scrub_duration":355,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234894029,"num_objects":59,"num_object_clones":0,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15073,"num_write_kb":289757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"23'2934","reported_seq":2965,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:04.828649+0000","last_change":"2024-08-31T12:29:04.828649+0000","last_active":"2024-08-31T12:29:04.828649+0000","last_peered":"2024-08-31T12:29:04.828649+0000","last_clean":"2024-08-31T12:29:04.828649+0000","last_became_active":"2024-08-31T12:15:57.300610+0000","last_became_peered":"2024-08-31T12:15:57.300610+0000","last_unstale":"2024-08-31T12:29:04.828649+0000","last_undegraded":"2024-08-31T12:29:04.828649+0000","last_fullsized":"2024-08-31T12:29:04.828649+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2934","last_scrub_stamp":"2024-08-31T12:29:04.828544+0000","last_deep_scrub":"23'2934","last_deep_scrub_stamp":"2024-08-31T12:29:04.828544+0000","last_clean_scrub_stamp":"2024-08-31T12:29:04.828544+0000","objects_scrubbed":48,"log_size":2934,"log_dups_size":0,"ondisk_log_size":2934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:33:45.754806+0000","scrub_duration":312,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201326592,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14139,"num_write_kb":253227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"23'2298","reported_seq":2329,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:59.361975+0000","last_change":"2024-08-31T12:28:59.361975+0000","last_active":"2024-08-31T12:28:59.361975+0000","last_peered":"2024-08-31T12:28:59.361975+0000","last_clean":"2024-08-31T12:28:59.361975+0000","last_became_active":"2024-08-31T12:15:57.302622+0000","last_became_peered":"2024-08-31T12:15:57.302622+0000","last_unstale":"2024-08-31T12:28:59.361975+0000","last_undegraded":"2024-08-31T12:28:59.361975+0000","last_fullsized":"2024-08-31T12:28:59.361975+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2298","last_scrub_stamp":"2024-08-31T12:28:59.361873+0000","last_deep_scrub":"23'2298","last_deep_scrub_stamp":"2024-08-31T12:28:59.361873+0000","last_clean_scrub_stamp":"2024-08-31T12:28:59.361873+0000","objects_scrubbed":45,"log_size":2298,"log_dups_size":0,"ondisk_log_size":2298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:53:07.427338+0000","scrub_duration":305,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176192645,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11403,"num_write_kb":217791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"23'3009","reported_seq":3041,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:02.934638+0000","last_change":"2024-08-31T12:29:02.934638+0000","last_active":"2024-08-31T12:29:02.934638+0000","last_peered":"2024-08-31T12:29:02.934638+0000","last_clean":"2024-08-31T12:29:02.934638+0000","last_became_active":"2024-08-31T12:15:57.305321+0000","last_became_peered":"2024-08-31T12:15:57.305321+0000","last_unstale":"2024-08-31T12:29:02.934638+0000","last_undegraded":"2024-08-31T12:29:02.934638+0000","last_fullsized":"2024-08-31T12:29:02.934638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3009","last_scrub_stamp":"2024-08-31T12:29:02.934538+0000","last_deep_scrub":"23'3009","last_deep_scrub_stamp":"2024-08-31T12:29:02.934538+0000","last_clean_scrub_stamp":"2024-08-31T12:29:02.934538+0000","objects_scrubbed":51,"log_size":3009,"log_dups_size":0,"ondisk_log_size":3009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:07:16.630867+0000","scrub_duration":445,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188840677,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15214,"num_write_kb":245235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"23'3184","reported_seq":3216,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:02.755088+0000","last_change":"2024-08-31T12:29:02.755088+0000","last_active":"2024-08-31T12:29:02.755088+0000","last_peered":"2024-08-31T12:29:02.755088+0000","last_clean":"2024-08-31T12:29:02.755088+0000","last_became_active":"2024-08-31T12:15:57.307013+0000","last_became_peered":"2024-08-31T12:15:57.307013+0000","last_unstale":"2024-08-31T12:29:02.755088+0000","last_undegraded":"2024-08-31T12:29:02.755088+0000","last_fullsized":"2024-08-31T12:29:02.755088+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3184","last_scrub_stamp":"2024-08-31T12:29:02.754968+0000","last_deep_scrub":"23'3184","last_deep_scrub_stamp":"2024-08-31T12:29:02.754968+0000","last_clean_scrub_stamp":"2024-08-31T12:29:02.754968+0000","objects_scrubbed":55,"log_size":3184,"log_dups_size":0,"ondisk_log_size":3184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:37:48.968997+0000","scrub_duration":515,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213986201,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15743,"num_write_kb":271989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"23'3012","reported_seq":3044,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:02.368847+0000","last_change":"2024-08-31T12:29:02.368847+0000","last_active":"2024-08-31T12:29:02.368847+0000","last_peered":"2024-08-31T12:29:02.368847+0000","last_clean":"2024-08-31T12:29:02.368847+0000","last_became_active":"2024-08-31T12:15:57.297182+0000","last_became_peered":"2024-08-31T12:15:57.297182+0000","last_unstale":"2024-08-31T12:29:02.368847+0000","last_undegraded":"2024-08-31T12:29:02.368847+0000","last_fullsized":"2024-08-31T12:29:02.368847+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3012","last_scrub_stamp":"2024-08-31T12:29:02.368743+0000","last_deep_scrub":"23'3012","last_deep_scrub_stamp":"2024-08-31T12:29:02.368743+0000","last_clean_scrub_stamp":"2024-08-31T12:29:02.368743+0000","objects_scrubbed":53,"log_size":3012,"log_dups_size":0,"ondisk_log_size":3012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:44:40.519549+0000","scrub_duration":519,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213913551,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14445,"num_write_kb":266758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"23'2592","reported_seq":2617,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:00.857087+0000","last_change":"2024-08-31T12:29:00.857087+0000","last_active":"2024-08-31T12:29:00.857087+0000","last_peered":"2024-08-31T12:29:00.857087+0000","last_clean":"2024-08-31T12:29:00.857087+0000","last_became_active":"2024-08-31T12:16:41.074773+0000","last_became_peered":"2024-08-31T12:16:41.074773+0000","last_unstale":"2024-08-31T12:29:00.857087+0000","last_undegraded":"2024-08-31T12:29:00.857087+0000","last_fullsized":"2024-08-31T12:29:00.857087+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2592","last_scrub_stamp":"2024-08-31T12:29:00.856966+0000","last_deep_scrub":"23'2592","last_deep_scrub_stamp":"2024-08-31T12:29:00.856966+0000","last_clean_scrub_stamp":"2024-08-31T12:29:00.856966+0000","objects_scrubbed":46,"log_size":2592,"log_dups_size":0,"ondisk_log_size":2592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:51:56.203878+0000","scrub_duration":627,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184599153,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12917,"num_write_kb":231981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"23'3786","reported_seq":3818,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:06.666056+0000","last_change":"2024-08-31T12:29:06.666056+0000","last_active":"2024-08-31T12:29:06.666056+0000","last_peered":"2024-08-31T12:29:06.666056+0000","last_clean":"2024-08-31T12:29:06.666056+0000","last_became_active":"2024-08-31T12:15:57.308388+0000","last_became_peered":"2024-08-31T12:15:57.308388+0000","last_unstale":"2024-08-31T12:29:06.666056+0000","last_undegraded":"2024-08-31T12:29:06.666056+0000","last_fullsized":"2024-08-31T12:29:06.666056+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3786","last_scrub_stamp":"2024-08-31T12:29:06.665964+0000","last_deep_scrub":"23'3786","last_deep_scrub_stamp":"2024-08-31T12:29:06.665964+0000","last_clean_scrub_stamp":"2024-08-31T12:29:06.665964+0000","objects_scrubbed":63,"log_size":3786,"log_dups_size":0,"ondisk_log_size":3786,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:01:28.190845+0000","scrub_duration":403,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251747648,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18708,"num_write_kb":320814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"23'2196","reported_seq":2227,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:57.484759+0000","last_change":"2024-08-31T12:28:57.484759+0000","last_active":"2024-08-31T12:28:57.484759+0000","last_peered":"2024-08-31T12:28:57.484759+0000","last_clean":"2024-08-31T12:28:57.484759+0000","last_became_active":"2024-08-31T12:15:57.299826+0000","last_became_peered":"2024-08-31T12:15:57.299826+0000","last_unstale":"2024-08-31T12:28:57.484759+0000","last_undegraded":"2024-08-31T12:28:57.484759+0000","last_fullsized":"2024-08-31T12:28:57.484759+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2196","last_scrub_stamp":"2024-08-31T12:28:57.484645+0000","last_deep_scrub":"23'2196","last_deep_scrub_stamp":"2024-08-31T12:28:57.484645+0000","last_clean_scrub_stamp":"2024-08-31T12:28:57.484645+0000","objects_scrubbed":48,"log_size":2196,"log_dups_size":0,"ondisk_log_size":2196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:45:16.612056+0000","scrub_duration":318,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167791563,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10514,"num_write_kb":205964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"23'2727","reported_seq":2759,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:09.358861+0000","last_change":"2024-08-31T12:29:09.358861+0000","last_active":"2024-08-31T12:29:09.358861+0000","last_peered":"2024-08-31T12:29:09.358861+0000","last_clean":"2024-08-31T12:29:09.358861+0000","last_became_active":"2024-08-31T12:15:57.317185+0000","last_became_peered":"2024-08-31T12:15:57.317185+0000","last_unstale":"2024-08-31T12:29:09.358861+0000","last_undegraded":"2024-08-31T12:29:09.358861+0000","last_fullsized":"2024-08-31T12:29:09.358861+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2727","last_scrub_stamp":"2024-08-31T12:29:09.358776+0000","last_deep_scrub":"23'2727","last_deep_scrub_stamp":"2024-08-31T12:29:09.358776+0000","last_clean_scrub_stamp":"2024-08-31T12:29:09.358776+0000","objects_scrubbed":52,"log_size":2727,"log_dups_size":0,"ondisk_log_size":2727,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:51:42.749382+0000","scrub_duration":366,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209762270,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13163,"num_write_kb":257524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"23'2501","reported_seq":2533,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:03.880357+0000","last_change":"2024-08-31T12:29:03.880357+0000","last_active":"2024-08-31T12:29:03.880357+0000","last_peered":"2024-08-31T12:29:03.880357+0000","last_clean":"2024-08-31T12:29:03.880357+0000","last_became_active":"2024-08-31T12:15:57.298430+0000","last_became_peered":"2024-08-31T12:15:57.298430+0000","last_unstale":"2024-08-31T12:29:03.880357+0000","last_undegraded":"2024-08-31T12:29:03.880357+0000","last_fullsized":"2024-08-31T12:29:03.880357+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2501","last_scrub_stamp":"2024-08-31T12:29:03.880275+0000","last_deep_scrub":"23'2501","last_deep_scrub_stamp":"2024-08-31T12:29:03.880275+0000","last_clean_scrub_stamp":"2024-08-31T12:29:03.880275+0000","objects_scrubbed":52,"log_size":2501,"log_dups_size":0,"ondisk_log_size":2501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:33:58.771109+0000","scrub_duration":368,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205530518,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12230,"num_write_kb":249721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"23'3007","reported_seq":3039,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:01.813452+0000","last_change":"2024-08-31T12:29:01.813452+0000","last_active":"2024-08-31T12:29:01.813452+0000","last_peered":"2024-08-31T12:29:01.813452+0000","last_clean":"2024-08-31T12:29:01.813452+0000","last_became_active":"2024-08-31T12:15:57.316870+0000","last_became_peered":"2024-08-31T12:15:57.316870+0000","last_unstale":"2024-08-31T12:29:01.813452+0000","last_undegraded":"2024-08-31T12:29:01.813452+0000","last_fullsized":"2024-08-31T12:29:01.813452+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3007","last_scrub_stamp":"2024-08-31T12:29:01.813349+0000","last_deep_scrub":"23'3007","last_deep_scrub_stamp":"2024-08-31T12:29:01.813349+0000","last_clean_scrub_stamp":"2024-08-31T12:29:01.813349+0000","objects_scrubbed":57,"log_size":3007,"log_dups_size":0,"ondisk_log_size":3007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:25:18.194613+0000","scrub_duration":546,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213923005,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14700,"num_write_kb":267872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"23'3121","reported_seq":3147,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:54.784122+0000","last_change":"2024-08-31T12:28:54.784122+0000","last_active":"2024-08-31T12:28:54.784122+0000","last_peered":"2024-08-31T12:28:54.784122+0000","last_clean":"2024-08-31T12:28:54.784122+0000","last_became_active":"2024-08-31T12:16:40.784204+0000","last_became_peered":"2024-08-31T12:16:40.784204+0000","last_unstale":"2024-08-31T12:28:54.784122+0000","last_undegraded":"2024-08-31T12:28:54.784122+0000","last_fullsized":"2024-08-31T12:28:54.784122+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3121","last_scrub_stamp":"2024-08-31T12:28:54.783999+0000","last_deep_scrub":"23'3121","last_deep_scrub_stamp":"2024-08-31T12:28:54.783999+0000","last_clean_scrub_stamp":"2024-08-31T12:28:54.783999+0000","objects_scrubbed":55,"log_size":3121,"log_dups_size":0,"ondisk_log_size":3121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:46:37.064206+0000","scrub_duration":509,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226493660,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15308,"num_write_kb":282531,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"23'2885","reported_seq":2917,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:03.884681+0000","last_change":"2024-08-31T12:29:03.884681+0000","last_active":"2024-08-31T12:29:03.884681+0000","last_peered":"2024-08-31T12:29:03.884681+0000","last_clean":"2024-08-31T12:29:03.884681+0000","last_became_active":"2024-08-31T12:15:57.303401+0000","last_became_peered":"2024-08-31T12:15:57.303401+0000","last_unstale":"2024-08-31T12:29:03.884681+0000","last_undegraded":"2024-08-31T12:29:03.884681+0000","last_fullsized":"2024-08-31T12:29:03.884681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2885","last_scrub_stamp":"2024-08-31T12:29:03.884604+0000","last_deep_scrub":"23'2885","last_deep_scrub_stamp":"2024-08-31T12:29:03.884604+0000","last_clean_scrub_stamp":"2024-08-31T12:29:03.884604+0000","objects_scrubbed":50,"log_size":2885,"log_dups_size":0,"ondisk_log_size":2885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:30:33.314909+0000","scrub_duration":417,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205588184,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14070,"num_write_kb":257066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"23'2521","reported_seq":2553,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:04.637461+0000","last_change":"2024-08-31T12:29:04.637461+0000","last_active":"2024-08-31T12:29:04.637461+0000","last_peered":"2024-08-31T12:29:04.637461+0000","last_clean":"2024-08-31T12:29:04.637461+0000","last_became_active":"2024-08-31T12:15:57.305075+0000","last_became_peered":"2024-08-31T12:15:57.305075+0000","last_unstale":"2024-08-31T12:29:04.637461+0000","last_undegraded":"2024-08-31T12:29:04.637461+0000","last_fullsized":"2024-08-31T12:29:04.637461+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2521","last_scrub_stamp":"2024-08-31T12:29:04.637273+0000","last_deep_scrub":"23'2521","last_deep_scrub_stamp":"2024-08-31T12:29:04.637273+0000","last_clean_scrub_stamp":"2024-08-31T12:29:04.637273+0000","objects_scrubbed":47,"log_size":2521,"log_dups_size":0,"ondisk_log_size":2521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:47:40.577786+0000","scrub_duration":436,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184555785,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12290,"num_write_kb":229419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"23'3229","reported_seq":3262,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:02.129504+0000","last_change":"2024-08-31T12:29:02.129504+0000","last_active":"2024-08-31T12:29:02.129504+0000","last_peered":"2024-08-31T12:29:02.129504+0000","last_clean":"2024-08-31T12:29:02.129504+0000","last_became_active":"2024-08-31T12:15:57.299861+0000","last_became_peered":"2024-08-31T12:15:57.299861+0000","last_unstale":"2024-08-31T12:29:02.129504+0000","last_undegraded":"2024-08-31T12:29:02.129504+0000","last_fullsized":"2024-08-31T12:29:02.129504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3229","last_scrub_stamp":"2024-08-31T12:29:02.129410+0000","last_deep_scrub":"23'3229","last_deep_scrub_stamp":"2024-08-31T12:29:02.129410+0000","last_clean_scrub_stamp":"2024-08-31T12:29:02.129410+0000","objects_scrubbed":63,"log_size":3229,"log_dups_size":0,"ondisk_log_size":3229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:02:39.277974+0000","scrub_duration":629,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243322604,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15709,"num_write_kb":300530,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"23'2792","reported_seq":2824,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:02.900464+0000","last_change":"2024-08-31T12:29:02.900464+0000","last_active":"2024-08-31T12:29:02.900464+0000","last_peered":"2024-08-31T12:29:02.900464+0000","last_clean":"2024-08-31T12:29:02.900464+0000","last_became_active":"2024-08-31T12:15:57.308189+0000","last_became_peered":"2024-08-31T12:15:57.308189+0000","last_unstale":"2024-08-31T12:29:02.900464+0000","last_undegraded":"2024-08-31T12:29:02.900464+0000","last_fullsized":"2024-08-31T12:29:02.900464+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2792","last_scrub_stamp":"2024-08-31T12:29:02.900351+0000","last_deep_scrub":"23'2792","last_deep_scrub_stamp":"2024-08-31T12:29:02.900351+0000","last_clean_scrub_stamp":"2024-08-31T12:29:02.900351+0000","objects_scrubbed":48,"log_size":2792,"log_dups_size":0,"ondisk_log_size":2792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:32:40.539776+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189169648,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13404,"num_write_kb":238420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"23'3142","reported_seq":3175,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:01.447679+0000","last_change":"2024-08-31T12:29:01.447679+0000","last_active":"2024-08-31T12:29:01.447679+0000","last_peered":"2024-08-31T12:29:01.447679+0000","last_clean":"2024-08-31T12:29:01.447679+0000","last_became_active":"2024-08-31T12:15:57.304514+0000","last_became_peered":"2024-08-31T12:15:57.304514+0000","last_unstale":"2024-08-31T12:29:01.447679+0000","last_undegraded":"2024-08-31T12:29:01.447679+0000","last_fullsized":"2024-08-31T12:29:01.447679+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3142","last_scrub_stamp":"2024-08-31T12:29:01.447580+0000","last_deep_scrub":"23'3142","last_deep_scrub_stamp":"2024-08-31T12:29:01.447580+0000","last_clean_scrub_stamp":"2024-08-31T12:29:01.447580+0000","objects_scrubbed":54,"log_size":3142,"log_dups_size":0,"ondisk_log_size":3142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:37:47.661609+0000","scrub_duration":645,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222298838,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15326,"num_write_kb":278485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"23'2621","reported_seq":2653,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:01.176336+0000","last_change":"2024-08-31T12:29:01.176264+0000","last_active":"2024-08-31T12:29:01.176336+0000","last_peered":"2024-08-31T12:29:01.176336+0000","last_clean":"2024-08-31T12:29:01.176336+0000","last_became_active":"2024-08-31T12:15:57.298766+0000","last_became_peered":"2024-08-31T12:15:57.298766+0000","last_unstale":"2024-08-31T12:29:01.176336+0000","last_undegraded":"2024-08-31T12:29:01.176336+0000","last_fullsized":"2024-08-31T12:29:01.176336+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2621","last_scrub_stamp":"2024-08-31T12:29:01.176105+0000","last_deep_scrub":"23'2621","last_deep_scrub_stamp":"2024-08-31T12:29:01.176105+0000","last_clean_scrub_stamp":"2024-08-31T12:29:01.176105+0000","objects_scrubbed":47,"log_size":2621,"log_dups_size":0,"ondisk_log_size":2621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:07:36.116413+0000","scrub_duration":653,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188773348,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12878,"num_write_kb":235916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"23'2373","reported_seq":2405,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:59.672533+0000","last_change":"2024-08-31T12:28:59.672533+0000","last_active":"2024-08-31T12:28:59.672533+0000","last_peered":"2024-08-31T12:28:59.672533+0000","last_clean":"2024-08-31T12:28:59.672533+0000","last_became_active":"2024-08-31T12:15:57.309440+0000","last_became_peered":"2024-08-31T12:15:57.309440+0000","last_unstale":"2024-08-31T12:28:59.672533+0000","last_undegraded":"2024-08-31T12:28:59.672533+0000","last_fullsized":"2024-08-31T12:28:59.672533+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2373","last_scrub_stamp":"2024-08-31T12:28:59.672418+0000","last_deep_scrub":"23'2373","last_deep_scrub_stamp":"2024-08-31T12:28:59.672418+0000","last_clean_scrub_stamp":"2024-08-31T12:28:59.672418+0000","objects_scrubbed":45,"log_size":2373,"log_dups_size":0,"ondisk_log_size":2373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:36:49.635342+0000","scrub_duration":387,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180357122,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11238,"num_write_kb":221098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"23'3509","reported_seq":3542,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:03.766589+0000","last_change":"2024-08-31T12:29:03.766589+0000","last_active":"2024-08-31T12:29:03.766589+0000","last_peered":"2024-08-31T12:29:03.766589+0000","last_clean":"2024-08-31T12:29:03.766589+0000","last_became_active":"2024-08-31T12:15:57.305878+0000","last_became_peered":"2024-08-31T12:15:57.305878+0000","last_unstale":"2024-08-31T12:29:03.766589+0000","last_undegraded":"2024-08-31T12:29:03.766589+0000","last_fullsized":"2024-08-31T12:29:03.766589+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3509","last_scrub_stamp":"2024-08-31T12:29:03.766459+0000","last_deep_scrub":"23'3509","last_deep_scrub_stamp":"2024-08-31T12:29:03.766459+0000","last_clean_scrub_stamp":"2024-08-31T12:29:03.766459+0000","objects_scrubbed":62,"log_size":3509,"log_dups_size":0,"ondisk_log_size":3509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:03:30.959671+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251668879,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17281,"num_write_kb":314954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"23'2933","reported_seq":2966,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:56.696020+0000","last_change":"2024-08-31T12:28:56.696020+0000","last_active":"2024-08-31T12:28:56.696020+0000","last_peered":"2024-08-31T12:28:56.696020+0000","last_clean":"2024-08-31T12:28:56.696020+0000","last_became_active":"2024-08-31T12:15:57.307056+0000","last_became_peered":"2024-08-31T12:15:57.307056+0000","last_unstale":"2024-08-31T12:28:56.696020+0000","last_undegraded":"2024-08-31T12:28:56.696020+0000","last_fullsized":"2024-08-31T12:28:56.696020+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2933","last_scrub_stamp":"2024-08-31T12:28:56.695930+0000","last_deep_scrub":"23'2933","last_deep_scrub_stamp":"2024-08-31T12:28:56.695930+0000","last_clean_scrub_stamp":"2024-08-31T12:28:56.695930+0000","objects_scrubbed":53,"log_size":2933,"log_dups_size":0,"ondisk_log_size":2933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:39:28.074678+0000","scrub_duration":507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222298112,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14386,"num_write_kb":274682,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"23'3186","reported_seq":3219,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:02.187805+0000","last_change":"2024-08-31T12:29:02.187805+0000","last_active":"2024-08-31T12:29:02.187805+0000","last_peered":"2024-08-31T12:29:02.187805+0000","last_clean":"2024-08-31T12:29:02.187805+0000","last_became_active":"2024-08-31T12:15:57.313463+0000","last_became_peered":"2024-08-31T12:15:57.313463+0000","last_unstale":"2024-08-31T12:29:02.187805+0000","last_undegraded":"2024-08-31T12:29:02.187805+0000","last_fullsized":"2024-08-31T12:29:02.187805+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3186","last_scrub_stamp":"2024-08-31T12:29:02.187686+0000","last_deep_scrub":"23'3186","last_deep_scrub_stamp":"2024-08-31T12:29:02.187686+0000","last_clean_scrub_stamp":"2024-08-31T12:29:02.187686+0000","objects_scrubbed":64,"log_size":3186,"log_dups_size":0,"ondisk_log_size":3186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:43:44.195597+0000","scrub_duration":668,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260049613,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15263,"num_write_kb":315030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"23'2333","reported_seq":2364,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:58.487797+0000","last_change":"2024-08-31T12:28:58.487797+0000","last_active":"2024-08-31T12:28:58.487797+0000","last_peered":"2024-08-31T12:28:58.487797+0000","last_clean":"2024-08-31T12:28:58.487797+0000","last_became_active":"2024-08-31T12:15:57.292981+0000","last_became_peered":"2024-08-31T12:15:57.292981+0000","last_unstale":"2024-08-31T12:28:58.487797+0000","last_undegraded":"2024-08-31T12:28:58.487797+0000","last_fullsized":"2024-08-31T12:28:58.487797+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2333","last_scrub_stamp":"2024-08-31T12:28:58.487698+0000","last_deep_scrub":"23'2333","last_deep_scrub_stamp":"2024-08-31T12:28:58.487698+0000","last_clean_scrub_stamp":"2024-08-31T12:28:58.487698+0000","objects_scrubbed":46,"log_size":2333,"log_dups_size":0,"ondisk_log_size":2333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:18:39.664026+0000","scrub_duration":388,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180377946,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11142,"num_write_kb":220690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"23'572","reported_seq":604,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:54.576817+0000","last_change":"2024-08-31T12:28:54.576817+0000","last_active":"2024-08-31T12:28:54.576817+0000","last_peered":"2024-08-31T12:28:54.576817+0000","last_clean":"2024-08-31T12:28:54.576817+0000","last_became_active":"2024-08-31T12:15:56.293164+0000","last_became_peered":"2024-08-31T12:15:56.293164+0000","last_unstale":"2024-08-31T12:28:54.576817+0000","last_undegraded":"2024-08-31T12:28:54.576817+0000","last_fullsized":"2024-08-31T12:28:54.576817+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'572","last_scrub_stamp":"2024-08-31T12:28:54.576677+0000","last_deep_scrub":"23'572","last_deep_scrub_stamp":"2024-08-31T12:28:54.576677+0000","last_clean_scrub_stamp":"2024-08-31T12:28:54.576677+0000","objects_scrubbed":3,"log_size":572,"log_dups_size":0,"ondisk_log_size":572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:12:36.474157+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4195508,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":567,"num_write_kb":4211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"23'2437","reported_seq":2468,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:58.755425+0000","last_change":"2024-08-31T12:28:58.755425+0000","last_active":"2024-08-31T12:28:58.755425+0000","last_peered":"2024-08-31T12:28:58.755425+0000","last_clean":"2024-08-31T12:28:58.755425+0000","last_became_active":"2024-08-31T12:15:57.312360+0000","last_became_peered":"2024-08-31T12:15:57.312360+0000","last_unstale":"2024-08-31T12:28:58.755425+0000","last_undegraded":"2024-08-31T12:28:58.755425+0000","last_fullsized":"2024-08-31T12:28:58.755425+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2437","last_scrub_stamp":"2024-08-31T12:28:58.755318+0000","last_deep_scrub":"23'2437","last_deep_scrub_stamp":"2024-08-31T12:28:58.755318+0000","last_clean_scrub_stamp":"2024-08-31T12:28:58.755318+0000","objects_scrubbed":41,"log_size":2437,"log_dups_size":0,"ondisk_log_size":2437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:30:07.510938+0000","scrub_duration":425,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163584295,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11807,"num_write_kb":207035,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"23'7","reported_seq":39,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:58.542622+0000","last_change":"2024-08-31T12:28:58.542622+0000","last_active":"2024-08-31T12:28:58.542622+0000","last_peered":"2024-08-31T12:28:58.542622+0000","last_clean":"2024-08-31T12:28:58.542622+0000","last_became_active":"2024-08-31T12:15:56.283147+0000","last_became_peered":"2024-08-31T12:15:56.283147+0000","last_unstale":"2024-08-31T12:28:58.542622+0000","last_undegraded":"2024-08-31T12:28:58.542622+0000","last_fullsized":"2024-08-31T12:28:58.542622+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'7","last_scrub_stamp":"2024-08-31T12:28:58.542515+0000","last_deep_scrub":"23'7","last_deep_scrub_stamp":"2024-08-31T12:28:58.542515+0000","last_clean_scrub_stamp":"2024-08-31T12:28:58.542515+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:22:30.474634+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5538,"num_omap_keys":11,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"23'5","reported_seq":37,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:53.595705+0000","last_change":"2024-08-31T12:28:53.595705+0000","last_active":"2024-08-31T12:28:53.595705+0000","last_peered":"2024-08-31T12:28:53.595705+0000","last_clean":"2024-08-31T12:28:53.595705+0000","last_became_active":"2024-08-31T12:15:56.290835+0000","last_became_peered":"2024-08-31T12:15:56.290835+0000","last_unstale":"2024-08-31T12:28:53.595705+0000","last_undegraded":"2024-08-31T12:28:53.595705+0000","last_fullsized":"2024-08-31T12:28:53.595705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5","last_scrub_stamp":"2024-08-31T12:28:53.595609+0000","last_deep_scrub":"23'5","last_deep_scrub_stamp":"2024-08-31T12:28:53.595609+0000","last_clean_scrub_stamp":"2024-08-31T12:28:53.595609+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:27:35.816726+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"23'2595","reported_seq":2627,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:05.944823+0000","last_change":"2024-08-31T12:29:05.944823+0000","last_active":"2024-08-31T12:29:05.944823+0000","last_peered":"2024-08-31T12:29:05.944823+0000","last_clean":"2024-08-31T12:29:05.944823+0000","last_became_active":"2024-08-31T12:15:57.306962+0000","last_became_peered":"2024-08-31T12:15:57.306962+0000","last_unstale":"2024-08-31T12:29:05.944823+0000","last_undegraded":"2024-08-31T12:29:05.944823+0000","last_fullsized":"2024-08-31T12:29:05.944823+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2595","last_scrub_stamp":"2024-08-31T12:29:05.944705+0000","last_deep_scrub":"23'2595","last_deep_scrub_stamp":"2024-08-31T12:29:05.944705+0000","last_clean_scrub_stamp":"2024-08-31T12:29:05.944705+0000","objects_scrubbed":52,"log_size":2595,"log_dups_size":0,"ondisk_log_size":2595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:35:15.992000+0000","scrub_duration":440,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205541461,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":251825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"23'3","reported_seq":35,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:58.192505+0000","last_change":"2024-08-31T12:28:58.192505+0000","last_active":"2024-08-31T12:28:58.192505+0000","last_peered":"2024-08-31T12:28:58.192505+0000","last_clean":"2024-08-31T12:28:58.192505+0000","last_became_active":"2024-08-31T12:15:56.287552+0000","last_became_peered":"2024-08-31T12:15:56.287552+0000","last_unstale":"2024-08-31T12:28:58.192505+0000","last_undegraded":"2024-08-31T12:28:58.192505+0000","last_fullsized":"2024-08-31T12:28:58.192505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3","last_scrub_stamp":"2024-08-31T12:28:58.192377+0000","last_deep_scrub":"23'3","last_deep_scrub_stamp":"2024-08-31T12:28:58.192377+0000","last_clean_scrub_stamp":"2024-08-31T12:28:58.192377+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:02:00.301104+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8000,"num_omap_keys":16,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"23'3514","reported_seq":3546,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:00.496904+0000","last_change":"2024-08-31T12:29:00.496904+0000","last_active":"2024-08-31T12:29:00.496904+0000","last_peered":"2024-08-31T12:29:00.496904+0000","last_clean":"2024-08-31T12:29:00.496904+0000","last_became_active":"2024-08-31T12:15:57.304700+0000","last_became_peered":"2024-08-31T12:15:57.304700+0000","last_unstale":"2024-08-31T12:29:00.496904+0000","last_undegraded":"2024-08-31T12:29:00.496904+0000","last_fullsized":"2024-08-31T12:29:00.496904+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3514","last_scrub_stamp":"2024-08-31T12:29:00.496725+0000","last_deep_scrub":"23'3514","last_deep_scrub_stamp":"2024-08-31T12:29:00.496725+0000","last_clean_scrub_stamp":"2024-08-31T12:29:00.496725+0000","objects_scrubbed":64,"log_size":3514,"log_dups_size":0,"ondisk_log_size":3514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:32:38.136150+0000","scrub_duration":733,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243316666,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17415,"num_write_kb":307366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"23'6","reported_seq":38,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:54.515829+0000","last_change":"2024-08-31T12:28:54.515829+0000","last_active":"2024-08-31T12:28:54.515829+0000","last_peered":"2024-08-31T12:28:54.515829+0000","last_clean":"2024-08-31T12:28:54.515829+0000","last_became_active":"2024-08-31T12:15:56.284558+0000","last_became_peered":"2024-08-31T12:15:56.284558+0000","last_unstale":"2024-08-31T12:28:54.515829+0000","last_undegraded":"2024-08-31T12:28:54.515829+0000","last_fullsized":"2024-08-31T12:28:54.515829+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'6","last_scrub_stamp":"2024-08-31T12:28:54.515723+0000","last_deep_scrub":"23'6","last_deep_scrub_stamp":"2024-08-31T12:28:54.515723+0000","last_clean_scrub_stamp":"2024-08-31T12:28:54.515723+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:38:57.450315+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"23'2962","reported_seq":2994,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:59.580677+0000","last_change":"2024-08-31T12:28:59.580677+0000","last_active":"2024-08-31T12:28:59.580677+0000","last_peered":"2024-08-31T12:28:59.580677+0000","last_clean":"2024-08-31T12:28:59.580677+0000","last_became_active":"2024-08-31T12:15:57.308260+0000","last_became_peered":"2024-08-31T12:15:57.308260+0000","last_unstale":"2024-08-31T12:28:59.580677+0000","last_undegraded":"2024-08-31T12:28:59.580677+0000","last_fullsized":"2024-08-31T12:28:59.580677+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2962","last_scrub_stamp":"2024-08-31T12:28:59.580567+0000","last_deep_scrub":"23'2962","last_deep_scrub_stamp":"2024-08-31T12:28:59.580567+0000","last_clean_scrub_stamp":"2024-08-31T12:28:59.580567+0000","objects_scrubbed":58,"log_size":2962,"log_dups_size":0,"ondisk_log_size":2962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:21:54.817438+0000","scrub_duration":376,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209955924,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14127,"num_write_kb":261546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"23'3","reported_seq":35,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:57.154575+0000","last_change":"2024-08-31T12:28:57.154575+0000","last_active":"2024-08-31T12:28:57.154575+0000","last_peered":"2024-08-31T12:28:57.154575+0000","last_clean":"2024-08-31T12:28:57.154575+0000","last_became_active":"2024-08-31T12:15:56.290400+0000","last_became_peered":"2024-08-31T12:15:56.290400+0000","last_unstale":"2024-08-31T12:28:57.154575+0000","last_undegraded":"2024-08-31T12:28:57.154575+0000","last_fullsized":"2024-08-31T12:28:57.154575+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3","last_scrub_stamp":"2024-08-31T12:28:57.154456+0000","last_deep_scrub":"23'3","last_deep_scrub_stamp":"2024-08-31T12:28:57.154456+0000","last_clean_scrub_stamp":"2024-08-31T12:28:57.154456+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:51:57.123031+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"23'2713","reported_seq":2744,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:56.748191+0000","last_change":"2024-08-31T12:28:56.748191+0000","last_active":"2024-08-31T12:28:56.748191+0000","last_peered":"2024-08-31T12:28:56.748191+0000","last_clean":"2024-08-31T12:28:56.748191+0000","last_became_active":"2024-08-31T12:15:57.310564+0000","last_became_peered":"2024-08-31T12:15:57.310564+0000","last_unstale":"2024-08-31T12:28:56.748191+0000","last_undegraded":"2024-08-31T12:28:56.748191+0000","last_fullsized":"2024-08-31T12:28:56.748191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2713","last_scrub_stamp":"2024-08-31T12:28:56.748076+0000","last_deep_scrub":"23'2713","last_deep_scrub_stamp":"2024-08-31T12:28:56.748076+0000","last_clean_scrub_stamp":"2024-08-31T12:28:56.748076+0000","objects_scrubbed":48,"log_size":2713,"log_dups_size":0,"ondisk_log_size":2713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:22:57.469228+0000","scrub_duration":438,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180410852,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13728,"num_write_kb":231123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"23'22","reported_seq":54,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:57.738642+0000","last_change":"2024-08-31T12:28:57.738642+0000","last_active":"2024-08-31T12:28:57.738642+0000","last_peered":"2024-08-31T12:28:57.738642+0000","last_clean":"2024-08-31T12:28:57.738642+0000","last_became_active":"2024-08-31T12:15:56.286458+0000","last_became_peered":"2024-08-31T12:15:56.286458+0000","last_unstale":"2024-08-31T12:28:57.738642+0000","last_undegraded":"2024-08-31T12:28:57.738642+0000","last_fullsized":"2024-08-31T12:28:57.738642+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'22","last_scrub_stamp":"2024-08-31T12:28:57.738518+0000","last_deep_scrub":"23'22","last_deep_scrub_stamp":"2024-08-31T12:28:57.738518+0000","last_clean_scrub_stamp":"2024-08-31T12:28:57.738518+0000","objects_scrubbed":3,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:57:18.106754+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"23'3738","reported_seq":3770,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:57.569166+0000","last_change":"2024-08-31T12:28:57.569166+0000","last_active":"2024-08-31T12:28:57.569166+0000","last_peered":"2024-08-31T12:28:57.569166+0000","last_clean":"2024-08-31T12:28:57.569166+0000","last_became_active":"2024-08-31T12:15:57.309102+0000","last_became_peered":"2024-08-31T12:15:57.309102+0000","last_unstale":"2024-08-31T12:28:57.569166+0000","last_undegraded":"2024-08-31T12:28:57.569166+0000","last_fullsized":"2024-08-31T12:28:57.569166+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3738","last_scrub_stamp":"2024-08-31T12:28:57.569054+0000","last_deep_scrub":"23'3738","last_deep_scrub_stamp":"2024-08-31T12:28:57.569054+0000","last_clean_scrub_stamp":"2024-08-31T12:28:57.569054+0000","objects_scrubbed":69,"log_size":3738,"log_dups_size":0,"ondisk_log_size":3738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:08:32.002870+0000","scrub_duration":507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":277024326,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17723,"num_write_kb":341816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"23'322","reported_seq":354,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:55.537277+0000","last_change":"2024-08-31T12:28:55.537204+0000","last_active":"2024-08-31T12:28:55.537277+0000","last_peered":"2024-08-31T12:28:55.537277+0000","last_clean":"2024-08-31T12:28:55.537277+0000","last_became_active":"2024-08-31T12:15:56.289096+0000","last_became_peered":"2024-08-31T12:15:56.289096+0000","last_unstale":"2024-08-31T12:28:55.537277+0000","last_undegraded":"2024-08-31T12:28:55.537277+0000","last_fullsized":"2024-08-31T12:28:55.537277+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'321","last_scrub_stamp":"2024-08-31T12:28:55.537113+0000","last_deep_scrub":"23'321","last_deep_scrub_stamp":"2024-08-31T12:28:55.537113+0000","last_clean_scrub_stamp":"2024-08-31T12:28:55.537113+0000","objects_scrubbed":5,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:59:21.280703+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":318,"num_write_kb":4237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"23'2794","reported_seq":2825,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:06.314905+0000","last_change":"2024-08-31T12:29:06.314905+0000","last_active":"2024-08-31T12:29:06.314905+0000","last_peered":"2024-08-31T12:29:06.314905+0000","last_clean":"2024-08-31T12:29:06.314905+0000","last_became_active":"2024-08-31T12:15:57.317632+0000","last_became_peered":"2024-08-31T12:15:57.317632+0000","last_unstale":"2024-08-31T12:29:06.314905+0000","last_undegraded":"2024-08-31T12:29:06.314905+0000","last_fullsized":"2024-08-31T12:29:06.314905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2794","last_scrub_stamp":"2024-08-31T12:29:06.314759+0000","last_deep_scrub":"23'2794","last_deep_scrub_stamp":"2024-08-31T12:29:06.314759+0000","last_clean_scrub_stamp":"2024-08-31T12:29:06.314759+0000","objects_scrubbed":47,"log_size":2794,"log_dups_size":0,"ondisk_log_size":2794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:02:15.197124+0000","scrub_duration":343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197132288,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13667,"num_write_kb":247166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"23'10","reported_seq":42,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:51.225974+0000","last_change":"2024-08-31T12:28:51.225974+0000","last_active":"2024-08-31T12:28:51.225974+0000","last_peered":"2024-08-31T12:28:51.225974+0000","last_clean":"2024-08-31T12:28:51.225974+0000","last_became_active":"2024-08-31T12:15:56.283675+0000","last_became_peered":"2024-08-31T12:15:56.283675+0000","last_unstale":"2024-08-31T12:28:51.225974+0000","last_undegraded":"2024-08-31T12:28:51.225974+0000","last_fullsized":"2024-08-31T12:28:51.225974+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'10","last_scrub_stamp":"2024-08-31T12:28:51.225832+0000","last_deep_scrub":"23'10","last_deep_scrub_stamp":"2024-08-31T12:28:51.225832+0000","last_clean_scrub_stamp":"2024-08-31T12:28:51.225832+0000","objects_scrubbed":4,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:46:50.223589+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"23'3109","reported_seq":3140,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:54.702050+0000","last_change":"2024-08-31T12:28:54.702050+0000","last_active":"2024-08-31T12:28:54.702050+0000","last_peered":"2024-08-31T12:28:54.702050+0000","last_clean":"2024-08-31T12:28:54.702050+0000","last_became_active":"2024-08-31T12:15:57.312007+0000","last_became_peered":"2024-08-31T12:15:57.312007+0000","last_unstale":"2024-08-31T12:28:54.702050+0000","last_undegraded":"2024-08-31T12:28:54.702050+0000","last_fullsized":"2024-08-31T12:28:54.702050+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3109","last_scrub_stamp":"2024-08-31T12:28:54.701881+0000","last_deep_scrub":"23'3109","last_deep_scrub_stamp":"2024-08-31T12:28:54.701881+0000","last_clean_scrub_stamp":"2024-08-31T12:28:54.701881+0000","objects_scrubbed":48,"log_size":3109,"log_dups_size":0,"ondisk_log_size":3109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:59:20.445471+0000","scrub_duration":442,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197138377,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15100,"num_write_kb":253041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"23'6","reported_seq":38,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:56.137793+0000","last_change":"2024-08-31T12:28:56.137793+0000","last_active":"2024-08-31T12:28:56.137793+0000","last_peered":"2024-08-31T12:28:56.137793+0000","last_clean":"2024-08-31T12:28:56.137793+0000","last_became_active":"2024-08-31T12:15:56.290603+0000","last_became_peered":"2024-08-31T12:15:56.290603+0000","last_unstale":"2024-08-31T12:28:56.137793+0000","last_undegraded":"2024-08-31T12:28:56.137793+0000","last_fullsized":"2024-08-31T12:28:56.137793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'6","last_scrub_stamp":"2024-08-31T12:28:56.137715+0000","last_deep_scrub":"23'6","last_deep_scrub_stamp":"2024-08-31T12:28:56.137715+0000","last_clean_scrub_stamp":"2024-08-31T12:28:56.137715+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:38:58.476394+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10500,"num_omap_keys":21,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"23'3354","reported_seq":3386,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:03.386881+0000","last_change":"2024-08-31T12:29:03.386881+0000","last_active":"2024-08-31T12:29:03.386881+0000","last_peered":"2024-08-31T12:29:03.386881+0000","last_clean":"2024-08-31T12:29:03.386881+0000","last_became_active":"2024-08-31T12:15:57.312383+0000","last_became_peered":"2024-08-31T12:15:57.312383+0000","last_unstale":"2024-08-31T12:29:03.386881+0000","last_undegraded":"2024-08-31T12:29:03.386881+0000","last_fullsized":"2024-08-31T12:29:03.386881+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3354","last_scrub_stamp":"2024-08-31T12:29:03.386793+0000","last_deep_scrub":"23'3354","last_deep_scrub_stamp":"2024-08-31T12:29:03.386793+0000","last_clean_scrub_stamp":"2024-08-31T12:29:03.386793+0000","objects_scrubbed":60,"log_size":3354,"log_dups_size":0,"ondisk_log_size":3354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:14:18.113806+0000","scrub_duration":421,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234891518,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16277,"num_write_kb":294634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"23'8","reported_seq":40,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:56.525994+0000","last_change":"2024-08-31T12:28:56.525994+0000","last_active":"2024-08-31T12:28:56.525994+0000","last_peered":"2024-08-31T12:28:56.525994+0000","last_clean":"2024-08-31T12:28:56.525994+0000","last_became_active":"2024-08-31T12:15:56.289477+0000","last_became_peered":"2024-08-31T12:15:56.289477+0000","last_unstale":"2024-08-31T12:28:56.525994+0000","last_undegraded":"2024-08-31T12:28:56.525994+0000","last_fullsized":"2024-08-31T12:28:56.525994+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'8","last_scrub_stamp":"2024-08-31T12:28:56.525900+0000","last_deep_scrub":"23'8","last_deep_scrub_stamp":"2024-08-31T12:28:56.525900+0000","last_clean_scrub_stamp":"2024-08-31T12:28:56.525900+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:48:04.313201+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":54,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"23'2892","reported_seq":2924,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:00.155915+0000","last_change":"2024-08-31T12:29:00.155915+0000","last_active":"2024-08-31T12:29:00.155915+0000","last_peered":"2024-08-31T12:29:00.155915+0000","last_clean":"2024-08-31T12:29:00.155915+0000","last_became_active":"2024-08-31T12:15:57.312462+0000","last_became_peered":"2024-08-31T12:15:57.312462+0000","last_unstale":"2024-08-31T12:29:00.155915+0000","last_undegraded":"2024-08-31T12:29:00.155915+0000","last_fullsized":"2024-08-31T12:29:00.155915+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2892","last_scrub_stamp":"2024-08-31T12:29:00.155821+0000","last_deep_scrub":"23'2892","last_deep_scrub_stamp":"2024-08-31T12:29:00.155821+0000","last_clean_scrub_stamp":"2024-08-31T12:29:00.155821+0000","objects_scrubbed":58,"log_size":2892,"log_dups_size":0,"ondisk_log_size":2892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:30:08.911441+0000","scrub_duration":578,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222316815,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13883,"num_write_kb":272702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"23'23","reported_seq":55,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:59.250583+0000","last_change":"2024-08-31T12:28:59.250583+0000","last_active":"2024-08-31T12:28:59.250583+0000","last_peered":"2024-08-31T12:28:59.250583+0000","last_clean":"2024-08-31T12:28:59.250583+0000","last_became_active":"2024-08-31T12:15:56.285108+0000","last_became_peered":"2024-08-31T12:15:56.285108+0000","last_unstale":"2024-08-31T12:28:59.250583+0000","last_undegraded":"2024-08-31T12:28:59.250583+0000","last_fullsized":"2024-08-31T12:28:59.250583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'23","last_scrub_stamp":"2024-08-31T12:28:59.250486+0000","last_deep_scrub":"23'23","last_deep_scrub_stamp":"2024-08-31T12:28:59.250486+0000","last_clean_scrub_stamp":"2024-08-31T12:28:59.250486+0000","objects_scrubbed":8,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:00:17.841221+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"23'2902","reported_seq":2934,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:02.360939+0000","last_change":"2024-08-31T12:29:02.360939+0000","last_active":"2024-08-31T12:29:02.360939+0000","last_peered":"2024-08-31T12:29:02.360939+0000","last_clean":"2024-08-31T12:29:02.360939+0000","last_became_active":"2024-08-31T12:15:57.311368+0000","last_became_peered":"2024-08-31T12:15:57.311368+0000","last_unstale":"2024-08-31T12:29:02.360939+0000","last_undegraded":"2024-08-31T12:29:02.360939+0000","last_fullsized":"2024-08-31T12:29:02.360939+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2902","last_scrub_stamp":"2024-08-31T12:29:02.360797+0000","last_deep_scrub":"23'2902","last_deep_scrub_stamp":"2024-08-31T12:29:02.360797+0000","last_clean_scrub_stamp":"2024-08-31T12:29:02.360797+0000","objects_scrubbed":53,"log_size":2902,"log_dups_size":0,"ondisk_log_size":2902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:43:22.205823+0000","scrub_duration":425,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205547202,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14167,"num_write_kb":257592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"23'6","reported_seq":38,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:54.024330+0000","last_change":"2024-08-31T12:28:54.024330+0000","last_active":"2024-08-31T12:28:54.024330+0000","last_peered":"2024-08-31T12:28:54.024330+0000","last_clean":"2024-08-31T12:28:54.024330+0000","last_became_active":"2024-08-31T12:15:56.288289+0000","last_became_peered":"2024-08-31T12:15:56.288289+0000","last_unstale":"2024-08-31T12:28:54.024330+0000","last_undegraded":"2024-08-31T12:28:54.024330+0000","last_fullsized":"2024-08-31T12:28:54.024330+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'6","last_scrub_stamp":"2024-08-31T12:28:54.024217+0000","last_deep_scrub":"23'6","last_deep_scrub_stamp":"2024-08-31T12:28:54.024217+0000","last_clean_scrub_stamp":"2024-08-31T12:28:54.024217+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:08:22.684603+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5500,"num_omap_keys":11,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"23'2813","reported_seq":2835,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:47.480495+0000","last_change":"2024-08-31T12:15:57.311580+0000","last_active":"2024-08-31T12:28:47.480495+0000","last_peered":"2024-08-31T12:28:47.480495+0000","last_clean":"2024-08-31T12:28:47.480495+0000","last_became_active":"2024-08-31T12:15:57.311323+0000","last_became_peered":"2024-08-31T12:15:57.311323+0000","last_unstale":"2024-08-31T12:28:47.480495+0000","last_undegraded":"2024-08-31T12:28:47.480495+0000","last_fullsized":"2024-08-31T12:28:47.480495+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-31T12:15:56.265088+0000","last_clean_scrub_stamp":"2024-08-31T12:15:56.265088+0000","objects_scrubbed":0,"log_size":2813,"log_dups_size":0,"ondisk_log_size":2813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:54:03.871840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201522906,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13808,"num_write_kb":252081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"23'4","reported_seq":36,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:55.235390+0000","last_change":"2024-08-31T12:28:55.235390+0000","last_active":"2024-08-31T12:28:55.235390+0000","last_peered":"2024-08-31T12:28:55.235390+0000","last_clean":"2024-08-31T12:28:55.235390+0000","last_became_active":"2024-08-31T12:15:56.290196+0000","last_became_peered":"2024-08-31T12:15:56.290196+0000","last_unstale":"2024-08-31T12:28:55.235390+0000","last_undegraded":"2024-08-31T12:28:55.235390+0000","last_fullsized":"2024-08-31T12:28:55.235390+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'4","last_scrub_stamp":"2024-08-31T12:28:55.235278+0000","last_deep_scrub":"23'4","last_deep_scrub_stamp":"2024-08-31T12:28:55.235278+0000","last_clean_scrub_stamp":"2024-08-31T12:28:55.235278+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:03:34.089107+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"23'3351","reported_seq":3383,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:04.312074+0000","last_change":"2024-08-31T12:29:04.312074+0000","last_active":"2024-08-31T12:29:04.312074+0000","last_peered":"2024-08-31T12:29:04.312074+0000","last_clean":"2024-08-31T12:29:04.312074+0000","last_became_active":"2024-08-31T12:15:57.312145+0000","last_became_peered":"2024-08-31T12:15:57.312145+0000","last_unstale":"2024-08-31T12:29:04.312074+0000","last_undegraded":"2024-08-31T12:29:04.312074+0000","last_fullsized":"2024-08-31T12:29:04.312074+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3351","last_scrub_stamp":"2024-08-31T12:29:04.311967+0000","last_deep_scrub":"23'3351","last_deep_scrub_stamp":"2024-08-31T12:29:04.311967+0000","last_clean_scrub_stamp":"2024-08-31T12:29:04.311967+0000","objects_scrubbed":60,"log_size":3351,"log_dups_size":0,"ondisk_log_size":3351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:56:28.264029+0000","scrub_duration":386,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243277619,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16586,"num_write_kb":304033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"23'2","reported_seq":34,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:55.708519+0000","last_change":"2024-08-31T12:28:55.708519+0000","last_active":"2024-08-31T12:28:55.708519+0000","last_peered":"2024-08-31T12:28:55.708519+0000","last_clean":"2024-08-31T12:28:55.708519+0000","last_became_active":"2024-08-31T12:15:56.281368+0000","last_became_peered":"2024-08-31T12:15:56.281368+0000","last_unstale":"2024-08-31T12:28:55.708519+0000","last_undegraded":"2024-08-31T12:28:55.708519+0000","last_fullsized":"2024-08-31T12:28:55.708519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2","last_scrub_stamp":"2024-08-31T12:28:55.708441+0000","last_deep_scrub":"23'2","last_deep_scrub_stamp":"2024-08-31T12:28:55.708441+0000","last_clean_scrub_stamp":"2024-08-31T12:28:55.708441+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:22:27.640560+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"23'3353","reported_seq":3386,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:01.001446+0000","last_change":"2024-08-31T12:29:01.001446+0000","last_active":"2024-08-31T12:29:01.001446+0000","last_peered":"2024-08-31T12:29:01.001446+0000","last_clean":"2024-08-31T12:29:01.001446+0000","last_became_active":"2024-08-31T12:15:57.305507+0000","last_became_peered":"2024-08-31T12:15:57.305507+0000","last_unstale":"2024-08-31T12:29:01.001446+0000","last_undegraded":"2024-08-31T12:29:01.001446+0000","last_fullsized":"2024-08-31T12:29:01.001446+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3353","last_scrub_stamp":"2024-08-31T12:29:01.001362+0000","last_deep_scrub":"23'3353","last_deep_scrub_stamp":"2024-08-31T12:29:01.001362+0000","last_clean_scrub_stamp":"2024-08-31T12:29:01.001362+0000","objects_scrubbed":58,"log_size":3353,"log_dups_size":0,"ondisk_log_size":3353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:30:34.897037+0000","scrub_duration":777,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218117314,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16432,"num_write_kb":278837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"23'6","reported_seq":38,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:53.311960+0000","last_change":"2024-08-31T12:28:53.311960+0000","last_active":"2024-08-31T12:28:53.311960+0000","last_peered":"2024-08-31T12:28:53.311960+0000","last_clean":"2024-08-31T12:28:53.311960+0000","last_became_active":"2024-08-31T12:15:56.291446+0000","last_became_peered":"2024-08-31T12:15:56.291446+0000","last_unstale":"2024-08-31T12:28:53.311960+0000","last_undegraded":"2024-08-31T12:28:53.311960+0000","last_fullsized":"2024-08-31T12:28:53.311960+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'6","last_scrub_stamp":"2024-08-31T12:28:53.311855+0000","last_deep_scrub":"23'6","last_deep_scrub_stamp":"2024-08-31T12:28:53.311855+0000","last_clean_scrub_stamp":"2024-08-31T12:28:53.311855+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:09:33.112431+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"23'2799","reported_seq":2832,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:05.295572+0000","last_change":"2024-08-31T12:29:05.295572+0000","last_active":"2024-08-31T12:29:05.295572+0000","last_peered":"2024-08-31T12:29:05.295572+0000","last_clean":"2024-08-31T12:29:05.295572+0000","last_became_active":"2024-08-31T12:15:57.315830+0000","last_became_peered":"2024-08-31T12:15:57.315830+0000","last_unstale":"2024-08-31T12:29:05.295572+0000","last_undegraded":"2024-08-31T12:29:05.295572+0000","last_fullsized":"2024-08-31T12:29:05.295572+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2799","last_scrub_stamp":"2024-08-31T12:29:05.295434+0000","last_deep_scrub":"23'2799","last_deep_scrub_stamp":"2024-08-31T12:29:05.295434+0000","last_clean_scrub_stamp":"2024-08-31T12:29:05.295434+0000","objects_scrubbed":55,"log_size":2799,"log_dups_size":0,"ondisk_log_size":2799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:01:26.820314+0000","scrub_duration":329,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209813064,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13700,"num_write_kb":259755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"23'74","reported_seq":131,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:55.002617+0000","last_change":"2024-08-31T12:28:55.002617+0000","last_active":"2024-08-31T12:28:55.002617+0000","last_peered":"2024-08-31T12:28:55.002617+0000","last_clean":"2024-08-31T12:28:55.002617+0000","last_became_active":"2024-08-31T12:15:47.083771+0000","last_became_peered":"2024-08-31T12:15:47.083771+0000","last_unstale":"2024-08-31T12:28:55.002617+0000","last_undegraded":"2024-08-31T12:28:55.002617+0000","last_fullsized":"2024-08-31T12:28:55.002617+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'74","last_scrub_stamp":"2024-08-31T12:28:55.002525+0000","last_deep_scrub":"23'74","last_deep_scrub_stamp":"2024-08-31T12:28:55.002525+0000","last_clean_scrub_stamp":"2024-08-31T12:28:55.002525+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:30:38.520538+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"23'7","reported_seq":40,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:55.178238+0000","last_change":"2024-08-31T12:28:55.178238+0000","last_active":"2024-08-31T12:28:55.178238+0000","last_peered":"2024-08-31T12:28:55.178238+0000","last_clean":"2024-08-31T12:28:55.178238+0000","last_became_active":"2024-08-31T12:15:56.284470+0000","last_became_peered":"2024-08-31T12:15:56.284470+0000","last_unstale":"2024-08-31T12:28:55.178238+0000","last_undegraded":"2024-08-31T12:28:55.178238+0000","last_fullsized":"2024-08-31T12:28:55.178238+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'7","last_scrub_stamp":"2024-08-31T12:28:55.178131+0000","last_deep_scrub":"23'7","last_deep_scrub_stamp":"2024-08-31T12:28:55.178131+0000","last_clean_scrub_stamp":"2024-08-31T12:28:55.178131+0000","objects_scrubbed":3,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:28:12.647620+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"23'2828","reported_seq":2861,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:59.048135+0000","last_change":"2024-08-31T12:28:59.048135+0000","last_active":"2024-08-31T12:28:59.048135+0000","last_peered":"2024-08-31T12:28:59.048135+0000","last_clean":"2024-08-31T12:28:59.048135+0000","last_became_active":"2024-08-31T12:15:57.308721+0000","last_became_peered":"2024-08-31T12:15:57.308721+0000","last_unstale":"2024-08-31T12:28:59.048135+0000","last_undegraded":"2024-08-31T12:28:59.048135+0000","last_fullsized":"2024-08-31T12:28:59.048135+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2828","last_scrub_stamp":"2024-08-31T12:28:59.048032+0000","last_deep_scrub":"23'2828","last_deep_scrub_stamp":"2024-08-31T12:28:59.048032+0000","last_clean_scrub_stamp":"2024-08-31T12:28:59.048032+0000","objects_scrubbed":53,"log_size":2828,"log_dups_size":0,"ondisk_log_size":2828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:25:15.429296+0000","scrub_duration":536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218112028,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13723,"num_write_kb":267984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"23'5","reported_seq":38,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:56.040275+0000","last_change":"2024-08-31T12:28:56.040275+0000","last_active":"2024-08-31T12:28:56.040275+0000","last_peered":"2024-08-31T12:28:56.040275+0000","last_clean":"2024-08-31T12:28:56.040275+0000","last_became_active":"2024-08-31T12:15:56.292910+0000","last_became_peered":"2024-08-31T12:15:56.292910+0000","last_unstale":"2024-08-31T12:28:56.040275+0000","last_undegraded":"2024-08-31T12:28:56.040275+0000","last_fullsized":"2024-08-31T12:28:56.040275+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5","last_scrub_stamp":"2024-08-31T12:28:56.040147+0000","last_deep_scrub":"23'5","last_deep_scrub_stamp":"2024-08-31T12:28:56.040147+0000","last_clean_scrub_stamp":"2024-08-31T12:28:56.040147+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:58:55.930106+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"23'2813","reported_seq":2846,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:52.584464+0000","last_change":"2024-08-31T12:28:52.584464+0000","last_active":"2024-08-31T12:28:52.584464+0000","last_peered":"2024-08-31T12:28:52.584464+0000","last_clean":"2024-08-31T12:28:52.584464+0000","last_became_active":"2024-08-31T12:15:57.306529+0000","last_became_peered":"2024-08-31T12:15:57.306529+0000","last_unstale":"2024-08-31T12:28:52.584464+0000","last_undegraded":"2024-08-31T12:28:52.584464+0000","last_fullsized":"2024-08-31T12:28:52.584464+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2813","last_scrub_stamp":"2024-08-31T12:28:52.584390+0000","last_deep_scrub":"23'2813","last_deep_scrub_stamp":"2024-08-31T12:28:52.584390+0000","last_clean_scrub_stamp":"2024-08-31T12:28:52.584390+0000","objects_scrubbed":51,"log_size":2813,"log_dups_size":0,"ondisk_log_size":2813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:30:35.075853+0000","scrub_duration":362,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201337916,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14174,"num_write_kb":253444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"23'38","reported_seq":101,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:52.310657+0000","last_change":"2024-08-31T12:28:52.310657+0000","last_active":"2024-08-31T12:28:52.310657+0000","last_peered":"2024-08-31T12:28:52.310657+0000","last_clean":"2024-08-31T12:28:52.310657+0000","last_became_active":"2024-08-31T12:15:56.291064+0000","last_became_peered":"2024-08-31T12:15:56.291064+0000","last_unstale":"2024-08-31T12:28:52.310657+0000","last_undegraded":"2024-08-31T12:28:52.310657+0000","last_fullsized":"2024-08-31T12:28:52.310657+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'38","last_scrub_stamp":"2024-08-31T12:28:52.310562+0000","last_deep_scrub":"23'38","last_deep_scrub_stamp":"2024-08-31T12:28:52.310562+0000","last_clean_scrub_stamp":"2024-08-31T12:28:52.310562+0000","objects_scrubbed":3,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:52:09.840889+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47459,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":30,"num_read_kb":64,"num_write":34,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"23'2997","reported_seq":3030,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:59.228323+0000","last_change":"2024-08-31T12:28:59.228323+0000","last_active":"2024-08-31T12:28:59.228323+0000","last_peered":"2024-08-31T12:28:59.228323+0000","last_clean":"2024-08-31T12:28:59.228323+0000","last_became_active":"2024-08-31T12:15:57.298892+0000","last_became_peered":"2024-08-31T12:15:57.298892+0000","last_unstale":"2024-08-31T12:28:59.228323+0000","last_undegraded":"2024-08-31T12:28:59.228323+0000","last_fullsized":"2024-08-31T12:28:59.228323+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2997","last_scrub_stamp":"2024-08-31T12:28:59.228243+0000","last_deep_scrub":"23'2997","last_deep_scrub_stamp":"2024-08-31T12:28:59.228243+0000","last_clean_scrub_stamp":"2024-08-31T12:28:59.228243+0000","objects_scrubbed":51,"log_size":2997,"log_dups_size":0,"ondisk_log_size":2997,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:43:41.236154+0000","scrub_duration":454,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197144226,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14947,"num_write_kb":252544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"23'2","reported_seq":35,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:54.218507+0000","last_change":"2024-08-31T12:28:54.218507+0000","last_active":"2024-08-31T12:28:54.218507+0000","last_peered":"2024-08-31T12:28:54.218507+0000","last_clean":"2024-08-31T12:28:54.218507+0000","last_became_active":"2024-08-31T12:15:56.285648+0000","last_became_peered":"2024-08-31T12:15:56.285648+0000","last_unstale":"2024-08-31T12:28:54.218507+0000","last_undegraded":"2024-08-31T12:28:54.218507+0000","last_fullsized":"2024-08-31T12:28:54.218507+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2","last_scrub_stamp":"2024-08-31T12:28:54.218372+0000","last_deep_scrub":"23'2","last_deep_scrub_stamp":"2024-08-31T12:28:54.218372+0000","last_clean_scrub_stamp":"2024-08-31T12:28:54.218372+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:17:40.309134+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"23'2818","reported_seq":2851,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:55.445307+0000","last_change":"2024-08-31T12:28:55.445307+0000","last_active":"2024-08-31T12:28:55.445307+0000","last_peered":"2024-08-31T12:28:55.445307+0000","last_clean":"2024-08-31T12:28:55.445307+0000","last_became_active":"2024-08-31T12:15:57.311469+0000","last_became_peered":"2024-08-31T12:15:57.311469+0000","last_unstale":"2024-08-31T12:28:55.445307+0000","last_undegraded":"2024-08-31T12:28:55.445307+0000","last_fullsized":"2024-08-31T12:28:55.445307+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2818","last_scrub_stamp":"2024-08-31T12:28:55.445200+0000","last_deep_scrub":"23'2818","last_deep_scrub_stamp":"2024-08-31T12:28:55.445200+0000","last_clean_scrub_stamp":"2024-08-31T12:28:55.445200+0000","objects_scrubbed":49,"log_size":2818,"log_dups_size":0,"ondisk_log_size":2818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:42:39.242246+0000","scrub_duration":414,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197139313,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14081,"num_write_kb":248914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"23'2","reported_seq":35,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:52.650358+0000","last_change":"2024-08-31T12:28:52.650358+0000","last_active":"2024-08-31T12:28:52.650358+0000","last_peered":"2024-08-31T12:28:52.650358+0000","last_clean":"2024-08-31T12:28:52.650358+0000","last_became_active":"2024-08-31T12:15:56.283979+0000","last_became_peered":"2024-08-31T12:15:56.283979+0000","last_unstale":"2024-08-31T12:28:52.650358+0000","last_undegraded":"2024-08-31T12:28:52.650358+0000","last_fullsized":"2024-08-31T12:28:52.650358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2","last_scrub_stamp":"2024-08-31T12:28:52.650280+0000","last_deep_scrub":"23'2","last_deep_scrub_stamp":"2024-08-31T12:28:52.650280+0000","last_clean_scrub_stamp":"2024-08-31T12:28:52.650280+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:59:18.393870+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7000,"num_omap_keys":14,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"23'2575","reported_seq":2601,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:58.709640+0000","last_change":"2024-08-31T12:28:58.709640+0000","last_active":"2024-08-31T12:28:58.709640+0000","last_peered":"2024-08-31T12:28:58.709640+0000","last_clean":"2024-08-31T12:28:58.709640+0000","last_became_active":"2024-08-31T12:16:41.194724+0000","last_became_peered":"2024-08-31T12:16:41.194724+0000","last_unstale":"2024-08-31T12:28:58.709640+0000","last_undegraded":"2024-08-31T12:28:58.709640+0000","last_fullsized":"2024-08-31T12:28:58.709640+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2575","last_scrub_stamp":"2024-08-31T12:28:58.709472+0000","last_deep_scrub":"23'2575","last_deep_scrub_stamp":"2024-08-31T12:28:58.709472+0000","last_clean_scrub_stamp":"2024-08-31T12:28:58.709472+0000","objects_scrubbed":48,"log_size":2575,"log_dups_size":0,"ondisk_log_size":2575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:59:13.626464+0000","scrub_duration":507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184558382,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13011,"num_write_kb":232344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"23'2","reported_seq":35,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:00.976677+0000","last_change":"2024-08-31T12:29:00.976677+0000","last_active":"2024-08-31T12:29:00.976677+0000","last_peered":"2024-08-31T12:29:00.976677+0000","last_clean":"2024-08-31T12:29:00.976677+0000","last_became_active":"2024-08-31T12:15:56.297306+0000","last_became_peered":"2024-08-31T12:15:56.297306+0000","last_unstale":"2024-08-31T12:29:00.976677+0000","last_undegraded":"2024-08-31T12:29:00.976677+0000","last_fullsized":"2024-08-31T12:29:00.976677+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2","last_scrub_stamp":"2024-08-31T12:29:00.976561+0000","last_deep_scrub":"23'2","last_deep_scrub_stamp":"2024-08-31T12:29:00.976561+0000","last_clean_scrub_stamp":"2024-08-31T12:29:00.976561+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:23:50.922340+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"23'3228","reported_seq":3261,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:55.228844+0000","last_change":"2024-08-31T12:28:55.228844+0000","last_active":"2024-08-31T12:28:55.228844+0000","last_peered":"2024-08-31T12:28:55.228844+0000","last_clean":"2024-08-31T12:28:55.228844+0000","last_became_active":"2024-08-31T12:15:57.296892+0000","last_became_peered":"2024-08-31T12:15:57.296892+0000","last_unstale":"2024-08-31T12:28:55.228844+0000","last_undegraded":"2024-08-31T12:28:55.228844+0000","last_fullsized":"2024-08-31T12:28:55.228844+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3228","last_scrub_stamp":"2024-08-31T12:28:55.228711+0000","last_deep_scrub":"23'3228","last_deep_scrub_stamp":"2024-08-31T12:28:55.228711+0000","last_clean_scrub_stamp":"2024-08-31T12:28:55.228711+0000","objects_scrubbed":55,"log_size":3228,"log_dups_size":0,"ondisk_log_size":3228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:22:55.949863+0000","scrub_duration":535,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213956769,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16152,"num_write_kb":273638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"23'431","reported_seq":464,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:51.651294+0000","last_change":"2024-08-31T12:28:51.651231+0000","last_active":"2024-08-31T12:28:51.651294+0000","last_peered":"2024-08-31T12:28:51.651294+0000","last_clean":"2024-08-31T12:28:51.651294+0000","last_became_active":"2024-08-31T12:15:56.286173+0000","last_became_peered":"2024-08-31T12:15:56.286173+0000","last_unstale":"2024-08-31T12:28:51.651294+0000","last_undegraded":"2024-08-31T12:28:51.651294+0000","last_fullsized":"2024-08-31T12:28:51.651294+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'431","last_scrub_stamp":"2024-08-31T12:28:51.651149+0000","last_deep_scrub":"23'431","last_deep_scrub_stamp":"2024-08-31T12:28:51.651149+0000","last_clean_scrub_stamp":"2024-08-31T12:28:51.651149+0000","objects_scrubbed":4,"log_size":431,"log_dups_size":0,"ondisk_log_size":431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:19:48.734353+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":977,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9038,"num_omap_keys":18,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"23'2972","reported_seq":3005,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:57.965445+0000","last_change":"2024-08-31T12:28:57.965381+0000","last_active":"2024-08-31T12:28:57.965445+0000","last_peered":"2024-08-31T12:28:57.965445+0000","last_clean":"2024-08-31T12:28:57.965445+0000","last_became_active":"2024-08-31T12:15:57.307701+0000","last_became_peered":"2024-08-31T12:15:57.307701+0000","last_unstale":"2024-08-31T12:28:57.965445+0000","last_undegraded":"2024-08-31T12:28:57.965445+0000","last_fullsized":"2024-08-31T12:28:57.965445+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2972","last_scrub_stamp":"2024-08-31T12:28:57.965299+0000","last_deep_scrub":"23'2972","last_deep_scrub_stamp":"2024-08-31T12:28:57.965299+0000","last_clean_scrub_stamp":"2024-08-31T12:28:57.965299+0000","objects_scrubbed":51,"log_size":2972,"log_dups_size":0,"ondisk_log_size":2972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:27:46.349413+0000","scrub_duration":468,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205521115,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14225,"num_write_kb":257747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"23'6","reported_seq":39,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:57.493049+0000","last_change":"2024-08-31T12:28:57.493049+0000","last_active":"2024-08-31T12:28:57.493049+0000","last_peered":"2024-08-31T12:28:57.493049+0000","last_clean":"2024-08-31T12:28:57.493049+0000","last_became_active":"2024-08-31T12:15:56.294496+0000","last_became_peered":"2024-08-31T12:15:56.294496+0000","last_unstale":"2024-08-31T12:28:57.493049+0000","last_undegraded":"2024-08-31T12:28:57.493049+0000","last_fullsized":"2024-08-31T12:28:57.493049+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'6","last_scrub_stamp":"2024-08-31T12:28:57.492917+0000","last_deep_scrub":"23'6","last_deep_scrub_stamp":"2024-08-31T12:28:57.492917+0000","last_clean_scrub_stamp":"2024-08-31T12:28:57.492917+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:22:58.214069+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"23'2589","reported_seq":2621,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:53.611819+0000","last_change":"2024-08-31T12:28:53.611757+0000","last_active":"2024-08-31T12:28:53.611819+0000","last_peered":"2024-08-31T12:28:53.611819+0000","last_clean":"2024-08-31T12:28:53.611819+0000","last_became_active":"2024-08-31T12:15:57.304602+0000","last_became_peered":"2024-08-31T12:15:57.304602+0000","last_unstale":"2024-08-31T12:28:53.611819+0000","last_undegraded":"2024-08-31T12:28:53.611819+0000","last_fullsized":"2024-08-31T12:28:53.611819+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2589","last_scrub_stamp":"2024-08-31T12:28:53.611658+0000","last_deep_scrub":"23'2589","last_deep_scrub_stamp":"2024-08-31T12:28:53.611658+0000","last_clean_scrub_stamp":"2024-08-31T12:28:53.611658+0000","objects_scrubbed":45,"log_size":2589,"log_dups_size":0,"ondisk_log_size":2589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:38:56.546250+0000","scrub_duration":400,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176167762,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12536,"num_write_kb":222149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"23'4","reported_seq":37,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:59.994908+0000","last_change":"2024-08-31T12:28:59.994848+0000","last_active":"2024-08-31T12:28:59.994908+0000","last_peered":"2024-08-31T12:28:59.994908+0000","last_clean":"2024-08-31T12:28:59.994908+0000","last_became_active":"2024-08-31T12:15:56.297168+0000","last_became_peered":"2024-08-31T12:15:56.297168+0000","last_unstale":"2024-08-31T12:28:59.994908+0000","last_undegraded":"2024-08-31T12:28:59.994908+0000","last_fullsized":"2024-08-31T12:28:59.994908+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'4","last_scrub_stamp":"2024-08-31T12:28:59.994769+0000","last_deep_scrub":"23'4","last_deep_scrub_stamp":"2024-08-31T12:28:59.994769+0000","last_clean_scrub_stamp":"2024-08-31T12:28:59.994769+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:50:05.473008+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"23'2372","reported_seq":2405,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:56.443443+0000","last_change":"2024-08-31T12:28:56.443379+0000","last_active":"2024-08-31T12:28:56.443443+0000","last_peered":"2024-08-31T12:28:56.443443+0000","last_clean":"2024-08-31T12:28:56.443443+0000","last_became_active":"2024-08-31T12:15:57.298520+0000","last_became_peered":"2024-08-31T12:15:57.298520+0000","last_unstale":"2024-08-31T12:28:56.443443+0000","last_undegraded":"2024-08-31T12:28:56.443443+0000","last_fullsized":"2024-08-31T12:28:56.443443+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2372","last_scrub_stamp":"2024-08-31T12:28:56.443305+0000","last_deep_scrub":"23'2372","last_deep_scrub_stamp":"2024-08-31T12:28:56.443305+0000","last_clean_scrub_stamp":"2024-08-31T12:28:56.443305+0000","objects_scrubbed":50,"log_size":2372,"log_dups_size":0,"ondisk_log_size":2372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:15:10.275590+0000","scrub_duration":422,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197144309,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11287,"num_write_kb":237691,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"23'5","reported_seq":38,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:56.716501+0000","last_change":"2024-08-31T12:28:56.716501+0000","last_active":"2024-08-31T12:28:56.716501+0000","last_peered":"2024-08-31T12:28:56.716501+0000","last_clean":"2024-08-31T12:28:56.716501+0000","last_became_active":"2024-08-31T12:15:56.289960+0000","last_became_peered":"2024-08-31T12:15:56.289960+0000","last_unstale":"2024-08-31T12:28:56.716501+0000","last_undegraded":"2024-08-31T12:28:56.716501+0000","last_fullsized":"2024-08-31T12:28:56.716501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5","last_scrub_stamp":"2024-08-31T12:28:56.716407+0000","last_deep_scrub":"23'5","last_deep_scrub_stamp":"2024-08-31T12:28:56.716407+0000","last_clean_scrub_stamp":"2024-08-31T12:28:56.716407+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:30:05.472027+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"23'3030","reported_seq":3063,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:04.944510+0000","last_change":"2024-08-31T12:29:04.944510+0000","last_active":"2024-08-31T12:29:04.944510+0000","last_peered":"2024-08-31T12:29:04.944510+0000","last_clean":"2024-08-31T12:29:04.944510+0000","last_became_active":"2024-08-31T12:15:57.301157+0000","last_became_peered":"2024-08-31T12:15:57.301157+0000","last_unstale":"2024-08-31T12:29:04.944510+0000","last_undegraded":"2024-08-31T12:29:04.944510+0000","last_fullsized":"2024-08-31T12:29:04.944510+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3030","last_scrub_stamp":"2024-08-31T12:29:04.944425+0000","last_deep_scrub":"23'3030","last_deep_scrub_stamp":"2024-08-31T12:29:04.944425+0000","last_clean_scrub_stamp":"2024-08-31T12:29:04.944425+0000","objects_scrubbed":51,"log_size":3030,"log_dups_size":0,"ondisk_log_size":3030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:20:43.219093+0000","scrub_duration":462,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209728881,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15223,"num_write_kb":265883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"23'2","reported_seq":35,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:53.651283+0000","last_change":"2024-08-31T12:28:53.651216+0000","last_active":"2024-08-31T12:28:53.651283+0000","last_peered":"2024-08-31T12:28:53.651283+0000","last_clean":"2024-08-31T12:28:53.651283+0000","last_became_active":"2024-08-31T12:15:56.289332+0000","last_became_peered":"2024-08-31T12:15:56.289332+0000","last_unstale":"2024-08-31T12:28:53.651283+0000","last_undegraded":"2024-08-31T12:28:53.651283+0000","last_fullsized":"2024-08-31T12:28:53.651283+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2","last_scrub_stamp":"2024-08-31T12:28:53.651142+0000","last_deep_scrub":"23'2","last_deep_scrub_stamp":"2024-08-31T12:28:53.651142+0000","last_clean_scrub_stamp":"2024-08-31T12:28:53.651142+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:19:43.725168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"23'2468","reported_seq":2500,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:56.968970+0000","last_change":"2024-08-31T12:28:56.968970+0000","last_active":"2024-08-31T12:28:56.968970+0000","last_peered":"2024-08-31T12:28:56.968970+0000","last_clean":"2024-08-31T12:28:56.968970+0000","last_became_active":"2024-08-31T12:15:57.313428+0000","last_became_peered":"2024-08-31T12:15:57.313428+0000","last_unstale":"2024-08-31T12:28:56.968970+0000","last_undegraded":"2024-08-31T12:28:56.968970+0000","last_fullsized":"2024-08-31T12:28:56.968970+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2468","last_scrub_stamp":"2024-08-31T12:28:56.968836+0000","last_deep_scrub":"23'2468","last_deep_scrub_stamp":"2024-08-31T12:28:56.968836+0000","last_clean_scrub_stamp":"2024-08-31T12:28:56.968836+0000","objects_scrubbed":45,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:57:17.337071+0000","scrub_duration":437,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176173175,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11447,"num_write_kb":217885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"23'1071","reported_seq":1103,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:57.057690+0000","last_change":"2024-08-31T12:28:57.057690+0000","last_active":"2024-08-31T12:28:57.057690+0000","last_peered":"2024-08-31T12:28:57.057690+0000","last_clean":"2024-08-31T12:28:57.057690+0000","last_became_active":"2024-08-31T12:15:56.295524+0000","last_became_peered":"2024-08-31T12:15:56.295524+0000","last_unstale":"2024-08-31T12:28:57.057690+0000","last_undegraded":"2024-08-31T12:28:57.057690+0000","last_fullsized":"2024-08-31T12:28:57.057690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'1071","last_scrub_stamp":"2024-08-31T12:28:57.057583+0000","last_deep_scrub":"23'1071","last_deep_scrub_stamp":"2024-08-31T12:28:57.057583+0000","last_clean_scrub_stamp":"2024-08-31T12:28:57.057583+0000","objects_scrubbed":3,"log_size":1071,"log_dups_size":0,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:05:01.152499+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1064,"num_write_kb":4401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"23'2553","reported_seq":2585,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:00.889097+0000","last_change":"2024-08-31T12:29:00.888999+0000","last_active":"2024-08-31T12:29:00.889097+0000","last_peered":"2024-08-31T12:29:00.889097+0000","last_clean":"2024-08-31T12:29:00.889097+0000","last_became_active":"2024-08-31T12:15:57.299436+0000","last_became_peered":"2024-08-31T12:15:57.299436+0000","last_unstale":"2024-08-31T12:29:00.889097+0000","last_undegraded":"2024-08-31T12:29:00.889097+0000","last_fullsized":"2024-08-31T12:29:00.889097+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2553","last_scrub_stamp":"2024-08-31T12:29:00.888868+0000","last_deep_scrub":"23'2553","last_deep_scrub_stamp":"2024-08-31T12:29:00.888868+0000","last_clean_scrub_stamp":"2024-08-31T12:29:00.888868+0000","objects_scrubbed":46,"log_size":2553,"log_dups_size":0,"ondisk_log_size":2553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:57:21.257104+0000","scrub_duration":321,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188744397,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12696,"num_write_kb":235198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"23'3436","reported_seq":3468,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:02.776669+0000","last_change":"2024-08-31T12:29:02.776669+0000","last_active":"2024-08-31T12:29:02.776669+0000","last_peered":"2024-08-31T12:29:02.776669+0000","last_clean":"2024-08-31T12:29:02.776669+0000","last_became_active":"2024-08-31T12:15:57.300159+0000","last_became_peered":"2024-08-31T12:15:57.300159+0000","last_unstale":"2024-08-31T12:29:02.776669+0000","last_undegraded":"2024-08-31T12:29:02.776669+0000","last_fullsized":"2024-08-31T12:29:02.776669+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3436","last_scrub_stamp":"2024-08-31T12:29:02.776519+0000","last_deep_scrub":"23'3436","last_deep_scrub_stamp":"2024-08-31T12:29:02.776519+0000","last_clean_scrub_stamp":"2024-08-31T12:29:02.776519+0000","objects_scrubbed":63,"log_size":3436,"log_dups_size":0,"ondisk_log_size":3436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:23:52.722298+0000","scrub_duration":514,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251730727,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16759,"num_write_kb":312885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"23'3","reported_seq":35,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:58.015823+0000","last_change":"2024-08-31T12:28:58.015823+0000","last_active":"2024-08-31T12:28:58.015823+0000","last_peered":"2024-08-31T12:28:58.015823+0000","last_clean":"2024-08-31T12:28:58.015823+0000","last_became_active":"2024-08-31T12:15:56.296976+0000","last_became_peered":"2024-08-31T12:15:56.296976+0000","last_unstale":"2024-08-31T12:28:58.015823+0000","last_undegraded":"2024-08-31T12:28:58.015823+0000","last_fullsized":"2024-08-31T12:28:58.015823+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3","last_scrub_stamp":"2024-08-31T12:28:58.015746+0000","last_deep_scrub":"23'3","last_deep_scrub_stamp":"2024-08-31T12:28:58.015746+0000","last_clean_scrub_stamp":"2024-08-31T12:28:58.015746+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:57:21.675079+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"23'2213","reported_seq":2244,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:01.520012+0000","last_change":"2024-08-31T12:29:01.520012+0000","last_active":"2024-08-31T12:29:01.520012+0000","last_peered":"2024-08-31T12:29:01.520012+0000","last_clean":"2024-08-31T12:29:01.520012+0000","last_became_active":"2024-08-31T12:15:57.300646+0000","last_became_peered":"2024-08-31T12:15:57.300646+0000","last_unstale":"2024-08-31T12:29:01.520012+0000","last_undegraded":"2024-08-31T12:29:01.520012+0000","last_fullsized":"2024-08-31T12:29:01.520012+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2213","last_scrub_stamp":"2024-08-31T12:29:01.519875+0000","last_deep_scrub":"23'2213","last_deep_scrub_stamp":"2024-08-31T12:29:01.519875+0000","last_clean_scrub_stamp":"2024-08-31T12:29:01.519875+0000","objects_scrubbed":37,"log_size":2213,"log_dups_size":0,"ondisk_log_size":2213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:35:49.816083+0000","scrub_duration":251,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150996671,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10769,"num_write_kb":190633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"23'11","reported_seq":43,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:58.971224+0000","last_change":"2024-08-31T12:28:58.971224+0000","last_active":"2024-08-31T12:28:58.971224+0000","last_peered":"2024-08-31T12:28:58.971224+0000","last_clean":"2024-08-31T12:28:58.971224+0000","last_became_active":"2024-08-31T12:15:56.294560+0000","last_became_peered":"2024-08-31T12:15:56.294560+0000","last_unstale":"2024-08-31T12:28:58.971224+0000","last_undegraded":"2024-08-31T12:28:58.971224+0000","last_fullsized":"2024-08-31T12:28:58.971224+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'11","last_scrub_stamp":"2024-08-31T12:28:58.971147+0000","last_deep_scrub":"23'11","last_deep_scrub_stamp":"2024-08-31T12:28:58.971147+0000","last_clean_scrub_stamp":"2024-08-31T12:28:58.971147+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:32:02.655635+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"23'3421","reported_seq":3454,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:57.914506+0000","last_change":"2024-08-31T12:28:57.914506+0000","last_active":"2024-08-31T12:28:57.914506+0000","last_peered":"2024-08-31T12:28:57.914506+0000","last_clean":"2024-08-31T12:28:57.914506+0000","last_became_active":"2024-08-31T12:15:57.310262+0000","last_became_peered":"2024-08-31T12:15:57.310262+0000","last_unstale":"2024-08-31T12:28:57.914506+0000","last_undegraded":"2024-08-31T12:28:57.914506+0000","last_fullsized":"2024-08-31T12:28:57.914506+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3421","last_scrub_stamp":"2024-08-31T12:28:57.914399+0000","last_deep_scrub":"23'3421","last_deep_scrub_stamp":"2024-08-31T12:28:57.914399+0000","last_clean_scrub_stamp":"2024-08-31T12:28:57.914399+0000","objects_scrubbed":67,"log_size":3421,"log_dups_size":0,"ondisk_log_size":3421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:22:29.846518+0000","scrub_duration":560,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251719163,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16721,"num_write_kb":312844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"23'4","reported_seq":37,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:55.530644+0000","last_change":"2024-08-31T12:28:55.530644+0000","last_active":"2024-08-31T12:28:55.530644+0000","last_peered":"2024-08-31T12:28:55.530644+0000","last_clean":"2024-08-31T12:28:55.530644+0000","last_became_active":"2024-08-31T12:15:56.284884+0000","last_became_peered":"2024-08-31T12:15:56.284884+0000","last_unstale":"2024-08-31T12:28:55.530644+0000","last_undegraded":"2024-08-31T12:28:55.530644+0000","last_fullsized":"2024-08-31T12:28:55.530644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'4","last_scrub_stamp":"2024-08-31T12:28:55.530523+0000","last_deep_scrub":"23'4","last_deep_scrub_stamp":"2024-08-31T12:28:55.530523+0000","last_clean_scrub_stamp":"2024-08-31T12:28:55.530523+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:27:24.024019+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5538,"num_omap_keys":11,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"23'3225","reported_seq":3258,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:00.137341+0000","last_change":"2024-08-31T12:29:00.137341+0000","last_active":"2024-08-31T12:29:00.137341+0000","last_peered":"2024-08-31T12:29:00.137341+0000","last_clean":"2024-08-31T12:29:00.137341+0000","last_became_active":"2024-08-31T12:15:57.305409+0000","last_became_peered":"2024-08-31T12:15:57.305409+0000","last_unstale":"2024-08-31T12:29:00.137341+0000","last_undegraded":"2024-08-31T12:29:00.137341+0000","last_fullsized":"2024-08-31T12:29:00.137341+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3225","last_scrub_stamp":"2024-08-31T12:29:00.137249+0000","last_deep_scrub":"23'3225","last_deep_scrub_stamp":"2024-08-31T12:29:00.137249+0000","last_clean_scrub_stamp":"2024-08-31T12:29:00.137249+0000","objects_scrubbed":53,"log_size":3225,"log_dups_size":0,"ondisk_log_size":3225,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:37:46.351278+0000","scrub_duration":589,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222298112,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15945,"num_write_kb":281070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"23'241","reported_seq":274,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:57.337647+0000","last_change":"2024-08-31T12:28:55.304129+0000","last_active":"2024-08-31T12:28:57.337647+0000","last_peered":"2024-08-31T12:28:57.337647+0000","last_clean":"2024-08-31T12:28:57.337647+0000","last_became_active":"2024-08-31T12:15:56.293864+0000","last_became_peered":"2024-08-31T12:15:56.293864+0000","last_unstale":"2024-08-31T12:28:57.337647+0000","last_undegraded":"2024-08-31T12:28:57.337647+0000","last_fullsized":"2024-08-31T12:28:57.337647+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'237","last_scrub_stamp":"2024-08-31T12:28:55.304041+0000","last_deep_scrub":"23'237","last_deep_scrub_stamp":"2024-08-31T12:28:55.304041+0000","last_clean_scrub_stamp":"2024-08-31T12:28:55.304041+0000","objects_scrubbed":5,"log_size":241,"log_dups_size":0,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:48:03.091342+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1390922,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":236,"num_write_kb":1494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":13424230175,"num_objects":3391,"num_object_clones":0,"num_object_copies":6782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":918154,"num_write_kb":16787942,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26849361920,"data_stored":26848441814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":187403,"ondisk_log_size":187403,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":14025485,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":30,"num_read_kb":64,"num_write":3320,"num_write_kb":14861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":66114,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28205056,"data_stored":28050970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2839,"ondisk_log_size":2839,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509605,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2618576,"kb_used_data":2384832,"kb_used_omap":1,"kb_used_meta":233726,"kb_avail":91753264,"statfs":{"total":96636764160,"available":93955342336,"internally_reserved":0,"allocated":2442067968,"data_stored":2441803901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":239335878},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509604,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4211060,"kb_used_data":3851044,"kb_used_omap":1,"kb_used_meta":359998,"kb_avail":90160780,"statfs":{"total":96636764160,"available":92324638720,"internally_reserved":0,"allocated":3943469056,"data_stored":3943171671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":368638407},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509605,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2548680,"kb_used_data":2306616,"kb_used_omap":1,"kb_used_meta":242046,"kb_avail":91823160,"statfs":{"total":96636764160,"available":94026915840,"internally_reserved":0,"allocated":2361974784,"data_stored":2361727195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":247855557},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509605,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4129328,"kb_used_data":3795680,"kb_used_omap":1,"kb_used_meta":333630,"kb_avail":90242512,"statfs":{"total":96636764160,"available":92408332288,"internally_reserved":0,"allocated":3886776320,"data_stored":3886460583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341637573},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542308,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4427080,"kb_used_data":4067000,"kb_used_omap":1,"kb_used_meta":360062,"kb_avail":89944760,"statfs":{"total":96636764160,"available":92103434240,"internally_reserved":0,"allocated":4164608000,"data_stored":4164280265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":368703946},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542310,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4149768,"kb_used_data":3793912,"kb_used_omap":1,"kb_used_meta":355838,"kb_avail":90222072,"statfs":{"total":96636764160,"available":92387401728,"internally_reserved":0,"allocated":3884965888,"data_stored":3884692848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364378565},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542308,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4140896,"kb_used_data":3839056,"kb_used_omap":1,"kb_used_meta":301822,"kb_avail":90230944,"statfs":{"total":96636764160,"available":92396486656,"internally_reserved":0,"allocated":3931193344,"data_stored":3930920211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":309066182},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575013,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2446212,"kb_used_data":2212468,"kb_used_omap":1,"kb_used_meta":233726,"kb_avail":91925628,"statfs":{"total":96636764160,"available":94131843072,"internally_reserved":0,"allocated":2265567232,"data_stored":2265299110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":239335877},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1409024,"data_stored":1391000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":48187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8458240,"data_stored":8437943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4210688,"data_stored":4195018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4210688,"data_stored":4195654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":9801728,"data_stored":9780846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3884539904,"data_stored":3884393932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2361716736,"data_stored":2361640968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2265264128,"data_stored":2265165640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3934777344,"data_stored":3934648445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3926749184,"data_stored":3926639910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3884711936,"data_stored":3884606187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2437623808,"data_stored":2437522964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4153978880,"data_stored":4153823768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-31T12:29:15.516 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:29:15.516 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:29:15.517 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-08-31T12:15:56.265088+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=15, tm_sec=56, tm_wday=5, tm_yday=244, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=31, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=5, tm_yday=244, tm_isdst=0) 2024-08-31T12:29:15.518 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-08-31T12:29:28.454 DEBUG:teuthology.orchestra.run.smithi019:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:29:28.462 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:29:35.518 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-31T12:29:35.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.646+0000 7ff92e990640 1 -- 172.21.15.19:0/511268614 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff92806e0e0 msgr2=0x7ff9280c9850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:35.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.646+0000 7ff92e990640 1 --2- 172.21.15.19:0/511268614 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff92806e0e0 0x7ff9280c9850 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ff91c005fd0 tx=0x7ff91c02a670 comp rx=0 tx=0).stop 2024-08-31T12:29:35.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.646+0000 7ff92e990640 1 -- 172.21.15.19:0/511268614 shutdown_connections 2024-08-31T12:29:35.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.646+0000 7ff92e990640 1 --2- 172.21.15.19:0/511268614 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff928155300 0x7ff92806d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:35.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.646+0000 7ff92e990640 1 --2- 172.21.15.19:0/511268614 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff92806e0e0 0x7ff9280c9850 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:35.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.646+0000 7ff92e990640 1 -- 172.21.15.19:0/511268614 >> 172.21.15.19:0/511268614 conn(0x7ff9280690d0 msgr2=0x7ff9280d0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:29:35.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.646+0000 7ff92e990640 1 -- 172.21.15.19:0/511268614 shutdown_connections 2024-08-31T12:29:35.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.646+0000 7ff92e990640 1 -- 172.21.15.19:0/511268614 wait complete. 2024-08-31T12:29:35.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.647+0000 7ff92e990640 1 Processor -- start 2024-08-31T12:29:35.650 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.648+0000 7ff92e990640 1 -- start start 2024-08-31T12:29:35.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.649+0000 7ff92e990640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff92806e0e0 0x7ff9281e5c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:29:35.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.649+0000 7ff92e990640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff9280c8a40 0x7ff9281f1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:29:35.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.649+0000 7ff92e990640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff928155300 0x7ff928157090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:29:35.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.649+0000 7ff92e990640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7ff9280c8330 con 0x7ff92806e0e0 2024-08-31T12:29:35.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.649+0000 7ff92e990640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff92815b250 con 0x7ff928155300 2024-08-31T12:29:35.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.649+0000 7ff92e990640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7ff9280c8610 con 0x7ff9280c8a40 2024-08-31T12:29:35.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.649+0000 7ff927fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff92806e0e0 0x7ff9281e5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:29:35.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.649+0000 7ff927fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff92806e0e0 0x7ff9281e5c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:53312/0 (socket says 172.21.15.19:53312) 2024-08-31T12:29:35.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.649+0000 7ff927fff640 1 -- 172.21.15.19:0/2692138876 learned_addr learned my addr 172.21.15.19:0/2692138876 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:29:35.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.649+0000 7ff9277fe640 1 --2- 172.21.15.19:0/2692138876 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff9280c8a40 0x7ff9281f1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:29:35.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.649+0000 7ff92cf2d640 1 --2- 172.21.15.19:0/2692138876 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff928155300 0x7ff928157090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:29:35.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.649+0000 7ff927fff640 1 -- 172.21.15.19:0/2692138876 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff9280c8a40 msgr2=0x7ff9281f1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:35.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.650+0000 7ff927fff640 1 --2- 172.21.15.19:0/2692138876 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff9280c8a40 0x7ff9281f1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:35.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.650+0000 7ff927fff640 1 -- 172.21.15.19:0/2692138876 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff928155300 msgr2=0x7ff928157090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:35.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.650+0000 7ff927fff640 1 --2- 172.21.15.19:0/2692138876 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff928155300 0x7ff928157090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:35.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.650+0000 7ff927fff640 1 -- 172.21.15.19:0/2692138876 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9281f2260 con 0x7ff92806e0e0 2024-08-31T12:29:35.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.650+0000 7ff927fff640 1 --2- 172.21.15.19:0/2692138876 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff92806e0e0 0x7ff9281e5c30 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7ff91800c3c0 tx=0x7ff91800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:29:35.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.650+0000 7ff924ff9640 1 -- 172.21.15.19:0/2692138876 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff918017070 con 0x7ff92806e0e0 2024-08-31T12:29:35.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.650+0000 7ff924ff9640 1 -- 172.21.15.19:0/2692138876 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff9180053c0 con 0x7ff92806e0e0 2024-08-31T12:29:35.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.650+0000 7ff92e990640 1 -- 172.21.15.19:0/2692138876 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9281e5670 con 0x7ff92806e0e0 2024-08-31T12:29:35.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.650+0000 7ff924ff9640 1 -- 172.21.15.19:0/2692138876 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff918003850 con 0x7ff92806e0e0 2024-08-31T12:29:35.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.650+0000 7ff92e990640 1 -- 172.21.15.19:0/2692138876 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9281e6560 con 0x7ff92806e0e0 2024-08-31T12:29:35.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.651+0000 7ff92e990640 1 -- 172.21.15.19:0/2692138876 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff92815a980 con 0x7ff92806e0e0 2024-08-31T12:29:35.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.652+0000 7ff924ff9640 1 -- 172.21.15.19:0/2692138876 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7ff91801a070 con 0x7ff92806e0e0 2024-08-31T12:29:35.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.652+0000 7ff924ff9640 1 --2- 172.21.15.19:0/2692138876 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff914077a50 0x7ff914079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:29:35.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.653+0000 7ff924ff9640 1 -- 172.21.15.19:0/2692138876 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7ff918098a20 con 0x7ff92806e0e0 2024-08-31T12:29:35.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.653+0000 7ff9277fe640 1 --2- 172.21.15.19:0/2692138876 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff914077a50 0x7ff914079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:29:35.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.653+0000 7ff9277fe640 1 --2- 172.21.15.19:0/2692138876 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff914077a50 0x7ff914079f60 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff91c033000 tx=0x7ff91c004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:29:35.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.657+0000 7ff924ff9640 1 -- 172.21.15.19:0/2692138876 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7ff9180610b0 con 0x7ff92806e0e0 2024-08-31T12:29:35.839 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.838+0000 7ff92e990640 1 -- 172.21.15.19:0/2692138876 --> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff9280ccf20 con 0x7ff914077a50 2024-08-31T12:29:35.844 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.843+0000 7ff924ff9640 1 -- 172.21.15.19:0/2692138876 <== mgr.4100 v2:172.21.15.43:6800/2824336655 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+255353 (secure 0 0 0) 0x7ff9280ccf20 con 0x7ff914077a50 2024-08-31T12:29:35.848 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:29:35.849 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-08-31T12:29:35.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.846+0000 7ff92e990640 1 -- 172.21.15.19:0/2692138876 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff914077a50 msgr2=0x7ff914079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:35.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.846+0000 7ff92e990640 1 --2- 172.21.15.19:0/2692138876 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff914077a50 0x7ff914079f60 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff91c033000 tx=0x7ff91c004730 comp rx=0 tx=0).stop 2024-08-31T12:29:35.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.846+0000 7ff92e990640 1 -- 172.21.15.19:0/2692138876 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff92806e0e0 msgr2=0x7ff9281e5c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:35.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.846+0000 7ff92e990640 1 --2- 172.21.15.19:0/2692138876 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff92806e0e0 0x7ff9281e5c30 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7ff91800c3c0 tx=0x7ff91800c880 comp rx=0 tx=0).stop 2024-08-31T12:29:35.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.846+0000 7ff92e990640 1 -- 172.21.15.19:0/2692138876 shutdown_connections 2024-08-31T12:29:35.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.846+0000 7ff92e990640 1 --2- 172.21.15.19:0/2692138876 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff914077a50 0x7ff914079f60 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:35.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.846+0000 7ff92e990640 1 --2- 172.21.15.19:0/2692138876 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff928155300 0x7ff928157090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:35.852 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.846+0000 7ff92e990640 1 --2- 172.21.15.19:0/2692138876 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff9280c8a40 0x7ff9281f1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:35.852 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.846+0000 7ff92e990640 1 --2- 172.21.15.19:0/2692138876 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff92806e0e0 0x7ff9281e5c30 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:35.852 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.846+0000 7ff92e990640 1 -- 172.21.15.19:0/2692138876 >> 172.21.15.19:0/2692138876 conn(0x7ff9280690d0 msgr2=0x7ff928158de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:29:35.852 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.846+0000 7ff92e990640 1 -- 172.21.15.19:0/2692138876 shutdown_connections 2024-08-31T12:29:35.852 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:35.846+0000 7ff92e990640 1 -- 172.21.15.19:0/2692138876 wait complete. 2024-08-31T12:29:35.859 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":432,"stamp":"2024-08-31T12:29:34.720082+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":13438846028,"num_objects":3466,"num_object_clones":0,"num_object_copies":6932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3466,"num_whiteouts":0,"num_read":96,"num_read_kb":119,"num_write":921575,"num_write_kb":16804431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":66114,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":190456,"ondisk_log_size":190456,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":28671600,"kb_used_data":26250608,"kb_used_omap":12,"kb_used_meta":2420851,"kb_avail":726303120,"statfs":{"total":773094113280,"available":743734394880,"internally_reserved":0,"allocated":26880622592,"data_stored":26878355784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":2478951985},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003883"},"pg_stats":[{"pgid":"3.3f","version":"23'2779","reported_seq":2811,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:12.332910+0000","last_change":"2024-08-31T12:29:12.332910+0000","last_active":"2024-08-31T12:29:12.332910+0000","last_peered":"2024-08-31T12:29:12.332910+0000","last_clean":"2024-08-31T12:29:12.332910+0000","last_became_active":"2024-08-31T12:15:57.306148+0000","last_became_peered":"2024-08-31T12:15:57.306148+0000","last_unstale":"2024-08-31T12:29:12.332910+0000","last_undegraded":"2024-08-31T12:29:12.332910+0000","last_fullsized":"2024-08-31T12:29:12.332910+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2779","last_scrub_stamp":"2024-08-31T12:29:12.332764+0000","last_deep_scrub":"23'2779","last_deep_scrub_stamp":"2024-08-31T12:29:12.332764+0000","last_clean_scrub_stamp":"2024-08-31T12:29:12.332764+0000","objects_scrubbed":47,"log_size":2779,"log_dups_size":0,"ondisk_log_size":2779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:21:52.819356+0000","scrub_duration":290,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180377297,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13788,"num_write_kb":231441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"23'3518","reported_seq":3551,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:04.479860+0000","last_change":"2024-08-31T12:29:04.479860+0000","last_active":"2024-08-31T12:29:04.479860+0000","last_peered":"2024-08-31T12:29:04.479860+0000","last_clean":"2024-08-31T12:29:04.479860+0000","last_became_active":"2024-08-31T12:15:57.306940+0000","last_became_peered":"2024-08-31T12:15:57.306940+0000","last_unstale":"2024-08-31T12:29:04.479860+0000","last_undegraded":"2024-08-31T12:29:04.479860+0000","last_fullsized":"2024-08-31T12:29:04.479860+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3518","last_scrub_stamp":"2024-08-31T12:29:04.479738+0000","last_deep_scrub":"23'3518","last_deep_scrub_stamp":"2024-08-31T12:29:04.479738+0000","last_clean_scrub_stamp":"2024-08-31T12:29:04.479738+0000","objects_scrubbed":66,"log_size":3518,"log_dups_size":0,"ondisk_log_size":3518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:20:30.975452+0000","scrub_duration":591,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264252554,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17611,"num_write_kb":328696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"23'3196","reported_seq":3229,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:01.068808+0000","last_change":"2024-08-31T12:29:01.068808+0000","last_active":"2024-08-31T12:29:01.068808+0000","last_peered":"2024-08-31T12:29:01.068808+0000","last_clean":"2024-08-31T12:29:01.068808+0000","last_became_active":"2024-08-31T12:15:57.305795+0000","last_became_peered":"2024-08-31T12:15:57.305795+0000","last_unstale":"2024-08-31T12:29:01.068808+0000","last_undegraded":"2024-08-31T12:29:01.068808+0000","last_fullsized":"2024-08-31T12:29:01.068808+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3196","last_scrub_stamp":"2024-08-31T12:29:01.068596+0000","last_deep_scrub":"23'3196","last_deep_scrub_stamp":"2024-08-31T12:29:01.068596+0000","last_clean_scrub_stamp":"2024-08-31T12:29:01.068596+0000","objects_scrubbed":55,"log_size":3196,"log_dups_size":0,"ondisk_log_size":3196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:43:43.076508+0000","scrub_duration":768,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213942725,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15777,"num_write_kb":272211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"23'2838","reported_seq":2870,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:07.638189+0000","last_change":"2024-08-31T12:29:07.638189+0000","last_active":"2024-08-31T12:29:07.638189+0000","last_peered":"2024-08-31T12:29:07.638189+0000","last_clean":"2024-08-31T12:29:07.638189+0000","last_became_active":"2024-08-31T12:15:57.306957+0000","last_became_peered":"2024-08-31T12:15:57.306957+0000","last_unstale":"2024-08-31T12:29:07.638189+0000","last_undegraded":"2024-08-31T12:29:07.638189+0000","last_fullsized":"2024-08-31T12:29:07.638189+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2838","last_scrub_stamp":"2024-08-31T12:29:07.638081+0000","last_deep_scrub":"23'2838","last_deep_scrub_stamp":"2024-08-31T12:29:07.638081+0000","last_clean_scrub_stamp":"2024-08-31T12:29:07.638081+0000","objects_scrubbed":45,"log_size":2838,"log_dups_size":0,"ondisk_log_size":2838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:02:16.520446+0000","scrub_duration":366,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14240,"num_write_kb":241389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"23'3348","reported_seq":3381,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:10.480204+0000","last_change":"2024-08-31T12:29:10.480204+0000","last_active":"2024-08-31T12:29:10.480204+0000","last_peered":"2024-08-31T12:29:10.480204+0000","last_clean":"2024-08-31T12:29:10.480204+0000","last_became_active":"2024-08-31T12:15:57.317079+0000","last_became_peered":"2024-08-31T12:15:57.317079+0000","last_unstale":"2024-08-31T12:29:10.480204+0000","last_undegraded":"2024-08-31T12:29:10.480204+0000","last_fullsized":"2024-08-31T12:29:10.480204+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3348","last_scrub_stamp":"2024-08-31T12:29:10.480111+0000","last_deep_scrub":"23'3348","last_deep_scrub_stamp":"2024-08-31T12:29:10.480111+0000","last_clean_scrub_stamp":"2024-08-31T12:29:10.480111+0000","objects_scrubbed":67,"log_size":3348,"log_dups_size":0,"ondisk_log_size":3348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:54:08.477510+0000","scrub_duration":444,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247474530,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15722,"num_write_kb":304743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"23'2562","reported_seq":2594,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:05.561895+0000","last_change":"2024-08-31T12:29:05.561895+0000","last_active":"2024-08-31T12:29:05.561895+0000","last_peered":"2024-08-31T12:29:05.561895+0000","last_clean":"2024-08-31T12:29:05.561895+0000","last_became_active":"2024-08-31T12:15:57.307010+0000","last_became_peered":"2024-08-31T12:15:57.307010+0000","last_unstale":"2024-08-31T12:29:05.561895+0000","last_undegraded":"2024-08-31T12:29:05.561895+0000","last_fullsized":"2024-08-31T12:29:05.561895+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2562","last_scrub_stamp":"2024-08-31T12:29:05.561780+0000","last_deep_scrub":"23'2562","last_deep_scrub_stamp":"2024-08-31T12:29:05.561780+0000","last_clean_scrub_stamp":"2024-08-31T12:29:05.561780+0000","objects_scrubbed":44,"log_size":2562,"log_dups_size":0,"ondisk_log_size":2562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:43:35.106630+0000","scrub_duration":324,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180356168,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12564,"num_write_kb":226449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"23'2683","reported_seq":2715,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:07.339239+0000","last_change":"2024-08-31T12:29:07.339239+0000","last_active":"2024-08-31T12:29:07.339239+0000","last_peered":"2024-08-31T12:29:07.339239+0000","last_clean":"2024-08-31T12:29:07.339239+0000","last_became_active":"2024-08-31T12:15:57.316504+0000","last_became_peered":"2024-08-31T12:15:57.316504+0000","last_unstale":"2024-08-31T12:29:07.339239+0000","last_undegraded":"2024-08-31T12:29:07.339239+0000","last_fullsized":"2024-08-31T12:29:07.339239+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2683","last_scrub_stamp":"2024-08-31T12:29:07.339100+0000","last_deep_scrub":"23'2683","last_deep_scrub_stamp":"2024-08-31T12:29:07.339100+0000","last_clean_scrub_stamp":"2024-08-31T12:29:07.339100+0000","objects_scrubbed":52,"log_size":2683,"log_dups_size":0,"ondisk_log_size":2683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:16:32.863297+0000","scrub_duration":370,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201398355,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13306,"num_write_kb":250051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"23'3222","reported_seq":3248,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:03.296289+0000","last_change":"2024-08-31T12:29:03.296217+0000","last_active":"2024-08-31T12:29:03.296289+0000","last_peered":"2024-08-31T12:29:03.296289+0000","last_clean":"2024-08-31T12:29:03.296289+0000","last_became_active":"2024-08-31T12:16:40.694165+0000","last_became_peered":"2024-08-31T12:16:40.694165+0000","last_unstale":"2024-08-31T12:29:03.296289+0000","last_undegraded":"2024-08-31T12:29:03.296289+0000","last_fullsized":"2024-08-31T12:29:03.296289+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3222","last_scrub_stamp":"2024-08-31T12:29:03.296130+0000","last_deep_scrub":"23'3222","last_deep_scrub_stamp":"2024-08-31T12:29:03.296130+0000","last_clean_scrub_stamp":"2024-08-31T12:29:03.296130+0000","objects_scrubbed":55,"log_size":3222,"log_dups_size":0,"ondisk_log_size":3222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:18:47.300381+0000","scrub_duration":434,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226497620,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15619,"num_write_kb":283880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.37","version":"23'3563","reported_seq":3595,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:08.348653+0000","last_change":"2024-08-31T12:29:08.348653+0000","last_active":"2024-08-31T12:29:08.348653+0000","last_peered":"2024-08-31T12:29:08.348653+0000","last_clean":"2024-08-31T12:29:08.348653+0000","last_became_active":"2024-08-31T12:15:57.316354+0000","last_became_peered":"2024-08-31T12:15:57.316354+0000","last_unstale":"2024-08-31T12:29:08.348653+0000","last_undegraded":"2024-08-31T12:29:08.348653+0000","last_fullsized":"2024-08-31T12:29:08.348653+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3563","last_scrub_stamp":"2024-08-31T12:29:08.348558+0000","last_deep_scrub":"23'3563","last_deep_scrub_stamp":"2024-08-31T12:29:08.348558+0000","last_clean_scrub_stamp":"2024-08-31T12:29:08.348558+0000","objects_scrubbed":61,"log_size":3563,"log_dups_size":0,"ondisk_log_size":3563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:21:41.849206+0000","scrub_duration":397,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251662082,"num_objects":61,"num_object_clones":0,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18100,"num_write_kb":318223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"23'3029","reported_seq":3061,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:05.887322+0000","last_change":"2024-08-31T12:29:05.887322+0000","last_active":"2024-08-31T12:29:05.887322+0000","last_peered":"2024-08-31T12:29:05.887322+0000","last_clean":"2024-08-31T12:29:05.887322+0000","last_became_active":"2024-08-31T12:15:57.310807+0000","last_became_peered":"2024-08-31T12:15:57.310807+0000","last_unstale":"2024-08-31T12:29:05.887322+0000","last_undegraded":"2024-08-31T12:29:05.887322+0000","last_fullsized":"2024-08-31T12:29:05.887322+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3029","last_scrub_stamp":"2024-08-31T12:29:05.887218+0000","last_deep_scrub":"23'3029","last_deep_scrub_stamp":"2024-08-31T12:29:05.887218+0000","last_clean_scrub_stamp":"2024-08-31T12:29:05.887218+0000","objects_scrubbed":59,"log_size":3029,"log_dups_size":0,"ondisk_log_size":3029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:18:49.891469+0000","scrub_duration":355,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234894029,"num_objects":59,"num_object_clones":0,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15073,"num_write_kb":289757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"23'2934","reported_seq":2965,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:04.828649+0000","last_change":"2024-08-31T12:29:04.828649+0000","last_active":"2024-08-31T12:29:04.828649+0000","last_peered":"2024-08-31T12:29:04.828649+0000","last_clean":"2024-08-31T12:29:04.828649+0000","last_became_active":"2024-08-31T12:15:57.300610+0000","last_became_peered":"2024-08-31T12:15:57.300610+0000","last_unstale":"2024-08-31T12:29:04.828649+0000","last_undegraded":"2024-08-31T12:29:04.828649+0000","last_fullsized":"2024-08-31T12:29:04.828649+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2934","last_scrub_stamp":"2024-08-31T12:29:04.828544+0000","last_deep_scrub":"23'2934","last_deep_scrub_stamp":"2024-08-31T12:29:04.828544+0000","last_clean_scrub_stamp":"2024-08-31T12:29:04.828544+0000","objects_scrubbed":48,"log_size":2934,"log_dups_size":0,"ondisk_log_size":2934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:33:45.754806+0000","scrub_duration":312,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201326592,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14139,"num_write_kb":253227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"23'2298","reported_seq":2329,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:59.361975+0000","last_change":"2024-08-31T12:28:59.361975+0000","last_active":"2024-08-31T12:28:59.361975+0000","last_peered":"2024-08-31T12:28:59.361975+0000","last_clean":"2024-08-31T12:28:59.361975+0000","last_became_active":"2024-08-31T12:15:57.302622+0000","last_became_peered":"2024-08-31T12:15:57.302622+0000","last_unstale":"2024-08-31T12:28:59.361975+0000","last_undegraded":"2024-08-31T12:28:59.361975+0000","last_fullsized":"2024-08-31T12:28:59.361975+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2298","last_scrub_stamp":"2024-08-31T12:28:59.361873+0000","last_deep_scrub":"23'2298","last_deep_scrub_stamp":"2024-08-31T12:28:59.361873+0000","last_clean_scrub_stamp":"2024-08-31T12:28:59.361873+0000","objects_scrubbed":45,"log_size":2298,"log_dups_size":0,"ondisk_log_size":2298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:53:07.427338+0000","scrub_duration":305,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176192645,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11403,"num_write_kb":217791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"23'3009","reported_seq":3041,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:02.934638+0000","last_change":"2024-08-31T12:29:02.934638+0000","last_active":"2024-08-31T12:29:02.934638+0000","last_peered":"2024-08-31T12:29:02.934638+0000","last_clean":"2024-08-31T12:29:02.934638+0000","last_became_active":"2024-08-31T12:15:57.305321+0000","last_became_peered":"2024-08-31T12:15:57.305321+0000","last_unstale":"2024-08-31T12:29:02.934638+0000","last_undegraded":"2024-08-31T12:29:02.934638+0000","last_fullsized":"2024-08-31T12:29:02.934638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3009","last_scrub_stamp":"2024-08-31T12:29:02.934538+0000","last_deep_scrub":"23'3009","last_deep_scrub_stamp":"2024-08-31T12:29:02.934538+0000","last_clean_scrub_stamp":"2024-08-31T12:29:02.934538+0000","objects_scrubbed":51,"log_size":3009,"log_dups_size":0,"ondisk_log_size":3009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:07:16.630867+0000","scrub_duration":445,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188840677,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15214,"num_write_kb":245235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"23'3184","reported_seq":3216,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:02.755088+0000","last_change":"2024-08-31T12:29:02.755088+0000","last_active":"2024-08-31T12:29:02.755088+0000","last_peered":"2024-08-31T12:29:02.755088+0000","last_clean":"2024-08-31T12:29:02.755088+0000","last_became_active":"2024-08-31T12:15:57.307013+0000","last_became_peered":"2024-08-31T12:15:57.307013+0000","last_unstale":"2024-08-31T12:29:02.755088+0000","last_undegraded":"2024-08-31T12:29:02.755088+0000","last_fullsized":"2024-08-31T12:29:02.755088+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3184","last_scrub_stamp":"2024-08-31T12:29:02.754968+0000","last_deep_scrub":"23'3184","last_deep_scrub_stamp":"2024-08-31T12:29:02.754968+0000","last_clean_scrub_stamp":"2024-08-31T12:29:02.754968+0000","objects_scrubbed":55,"log_size":3184,"log_dups_size":0,"ondisk_log_size":3184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:37:48.968997+0000","scrub_duration":515,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213986201,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15743,"num_write_kb":271989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"23'3012","reported_seq":3044,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:02.368847+0000","last_change":"2024-08-31T12:29:02.368847+0000","last_active":"2024-08-31T12:29:02.368847+0000","last_peered":"2024-08-31T12:29:02.368847+0000","last_clean":"2024-08-31T12:29:02.368847+0000","last_became_active":"2024-08-31T12:15:57.297182+0000","last_became_peered":"2024-08-31T12:15:57.297182+0000","last_unstale":"2024-08-31T12:29:02.368847+0000","last_undegraded":"2024-08-31T12:29:02.368847+0000","last_fullsized":"2024-08-31T12:29:02.368847+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3012","last_scrub_stamp":"2024-08-31T12:29:02.368743+0000","last_deep_scrub":"23'3012","last_deep_scrub_stamp":"2024-08-31T12:29:02.368743+0000","last_clean_scrub_stamp":"2024-08-31T12:29:02.368743+0000","objects_scrubbed":53,"log_size":3012,"log_dups_size":0,"ondisk_log_size":3012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:44:40.519549+0000","scrub_duration":519,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213913551,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14445,"num_write_kb":266758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"23'2592","reported_seq":2617,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:00.857087+0000","last_change":"2024-08-31T12:29:00.857087+0000","last_active":"2024-08-31T12:29:00.857087+0000","last_peered":"2024-08-31T12:29:00.857087+0000","last_clean":"2024-08-31T12:29:00.857087+0000","last_became_active":"2024-08-31T12:16:41.074773+0000","last_became_peered":"2024-08-31T12:16:41.074773+0000","last_unstale":"2024-08-31T12:29:00.857087+0000","last_undegraded":"2024-08-31T12:29:00.857087+0000","last_fullsized":"2024-08-31T12:29:00.857087+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2592","last_scrub_stamp":"2024-08-31T12:29:00.856966+0000","last_deep_scrub":"23'2592","last_deep_scrub_stamp":"2024-08-31T12:29:00.856966+0000","last_clean_scrub_stamp":"2024-08-31T12:29:00.856966+0000","objects_scrubbed":46,"log_size":2592,"log_dups_size":0,"ondisk_log_size":2592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:51:56.203878+0000","scrub_duration":627,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184599153,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12917,"num_write_kb":231981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"23'3786","reported_seq":3818,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:06.666056+0000","last_change":"2024-08-31T12:29:06.666056+0000","last_active":"2024-08-31T12:29:06.666056+0000","last_peered":"2024-08-31T12:29:06.666056+0000","last_clean":"2024-08-31T12:29:06.666056+0000","last_became_active":"2024-08-31T12:15:57.308388+0000","last_became_peered":"2024-08-31T12:15:57.308388+0000","last_unstale":"2024-08-31T12:29:06.666056+0000","last_undegraded":"2024-08-31T12:29:06.666056+0000","last_fullsized":"2024-08-31T12:29:06.666056+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3786","last_scrub_stamp":"2024-08-31T12:29:06.665964+0000","last_deep_scrub":"23'3786","last_deep_scrub_stamp":"2024-08-31T12:29:06.665964+0000","last_clean_scrub_stamp":"2024-08-31T12:29:06.665964+0000","objects_scrubbed":63,"log_size":3786,"log_dups_size":0,"ondisk_log_size":3786,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:01:28.190845+0000","scrub_duration":403,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251747648,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18708,"num_write_kb":320814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"23'2196","reported_seq":2227,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:57.484759+0000","last_change":"2024-08-31T12:28:57.484759+0000","last_active":"2024-08-31T12:28:57.484759+0000","last_peered":"2024-08-31T12:28:57.484759+0000","last_clean":"2024-08-31T12:28:57.484759+0000","last_became_active":"2024-08-31T12:15:57.299826+0000","last_became_peered":"2024-08-31T12:15:57.299826+0000","last_unstale":"2024-08-31T12:28:57.484759+0000","last_undegraded":"2024-08-31T12:28:57.484759+0000","last_fullsized":"2024-08-31T12:28:57.484759+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2196","last_scrub_stamp":"2024-08-31T12:28:57.484645+0000","last_deep_scrub":"23'2196","last_deep_scrub_stamp":"2024-08-31T12:28:57.484645+0000","last_clean_scrub_stamp":"2024-08-31T12:28:57.484645+0000","objects_scrubbed":48,"log_size":2196,"log_dups_size":0,"ondisk_log_size":2196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:45:16.612056+0000","scrub_duration":318,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167791563,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10514,"num_write_kb":205964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"23'2727","reported_seq":2760,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:09.358925+0000","last_change":"2024-08-31T12:29:09.358861+0000","last_active":"2024-08-31T12:29:09.358925+0000","last_peered":"2024-08-31T12:29:09.358925+0000","last_clean":"2024-08-31T12:29:09.358925+0000","last_became_active":"2024-08-31T12:15:57.317185+0000","last_became_peered":"2024-08-31T12:15:57.317185+0000","last_unstale":"2024-08-31T12:29:09.358925+0000","last_undegraded":"2024-08-31T12:29:09.358925+0000","last_fullsized":"2024-08-31T12:29:09.358925+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2727","last_scrub_stamp":"2024-08-31T12:29:09.358776+0000","last_deep_scrub":"23'2727","last_deep_scrub_stamp":"2024-08-31T12:29:09.358776+0000","last_clean_scrub_stamp":"2024-08-31T12:29:09.358776+0000","objects_scrubbed":52,"log_size":2727,"log_dups_size":0,"ondisk_log_size":2727,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:51:42.749382+0000","scrub_duration":366,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209762270,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13163,"num_write_kb":257524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"23'2501","reported_seq":2534,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:03.880428+0000","last_change":"2024-08-31T12:29:03.880357+0000","last_active":"2024-08-31T12:29:03.880428+0000","last_peered":"2024-08-31T12:29:03.880428+0000","last_clean":"2024-08-31T12:29:03.880428+0000","last_became_active":"2024-08-31T12:15:57.298430+0000","last_became_peered":"2024-08-31T12:15:57.298430+0000","last_unstale":"2024-08-31T12:29:03.880428+0000","last_undegraded":"2024-08-31T12:29:03.880428+0000","last_fullsized":"2024-08-31T12:29:03.880428+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2501","last_scrub_stamp":"2024-08-31T12:29:03.880275+0000","last_deep_scrub":"23'2501","last_deep_scrub_stamp":"2024-08-31T12:29:03.880275+0000","last_clean_scrub_stamp":"2024-08-31T12:29:03.880275+0000","objects_scrubbed":52,"log_size":2501,"log_dups_size":0,"ondisk_log_size":2501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:33:58.771109+0000","scrub_duration":368,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205530518,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12230,"num_write_kb":249721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"23'3007","reported_seq":3040,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:01.813513+0000","last_change":"2024-08-31T12:29:01.813452+0000","last_active":"2024-08-31T12:29:01.813513+0000","last_peered":"2024-08-31T12:29:01.813513+0000","last_clean":"2024-08-31T12:29:01.813513+0000","last_became_active":"2024-08-31T12:15:57.316870+0000","last_became_peered":"2024-08-31T12:15:57.316870+0000","last_unstale":"2024-08-31T12:29:01.813513+0000","last_undegraded":"2024-08-31T12:29:01.813513+0000","last_fullsized":"2024-08-31T12:29:01.813513+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3007","last_scrub_stamp":"2024-08-31T12:29:01.813349+0000","last_deep_scrub":"23'3007","last_deep_scrub_stamp":"2024-08-31T12:29:01.813349+0000","last_clean_scrub_stamp":"2024-08-31T12:29:01.813349+0000","objects_scrubbed":57,"log_size":3007,"log_dups_size":0,"ondisk_log_size":3007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:25:18.194613+0000","scrub_duration":546,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213923005,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14700,"num_write_kb":267872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"23'3121","reported_seq":3147,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:54.784122+0000","last_change":"2024-08-31T12:28:54.784122+0000","last_active":"2024-08-31T12:28:54.784122+0000","last_peered":"2024-08-31T12:28:54.784122+0000","last_clean":"2024-08-31T12:28:54.784122+0000","last_became_active":"2024-08-31T12:16:40.784204+0000","last_became_peered":"2024-08-31T12:16:40.784204+0000","last_unstale":"2024-08-31T12:28:54.784122+0000","last_undegraded":"2024-08-31T12:28:54.784122+0000","last_fullsized":"2024-08-31T12:28:54.784122+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3121","last_scrub_stamp":"2024-08-31T12:28:54.783999+0000","last_deep_scrub":"23'3121","last_deep_scrub_stamp":"2024-08-31T12:28:54.783999+0000","last_clean_scrub_stamp":"2024-08-31T12:28:54.783999+0000","objects_scrubbed":55,"log_size":3121,"log_dups_size":0,"ondisk_log_size":3121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:46:37.064206+0000","scrub_duration":509,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226493660,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15308,"num_write_kb":282531,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"23'2885","reported_seq":2918,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:03.884762+0000","last_change":"2024-08-31T12:29:03.884681+0000","last_active":"2024-08-31T12:29:03.884762+0000","last_peered":"2024-08-31T12:29:03.884762+0000","last_clean":"2024-08-31T12:29:03.884762+0000","last_became_active":"2024-08-31T12:15:57.303401+0000","last_became_peered":"2024-08-31T12:15:57.303401+0000","last_unstale":"2024-08-31T12:29:03.884762+0000","last_undegraded":"2024-08-31T12:29:03.884762+0000","last_fullsized":"2024-08-31T12:29:03.884762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2885","last_scrub_stamp":"2024-08-31T12:29:03.884604+0000","last_deep_scrub":"23'2885","last_deep_scrub_stamp":"2024-08-31T12:29:03.884604+0000","last_clean_scrub_stamp":"2024-08-31T12:29:03.884604+0000","objects_scrubbed":50,"log_size":2885,"log_dups_size":0,"ondisk_log_size":2885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:30:33.314909+0000","scrub_duration":417,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205588184,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14070,"num_write_kb":257066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"23'2521","reported_seq":2553,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:04.637461+0000","last_change":"2024-08-31T12:29:04.637461+0000","last_active":"2024-08-31T12:29:04.637461+0000","last_peered":"2024-08-31T12:29:04.637461+0000","last_clean":"2024-08-31T12:29:04.637461+0000","last_became_active":"2024-08-31T12:15:57.305075+0000","last_became_peered":"2024-08-31T12:15:57.305075+0000","last_unstale":"2024-08-31T12:29:04.637461+0000","last_undegraded":"2024-08-31T12:29:04.637461+0000","last_fullsized":"2024-08-31T12:29:04.637461+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2521","last_scrub_stamp":"2024-08-31T12:29:04.637273+0000","last_deep_scrub":"23'2521","last_deep_scrub_stamp":"2024-08-31T12:29:04.637273+0000","last_clean_scrub_stamp":"2024-08-31T12:29:04.637273+0000","objects_scrubbed":47,"log_size":2521,"log_dups_size":0,"ondisk_log_size":2521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:47:40.577786+0000","scrub_duration":436,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184555785,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12290,"num_write_kb":229419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"23'3229","reported_seq":3262,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:02.129504+0000","last_change":"2024-08-31T12:29:02.129504+0000","last_active":"2024-08-31T12:29:02.129504+0000","last_peered":"2024-08-31T12:29:02.129504+0000","last_clean":"2024-08-31T12:29:02.129504+0000","last_became_active":"2024-08-31T12:15:57.299861+0000","last_became_peered":"2024-08-31T12:15:57.299861+0000","last_unstale":"2024-08-31T12:29:02.129504+0000","last_undegraded":"2024-08-31T12:29:02.129504+0000","last_fullsized":"2024-08-31T12:29:02.129504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3229","last_scrub_stamp":"2024-08-31T12:29:02.129410+0000","last_deep_scrub":"23'3229","last_deep_scrub_stamp":"2024-08-31T12:29:02.129410+0000","last_clean_scrub_stamp":"2024-08-31T12:29:02.129410+0000","objects_scrubbed":63,"log_size":3229,"log_dups_size":0,"ondisk_log_size":3229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:02:39.277974+0000","scrub_duration":629,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243322604,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15709,"num_write_kb":300530,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"23'2792","reported_seq":2824,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:02.900464+0000","last_change":"2024-08-31T12:29:02.900464+0000","last_active":"2024-08-31T12:29:02.900464+0000","last_peered":"2024-08-31T12:29:02.900464+0000","last_clean":"2024-08-31T12:29:02.900464+0000","last_became_active":"2024-08-31T12:15:57.308189+0000","last_became_peered":"2024-08-31T12:15:57.308189+0000","last_unstale":"2024-08-31T12:29:02.900464+0000","last_undegraded":"2024-08-31T12:29:02.900464+0000","last_fullsized":"2024-08-31T12:29:02.900464+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2792","last_scrub_stamp":"2024-08-31T12:29:02.900351+0000","last_deep_scrub":"23'2792","last_deep_scrub_stamp":"2024-08-31T12:29:02.900351+0000","last_clean_scrub_stamp":"2024-08-31T12:29:02.900351+0000","objects_scrubbed":48,"log_size":2792,"log_dups_size":0,"ondisk_log_size":2792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:32:40.539776+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189169648,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13404,"num_write_kb":238420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"23'3142","reported_seq":3175,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:01.447679+0000","last_change":"2024-08-31T12:29:01.447679+0000","last_active":"2024-08-31T12:29:01.447679+0000","last_peered":"2024-08-31T12:29:01.447679+0000","last_clean":"2024-08-31T12:29:01.447679+0000","last_became_active":"2024-08-31T12:15:57.304514+0000","last_became_peered":"2024-08-31T12:15:57.304514+0000","last_unstale":"2024-08-31T12:29:01.447679+0000","last_undegraded":"2024-08-31T12:29:01.447679+0000","last_fullsized":"2024-08-31T12:29:01.447679+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3142","last_scrub_stamp":"2024-08-31T12:29:01.447580+0000","last_deep_scrub":"23'3142","last_deep_scrub_stamp":"2024-08-31T12:29:01.447580+0000","last_clean_scrub_stamp":"2024-08-31T12:29:01.447580+0000","objects_scrubbed":54,"log_size":3142,"log_dups_size":0,"ondisk_log_size":3142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:37:47.661609+0000","scrub_duration":645,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222298838,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15326,"num_write_kb":278485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"23'2621","reported_seq":2653,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:01.176336+0000","last_change":"2024-08-31T12:29:01.176264+0000","last_active":"2024-08-31T12:29:01.176336+0000","last_peered":"2024-08-31T12:29:01.176336+0000","last_clean":"2024-08-31T12:29:01.176336+0000","last_became_active":"2024-08-31T12:15:57.298766+0000","last_became_peered":"2024-08-31T12:15:57.298766+0000","last_unstale":"2024-08-31T12:29:01.176336+0000","last_undegraded":"2024-08-31T12:29:01.176336+0000","last_fullsized":"2024-08-31T12:29:01.176336+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2621","last_scrub_stamp":"2024-08-31T12:29:01.176105+0000","last_deep_scrub":"23'2621","last_deep_scrub_stamp":"2024-08-31T12:29:01.176105+0000","last_clean_scrub_stamp":"2024-08-31T12:29:01.176105+0000","objects_scrubbed":47,"log_size":2621,"log_dups_size":0,"ondisk_log_size":2621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:07:36.116413+0000","scrub_duration":653,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188773348,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12878,"num_write_kb":235916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"23'2373","reported_seq":2405,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:59.672533+0000","last_change":"2024-08-31T12:28:59.672533+0000","last_active":"2024-08-31T12:28:59.672533+0000","last_peered":"2024-08-31T12:28:59.672533+0000","last_clean":"2024-08-31T12:28:59.672533+0000","last_became_active":"2024-08-31T12:15:57.309440+0000","last_became_peered":"2024-08-31T12:15:57.309440+0000","last_unstale":"2024-08-31T12:28:59.672533+0000","last_undegraded":"2024-08-31T12:28:59.672533+0000","last_fullsized":"2024-08-31T12:28:59.672533+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2373","last_scrub_stamp":"2024-08-31T12:28:59.672418+0000","last_deep_scrub":"23'2373","last_deep_scrub_stamp":"2024-08-31T12:28:59.672418+0000","last_clean_scrub_stamp":"2024-08-31T12:28:59.672418+0000","objects_scrubbed":45,"log_size":2373,"log_dups_size":0,"ondisk_log_size":2373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:36:49.635342+0000","scrub_duration":387,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180357122,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11238,"num_write_kb":221098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"23'3509","reported_seq":3542,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:03.766589+0000","last_change":"2024-08-31T12:29:03.766589+0000","last_active":"2024-08-31T12:29:03.766589+0000","last_peered":"2024-08-31T12:29:03.766589+0000","last_clean":"2024-08-31T12:29:03.766589+0000","last_became_active":"2024-08-31T12:15:57.305878+0000","last_became_peered":"2024-08-31T12:15:57.305878+0000","last_unstale":"2024-08-31T12:29:03.766589+0000","last_undegraded":"2024-08-31T12:29:03.766589+0000","last_fullsized":"2024-08-31T12:29:03.766589+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3509","last_scrub_stamp":"2024-08-31T12:29:03.766459+0000","last_deep_scrub":"23'3509","last_deep_scrub_stamp":"2024-08-31T12:29:03.766459+0000","last_clean_scrub_stamp":"2024-08-31T12:29:03.766459+0000","objects_scrubbed":62,"log_size":3509,"log_dups_size":0,"ondisk_log_size":3509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:03:30.959671+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251668879,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17281,"num_write_kb":314954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"23'2933","reported_seq":2966,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:56.696020+0000","last_change":"2024-08-31T12:28:56.696020+0000","last_active":"2024-08-31T12:28:56.696020+0000","last_peered":"2024-08-31T12:28:56.696020+0000","last_clean":"2024-08-31T12:28:56.696020+0000","last_became_active":"2024-08-31T12:15:57.307056+0000","last_became_peered":"2024-08-31T12:15:57.307056+0000","last_unstale":"2024-08-31T12:28:56.696020+0000","last_undegraded":"2024-08-31T12:28:56.696020+0000","last_fullsized":"2024-08-31T12:28:56.696020+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2933","last_scrub_stamp":"2024-08-31T12:28:56.695930+0000","last_deep_scrub":"23'2933","last_deep_scrub_stamp":"2024-08-31T12:28:56.695930+0000","last_clean_scrub_stamp":"2024-08-31T12:28:56.695930+0000","objects_scrubbed":53,"log_size":2933,"log_dups_size":0,"ondisk_log_size":2933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:39:28.074678+0000","scrub_duration":507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222298112,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14386,"num_write_kb":274682,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"23'3186","reported_seq":3219,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:02.187805+0000","last_change":"2024-08-31T12:29:02.187805+0000","last_active":"2024-08-31T12:29:02.187805+0000","last_peered":"2024-08-31T12:29:02.187805+0000","last_clean":"2024-08-31T12:29:02.187805+0000","last_became_active":"2024-08-31T12:15:57.313463+0000","last_became_peered":"2024-08-31T12:15:57.313463+0000","last_unstale":"2024-08-31T12:29:02.187805+0000","last_undegraded":"2024-08-31T12:29:02.187805+0000","last_fullsized":"2024-08-31T12:29:02.187805+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3186","last_scrub_stamp":"2024-08-31T12:29:02.187686+0000","last_deep_scrub":"23'3186","last_deep_scrub_stamp":"2024-08-31T12:29:02.187686+0000","last_clean_scrub_stamp":"2024-08-31T12:29:02.187686+0000","objects_scrubbed":64,"log_size":3186,"log_dups_size":0,"ondisk_log_size":3186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:43:44.195597+0000","scrub_duration":668,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260049613,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15263,"num_write_kb":315030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"23'2333","reported_seq":2364,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:58.487797+0000","last_change":"2024-08-31T12:28:58.487797+0000","last_active":"2024-08-31T12:28:58.487797+0000","last_peered":"2024-08-31T12:28:58.487797+0000","last_clean":"2024-08-31T12:28:58.487797+0000","last_became_active":"2024-08-31T12:15:57.292981+0000","last_became_peered":"2024-08-31T12:15:57.292981+0000","last_unstale":"2024-08-31T12:28:58.487797+0000","last_undegraded":"2024-08-31T12:28:58.487797+0000","last_fullsized":"2024-08-31T12:28:58.487797+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2333","last_scrub_stamp":"2024-08-31T12:28:58.487698+0000","last_deep_scrub":"23'2333","last_deep_scrub_stamp":"2024-08-31T12:28:58.487698+0000","last_clean_scrub_stamp":"2024-08-31T12:28:58.487698+0000","objects_scrubbed":46,"log_size":2333,"log_dups_size":0,"ondisk_log_size":2333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:18:39.664026+0000","scrub_duration":388,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180377946,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11142,"num_write_kb":220690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"23'572","reported_seq":604,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:54.576817+0000","last_change":"2024-08-31T12:28:54.576817+0000","last_active":"2024-08-31T12:28:54.576817+0000","last_peered":"2024-08-31T12:28:54.576817+0000","last_clean":"2024-08-31T12:28:54.576817+0000","last_became_active":"2024-08-31T12:15:56.293164+0000","last_became_peered":"2024-08-31T12:15:56.293164+0000","last_unstale":"2024-08-31T12:28:54.576817+0000","last_undegraded":"2024-08-31T12:28:54.576817+0000","last_fullsized":"2024-08-31T12:28:54.576817+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'572","last_scrub_stamp":"2024-08-31T12:28:54.576677+0000","last_deep_scrub":"23'572","last_deep_scrub_stamp":"2024-08-31T12:28:54.576677+0000","last_clean_scrub_stamp":"2024-08-31T12:28:54.576677+0000","objects_scrubbed":3,"log_size":572,"log_dups_size":0,"ondisk_log_size":572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:12:36.474157+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4195508,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":567,"num_write_kb":4211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"23'2437","reported_seq":2468,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:58.755425+0000","last_change":"2024-08-31T12:28:58.755425+0000","last_active":"2024-08-31T12:28:58.755425+0000","last_peered":"2024-08-31T12:28:58.755425+0000","last_clean":"2024-08-31T12:28:58.755425+0000","last_became_active":"2024-08-31T12:15:57.312360+0000","last_became_peered":"2024-08-31T12:15:57.312360+0000","last_unstale":"2024-08-31T12:28:58.755425+0000","last_undegraded":"2024-08-31T12:28:58.755425+0000","last_fullsized":"2024-08-31T12:28:58.755425+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2437","last_scrub_stamp":"2024-08-31T12:28:58.755318+0000","last_deep_scrub":"23'2437","last_deep_scrub_stamp":"2024-08-31T12:28:58.755318+0000","last_clean_scrub_stamp":"2024-08-31T12:28:58.755318+0000","objects_scrubbed":41,"log_size":2437,"log_dups_size":0,"ondisk_log_size":2437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:30:07.510938+0000","scrub_duration":425,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163584295,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11807,"num_write_kb":207035,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"23'7","reported_seq":39,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:58.542622+0000","last_change":"2024-08-31T12:28:58.542622+0000","last_active":"2024-08-31T12:28:58.542622+0000","last_peered":"2024-08-31T12:28:58.542622+0000","last_clean":"2024-08-31T12:28:58.542622+0000","last_became_active":"2024-08-31T12:15:56.283147+0000","last_became_peered":"2024-08-31T12:15:56.283147+0000","last_unstale":"2024-08-31T12:28:58.542622+0000","last_undegraded":"2024-08-31T12:28:58.542622+0000","last_fullsized":"2024-08-31T12:28:58.542622+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'7","last_scrub_stamp":"2024-08-31T12:28:58.542515+0000","last_deep_scrub":"23'7","last_deep_scrub_stamp":"2024-08-31T12:28:58.542515+0000","last_clean_scrub_stamp":"2024-08-31T12:28:58.542515+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:22:30.474634+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5538,"num_omap_keys":11,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"23'5","reported_seq":37,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:53.595705+0000","last_change":"2024-08-31T12:28:53.595705+0000","last_active":"2024-08-31T12:28:53.595705+0000","last_peered":"2024-08-31T12:28:53.595705+0000","last_clean":"2024-08-31T12:28:53.595705+0000","last_became_active":"2024-08-31T12:15:56.290835+0000","last_became_peered":"2024-08-31T12:15:56.290835+0000","last_unstale":"2024-08-31T12:28:53.595705+0000","last_undegraded":"2024-08-31T12:28:53.595705+0000","last_fullsized":"2024-08-31T12:28:53.595705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5","last_scrub_stamp":"2024-08-31T12:28:53.595609+0000","last_deep_scrub":"23'5","last_deep_scrub_stamp":"2024-08-31T12:28:53.595609+0000","last_clean_scrub_stamp":"2024-08-31T12:28:53.595609+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:27:35.816726+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"23'2595","reported_seq":2627,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:05.944823+0000","last_change":"2024-08-31T12:29:05.944823+0000","last_active":"2024-08-31T12:29:05.944823+0000","last_peered":"2024-08-31T12:29:05.944823+0000","last_clean":"2024-08-31T12:29:05.944823+0000","last_became_active":"2024-08-31T12:15:57.306962+0000","last_became_peered":"2024-08-31T12:15:57.306962+0000","last_unstale":"2024-08-31T12:29:05.944823+0000","last_undegraded":"2024-08-31T12:29:05.944823+0000","last_fullsized":"2024-08-31T12:29:05.944823+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2595","last_scrub_stamp":"2024-08-31T12:29:05.944705+0000","last_deep_scrub":"23'2595","last_deep_scrub_stamp":"2024-08-31T12:29:05.944705+0000","last_clean_scrub_stamp":"2024-08-31T12:29:05.944705+0000","objects_scrubbed":52,"log_size":2595,"log_dups_size":0,"ondisk_log_size":2595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:35:15.992000+0000","scrub_duration":440,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205541461,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12767,"num_write_kb":251825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"23'3","reported_seq":35,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:58.192505+0000","last_change":"2024-08-31T12:28:58.192505+0000","last_active":"2024-08-31T12:28:58.192505+0000","last_peered":"2024-08-31T12:28:58.192505+0000","last_clean":"2024-08-31T12:28:58.192505+0000","last_became_active":"2024-08-31T12:15:56.287552+0000","last_became_peered":"2024-08-31T12:15:56.287552+0000","last_unstale":"2024-08-31T12:28:58.192505+0000","last_undegraded":"2024-08-31T12:28:58.192505+0000","last_fullsized":"2024-08-31T12:28:58.192505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3","last_scrub_stamp":"2024-08-31T12:28:58.192377+0000","last_deep_scrub":"23'3","last_deep_scrub_stamp":"2024-08-31T12:28:58.192377+0000","last_clean_scrub_stamp":"2024-08-31T12:28:58.192377+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:02:00.301104+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8000,"num_omap_keys":16,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"23'3514","reported_seq":3546,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:00.496904+0000","last_change":"2024-08-31T12:29:00.496904+0000","last_active":"2024-08-31T12:29:00.496904+0000","last_peered":"2024-08-31T12:29:00.496904+0000","last_clean":"2024-08-31T12:29:00.496904+0000","last_became_active":"2024-08-31T12:15:57.304700+0000","last_became_peered":"2024-08-31T12:15:57.304700+0000","last_unstale":"2024-08-31T12:29:00.496904+0000","last_undegraded":"2024-08-31T12:29:00.496904+0000","last_fullsized":"2024-08-31T12:29:00.496904+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3514","last_scrub_stamp":"2024-08-31T12:29:00.496725+0000","last_deep_scrub":"23'3514","last_deep_scrub_stamp":"2024-08-31T12:29:00.496725+0000","last_clean_scrub_stamp":"2024-08-31T12:29:00.496725+0000","objects_scrubbed":64,"log_size":3514,"log_dups_size":0,"ondisk_log_size":3514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:32:38.136150+0000","scrub_duration":733,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243316666,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17415,"num_write_kb":307366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"23'6","reported_seq":38,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:54.515829+0000","last_change":"2024-08-31T12:28:54.515829+0000","last_active":"2024-08-31T12:28:54.515829+0000","last_peered":"2024-08-31T12:28:54.515829+0000","last_clean":"2024-08-31T12:28:54.515829+0000","last_became_active":"2024-08-31T12:15:56.284558+0000","last_became_peered":"2024-08-31T12:15:56.284558+0000","last_unstale":"2024-08-31T12:28:54.515829+0000","last_undegraded":"2024-08-31T12:28:54.515829+0000","last_fullsized":"2024-08-31T12:28:54.515829+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'6","last_scrub_stamp":"2024-08-31T12:28:54.515723+0000","last_deep_scrub":"23'6","last_deep_scrub_stamp":"2024-08-31T12:28:54.515723+0000","last_clean_scrub_stamp":"2024-08-31T12:28:54.515723+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:38:57.450315+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"23'2962","reported_seq":2994,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:59.580677+0000","last_change":"2024-08-31T12:28:59.580677+0000","last_active":"2024-08-31T12:28:59.580677+0000","last_peered":"2024-08-31T12:28:59.580677+0000","last_clean":"2024-08-31T12:28:59.580677+0000","last_became_active":"2024-08-31T12:15:57.308260+0000","last_became_peered":"2024-08-31T12:15:57.308260+0000","last_unstale":"2024-08-31T12:28:59.580677+0000","last_undegraded":"2024-08-31T12:28:59.580677+0000","last_fullsized":"2024-08-31T12:28:59.580677+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2962","last_scrub_stamp":"2024-08-31T12:28:59.580567+0000","last_deep_scrub":"23'2962","last_deep_scrub_stamp":"2024-08-31T12:28:59.580567+0000","last_clean_scrub_stamp":"2024-08-31T12:28:59.580567+0000","objects_scrubbed":58,"log_size":2962,"log_dups_size":0,"ondisk_log_size":2962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:21:54.817438+0000","scrub_duration":376,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209955924,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14127,"num_write_kb":261546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"23'3","reported_seq":35,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:57.154575+0000","last_change":"2024-08-31T12:28:57.154575+0000","last_active":"2024-08-31T12:28:57.154575+0000","last_peered":"2024-08-31T12:28:57.154575+0000","last_clean":"2024-08-31T12:28:57.154575+0000","last_became_active":"2024-08-31T12:15:56.290400+0000","last_became_peered":"2024-08-31T12:15:56.290400+0000","last_unstale":"2024-08-31T12:28:57.154575+0000","last_undegraded":"2024-08-31T12:28:57.154575+0000","last_fullsized":"2024-08-31T12:28:57.154575+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3","last_scrub_stamp":"2024-08-31T12:28:57.154456+0000","last_deep_scrub":"23'3","last_deep_scrub_stamp":"2024-08-31T12:28:57.154456+0000","last_clean_scrub_stamp":"2024-08-31T12:28:57.154456+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T19:51:57.123031+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"23'2713","reported_seq":2744,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:56.748191+0000","last_change":"2024-08-31T12:28:56.748191+0000","last_active":"2024-08-31T12:28:56.748191+0000","last_peered":"2024-08-31T12:28:56.748191+0000","last_clean":"2024-08-31T12:28:56.748191+0000","last_became_active":"2024-08-31T12:15:57.310564+0000","last_became_peered":"2024-08-31T12:15:57.310564+0000","last_unstale":"2024-08-31T12:28:56.748191+0000","last_undegraded":"2024-08-31T12:28:56.748191+0000","last_fullsized":"2024-08-31T12:28:56.748191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2713","last_scrub_stamp":"2024-08-31T12:28:56.748076+0000","last_deep_scrub":"23'2713","last_deep_scrub_stamp":"2024-08-31T12:28:56.748076+0000","last_clean_scrub_stamp":"2024-08-31T12:28:56.748076+0000","objects_scrubbed":48,"log_size":2713,"log_dups_size":0,"ondisk_log_size":2713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:22:57.469228+0000","scrub_duration":438,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180410852,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13728,"num_write_kb":231123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"23'22","reported_seq":54,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:57.738642+0000","last_change":"2024-08-31T12:28:57.738642+0000","last_active":"2024-08-31T12:28:57.738642+0000","last_peered":"2024-08-31T12:28:57.738642+0000","last_clean":"2024-08-31T12:28:57.738642+0000","last_became_active":"2024-08-31T12:15:56.286458+0000","last_became_peered":"2024-08-31T12:15:56.286458+0000","last_unstale":"2024-08-31T12:28:57.738642+0000","last_undegraded":"2024-08-31T12:28:57.738642+0000","last_fullsized":"2024-08-31T12:28:57.738642+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'22","last_scrub_stamp":"2024-08-31T12:28:57.738518+0000","last_deep_scrub":"23'22","last_deep_scrub_stamp":"2024-08-31T12:28:57.738518+0000","last_clean_scrub_stamp":"2024-08-31T12:28:57.738518+0000","objects_scrubbed":3,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:57:18.106754+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"23'3738","reported_seq":3770,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:57.569166+0000","last_change":"2024-08-31T12:28:57.569166+0000","last_active":"2024-08-31T12:28:57.569166+0000","last_peered":"2024-08-31T12:28:57.569166+0000","last_clean":"2024-08-31T12:28:57.569166+0000","last_became_active":"2024-08-31T12:15:57.309102+0000","last_became_peered":"2024-08-31T12:15:57.309102+0000","last_unstale":"2024-08-31T12:28:57.569166+0000","last_undegraded":"2024-08-31T12:28:57.569166+0000","last_fullsized":"2024-08-31T12:28:57.569166+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3738","last_scrub_stamp":"2024-08-31T12:28:57.569054+0000","last_deep_scrub":"23'3738","last_deep_scrub_stamp":"2024-08-31T12:28:57.569054+0000","last_clean_scrub_stamp":"2024-08-31T12:28:57.569054+0000","objects_scrubbed":69,"log_size":3738,"log_dups_size":0,"ondisk_log_size":3738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:08:32.002870+0000","scrub_duration":507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":277024326,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17723,"num_write_kb":341816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"23'322","reported_seq":354,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:55.537277+0000","last_change":"2024-08-31T12:28:55.537204+0000","last_active":"2024-08-31T12:28:55.537277+0000","last_peered":"2024-08-31T12:28:55.537277+0000","last_clean":"2024-08-31T12:28:55.537277+0000","last_became_active":"2024-08-31T12:15:56.289096+0000","last_became_peered":"2024-08-31T12:15:56.289096+0000","last_unstale":"2024-08-31T12:28:55.537277+0000","last_undegraded":"2024-08-31T12:28:55.537277+0000","last_fullsized":"2024-08-31T12:28:55.537277+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'321","last_scrub_stamp":"2024-08-31T12:28:55.537113+0000","last_deep_scrub":"23'321","last_deep_scrub_stamp":"2024-08-31T12:28:55.537113+0000","last_clean_scrub_stamp":"2024-08-31T12:28:55.537113+0000","objects_scrubbed":5,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:59:21.280703+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":318,"num_write_kb":4237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"23'2794","reported_seq":2825,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:06.314905+0000","last_change":"2024-08-31T12:29:06.314905+0000","last_active":"2024-08-31T12:29:06.314905+0000","last_peered":"2024-08-31T12:29:06.314905+0000","last_clean":"2024-08-31T12:29:06.314905+0000","last_became_active":"2024-08-31T12:15:57.317632+0000","last_became_peered":"2024-08-31T12:15:57.317632+0000","last_unstale":"2024-08-31T12:29:06.314905+0000","last_undegraded":"2024-08-31T12:29:06.314905+0000","last_fullsized":"2024-08-31T12:29:06.314905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2794","last_scrub_stamp":"2024-08-31T12:29:06.314759+0000","last_deep_scrub":"23'2794","last_deep_scrub_stamp":"2024-08-31T12:29:06.314759+0000","last_clean_scrub_stamp":"2024-08-31T12:29:06.314759+0000","objects_scrubbed":47,"log_size":2794,"log_dups_size":0,"ondisk_log_size":2794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:02:15.197124+0000","scrub_duration":343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197132288,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13667,"num_write_kb":247166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"23'10","reported_seq":42,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:51.225974+0000","last_change":"2024-08-31T12:28:51.225974+0000","last_active":"2024-08-31T12:28:51.225974+0000","last_peered":"2024-08-31T12:28:51.225974+0000","last_clean":"2024-08-31T12:28:51.225974+0000","last_became_active":"2024-08-31T12:15:56.283675+0000","last_became_peered":"2024-08-31T12:15:56.283675+0000","last_unstale":"2024-08-31T12:28:51.225974+0000","last_undegraded":"2024-08-31T12:28:51.225974+0000","last_fullsized":"2024-08-31T12:28:51.225974+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'10","last_scrub_stamp":"2024-08-31T12:28:51.225832+0000","last_deep_scrub":"23'10","last_deep_scrub_stamp":"2024-08-31T12:28:51.225832+0000","last_clean_scrub_stamp":"2024-08-31T12:28:51.225832+0000","objects_scrubbed":4,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:46:50.223589+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"23'3109","reported_seq":3140,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:54.702050+0000","last_change":"2024-08-31T12:28:54.702050+0000","last_active":"2024-08-31T12:28:54.702050+0000","last_peered":"2024-08-31T12:28:54.702050+0000","last_clean":"2024-08-31T12:28:54.702050+0000","last_became_active":"2024-08-31T12:15:57.312007+0000","last_became_peered":"2024-08-31T12:15:57.312007+0000","last_unstale":"2024-08-31T12:28:54.702050+0000","last_undegraded":"2024-08-31T12:28:54.702050+0000","last_fullsized":"2024-08-31T12:28:54.702050+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3109","last_scrub_stamp":"2024-08-31T12:28:54.701881+0000","last_deep_scrub":"23'3109","last_deep_scrub_stamp":"2024-08-31T12:28:54.701881+0000","last_clean_scrub_stamp":"2024-08-31T12:28:54.701881+0000","objects_scrubbed":48,"log_size":3109,"log_dups_size":0,"ondisk_log_size":3109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:59:20.445471+0000","scrub_duration":442,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197138377,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15100,"num_write_kb":253041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"23'6","reported_seq":38,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:56.137793+0000","last_change":"2024-08-31T12:28:56.137793+0000","last_active":"2024-08-31T12:28:56.137793+0000","last_peered":"2024-08-31T12:28:56.137793+0000","last_clean":"2024-08-31T12:28:56.137793+0000","last_became_active":"2024-08-31T12:15:56.290603+0000","last_became_peered":"2024-08-31T12:15:56.290603+0000","last_unstale":"2024-08-31T12:28:56.137793+0000","last_undegraded":"2024-08-31T12:28:56.137793+0000","last_fullsized":"2024-08-31T12:28:56.137793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'6","last_scrub_stamp":"2024-08-31T12:28:56.137715+0000","last_deep_scrub":"23'6","last_deep_scrub_stamp":"2024-08-31T12:28:56.137715+0000","last_clean_scrub_stamp":"2024-08-31T12:28:56.137715+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:38:58.476394+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10500,"num_omap_keys":21,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"23'3354","reported_seq":3386,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:03.386881+0000","last_change":"2024-08-31T12:29:03.386881+0000","last_active":"2024-08-31T12:29:03.386881+0000","last_peered":"2024-08-31T12:29:03.386881+0000","last_clean":"2024-08-31T12:29:03.386881+0000","last_became_active":"2024-08-31T12:15:57.312383+0000","last_became_peered":"2024-08-31T12:15:57.312383+0000","last_unstale":"2024-08-31T12:29:03.386881+0000","last_undegraded":"2024-08-31T12:29:03.386881+0000","last_fullsized":"2024-08-31T12:29:03.386881+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3354","last_scrub_stamp":"2024-08-31T12:29:03.386793+0000","last_deep_scrub":"23'3354","last_deep_scrub_stamp":"2024-08-31T12:29:03.386793+0000","last_clean_scrub_stamp":"2024-08-31T12:29:03.386793+0000","objects_scrubbed":60,"log_size":3354,"log_dups_size":0,"ondisk_log_size":3354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:14:18.113806+0000","scrub_duration":421,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234891518,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16277,"num_write_kb":294634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"23'8","reported_seq":40,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:56.525994+0000","last_change":"2024-08-31T12:28:56.525994+0000","last_active":"2024-08-31T12:28:56.525994+0000","last_peered":"2024-08-31T12:28:56.525994+0000","last_clean":"2024-08-31T12:28:56.525994+0000","last_became_active":"2024-08-31T12:15:56.289477+0000","last_became_peered":"2024-08-31T12:15:56.289477+0000","last_unstale":"2024-08-31T12:28:56.525994+0000","last_undegraded":"2024-08-31T12:28:56.525994+0000","last_fullsized":"2024-08-31T12:28:56.525994+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'8","last_scrub_stamp":"2024-08-31T12:28:56.525900+0000","last_deep_scrub":"23'8","last_deep_scrub_stamp":"2024-08-31T12:28:56.525900+0000","last_clean_scrub_stamp":"2024-08-31T12:28:56.525900+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:48:04.313201+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":54,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"23'2892","reported_seq":2924,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:00.155915+0000","last_change":"2024-08-31T12:29:00.155915+0000","last_active":"2024-08-31T12:29:00.155915+0000","last_peered":"2024-08-31T12:29:00.155915+0000","last_clean":"2024-08-31T12:29:00.155915+0000","last_became_active":"2024-08-31T12:15:57.312462+0000","last_became_peered":"2024-08-31T12:15:57.312462+0000","last_unstale":"2024-08-31T12:29:00.155915+0000","last_undegraded":"2024-08-31T12:29:00.155915+0000","last_fullsized":"2024-08-31T12:29:00.155915+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2892","last_scrub_stamp":"2024-08-31T12:29:00.155821+0000","last_deep_scrub":"23'2892","last_deep_scrub_stamp":"2024-08-31T12:29:00.155821+0000","last_clean_scrub_stamp":"2024-08-31T12:29:00.155821+0000","objects_scrubbed":58,"log_size":2892,"log_dups_size":0,"ondisk_log_size":2892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:30:08.911441+0000","scrub_duration":578,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222316815,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13883,"num_write_kb":272702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"23'23","reported_seq":55,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:59.250583+0000","last_change":"2024-08-31T12:28:59.250583+0000","last_active":"2024-08-31T12:28:59.250583+0000","last_peered":"2024-08-31T12:28:59.250583+0000","last_clean":"2024-08-31T12:28:59.250583+0000","last_became_active":"2024-08-31T12:15:56.285108+0000","last_became_peered":"2024-08-31T12:15:56.285108+0000","last_unstale":"2024-08-31T12:28:59.250583+0000","last_undegraded":"2024-08-31T12:28:59.250583+0000","last_fullsized":"2024-08-31T12:28:59.250583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'23","last_scrub_stamp":"2024-08-31T12:28:59.250486+0000","last_deep_scrub":"23'23","last_deep_scrub_stamp":"2024-08-31T12:28:59.250486+0000","last_clean_scrub_stamp":"2024-08-31T12:28:59.250486+0000","objects_scrubbed":8,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:00:17.841221+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"23'2902","reported_seq":2934,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:02.360939+0000","last_change":"2024-08-31T12:29:02.360939+0000","last_active":"2024-08-31T12:29:02.360939+0000","last_peered":"2024-08-31T12:29:02.360939+0000","last_clean":"2024-08-31T12:29:02.360939+0000","last_became_active":"2024-08-31T12:15:57.311368+0000","last_became_peered":"2024-08-31T12:15:57.311368+0000","last_unstale":"2024-08-31T12:29:02.360939+0000","last_undegraded":"2024-08-31T12:29:02.360939+0000","last_fullsized":"2024-08-31T12:29:02.360939+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2902","last_scrub_stamp":"2024-08-31T12:29:02.360797+0000","last_deep_scrub":"23'2902","last_deep_scrub_stamp":"2024-08-31T12:29:02.360797+0000","last_clean_scrub_stamp":"2024-08-31T12:29:02.360797+0000","objects_scrubbed":53,"log_size":2902,"log_dups_size":0,"ondisk_log_size":2902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:43:22.205823+0000","scrub_duration":425,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205547202,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14167,"num_write_kb":257592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"23'6","reported_seq":39,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:54.024401+0000","last_change":"2024-08-31T12:28:54.024330+0000","last_active":"2024-08-31T12:28:54.024401+0000","last_peered":"2024-08-31T12:28:54.024401+0000","last_clean":"2024-08-31T12:28:54.024401+0000","last_became_active":"2024-08-31T12:15:56.288289+0000","last_became_peered":"2024-08-31T12:15:56.288289+0000","last_unstale":"2024-08-31T12:28:54.024401+0000","last_undegraded":"2024-08-31T12:28:54.024401+0000","last_fullsized":"2024-08-31T12:28:54.024401+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'6","last_scrub_stamp":"2024-08-31T12:28:54.024217+0000","last_deep_scrub":"23'6","last_deep_scrub_stamp":"2024-08-31T12:28:54.024217+0000","last_clean_scrub_stamp":"2024-08-31T12:28:54.024217+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:08:22.684603+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5500,"num_omap_keys":11,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"23'2863","reported_seq":2896,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:11.364867+0000","last_change":"2024-08-31T12:29:11.364793+0000","last_active":"2024-08-31T12:29:11.364867+0000","last_peered":"2024-08-31T12:29:11.364867+0000","last_clean":"2024-08-31T12:29:11.364867+0000","last_became_active":"2024-08-31T12:15:57.311323+0000","last_became_peered":"2024-08-31T12:15:57.311323+0000","last_unstale":"2024-08-31T12:29:11.364867+0000","last_undegraded":"2024-08-31T12:29:11.364867+0000","last_fullsized":"2024-08-31T12:29:11.364867+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2863","last_scrub_stamp":"2024-08-31T12:29:11.364684+0000","last_deep_scrub":"23'2863","last_deep_scrub_stamp":"2024-08-31T12:29:11.364684+0000","last_clean_scrub_stamp":"2024-08-31T12:29:11.364684+0000","objects_scrubbed":50,"log_size":2863,"log_dups_size":0,"ondisk_log_size":2863,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:20:17.955601+0000","scrub_duration":313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201522906,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13808,"num_write_kb":252081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"23'4","reported_seq":37,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:55.235490+0000","last_change":"2024-08-31T12:28:55.235390+0000","last_active":"2024-08-31T12:28:55.235490+0000","last_peered":"2024-08-31T12:28:55.235490+0000","last_clean":"2024-08-31T12:28:55.235490+0000","last_became_active":"2024-08-31T12:15:56.290196+0000","last_became_peered":"2024-08-31T12:15:56.290196+0000","last_unstale":"2024-08-31T12:28:55.235490+0000","last_undegraded":"2024-08-31T12:28:55.235490+0000","last_fullsized":"2024-08-31T12:28:55.235490+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'4","last_scrub_stamp":"2024-08-31T12:28:55.235278+0000","last_deep_scrub":"23'4","last_deep_scrub_stamp":"2024-08-31T12:28:55.235278+0000","last_clean_scrub_stamp":"2024-08-31T12:28:55.235278+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:03:34.089107+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"23'3351","reported_seq":3384,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:04.312166+0000","last_change":"2024-08-31T12:29:04.312074+0000","last_active":"2024-08-31T12:29:04.312166+0000","last_peered":"2024-08-31T12:29:04.312166+0000","last_clean":"2024-08-31T12:29:04.312166+0000","last_became_active":"2024-08-31T12:15:57.312145+0000","last_became_peered":"2024-08-31T12:15:57.312145+0000","last_unstale":"2024-08-31T12:29:04.312166+0000","last_undegraded":"2024-08-31T12:29:04.312166+0000","last_fullsized":"2024-08-31T12:29:04.312166+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3351","last_scrub_stamp":"2024-08-31T12:29:04.311967+0000","last_deep_scrub":"23'3351","last_deep_scrub_stamp":"2024-08-31T12:29:04.311967+0000","last_clean_scrub_stamp":"2024-08-31T12:29:04.311967+0000","objects_scrubbed":60,"log_size":3351,"log_dups_size":0,"ondisk_log_size":3351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:56:28.264029+0000","scrub_duration":386,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243277619,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16586,"num_write_kb":304033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"23'2","reported_seq":35,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:55.708587+0000","last_change":"2024-08-31T12:28:55.708519+0000","last_active":"2024-08-31T12:28:55.708587+0000","last_peered":"2024-08-31T12:28:55.708587+0000","last_clean":"2024-08-31T12:28:55.708587+0000","last_became_active":"2024-08-31T12:15:56.281368+0000","last_became_peered":"2024-08-31T12:15:56.281368+0000","last_unstale":"2024-08-31T12:28:55.708587+0000","last_undegraded":"2024-08-31T12:28:55.708587+0000","last_fullsized":"2024-08-31T12:28:55.708587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2","last_scrub_stamp":"2024-08-31T12:28:55.708441+0000","last_deep_scrub":"23'2","last_deep_scrub_stamp":"2024-08-31T12:28:55.708441+0000","last_clean_scrub_stamp":"2024-08-31T12:28:55.708441+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:22:27.640560+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"23'3353","reported_seq":3386,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:01.001446+0000","last_change":"2024-08-31T12:29:01.001446+0000","last_active":"2024-08-31T12:29:01.001446+0000","last_peered":"2024-08-31T12:29:01.001446+0000","last_clean":"2024-08-31T12:29:01.001446+0000","last_became_active":"2024-08-31T12:15:57.305507+0000","last_became_peered":"2024-08-31T12:15:57.305507+0000","last_unstale":"2024-08-31T12:29:01.001446+0000","last_undegraded":"2024-08-31T12:29:01.001446+0000","last_fullsized":"2024-08-31T12:29:01.001446+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3353","last_scrub_stamp":"2024-08-31T12:29:01.001362+0000","last_deep_scrub":"23'3353","last_deep_scrub_stamp":"2024-08-31T12:29:01.001362+0000","last_clean_scrub_stamp":"2024-08-31T12:29:01.001362+0000","objects_scrubbed":58,"log_size":3353,"log_dups_size":0,"ondisk_log_size":3353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:30:34.897037+0000","scrub_duration":777,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218117314,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16432,"num_write_kb":278837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"23'6","reported_seq":39,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:53.312078+0000","last_change":"2024-08-31T12:28:53.311960+0000","last_active":"2024-08-31T12:28:53.312078+0000","last_peered":"2024-08-31T12:28:53.312078+0000","last_clean":"2024-08-31T12:28:53.312078+0000","last_became_active":"2024-08-31T12:15:56.291446+0000","last_became_peered":"2024-08-31T12:15:56.291446+0000","last_unstale":"2024-08-31T12:28:53.312078+0000","last_undegraded":"2024-08-31T12:28:53.312078+0000","last_fullsized":"2024-08-31T12:28:53.312078+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'6","last_scrub_stamp":"2024-08-31T12:28:53.311855+0000","last_deep_scrub":"23'6","last_deep_scrub_stamp":"2024-08-31T12:28:53.311855+0000","last_clean_scrub_stamp":"2024-08-31T12:28:53.311855+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:09:33.112431+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"23'2799","reported_seq":2832,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:05.295572+0000","last_change":"2024-08-31T12:29:05.295572+0000","last_active":"2024-08-31T12:29:05.295572+0000","last_peered":"2024-08-31T12:29:05.295572+0000","last_clean":"2024-08-31T12:29:05.295572+0000","last_became_active":"2024-08-31T12:15:57.315830+0000","last_became_peered":"2024-08-31T12:15:57.315830+0000","last_unstale":"2024-08-31T12:29:05.295572+0000","last_undegraded":"2024-08-31T12:29:05.295572+0000","last_fullsized":"2024-08-31T12:29:05.295572+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2799","last_scrub_stamp":"2024-08-31T12:29:05.295434+0000","last_deep_scrub":"23'2799","last_deep_scrub_stamp":"2024-08-31T12:29:05.295434+0000","last_clean_scrub_stamp":"2024-08-31T12:29:05.295434+0000","objects_scrubbed":55,"log_size":2799,"log_dups_size":0,"ondisk_log_size":2799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:01:26.820314+0000","scrub_duration":329,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209813064,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13700,"num_write_kb":259755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"23'74","reported_seq":131,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:55.002617+0000","last_change":"2024-08-31T12:28:55.002617+0000","last_active":"2024-08-31T12:28:55.002617+0000","last_peered":"2024-08-31T12:28:55.002617+0000","last_clean":"2024-08-31T12:28:55.002617+0000","last_became_active":"2024-08-31T12:15:47.083771+0000","last_became_peered":"2024-08-31T12:15:47.083771+0000","last_unstale":"2024-08-31T12:28:55.002617+0000","last_undegraded":"2024-08-31T12:28:55.002617+0000","last_fullsized":"2024-08-31T12:28:55.002617+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'74","last_scrub_stamp":"2024-08-31T12:28:55.002525+0000","last_deep_scrub":"23'74","last_deep_scrub_stamp":"2024-08-31T12:28:55.002525+0000","last_clean_scrub_stamp":"2024-08-31T12:28:55.002525+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:30:38.520538+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"23'7","reported_seq":40,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:55.178238+0000","last_change":"2024-08-31T12:28:55.178238+0000","last_active":"2024-08-31T12:28:55.178238+0000","last_peered":"2024-08-31T12:28:55.178238+0000","last_clean":"2024-08-31T12:28:55.178238+0000","last_became_active":"2024-08-31T12:15:56.284470+0000","last_became_peered":"2024-08-31T12:15:56.284470+0000","last_unstale":"2024-08-31T12:28:55.178238+0000","last_undegraded":"2024-08-31T12:28:55.178238+0000","last_fullsized":"2024-08-31T12:28:55.178238+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'7","last_scrub_stamp":"2024-08-31T12:28:55.178131+0000","last_deep_scrub":"23'7","last_deep_scrub_stamp":"2024-08-31T12:28:55.178131+0000","last_clean_scrub_stamp":"2024-08-31T12:28:55.178131+0000","objects_scrubbed":3,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T14:28:12.647620+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"23'2828","reported_seq":2861,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:59.048135+0000","last_change":"2024-08-31T12:28:59.048135+0000","last_active":"2024-08-31T12:28:59.048135+0000","last_peered":"2024-08-31T12:28:59.048135+0000","last_clean":"2024-08-31T12:28:59.048135+0000","last_became_active":"2024-08-31T12:15:57.308721+0000","last_became_peered":"2024-08-31T12:15:57.308721+0000","last_unstale":"2024-08-31T12:28:59.048135+0000","last_undegraded":"2024-08-31T12:28:59.048135+0000","last_fullsized":"2024-08-31T12:28:59.048135+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2828","last_scrub_stamp":"2024-08-31T12:28:59.048032+0000","last_deep_scrub":"23'2828","last_deep_scrub_stamp":"2024-08-31T12:28:59.048032+0000","last_clean_scrub_stamp":"2024-08-31T12:28:59.048032+0000","objects_scrubbed":53,"log_size":2828,"log_dups_size":0,"ondisk_log_size":2828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:25:15.429296+0000","scrub_duration":536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218112028,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13723,"num_write_kb":267984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"23'5","reported_seq":38,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:56.040275+0000","last_change":"2024-08-31T12:28:56.040275+0000","last_active":"2024-08-31T12:28:56.040275+0000","last_peered":"2024-08-31T12:28:56.040275+0000","last_clean":"2024-08-31T12:28:56.040275+0000","last_became_active":"2024-08-31T12:15:56.292910+0000","last_became_peered":"2024-08-31T12:15:56.292910+0000","last_unstale":"2024-08-31T12:28:56.040275+0000","last_undegraded":"2024-08-31T12:28:56.040275+0000","last_fullsized":"2024-08-31T12:28:56.040275+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5","last_scrub_stamp":"2024-08-31T12:28:56.040147+0000","last_deep_scrub":"23'5","last_deep_scrub_stamp":"2024-08-31T12:28:56.040147+0000","last_clean_scrub_stamp":"2024-08-31T12:28:56.040147+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:58:55.930106+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"23'2813","reported_seq":2846,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:52.584464+0000","last_change":"2024-08-31T12:28:52.584464+0000","last_active":"2024-08-31T12:28:52.584464+0000","last_peered":"2024-08-31T12:28:52.584464+0000","last_clean":"2024-08-31T12:28:52.584464+0000","last_became_active":"2024-08-31T12:15:57.306529+0000","last_became_peered":"2024-08-31T12:15:57.306529+0000","last_unstale":"2024-08-31T12:28:52.584464+0000","last_undegraded":"2024-08-31T12:28:52.584464+0000","last_fullsized":"2024-08-31T12:28:52.584464+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2813","last_scrub_stamp":"2024-08-31T12:28:52.584390+0000","last_deep_scrub":"23'2813","last_deep_scrub_stamp":"2024-08-31T12:28:52.584390+0000","last_clean_scrub_stamp":"2024-08-31T12:28:52.584390+0000","objects_scrubbed":51,"log_size":2813,"log_dups_size":0,"ondisk_log_size":2813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:30:35.075853+0000","scrub_duration":362,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201337916,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14174,"num_write_kb":253444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"23'38","reported_seq":101,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:52.310657+0000","last_change":"2024-08-31T12:28:52.310657+0000","last_active":"2024-08-31T12:28:52.310657+0000","last_peered":"2024-08-31T12:28:52.310657+0000","last_clean":"2024-08-31T12:28:52.310657+0000","last_became_active":"2024-08-31T12:15:56.291064+0000","last_became_peered":"2024-08-31T12:15:56.291064+0000","last_unstale":"2024-08-31T12:28:52.310657+0000","last_undegraded":"2024-08-31T12:28:52.310657+0000","last_fullsized":"2024-08-31T12:28:52.310657+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'38","last_scrub_stamp":"2024-08-31T12:28:52.310562+0000","last_deep_scrub":"23'38","last_deep_scrub_stamp":"2024-08-31T12:28:52.310562+0000","last_clean_scrub_stamp":"2024-08-31T12:28:52.310562+0000","objects_scrubbed":3,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T20:52:09.840889+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47459,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":30,"num_read_kb":64,"num_write":34,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"23'2997","reported_seq":3030,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:59.228323+0000","last_change":"2024-08-31T12:28:59.228323+0000","last_active":"2024-08-31T12:28:59.228323+0000","last_peered":"2024-08-31T12:28:59.228323+0000","last_clean":"2024-08-31T12:28:59.228323+0000","last_became_active":"2024-08-31T12:15:57.298892+0000","last_became_peered":"2024-08-31T12:15:57.298892+0000","last_unstale":"2024-08-31T12:28:59.228323+0000","last_undegraded":"2024-08-31T12:28:59.228323+0000","last_fullsized":"2024-08-31T12:28:59.228323+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2997","last_scrub_stamp":"2024-08-31T12:28:59.228243+0000","last_deep_scrub":"23'2997","last_deep_scrub_stamp":"2024-08-31T12:28:59.228243+0000","last_clean_scrub_stamp":"2024-08-31T12:28:59.228243+0000","objects_scrubbed":51,"log_size":2997,"log_dups_size":0,"ondisk_log_size":2997,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:43:41.236154+0000","scrub_duration":454,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197144226,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14947,"num_write_kb":252544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"23'2","reported_seq":35,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:54.218507+0000","last_change":"2024-08-31T12:28:54.218507+0000","last_active":"2024-08-31T12:28:54.218507+0000","last_peered":"2024-08-31T12:28:54.218507+0000","last_clean":"2024-08-31T12:28:54.218507+0000","last_became_active":"2024-08-31T12:15:56.285648+0000","last_became_peered":"2024-08-31T12:15:56.285648+0000","last_unstale":"2024-08-31T12:28:54.218507+0000","last_undegraded":"2024-08-31T12:28:54.218507+0000","last_fullsized":"2024-08-31T12:28:54.218507+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2","last_scrub_stamp":"2024-08-31T12:28:54.218372+0000","last_deep_scrub":"23'2","last_deep_scrub_stamp":"2024-08-31T12:28:54.218372+0000","last_clean_scrub_stamp":"2024-08-31T12:28:54.218372+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:17:40.309134+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"23'2818","reported_seq":2851,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:55.445307+0000","last_change":"2024-08-31T12:28:55.445307+0000","last_active":"2024-08-31T12:28:55.445307+0000","last_peered":"2024-08-31T12:28:55.445307+0000","last_clean":"2024-08-31T12:28:55.445307+0000","last_became_active":"2024-08-31T12:15:57.311469+0000","last_became_peered":"2024-08-31T12:15:57.311469+0000","last_unstale":"2024-08-31T12:28:55.445307+0000","last_undegraded":"2024-08-31T12:28:55.445307+0000","last_fullsized":"2024-08-31T12:28:55.445307+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2818","last_scrub_stamp":"2024-08-31T12:28:55.445200+0000","last_deep_scrub":"23'2818","last_deep_scrub_stamp":"2024-08-31T12:28:55.445200+0000","last_clean_scrub_stamp":"2024-08-31T12:28:55.445200+0000","objects_scrubbed":49,"log_size":2818,"log_dups_size":0,"ondisk_log_size":2818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:42:39.242246+0000","scrub_duration":414,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197139313,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14081,"num_write_kb":248914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"23'2","reported_seq":35,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:52.650358+0000","last_change":"2024-08-31T12:28:52.650358+0000","last_active":"2024-08-31T12:28:52.650358+0000","last_peered":"2024-08-31T12:28:52.650358+0000","last_clean":"2024-08-31T12:28:52.650358+0000","last_became_active":"2024-08-31T12:15:56.283979+0000","last_became_peered":"2024-08-31T12:15:56.283979+0000","last_unstale":"2024-08-31T12:28:52.650358+0000","last_undegraded":"2024-08-31T12:28:52.650358+0000","last_fullsized":"2024-08-31T12:28:52.650358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2","last_scrub_stamp":"2024-08-31T12:28:52.650280+0000","last_deep_scrub":"23'2","last_deep_scrub_stamp":"2024-08-31T12:28:52.650280+0000","last_clean_scrub_stamp":"2024-08-31T12:28:52.650280+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:59:18.393870+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7000,"num_omap_keys":14,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"23'2575","reported_seq":2601,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:58.709640+0000","last_change":"2024-08-31T12:28:58.709640+0000","last_active":"2024-08-31T12:28:58.709640+0000","last_peered":"2024-08-31T12:28:58.709640+0000","last_clean":"2024-08-31T12:28:58.709640+0000","last_became_active":"2024-08-31T12:16:41.194724+0000","last_became_peered":"2024-08-31T12:16:41.194724+0000","last_unstale":"2024-08-31T12:28:58.709640+0000","last_undegraded":"2024-08-31T12:28:58.709640+0000","last_fullsized":"2024-08-31T12:28:58.709640+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2575","last_scrub_stamp":"2024-08-31T12:28:58.709472+0000","last_deep_scrub":"23'2575","last_deep_scrub_stamp":"2024-08-31T12:28:58.709472+0000","last_clean_scrub_stamp":"2024-08-31T12:28:58.709472+0000","objects_scrubbed":48,"log_size":2575,"log_dups_size":0,"ondisk_log_size":2575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T12:59:13.626464+0000","scrub_duration":507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184558382,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13011,"num_write_kb":232344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"23'2","reported_seq":35,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:00.976677+0000","last_change":"2024-08-31T12:29:00.976677+0000","last_active":"2024-08-31T12:29:00.976677+0000","last_peered":"2024-08-31T12:29:00.976677+0000","last_clean":"2024-08-31T12:29:00.976677+0000","last_became_active":"2024-08-31T12:15:56.297306+0000","last_became_peered":"2024-08-31T12:15:56.297306+0000","last_unstale":"2024-08-31T12:29:00.976677+0000","last_undegraded":"2024-08-31T12:29:00.976677+0000","last_fullsized":"2024-08-31T12:29:00.976677+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2","last_scrub_stamp":"2024-08-31T12:29:00.976561+0000","last_deep_scrub":"23'2","last_deep_scrub_stamp":"2024-08-31T12:29:00.976561+0000","last_clean_scrub_stamp":"2024-08-31T12:29:00.976561+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:23:50.922340+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"23'3228","reported_seq":3261,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:55.228844+0000","last_change":"2024-08-31T12:28:55.228844+0000","last_active":"2024-08-31T12:28:55.228844+0000","last_peered":"2024-08-31T12:28:55.228844+0000","last_clean":"2024-08-31T12:28:55.228844+0000","last_became_active":"2024-08-31T12:15:57.296892+0000","last_became_peered":"2024-08-31T12:15:57.296892+0000","last_unstale":"2024-08-31T12:28:55.228844+0000","last_undegraded":"2024-08-31T12:28:55.228844+0000","last_fullsized":"2024-08-31T12:28:55.228844+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3228","last_scrub_stamp":"2024-08-31T12:28:55.228711+0000","last_deep_scrub":"23'3228","last_deep_scrub_stamp":"2024-08-31T12:28:55.228711+0000","last_clean_scrub_stamp":"2024-08-31T12:28:55.228711+0000","objects_scrubbed":55,"log_size":3228,"log_dups_size":0,"ondisk_log_size":3228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:22:55.949863+0000","scrub_duration":535,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213956769,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16152,"num_write_kb":273638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"23'431","reported_seq":464,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:51.651294+0000","last_change":"2024-08-31T12:28:51.651231+0000","last_active":"2024-08-31T12:28:51.651294+0000","last_peered":"2024-08-31T12:28:51.651294+0000","last_clean":"2024-08-31T12:28:51.651294+0000","last_became_active":"2024-08-31T12:15:56.286173+0000","last_became_peered":"2024-08-31T12:15:56.286173+0000","last_unstale":"2024-08-31T12:28:51.651294+0000","last_undegraded":"2024-08-31T12:28:51.651294+0000","last_fullsized":"2024-08-31T12:28:51.651294+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'431","last_scrub_stamp":"2024-08-31T12:28:51.651149+0000","last_deep_scrub":"23'431","last_deep_scrub_stamp":"2024-08-31T12:28:51.651149+0000","last_clean_scrub_stamp":"2024-08-31T12:28:51.651149+0000","objects_scrubbed":4,"log_size":431,"log_dups_size":0,"ondisk_log_size":431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T16:19:48.734353+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":977,"num_write_kb":307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9038,"num_omap_keys":18,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"23'2972","reported_seq":3005,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:57.965445+0000","last_change":"2024-08-31T12:28:57.965381+0000","last_active":"2024-08-31T12:28:57.965445+0000","last_peered":"2024-08-31T12:28:57.965445+0000","last_clean":"2024-08-31T12:28:57.965445+0000","last_became_active":"2024-08-31T12:15:57.307701+0000","last_became_peered":"2024-08-31T12:15:57.307701+0000","last_unstale":"2024-08-31T12:28:57.965445+0000","last_undegraded":"2024-08-31T12:28:57.965445+0000","last_fullsized":"2024-08-31T12:28:57.965445+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2972","last_scrub_stamp":"2024-08-31T12:28:57.965299+0000","last_deep_scrub":"23'2972","last_deep_scrub_stamp":"2024-08-31T12:28:57.965299+0000","last_clean_scrub_stamp":"2024-08-31T12:28:57.965299+0000","objects_scrubbed":51,"log_size":2972,"log_dups_size":0,"ondisk_log_size":2972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:27:46.349413+0000","scrub_duration":468,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205521115,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14225,"num_write_kb":257747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"23'6","reported_seq":39,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:57.493049+0000","last_change":"2024-08-31T12:28:57.493049+0000","last_active":"2024-08-31T12:28:57.493049+0000","last_peered":"2024-08-31T12:28:57.493049+0000","last_clean":"2024-08-31T12:28:57.493049+0000","last_became_active":"2024-08-31T12:15:56.294496+0000","last_became_peered":"2024-08-31T12:15:56.294496+0000","last_unstale":"2024-08-31T12:28:57.493049+0000","last_undegraded":"2024-08-31T12:28:57.493049+0000","last_fullsized":"2024-08-31T12:28:57.493049+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'6","last_scrub_stamp":"2024-08-31T12:28:57.492917+0000","last_deep_scrub":"23'6","last_deep_scrub_stamp":"2024-08-31T12:28:57.492917+0000","last_clean_scrub_stamp":"2024-08-31T12:28:57.492917+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:22:58.214069+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"23'2589","reported_seq":2621,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:53.611819+0000","last_change":"2024-08-31T12:28:53.611757+0000","last_active":"2024-08-31T12:28:53.611819+0000","last_peered":"2024-08-31T12:28:53.611819+0000","last_clean":"2024-08-31T12:28:53.611819+0000","last_became_active":"2024-08-31T12:15:57.304602+0000","last_became_peered":"2024-08-31T12:15:57.304602+0000","last_unstale":"2024-08-31T12:28:53.611819+0000","last_undegraded":"2024-08-31T12:28:53.611819+0000","last_fullsized":"2024-08-31T12:28:53.611819+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2589","last_scrub_stamp":"2024-08-31T12:28:53.611658+0000","last_deep_scrub":"23'2589","last_deep_scrub_stamp":"2024-08-31T12:28:53.611658+0000","last_clean_scrub_stamp":"2024-08-31T12:28:53.611658+0000","objects_scrubbed":45,"log_size":2589,"log_dups_size":0,"ondisk_log_size":2589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:38:56.546250+0000","scrub_duration":400,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176167762,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12536,"num_write_kb":222149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"23'4","reported_seq":37,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:59.994908+0000","last_change":"2024-08-31T12:28:59.994848+0000","last_active":"2024-08-31T12:28:59.994908+0000","last_peered":"2024-08-31T12:28:59.994908+0000","last_clean":"2024-08-31T12:28:59.994908+0000","last_became_active":"2024-08-31T12:15:56.297168+0000","last_became_peered":"2024-08-31T12:15:56.297168+0000","last_unstale":"2024-08-31T12:28:59.994908+0000","last_undegraded":"2024-08-31T12:28:59.994908+0000","last_fullsized":"2024-08-31T12:28:59.994908+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'4","last_scrub_stamp":"2024-08-31T12:28:59.994769+0000","last_deep_scrub":"23'4","last_deep_scrub_stamp":"2024-08-31T12:28:59.994769+0000","last_clean_scrub_stamp":"2024-08-31T12:28:59.994769+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:50:05.473008+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"23'2372","reported_seq":2405,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:56.443443+0000","last_change":"2024-08-31T12:28:56.443379+0000","last_active":"2024-08-31T12:28:56.443443+0000","last_peered":"2024-08-31T12:28:56.443443+0000","last_clean":"2024-08-31T12:28:56.443443+0000","last_became_active":"2024-08-31T12:15:57.298520+0000","last_became_peered":"2024-08-31T12:15:57.298520+0000","last_unstale":"2024-08-31T12:28:56.443443+0000","last_undegraded":"2024-08-31T12:28:56.443443+0000","last_fullsized":"2024-08-31T12:28:56.443443+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2372","last_scrub_stamp":"2024-08-31T12:28:56.443305+0000","last_deep_scrub":"23'2372","last_deep_scrub_stamp":"2024-08-31T12:28:56.443305+0000","last_clean_scrub_stamp":"2024-08-31T12:28:56.443305+0000","objects_scrubbed":50,"log_size":2372,"log_dups_size":0,"ondisk_log_size":2372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:15:10.275590+0000","scrub_duration":422,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197144309,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11287,"num_write_kb":237691,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"23'5","reported_seq":38,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:56.716501+0000","last_change":"2024-08-31T12:28:56.716501+0000","last_active":"2024-08-31T12:28:56.716501+0000","last_peered":"2024-08-31T12:28:56.716501+0000","last_clean":"2024-08-31T12:28:56.716501+0000","last_became_active":"2024-08-31T12:15:56.289960+0000","last_became_peered":"2024-08-31T12:15:56.289960+0000","last_unstale":"2024-08-31T12:28:56.716501+0000","last_undegraded":"2024-08-31T12:28:56.716501+0000","last_fullsized":"2024-08-31T12:28:56.716501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5","last_scrub_stamp":"2024-08-31T12:28:56.716407+0000","last_deep_scrub":"23'5","last_deep_scrub_stamp":"2024-08-31T12:28:56.716407+0000","last_clean_scrub_stamp":"2024-08-31T12:28:56.716407+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:30:05.472027+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"23'3030","reported_seq":3063,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:04.944510+0000","last_change":"2024-08-31T12:29:04.944510+0000","last_active":"2024-08-31T12:29:04.944510+0000","last_peered":"2024-08-31T12:29:04.944510+0000","last_clean":"2024-08-31T12:29:04.944510+0000","last_became_active":"2024-08-31T12:15:57.301157+0000","last_became_peered":"2024-08-31T12:15:57.301157+0000","last_unstale":"2024-08-31T12:29:04.944510+0000","last_undegraded":"2024-08-31T12:29:04.944510+0000","last_fullsized":"2024-08-31T12:29:04.944510+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3030","last_scrub_stamp":"2024-08-31T12:29:04.944425+0000","last_deep_scrub":"23'3030","last_deep_scrub_stamp":"2024-08-31T12:29:04.944425+0000","last_clean_scrub_stamp":"2024-08-31T12:29:04.944425+0000","objects_scrubbed":51,"log_size":3030,"log_dups_size":0,"ondisk_log_size":3030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T18:20:43.219093+0000","scrub_duration":462,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209728881,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15223,"num_write_kb":265883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"23'2","reported_seq":35,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:53.651283+0000","last_change":"2024-08-31T12:28:53.651216+0000","last_active":"2024-08-31T12:28:53.651283+0000","last_peered":"2024-08-31T12:28:53.651283+0000","last_clean":"2024-08-31T12:28:53.651283+0000","last_became_active":"2024-08-31T12:15:56.289332+0000","last_became_peered":"2024-08-31T12:15:56.289332+0000","last_unstale":"2024-08-31T12:28:53.651283+0000","last_undegraded":"2024-08-31T12:28:53.651283+0000","last_fullsized":"2024-08-31T12:28:53.651283+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2","last_scrub_stamp":"2024-08-31T12:28:53.651142+0000","last_deep_scrub":"23'2","last_deep_scrub_stamp":"2024-08-31T12:28:53.651142+0000","last_clean_scrub_stamp":"2024-08-31T12:28:53.651142+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:19:43.725168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"23'2468","reported_seq":2500,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:56.968970+0000","last_change":"2024-08-31T12:28:56.968970+0000","last_active":"2024-08-31T12:28:56.968970+0000","last_peered":"2024-08-31T12:28:56.968970+0000","last_clean":"2024-08-31T12:28:56.968970+0000","last_became_active":"2024-08-31T12:15:57.313428+0000","last_became_peered":"2024-08-31T12:15:57.313428+0000","last_unstale":"2024-08-31T12:28:56.968970+0000","last_undegraded":"2024-08-31T12:28:56.968970+0000","last_fullsized":"2024-08-31T12:28:56.968970+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2468","last_scrub_stamp":"2024-08-31T12:28:56.968836+0000","last_deep_scrub":"23'2468","last_deep_scrub_stamp":"2024-08-31T12:28:56.968836+0000","last_clean_scrub_stamp":"2024-08-31T12:28:56.968836+0000","objects_scrubbed":45,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:57:17.337071+0000","scrub_duration":437,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176173175,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11447,"num_write_kb":217885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"23'1071","reported_seq":1104,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:57.057809+0000","last_change":"2024-08-31T12:28:57.057690+0000","last_active":"2024-08-31T12:28:57.057809+0000","last_peered":"2024-08-31T12:28:57.057809+0000","last_clean":"2024-08-31T12:28:57.057809+0000","last_became_active":"2024-08-31T12:15:56.295524+0000","last_became_peered":"2024-08-31T12:15:56.295524+0000","last_unstale":"2024-08-31T12:28:57.057809+0000","last_undegraded":"2024-08-31T12:28:57.057809+0000","last_fullsized":"2024-08-31T12:28:57.057809+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'1071","last_scrub_stamp":"2024-08-31T12:28:57.057583+0000","last_deep_scrub":"23'1071","last_deep_scrub_stamp":"2024-08-31T12:28:57.057583+0000","last_clean_scrub_stamp":"2024-08-31T12:28:57.057583+0000","objects_scrubbed":3,"log_size":1071,"log_dups_size":0,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:05:01.152499+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1064,"num_write_kb":4401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"23'2553","reported_seq":2585,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:00.889097+0000","last_change":"2024-08-31T12:29:00.888999+0000","last_active":"2024-08-31T12:29:00.889097+0000","last_peered":"2024-08-31T12:29:00.889097+0000","last_clean":"2024-08-31T12:29:00.889097+0000","last_became_active":"2024-08-31T12:15:57.299436+0000","last_became_peered":"2024-08-31T12:15:57.299436+0000","last_unstale":"2024-08-31T12:29:00.889097+0000","last_undegraded":"2024-08-31T12:29:00.889097+0000","last_fullsized":"2024-08-31T12:29:00.889097+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2553","last_scrub_stamp":"2024-08-31T12:29:00.888868+0000","last_deep_scrub":"23'2553","last_deep_scrub_stamp":"2024-08-31T12:29:00.888868+0000","last_clean_scrub_stamp":"2024-08-31T12:29:00.888868+0000","objects_scrubbed":46,"log_size":2553,"log_dups_size":0,"ondisk_log_size":2553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:57:21.257104+0000","scrub_duration":321,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188744397,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12696,"num_write_kb":235198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"23'3436","reported_seq":3468,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:02.776669+0000","last_change":"2024-08-31T12:29:02.776669+0000","last_active":"2024-08-31T12:29:02.776669+0000","last_peered":"2024-08-31T12:29:02.776669+0000","last_clean":"2024-08-31T12:29:02.776669+0000","last_became_active":"2024-08-31T12:15:57.300159+0000","last_became_peered":"2024-08-31T12:15:57.300159+0000","last_unstale":"2024-08-31T12:29:02.776669+0000","last_undegraded":"2024-08-31T12:29:02.776669+0000","last_fullsized":"2024-08-31T12:29:02.776669+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3436","last_scrub_stamp":"2024-08-31T12:29:02.776519+0000","last_deep_scrub":"23'3436","last_deep_scrub_stamp":"2024-08-31T12:29:02.776519+0000","last_clean_scrub_stamp":"2024-08-31T12:29:02.776519+0000","objects_scrubbed":63,"log_size":3436,"log_dups_size":0,"ondisk_log_size":3436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T17:23:52.722298+0000","scrub_duration":514,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251730727,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16759,"num_write_kb":312885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"23'3","reported_seq":35,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:58.015823+0000","last_change":"2024-08-31T12:28:58.015823+0000","last_active":"2024-08-31T12:28:58.015823+0000","last_peered":"2024-08-31T12:28:58.015823+0000","last_clean":"2024-08-31T12:28:58.015823+0000","last_became_active":"2024-08-31T12:15:56.296976+0000","last_became_peered":"2024-08-31T12:15:56.296976+0000","last_unstale":"2024-08-31T12:28:58.015823+0000","last_undegraded":"2024-08-31T12:28:58.015823+0000","last_fullsized":"2024-08-31T12:28:58.015823+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3","last_scrub_stamp":"2024-08-31T12:28:58.015746+0000","last_deep_scrub":"23'3","last_deep_scrub_stamp":"2024-08-31T12:28:58.015746+0000","last_clean_scrub_stamp":"2024-08-31T12:28:58.015746+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T21:57:21.675079+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"23'2213","reported_seq":2244,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:01.520012+0000","last_change":"2024-08-31T12:29:01.520012+0000","last_active":"2024-08-31T12:29:01.520012+0000","last_peered":"2024-08-31T12:29:01.520012+0000","last_clean":"2024-08-31T12:29:01.520012+0000","last_became_active":"2024-08-31T12:15:57.300646+0000","last_became_peered":"2024-08-31T12:15:57.300646+0000","last_unstale":"2024-08-31T12:29:01.520012+0000","last_undegraded":"2024-08-31T12:29:01.520012+0000","last_fullsized":"2024-08-31T12:29:01.520012+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'2213","last_scrub_stamp":"2024-08-31T12:29:01.519875+0000","last_deep_scrub":"23'2213","last_deep_scrub_stamp":"2024-08-31T12:29:01.519875+0000","last_clean_scrub_stamp":"2024-08-31T12:29:01.519875+0000","objects_scrubbed":37,"log_size":2213,"log_dups_size":0,"ondisk_log_size":2213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T23:35:49.816083+0000","scrub_duration":251,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150996671,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10769,"num_write_kb":190633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"23'11","reported_seq":43,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:58.971224+0000","last_change":"2024-08-31T12:28:58.971224+0000","last_active":"2024-08-31T12:28:58.971224+0000","last_peered":"2024-08-31T12:28:58.971224+0000","last_clean":"2024-08-31T12:28:58.971224+0000","last_became_active":"2024-08-31T12:15:56.294560+0000","last_became_peered":"2024-08-31T12:15:56.294560+0000","last_unstale":"2024-08-31T12:28:58.971224+0000","last_undegraded":"2024-08-31T12:28:58.971224+0000","last_fullsized":"2024-08-31T12:28:58.971224+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'11","last_scrub_stamp":"2024-08-31T12:28:58.971147+0000","last_deep_scrub":"23'11","last_deep_scrub_stamp":"2024-08-31T12:28:58.971147+0000","last_clean_scrub_stamp":"2024-08-31T12:28:58.971147+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T13:32:02.655635+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"23'3421","reported_seq":3454,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:57.914506+0000","last_change":"2024-08-31T12:28:57.914506+0000","last_active":"2024-08-31T12:28:57.914506+0000","last_peered":"2024-08-31T12:28:57.914506+0000","last_clean":"2024-08-31T12:28:57.914506+0000","last_became_active":"2024-08-31T12:15:57.310262+0000","last_became_peered":"2024-08-31T12:15:57.310262+0000","last_unstale":"2024-08-31T12:28:57.914506+0000","last_undegraded":"2024-08-31T12:28:57.914506+0000","last_fullsized":"2024-08-31T12:28:57.914506+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3421","last_scrub_stamp":"2024-08-31T12:28:57.914399+0000","last_deep_scrub":"23'3421","last_deep_scrub_stamp":"2024-08-31T12:28:57.914399+0000","last_clean_scrub_stamp":"2024-08-31T12:28:57.914399+0000","objects_scrubbed":67,"log_size":3421,"log_dups_size":0,"ondisk_log_size":3421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:22:29.846518+0000","scrub_duration":560,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251719163,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16721,"num_write_kb":312844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"23'4","reported_seq":37,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:55.530644+0000","last_change":"2024-08-31T12:28:55.530644+0000","last_active":"2024-08-31T12:28:55.530644+0000","last_peered":"2024-08-31T12:28:55.530644+0000","last_clean":"2024-08-31T12:28:55.530644+0000","last_became_active":"2024-08-31T12:15:56.284884+0000","last_became_peered":"2024-08-31T12:15:56.284884+0000","last_unstale":"2024-08-31T12:28:55.530644+0000","last_undegraded":"2024-08-31T12:28:55.530644+0000","last_fullsized":"2024-08-31T12:28:55.530644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'4","last_scrub_stamp":"2024-08-31T12:28:55.530523+0000","last_deep_scrub":"23'4","last_deep_scrub_stamp":"2024-08-31T12:28:55.530523+0000","last_clean_scrub_stamp":"2024-08-31T12:28:55.530523+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T00:27:24.024019+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5538,"num_omap_keys":11,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"23'3225","reported_seq":3258,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:29:00.137341+0000","last_change":"2024-08-31T12:29:00.137341+0000","last_active":"2024-08-31T12:29:00.137341+0000","last_peered":"2024-08-31T12:29:00.137341+0000","last_clean":"2024-08-31T12:29:00.137341+0000","last_became_active":"2024-08-31T12:15:57.305409+0000","last_became_peered":"2024-08-31T12:15:57.305409+0000","last_unstale":"2024-08-31T12:29:00.137341+0000","last_undegraded":"2024-08-31T12:29:00.137341+0000","last_fullsized":"2024-08-31T12:29:00.137341+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'3225","last_scrub_stamp":"2024-08-31T12:29:00.137249+0000","last_deep_scrub":"23'3225","last_deep_scrub_stamp":"2024-08-31T12:29:00.137249+0000","last_clean_scrub_stamp":"2024-08-31T12:29:00.137249+0000","objects_scrubbed":53,"log_size":3225,"log_dups_size":0,"ondisk_log_size":3225,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T15:37:46.351278+0000","scrub_duration":589,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222298112,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15945,"num_write_kb":281070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"23'241","reported_seq":274,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-08-31T12:28:57.337647+0000","last_change":"2024-08-31T12:28:55.304129+0000","last_active":"2024-08-31T12:28:57.337647+0000","last_peered":"2024-08-31T12:28:57.337647+0000","last_clean":"2024-08-31T12:28:57.337647+0000","last_became_active":"2024-08-31T12:15:56.293864+0000","last_became_peered":"2024-08-31T12:15:56.293864+0000","last_unstale":"2024-08-31T12:28:57.337647+0000","last_undegraded":"2024-08-31T12:28:57.337647+0000","last_fullsized":"2024-08-31T12:28:57.337647+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'237","last_scrub_stamp":"2024-08-31T12:28:55.304041+0000","last_deep_scrub":"23'237","last_deep_scrub_stamp":"2024-08-31T12:28:55.304041+0000","last_clean_scrub_stamp":"2024-08-31T12:28:55.304041+0000","objects_scrubbed":5,"log_size":241,"log_dups_size":0,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T22:48:03.091342+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1390922,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":236,"num_write_kb":1494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":13424230175,"num_objects":3391,"num_object_clones":0,"num_object_copies":6782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":918154,"num_write_kb":16787942,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26849361920,"data_stored":26848441814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":187543,"ondisk_log_size":187543,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":14025485,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":30,"num_read_kb":64,"num_write":3320,"num_write_kb":14861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":66114,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28205056,"data_stored":28050970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2839,"ondisk_log_size":2839,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509609,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2618576,"kb_used_data":2384832,"kb_used_omap":1,"kb_used_meta":233726,"kb_avail":91753264,"statfs":{"total":96636764160,"available":93955342336,"internally_reserved":0,"allocated":2442067968,"data_stored":2441803901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":239335878},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509608,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4211060,"kb_used_data":3851044,"kb_used_omap":1,"kb_used_meta":359998,"kb_avail":90160780,"statfs":{"total":96636764160,"available":92324638720,"internally_reserved":0,"allocated":3943469056,"data_stored":3943171671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":368638407},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509609,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2548680,"kb_used_data":2306616,"kb_used_omap":1,"kb_used_meta":242046,"kb_avail":91823160,"statfs":{"total":96636764160,"available":94026915840,"internally_reserved":0,"allocated":2361974784,"data_stored":2361727195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":247855557},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509609,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4129328,"kb_used_data":3795680,"kb_used_omap":1,"kb_used_meta":333630,"kb_avail":90242512,"statfs":{"total":96636764160,"available":92408332288,"internally_reserved":0,"allocated":3886776320,"data_stored":3886460583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341637573},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542312,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4427080,"kb_used_data":4067000,"kb_used_omap":1,"kb_used_meta":360062,"kb_avail":89944760,"statfs":{"total":96636764160,"available":92103434240,"internally_reserved":0,"allocated":4164608000,"data_stored":4164280265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":368703946},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542314,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4149768,"kb_used_data":3793912,"kb_used_omap":1,"kb_used_meta":355838,"kb_avail":90222072,"statfs":{"total":96636764160,"available":92387401728,"internally_reserved":0,"allocated":3884965888,"data_stored":3884692848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364378565},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542312,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4140896,"kb_used_data":3839056,"kb_used_omap":1,"kb_used_meta":301822,"kb_avail":90230944,"statfs":{"total":96636764160,"available":92396486656,"internally_reserved":0,"allocated":3931193344,"data_stored":3930920211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":309066182},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575017,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2446212,"kb_used_data":2212468,"kb_used_omap":1,"kb_used_meta":233726,"kb_avail":91925628,"statfs":{"total":96636764160,"available":94131843072,"internally_reserved":0,"allocated":2265567232,"data_stored":2265299110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":239335877},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1409024,"data_stored":1391000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":48187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8458240,"data_stored":8437943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4210688,"data_stored":4195018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4210688,"data_stored":4195654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":9801728,"data_stored":9780846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3884539904,"data_stored":3884393932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2361716736,"data_stored":2361640968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2265264128,"data_stored":2265165640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3934777344,"data_stored":3934648445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3926749184,"data_stored":3926639910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3884711936,"data_stored":3884606187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2437623808,"data_stored":2437522964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4153978880,"data_stored":4153823768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-31T12:29:35.865 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-08-31T12:29:36.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.012+0000 7ff646f2b640 1 -- 172.21.15.19:0/1599109967 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff628004800 msgr2=0x7ff628004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:36.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.012+0000 7ff646f2b640 1 --2- 172.21.15.19:0/1599109967 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff628004800 0x7ff628004c10 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7ff630009b20 tx=0x7ff628005830 comp rx=0 tx=0).stop 2024-08-31T12:29:36.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.012+0000 7ff6454c8640 1 -- 172.21.15.19:0/1599109967 reap_dead start 2024-08-31T12:29:36.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.012+0000 7ff646f2b640 1 -- 172.21.15.19:0/1599109967 shutdown_connections 2024-08-31T12:29:36.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.012+0000 7ff646f2b640 1 -- 172.21.15.19:0/1599109967 >> 172.21.15.19:0/1599109967 conn(0x7ff64006d110 msgr2=0x7ff640068aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:29:36.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.012+0000 7ff646f2b640 1 -- 172.21.15.19:0/1599109967 shutdown_connections 2024-08-31T12:29:36.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.012+0000 7ff646f2b640 1 -- 172.21.15.19:0/1599109967 wait complete. 2024-08-31T12:29:36.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.014+0000 7ff646f2b640 1 Processor -- start 2024-08-31T12:29:36.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.014+0000 7ff646f2b640 1 -- start start 2024-08-31T12:29:36.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.015+0000 7ff646f2b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff628004800 0x7ff64014ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:29:36.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.015+0000 7ff646f2b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff6401f3270 0x7ff6401e45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:29:36.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.015+0000 7ff646f2b640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6401e4b00 0x7ff6401470c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:29:36.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.015+0000 7ff646f2b640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7ff640159d80 con 0x7ff6401f3270 2024-08-31T12:29:36.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.015+0000 7ff646f2b640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff6400c9aa0 con 0x7ff6401e4b00 2024-08-31T12:29:36.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.015+0000 7ff646f2b640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7ff6400c97b0 con 0x7ff628004800 2024-08-31T12:29:36.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.015+0000 7ff637fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff6401f3270 0x7ff6401e45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:29:36.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.015+0000 7ff637fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff6401f3270 0x7ff6401e45c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:53322/0 (socket says 172.21.15.19:53322) 2024-08-31T12:29:36.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.015+0000 7ff637fff640 1 -- 172.21.15.19:0/3095183429 learned_addr learned my addr 172.21.15.19:0/3095183429 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:29:36.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.015+0000 7ff644cc7640 1 --2- 172.21.15.19:0/3095183429 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff628004800 0x7ff64014ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:29:36.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.015+0000 7ff6454c8640 1 --2- 172.21.15.19:0/3095183429 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6401e4b00 0x7ff6401470c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:29:36.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.016+0000 7ff637fff640 1 -- 172.21.15.19:0/3095183429 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff628004800 msgr2=0x7ff64014ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:36.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.016+0000 7ff637fff640 1 --2- 172.21.15.19:0/3095183429 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff628004800 0x7ff64014ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:36.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.016+0000 7ff637fff640 1 -- 172.21.15.19:0/3095183429 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6401e4b00 msgr2=0x7ff6401470c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:36.020 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.016+0000 7ff637fff640 1 --2- 172.21.15.19:0/3095183429 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6401e4b00 0x7ff6401470c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:36.020 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.016+0000 7ff637fff640 1 -- 172.21.15.19:0/3095183429 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6400ce760 con 0x7ff6401f3270 2024-08-31T12:29:36.020 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.016+0000 7ff637fff640 1 --2- 172.21.15.19:0/3095183429 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff6401f3270 0x7ff6401e45c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff630009a70 tx=0x7ff630033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:29:36.020 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.016+0000 7ff6357fa640 1 -- 172.21.15.19:0/3095183429 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff630038050 con 0x7ff6401f3270 2024-08-31T12:29:36.020 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.016+0000 7ff6357fa640 1 -- 172.21.15.19:0/3095183429 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff630039040 con 0x7ff6401f3270 2024-08-31T12:29:36.020 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.016+0000 7ff646f2b640 1 -- 172.21.15.19:0/3095183429 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6401f2790 con 0x7ff6401f3270 2024-08-31T12:29:36.020 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.016+0000 7ff6357fa640 1 -- 172.21.15.19:0/3095183429 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff630033c50 con 0x7ff6401f3270 2024-08-31T12:29:36.020 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.016+0000 7ff646f2b640 1 -- 172.21.15.19:0/3095183429 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6401f2d50 con 0x7ff6401f3270 2024-08-31T12:29:36.021 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.017+0000 7ff646f2b640 1 -- 172.21.15.19:0/3095183429 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6400c48e0 con 0x7ff6401f3270 2024-08-31T12:29:36.021 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.018+0000 7ff6357fa640 1 -- 172.21.15.19:0/3095183429 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7ff630046020 con 0x7ff6401f3270 2024-08-31T12:29:36.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.018+0000 7ff6357fa640 1 --2- 172.21.15.19:0/3095183429 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff62c077a50 0x7ff62c079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:29:36.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.018+0000 7ff6357fa640 1 -- 172.21.15.19:0/3095183429 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7ff6300ba210 con 0x7ff6401f3270 2024-08-31T12:29:36.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.019+0000 7ff644cc7640 1 --2- 172.21.15.19:0/3095183429 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff62c077a50 0x7ff62c079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:29:36.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.019+0000 7ff644cc7640 1 --2- 172.21.15.19:0/3095183429 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff62c077a50 0x7ff62c079f60 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7ff63c001d50 tx=0x7ff63c001c00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:29:36.024 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.023+0000 7ff6357fa640 1 -- 172.21.15.19:0/3095183429 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7ff63004d020 con 0x7ff6401f3270 2024-08-31T12:29:36.207 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.206+0000 7ff646f2b640 1 -- 172.21.15.19:0/3095183429 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7ff6400ca210 con 0x7ff6401f3270 2024-08-31T12:29:36.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.210+0000 7ff6357fa640 1 -- 172.21.15.19:0/3095183429 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v33)=0 v33) ==== 123+0+0 (secure 0 0 0) 0x7ff630086fe0 con 0x7ff6401f3270 2024-08-31T12:29:36.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.212+0000 7ff6357fa640 1 -- 172.21.15.19:0/3095183429 <== mon.0 v2:172.21.15.19:3300/0 8 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7ff6300823a0 con 0x7ff6401f3270 2024-08-31T12:29:36.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.213+0000 7ff646f2b640 1 -- 172.21.15.19:0/3095183429 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff62c077a50 msgr2=0x7ff62c079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:36.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.213+0000 7ff646f2b640 1 --2- 172.21.15.19:0/3095183429 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff62c077a50 0x7ff62c079f60 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7ff63c001d50 tx=0x7ff63c001c00 comp rx=0 tx=0).stop 2024-08-31T12:29:36.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.213+0000 7ff646f2b640 1 -- 172.21.15.19:0/3095183429 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff6401f3270 msgr2=0x7ff6401e45c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:36.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.213+0000 7ff646f2b640 1 --2- 172.21.15.19:0/3095183429 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff6401f3270 0x7ff6401e45c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff630009a70 tx=0x7ff630033670 comp rx=0 tx=0).stop 2024-08-31T12:29:36.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.213+0000 7ff646f2b640 1 -- 172.21.15.19:0/3095183429 shutdown_connections 2024-08-31T12:29:36.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.213+0000 7ff646f2b640 1 --2- 172.21.15.19:0/3095183429 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7ff62c077a50 0x7ff62c079f60 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:36.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.213+0000 7ff646f2b640 1 --2- 172.21.15.19:0/3095183429 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6401e4b00 0x7ff6401470c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:36.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.213+0000 7ff646f2b640 1 --2- 172.21.15.19:0/3095183429 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff6401f3270 0x7ff6401e45c0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:36.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.213+0000 7ff646f2b640 1 --2- 172.21.15.19:0/3095183429 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff628004800 0x7ff64014ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:36.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.213+0000 7ff646f2b640 1 -- 172.21.15.19:0/3095183429 >> 172.21.15.19:0/3095183429 conn(0x7ff64006d110 msgr2=0x7ff640149350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:29:36.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.214+0000 7ff646f2b640 1 -- 172.21.15.19:0/3095183429 shutdown_connections 2024-08-31T12:29:36.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.214+0000 7ff646f2b640 1 -- 172.21.15.19:0/3095183429 wait complete. 2024-08-31T12:29:36.227 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-08-31T12:29:36.227 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-31T12:29:36.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.340+0000 7f89d2a2f640 1 -- 172.21.15.19:0/1235570396 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f89cc071ea0 msgr2=0x7f89cc15c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:36.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.340+0000 7f89d2a2f640 1 --2- 172.21.15.19:0/1235570396 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f89cc071ea0 0x7f89cc15c4b0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f89cc072450 tx=0x7f89c402aae0 comp rx=0 tx=0).stop 2024-08-31T12:29:36.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.340+0000 7f89d2a2f640 1 -- 172.21.15.19:0/1235570396 shutdown_connections 2024-08-31T12:29:36.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.340+0000 7f89d2a2f640 1 --2- 172.21.15.19:0/1235570396 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f89cc071ea0 0x7f89cc15c4b0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:36.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.340+0000 7f89d2a2f640 1 --2- 172.21.15.19:0/1235570396 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f89cc0cac90 0x7f89cc071960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:36.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.340+0000 7f89d2a2f640 1 --2- 172.21.15.19:0/1235570396 >> v2:172.21.15.43:3300/0 conn(0x7f89cc0ca7d0 0x7f89cc1530f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:36.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.340+0000 7f89d2a2f640 1 -- 172.21.15.19:0/1235570396 >> 172.21.15.19:0/1235570396 conn(0x7f89cc0cef50 msgr2=0x7f89cc0cb340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:29:36.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.340+0000 7f89d2a2f640 1 -- 172.21.15.19:0/1235570396 shutdown_connections 2024-08-31T12:29:36.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.340+0000 7f89d2a2f640 1 -- 172.21.15.19:0/1235570396 wait complete. 2024-08-31T12:29:36.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.342+0000 7f89d2a2f640 1 Processor -- start 2024-08-31T12:29:36.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.342+0000 7f89d2a2f640 1 -- start start 2024-08-31T12:29:36.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.343+0000 7f89d2a2f640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f89cc071ea0 0x7f89cc1570d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:29:36.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.343+0000 7f89d2a2f640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f89cc0ca7d0 0x7f89cc15aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:29:36.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.343+0000 7f89d2a2f640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f89cc0cac90 0x7f89cc15a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:29:36.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.343+0000 7f89d2a2f640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f89cc15d330 con 0x7f89cc0ca7d0 2024-08-31T12:29:36.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.343+0000 7f89d2a2f640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f89cc071440 con 0x7f89cc0cac90 2024-08-31T12:29:36.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.343+0000 7f89d2a2f640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f89cc0c58d0 con 0x7f89cc071ea0 2024-08-31T12:29:36.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.343+0000 7f89cb7fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f89cc0ca7d0 0x7f89cc15aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:29:36.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.343+0000 7f89cb7fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f89cc0ca7d0 0x7f89cc15aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:53348/0 (socket says 172.21.15.19:53348) 2024-08-31T12:29:36.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.343+0000 7f89cb7fe640 1 -- 172.21.15.19:0/3978355730 learned_addr learned my addr 172.21.15.19:0/3978355730 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:29:36.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.343+0000 7f89cbfff640 1 --2- 172.21.15.19:0/3978355730 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f89cc071ea0 0x7f89cc1570d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:29:36.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.343+0000 7f89d0fcc640 1 --2- 172.21.15.19:0/3978355730 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f89cc0cac90 0x7f89cc15a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:29:36.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.343+0000 7f89cb7fe640 1 -- 172.21.15.19:0/3978355730 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f89cc071ea0 msgr2=0x7f89cc1570d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:36.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.344+0000 7f89cb7fe640 1 --2- 172.21.15.19:0/3978355730 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f89cc071ea0 0x7f89cc1570d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:36.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.344+0000 7f89cb7fe640 1 -- 172.21.15.19:0/3978355730 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f89cc0cac90 msgr2=0x7f89cc15a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:36.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.344+0000 7f89cb7fe640 1 --2- 172.21.15.19:0/3978355730 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f89cc0cac90 0x7f89cc15a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:36.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.344+0000 7f89cb7fe640 1 -- 172.21.15.19:0/3978355730 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89cc0b3500 con 0x7f89cc0ca7d0 2024-08-31T12:29:36.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.344+0000 7f89d0fcc640 1 --2- 172.21.15.19:0/3978355730 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f89cc0cac90 0x7f89cc15a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:29:36.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.344+0000 7f89cb7fe640 1 --2- 172.21.15.19:0/3978355730 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f89cc0ca7d0 0x7f89cc15aba0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f89c000c970 tx=0x7f89c000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:29:36.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.344+0000 7f89c8ff9640 1 -- 172.21.15.19:0/3978355730 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c0007bb0 con 0x7f89cc0ca7d0 2024-08-31T12:29:36.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.344+0000 7f89c8ff9640 1 -- 172.21.15.19:0/3978355730 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7f89c000f450 con 0x7f89cc0ca7d0 2024-08-31T12:29:36.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.344+0000 7f89c8ff9640 1 -- 172.21.15.19:0/3978355730 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c0007e60 con 0x7f89cc0ca7d0 2024-08-31T12:29:36.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.344+0000 7f89d2a2f640 1 -- 172.21.15.19:0/3978355730 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89cc15b0e0 con 0x7f89cc0ca7d0 2024-08-31T12:29:36.349 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.344+0000 7f89d2a2f640 1 -- 172.21.15.19:0/3978355730 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89cc0b3220 con 0x7f89cc0ca7d0 2024-08-31T12:29:36.349 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.345+0000 7f89d2a2f640 1 -- 172.21.15.19:0/3978355730 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89cc1f3610 con 0x7f89cc0ca7d0 2024-08-31T12:29:36.350 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.346+0000 7f89c8ff9640 1 -- 172.21.15.19:0/3978355730 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f89c000f8c0 con 0x7f89cc0ca7d0 2024-08-31T12:29:36.350 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.346+0000 7f89c8ff9640 1 --2- 172.21.15.19:0/3978355730 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f89b8077a80 0x7f89b8079f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:29:36.350 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.346+0000 7f89c8ff9640 1 -- 172.21.15.19:0/3978355730 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f89c0099e50 con 0x7f89cc0ca7d0 2024-08-31T12:29:36.350 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.346+0000 7f89cbfff640 1 --2- 172.21.15.19:0/3978355730 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f89b8077a80 0x7f89b8079f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:29:36.350 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.347+0000 7f89cbfff640 1 --2- 172.21.15.19:0/3978355730 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f89b8077a80 0x7f89b8079f90 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f89bc005fd0 tx=0x7f89bc00b260 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:29:36.351 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.351+0000 7f89c8ff9640 1 -- 172.21.15.19:0/3978355730 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f89c0014030 con 0x7f89cc0ca7d0 2024-08-31T12:29:36.572 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.571+0000 7f89d2a2f640 1 -- 172.21.15.19:0/3978355730 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f89cc0c43c0 con 0x7f89cc0ca7d0 2024-08-31T12:29:36.572 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.571+0000 7f89c8ff9640 1 -- 172.21.15.19:0/3978355730 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+6901 (secure 0 0 0) 0x7f89c0062b50 con 0x7f89cc0ca7d0 2024-08-31T12:29:36.573 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:29:36.573 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":12,"btime":"2024-08-31T12:16:13:439874+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4371,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6833/4215834946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":4215834946},{"type":"v1","addr":"172.21.15.19:6833","nonce":4215834946}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/543344732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":543344732},{"type":"v1","addr":"172.21.15.19:6835","nonce":543344732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4380,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6837/699077015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6836","nonce":699077015},{"type":"v1","addr":"172.21.15.19:6837","nonce":699077015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-31T12:15:56.652936+0000","modified":"2024-08-31T12:16:13.439859+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4429,"mds_1":4426,"mds_2":4398},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4398":{"gid":4398,"name":"f","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.43:6837/2103490885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6835","nonce":2103490885},{"type":"v1","addr":"172.21.15.43:6837","nonce":2103490885}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4426":{"gid":4426,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.43:6839/1196858061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1196858061},{"type":"v1","addr":"172.21.15.43:6839","nonce":1196858061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4429":{"gid":4429,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.43:6838/3981040655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":3981040655},{"type":"v1","addr":"172.21.15.43:6838","nonce":3981040655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4429,"qdb_cluster":[4398,4426,4429]},"id":1}]} 2024-08-31T12:29:36.573 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 12 2024-08-31T12:29:36.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.573+0000 7f89d2a2f640 1 -- 172.21.15.19:0/3978355730 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f89b8077a80 msgr2=0x7f89b8079f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:36.576 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.573+0000 7f89d2a2f640 1 --2- 172.21.15.19:0/3978355730 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f89b8077a80 0x7f89b8079f90 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f89bc005fd0 tx=0x7f89bc00b260 comp rx=0 tx=0).stop 2024-08-31T12:29:36.576 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.573+0000 7f89d2a2f640 1 -- 172.21.15.19:0/3978355730 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f89cc0ca7d0 msgr2=0x7f89cc15aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:36.576 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.573+0000 7f89d2a2f640 1 --2- 172.21.15.19:0/3978355730 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f89cc0ca7d0 0x7f89cc15aba0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f89c000c970 tx=0x7f89c000ce30 comp rx=0 tx=0).stop 2024-08-31T12:29:36.576 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.573+0000 7f89d2a2f640 1 -- 172.21.15.19:0/3978355730 shutdown_connections 2024-08-31T12:29:36.576 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.573+0000 7f89d2a2f640 1 --2- 172.21.15.19:0/3978355730 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f89b8077a80 0x7f89b8079f90 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:36.576 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.574+0000 7f89d2a2f640 1 --2- 172.21.15.19:0/3978355730 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f89cc0cac90 0x7f89cc15a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:36.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.574+0000 7f89d2a2f640 1 --2- 172.21.15.19:0/3978355730 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f89cc0ca7d0 0x7f89cc15aba0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:36.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.574+0000 7f89d2a2f640 1 --2- 172.21.15.19:0/3978355730 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f89cc071ea0 0x7f89cc1570d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:36.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.574+0000 7f89d2a2f640 1 -- 172.21.15.19:0/3978355730 >> 172.21.15.19:0/3978355730 conn(0x7f89cc0cef50 msgr2=0x7f89cc072e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:29:36.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.574+0000 7f89d2a2f640 1 -- 172.21.15.19:0/3978355730 shutdown_connections 2024-08-31T12:29:36.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.574+0000 7f89d2a2f640 1 -- 172.21.15.19:0/3978355730 wait complete. 2024-08-31T12:29:36.586 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-08-31T12:29:36.700 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.699+0000 7f7ad4da1640 1 -- 172.21.15.19:0/2587146575 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7ad0074130 msgr2=0x7f7ad00c9090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:36.700 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.699+0000 7f7ad4da1640 1 --2- 172.21.15.19:0/2587146575 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7ad0074130 0x7f7ad00c9090 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f7ad006fd00 tx=0x7f7ac802a530 comp rx=0 tx=0).stop 2024-08-31T12:29:36.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.699+0000 7f7ad4da1640 1 -- 172.21.15.19:0/2587146575 shutdown_connections 2024-08-31T12:29:36.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.699+0000 7f7ad4da1640 1 --2- 172.21.15.19:0/2587146575 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7ad0074130 0x7f7ad00c9090 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:36.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.699+0000 7f7ad4da1640 1 --2- 172.21.15.19:0/2587146575 >> v2:172.21.15.43:3300/0 conn(0x7f7ad00c7e20 0x7f7ad0074840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:36.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.699+0000 7f7ad4da1640 1 -- 172.21.15.19:0/2587146575 >> 172.21.15.19:0/2587146575 conn(0x7f7ad00c9f20 msgr2=0x7f7ad00ca330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:29:36.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.699+0000 7f7ad4da1640 1 -- 172.21.15.19:0/2587146575 shutdown_connections 2024-08-31T12:29:36.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.699+0000 7f7ad4da1640 1 -- 172.21.15.19:0/2587146575 wait complete. 2024-08-31T12:29:36.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.701+0000 7f7ad4da1640 1 Processor -- start 2024-08-31T12:29:36.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.701+0000 7f7ad4da1640 1 -- start start 2024-08-31T12:29:36.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.702+0000 7f7ad4da1640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ad0073d20 0x7f7ad01e2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:29:36.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.702+0000 7f7ad4da1640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7ad0074130 0x7f7ad014d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:29:36.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.702+0000 7f7ad4da1640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ad00c7e20 0x7f7ad01f0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:29:36.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.702+0000 7f7ad4da1640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad0074d80 con 0x7f7ad0074130 2024-08-31T12:29:36.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.702+0000 7f7ad4da1640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad00c7480 con 0x7f7ad00c7e20 2024-08-31T12:29:36.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.702+0000 7f7ad4da1640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f7ad00c5410 con 0x7f7ad0073d20 2024-08-31T12:29:36.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.702+0000 7f7acdd9b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7ad0074130 0x7f7ad014d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:29:36.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.702+0000 7f7acdd9b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7ad0074130 0x7f7ad014d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:53364/0 (socket says 172.21.15.19:53364) 2024-08-31T12:29:36.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.702+0000 7f7acdd9b640 1 -- 172.21.15.19:0/181170622 learned_addr learned my addr 172.21.15.19:0/181170622 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:29:36.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.702+0000 7f7aced9d640 1 --2- 172.21.15.19:0/181170622 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ad00c7e20 0x7f7ad01f0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:29:36.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.702+0000 7f7ace59c640 1 --2- 172.21.15.19:0/181170622 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ad0073d20 0x7f7ad01e2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:29:36.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.702+0000 7f7acdd9b640 1 -- 172.21.15.19:0/181170622 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ad0073d20 msgr2=0x7f7ad01e2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:36.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.702+0000 7f7acdd9b640 1 --2- 172.21.15.19:0/181170622 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ad0073d20 0x7f7ad01e2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:36.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.702+0000 7f7acdd9b640 1 -- 172.21.15.19:0/181170622 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ad00c7e20 msgr2=0x7f7ad01f0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:36.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.702+0000 7f7acdd9b640 1 --2- 172.21.15.19:0/181170622 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ad00c7e20 0x7f7ad01f0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:36.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.702+0000 7f7acdd9b640 1 -- 172.21.15.19:0/181170622 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ad01e26a0 con 0x7f7ad0074130 2024-08-31T12:29:36.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.702+0000 7f7aced9d640 1 --2- 172.21.15.19:0/181170622 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ad00c7e20 0x7f7ad01f0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:29:36.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.703+0000 7f7acdd9b640 1 --2- 172.21.15.19:0/181170622 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7ad0074130 0x7f7ad014d280 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f7ab800c3d0 tx=0x7f7ab800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:29:36.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.703+0000 7f7ac6ffd640 1 -- 172.21.15.19:0/181170622 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ab8011790 con 0x7f7ad0074130 2024-08-31T12:29:36.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.703+0000 7f7ac6ffd640 1 -- 172.21.15.19:0/181170622 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7f7ab8017070 con 0x7f7ad0074130 2024-08-31T12:29:36.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.703+0000 7f7ad4da1640 1 -- 172.21.15.19:0/181170622 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ad0156750 con 0x7f7ad0074130 2024-08-31T12:29:36.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.703+0000 7f7ac6ffd640 1 -- 172.21.15.19:0/181170622 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ab8011bf0 con 0x7f7ad0074130 2024-08-31T12:29:36.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.703+0000 7f7ad4da1640 1 -- 172.21.15.19:0/181170622 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ad01e1960 con 0x7f7ad0074130 2024-08-31T12:29:36.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.704+0000 7f7ad4da1640 1 -- 172.21.15.19:0/181170622 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ad01564c0 con 0x7f7ad0074130 2024-08-31T12:29:36.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.705+0000 7f7ac6ffd640 1 -- 172.21.15.19:0/181170622 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f7ab801c460 con 0x7f7ad0074130 2024-08-31T12:29:36.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.705+0000 7f7ac6ffd640 1 --2- 172.21.15.19:0/181170622 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7ab0077a50 0x7f7ab0079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:29:36.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.705+0000 7f7ac6ffd640 1 -- 172.21.15.19:0/181170622 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f7ab8099450 con 0x7f7ad0074130 2024-08-31T12:29:36.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.705+0000 7f7ace59c640 1 --2- 172.21.15.19:0/181170622 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7ab0077a50 0x7f7ab0079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:29:36.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.706+0000 7f7ace59c640 1 --2- 172.21.15.19:0/181170622 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7ab0077a50 0x7f7ab0079f60 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f7ac0005fd0 tx=0x7f7ac0009500 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:29:36.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.710+0000 7f7ac6ffd640 1 -- 172.21.15.19:0/181170622 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f7ab8062140 con 0x7f7ad0074130 2024-08-31T12:29:36.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.929+0000 7f7ad4da1640 1 -- 172.21.15.19:0/181170622 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ad0152170 con 0x7f7ad0074130 2024-08-31T12:29:36.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.929+0000 7f7ac6ffd640 1 -- 172.21.15.19:0/181170622 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+6901 (secure 0 0 0) 0x7f7ab8061a40 con 0x7f7ad0074130 2024-08-31T12:29:36.931 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:29:36.931 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":12,"btime":"2024-08-31T12:16:13:439874+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4371,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6833/4215834946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":4215834946},{"type":"v1","addr":"172.21.15.19:6833","nonce":4215834946}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/543344732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":543344732},{"type":"v1","addr":"172.21.15.19:6835","nonce":543344732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4380,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6837/699077015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6836","nonce":699077015},{"type":"v1","addr":"172.21.15.19:6837","nonce":699077015}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-08-31T12:15:56.652936+0000","modified":"2024-08-31T12:16:13.439859+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4429,"mds_1":4426,"mds_2":4398},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4398":{"gid":4398,"name":"f","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.43:6837/2103490885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6835","nonce":2103490885},{"type":"v1","addr":"172.21.15.43:6837","nonce":2103490885}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4426":{"gid":4426,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.43:6839/1196858061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1196858061},{"type":"v1","addr":"172.21.15.43:6839","nonce":1196858061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4429":{"gid":4429,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.43:6838/3981040655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":3981040655},{"type":"v1","addr":"172.21.15.43:6838","nonce":3981040655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4429,"qdb_cluster":[4398,4426,4429]},"id":1}]} 2024-08-31T12:29:36.931 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 12 2024-08-31T12:29:36.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.931+0000 7f7ad4da1640 1 -- 172.21.15.19:0/181170622 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7ab0077a50 msgr2=0x7f7ab0079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:36.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.931+0000 7f7ad4da1640 1 --2- 172.21.15.19:0/181170622 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7ab0077a50 0x7f7ab0079f60 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f7ac0005fd0 tx=0x7f7ac0009500 comp rx=0 tx=0).stop 2024-08-31T12:29:36.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.931+0000 7f7ad4da1640 1 -- 172.21.15.19:0/181170622 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7ad0074130 msgr2=0x7f7ad014d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:36.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.931+0000 7f7ad4da1640 1 --2- 172.21.15.19:0/181170622 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7ad0074130 0x7f7ad014d280 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f7ab800c3d0 tx=0x7f7ab800c890 comp rx=0 tx=0).stop 2024-08-31T12:29:36.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.931+0000 7f7ad4da1640 1 -- 172.21.15.19:0/181170622 shutdown_connections 2024-08-31T12:29:36.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.931+0000 7f7ad4da1640 1 --2- 172.21.15.19:0/181170622 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f7ab0077a50 0x7f7ab0079f60 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:36.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.931+0000 7f7ad4da1640 1 --2- 172.21.15.19:0/181170622 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ad00c7e20 0x7f7ad01f0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:36.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.931+0000 7f7ad4da1640 1 --2- 172.21.15.19:0/181170622 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7ad0074130 0x7f7ad014d280 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:36.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.931+0000 7f7ad4da1640 1 --2- 172.21.15.19:0/181170622 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ad0073d20 0x7f7ad01e2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:36.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.931+0000 7f7ad4da1640 1 -- 172.21.15.19:0/181170622 >> 172.21.15.19:0/181170622 conn(0x7f7ad00c9f20 msgr2=0x7f7ad0071380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:29:36.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.931+0000 7f7ad4da1640 1 -- 172.21.15.19:0/181170622 shutdown_connections 2024-08-31T12:29:36.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:36.931+0000 7f7ad4da1640 1 -- 172.21.15.19:0/181170622 wait complete. 2024-08-31T12:29:36.944 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-31T12:29:37.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.057+0000 7f1d42bf8640 1 -- 172.21.15.19:0/725609656 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d3c0cdab0 msgr2=0x7f1d3c0d0cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:37.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.057+0000 7f1d42bf8640 1 --2- 172.21.15.19:0/725609656 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d3c0cdab0 0x7f1d3c0d0cd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c009830 tx=0x7f1d2c02a640 comp rx=0 tx=0).stop 2024-08-31T12:29:37.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.057+0000 7f1d42bf8640 1 -- 172.21.15.19:0/725609656 shutdown_connections 2024-08-31T12:29:37.059 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.057+0000 7f1d42bf8640 1 --2- 172.21.15.19:0/725609656 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d3c0cdab0 0x7f1d3c0d0cd0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:37.059 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.057+0000 7f1d42bf8640 1 -- 172.21.15.19:0/725609656 >> 172.21.15.19:0/725609656 conn(0x7f1d3c06e6e0 msgr2=0x7f1d3c06eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:29:37.059 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.057+0000 7f1d42bf8640 1 -- 172.21.15.19:0/725609656 shutdown_connections 2024-08-31T12:29:37.059 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.057+0000 7f1d42bf8640 1 -- 172.21.15.19:0/725609656 wait complete. 2024-08-31T12:29:37.060 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.058+0000 7f1d42bf8640 1 Processor -- start 2024-08-31T12:29:37.060 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.058+0000 7f1d42bf8640 1 -- start start 2024-08-31T12:29:37.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.060+0000 7f1d42bf8640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d3c0cc620 0x7f1d3c1e6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:29:37.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.060+0000 7f1d42bf8640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d3c0cdab0 0x7f1d3c0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:29:37.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.060+0000 7f1d42bf8640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d3c0d1210 0x7f1d3c158020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:29:37.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.060+0000 7f1d42bf8640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f1d3c15cdb0 con 0x7f1d3c0cdab0 2024-08-31T12:29:37.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.060+0000 7f1d42bf8640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1d3c15cad0 con 0x7f1d3c0cc620 2024-08-31T12:29:37.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.060+0000 7f1d42bf8640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f1d3c15d090 con 0x7f1d3c0d1210 2024-08-31T12:29:37.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.060+0000 7f1d33fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d3c0cdab0 0x7f1d3c0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:29:37.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.060+0000 7f1d33fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d3c0cdab0 0x7f1d3c0b4d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:53376/0 (socket says 172.21.15.19:53376) 2024-08-31T12:29:37.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.060+0000 7f1d33fff640 1 -- 172.21.15.19:0/3367633384 learned_addr learned my addr 172.21.15.19:0/3367633384 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:29:37.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.060+0000 7f1d40994640 1 --2- 172.21.15.19:0/3367633384 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d3c0cc620 0x7f1d3c1e6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:29:37.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.060+0000 7f1d41195640 1 --2- 172.21.15.19:0/3367633384 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d3c0d1210 0x7f1d3c158020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:29:37.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.060+0000 7f1d33fff640 1 -- 172.21.15.19:0/3367633384 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d3c0d1210 msgr2=0x7f1d3c158020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:37.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.060+0000 7f1d33fff640 1 --2- 172.21.15.19:0/3367633384 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d3c0d1210 0x7f1d3c158020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:37.064 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.060+0000 7f1d33fff640 1 -- 172.21.15.19:0/3367633384 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d3c0cc620 msgr2=0x7f1d3c1e6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:37.064 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.060+0000 7f1d33fff640 1 --2- 172.21.15.19:0/3367633384 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d3c0cc620 0x7f1d3c1e6360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:37.064 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.060+0000 7f1d33fff640 1 -- 172.21.15.19:0/3367633384 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d3c1e4a60 con 0x7f1d3c0cdab0 2024-08-31T12:29:37.064 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.060+0000 7f1d33fff640 1 --2- 172.21.15.19:0/3367633384 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d3c0cdab0 0x7f1d3c0b4d90 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c009920 tx=0x7f1d2c0055e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:29:37.064 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.061+0000 7f1d317fa640 1 -- 172.21.15.19:0/3367633384 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c02ae60 con 0x7f1d3c0cdab0 2024-08-31T12:29:37.064 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.061+0000 7f1d317fa640 1 -- 172.21.15.19:0/3367633384 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7f1d2c031be0 con 0x7f1d3c0cdab0 2024-08-31T12:29:37.065 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.061+0000 7f1d42bf8640 1 -- 172.21.15.19:0/3367633384 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d3c1e44c0 con 0x7f1d3c0cdab0 2024-08-31T12:29:37.065 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.061+0000 7f1d317fa640 1 -- 172.21.15.19:0/3367633384 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c033070 con 0x7f1d3c0cdab0 2024-08-31T12:29:37.065 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.061+0000 7f1d42bf8640 1 -- 172.21.15.19:0/3367633384 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d3c0c6cb0 con 0x7f1d3c0cdab0 2024-08-31T12:29:37.066 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.062+0000 7f1d42bf8640 1 -- 172.21.15.19:0/3367633384 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d3c0cb2c0 con 0x7f1d3c0cdab0 2024-08-31T12:29:37.066 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.062+0000 7f1d317fa640 1 -- 172.21.15.19:0/3367633384 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f1d2c033210 con 0x7f1d3c0cdab0 2024-08-31T12:29:37.066 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.063+0000 7f1d317fa640 1 --2- 172.21.15.19:0/3367633384 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1d28077a50 0x7f1d28079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:29:37.066 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.063+0000 7f1d317fa640 1 -- 172.21.15.19:0/3367633384 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f1d2c0ba4c0 con 0x7f1d3c0cdab0 2024-08-31T12:29:37.066 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.063+0000 7f1d40994640 1 --2- 172.21.15.19:0/3367633384 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1d28077a50 0x7f1d28079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:29:37.066 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.064+0000 7f1d40994640 1 --2- 172.21.15.19:0/3367633384 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1d28077a50 0x7f1d28079f60 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f1d38000f60 tx=0x7f1d38009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:29:37.069 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.068+0000 7f1d317fa640 1 -- 172.21.15.19:0/3367633384 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f1d2c04a030 con 0x7f1d3c0cdab0 2024-08-31T12:29:37.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.254+0000 7f1d42bf8640 1 -- 172.21.15.19:0/3367633384 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1d3c1e7430 con 0x7f1d3c0cdab0 2024-08-31T12:29:37.256 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.255+0000 7f1d317fa640 1 -- 172.21.15.19:0/3367633384 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) ==== 74+0+16866 (secure 0 0 0) 0x7f1d2c03f030 con 0x7f1d3c0cdab0 2024-08-31T12:29:37.256 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-08-31T12:29:37.257 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":23,"fsid":"65c40ef5-9e6a-4782-8f93-4032c6800123","created":"2024-08-31T12:15:32.084309+0000","modified":"2024-08-31T12:16:40.637326+0000","last_up_change":"2024-08-31T12:15:45.213688+0000","last_in_change":"2024-08-31T12:15:35.545202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-31T12:15:44.508197+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-08-31T12:15:54.512584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-08-31T12:15:55.618251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b80d6a3f-7e4e-4d4d-b1b4-0df71c9176ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6800","nonce":680447146},{"type":"v1","addr":"172.21.15.19:6801","nonce":680447146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":680447146},{"type":"v1","addr":"172.21.15.19:6803","nonce":680447146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":680447146},{"type":"v1","addr":"172.21.15.19:6807","nonce":680447146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":680447146},{"type":"v1","addr":"172.21.15.19:6805","nonce":680447146}]},"public_addr":"172.21.15.19:6801/680447146","cluster_addr":"172.21.15.19:6803/680447146","heartbeat_back_addr":"172.21.15.19:6807/680447146","heartbeat_front_addr":"172.21.15.19:6805/680447146","state":["exists","up"]},{"osd":1,"uuid":"6f986366-82a6-4133-9f2e-43f84c7b03bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":1522610034},{"type":"v1","addr":"172.21.15.19:6817","nonce":1522610034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":1522610034},{"type":"v1","addr":"172.21.15.19:6819","nonce":1522610034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":1522610034},{"type":"v1","addr":"172.21.15.19:6823","nonce":1522610034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":1522610034},{"type":"v1","addr":"172.21.15.19:6821","nonce":1522610034}]},"public_addr":"172.21.15.19:6817/1522610034","cluster_addr":"172.21.15.19:6819/1522610034","heartbeat_back_addr":"172.21.15.19:6823/1522610034","heartbeat_front_addr":"172.21.15.19:6821/1522610034","state":["exists","up"]},{"osd":2,"uuid":"ce8c2068-845c-4cbb-a4d1-077375d605ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6809","nonce":2332368824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6811","nonce":2332368824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6815","nonce":2332368824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":2332368824},{"type":"v1","addr":"172.21.15.19:6813","nonce":2332368824}]},"public_addr":"172.21.15.19:6809/2332368824","cluster_addr":"172.21.15.19:6811/2332368824","heartbeat_back_addr":"172.21.15.19:6815/2332368824","heartbeat_front_addr":"172.21.15.19:6813/2332368824","state":["exists","up"]},{"osd":3,"uuid":"b1cb0ef4-aa9b-421b-a94e-82de0c46e5cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":3785550598},{"type":"v1","addr":"172.21.15.19:6825","nonce":3785550598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":3785550598},{"type":"v1","addr":"172.21.15.19:6827","nonce":3785550598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":3785550598},{"type":"v1","addr":"172.21.15.19:6831","nonce":3785550598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":3785550598},{"type":"v1","addr":"172.21.15.19:6829","nonce":3785550598}]},"public_addr":"172.21.15.19:6825/3785550598","cluster_addr":"172.21.15.19:6827/3785550598","heartbeat_back_addr":"172.21.15.19:6831/3785550598","heartbeat_front_addr":"172.21.15.19:6829/3785550598","state":["exists","up"]},{"osd":4,"uuid":"825b58ec-a6da-4515-b6a5-81debe327011","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":1985519349},{"type":"v1","addr":"172.21.15.43:6819","nonce":1985519349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":1985519349},{"type":"v1","addr":"172.21.15.43:6821","nonce":1985519349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":1985519349},{"type":"v1","addr":"172.21.15.43:6825","nonce":1985519349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":1985519349},{"type":"v1","addr":"172.21.15.43:6823","nonce":1985519349}]},"public_addr":"172.21.15.43:6819/1985519349","cluster_addr":"172.21.15.43:6821/1985519349","heartbeat_back_addr":"172.21.15.43:6825/1985519349","heartbeat_front_addr":"172.21.15.43:6823/1985519349","state":["exists","up"]},{"osd":5,"uuid":"7f34e537-5997-40e4-92e7-2570507160c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":4271971521},{"type":"v1","addr":"172.21.15.43:6803","nonce":4271971521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":4271971521},{"type":"v1","addr":"172.21.15.43:6805","nonce":4271971521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":4271971521},{"type":"v1","addr":"172.21.15.43:6809","nonce":4271971521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":4271971521},{"type":"v1","addr":"172.21.15.43:6807","nonce":4271971521}]},"public_addr":"172.21.15.43:6803/4271971521","cluster_addr":"172.21.15.43:6805/4271971521","heartbeat_back_addr":"172.21.15.43:6809/4271971521","heartbeat_front_addr":"172.21.15.43:6807/4271971521","state":["exists","up"]},{"osd":6,"uuid":"195dd8ad-640a-4fa3-9ee4-06e4172e0464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":2863384694},{"type":"v1","addr":"172.21.15.43:6827","nonce":2863384694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":2863384694},{"type":"v1","addr":"172.21.15.43:6829","nonce":2863384694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":2863384694},{"type":"v1","addr":"172.21.15.43:6833","nonce":2863384694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":2863384694},{"type":"v1","addr":"172.21.15.43:6831","nonce":2863384694}]},"public_addr":"172.21.15.43:6827/2863384694","cluster_addr":"172.21.15.43:6829/2863384694","heartbeat_back_addr":"172.21.15.43:6833/2863384694","heartbeat_front_addr":"172.21.15.43:6831/2863384694","state":["exists","up"]},{"osd":7,"uuid":"4645661f-f430-4f05-bdd1-69138dcd1bbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":792572234},{"type":"v1","addr":"172.21.15.43:6811","nonce":792572234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":792572234},{"type":"v1","addr":"172.21.15.43:6813","nonce":792572234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":792572234},{"type":"v1","addr":"172.21.15.43:6817","nonce":792572234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":792572234},{"type":"v1","addr":"172.21.15.43:6815","nonce":792572234}]},"public_addr":"172.21.15.43:6811/792572234","cluster_addr":"172.21.15.43:6813/792572234","heartbeat_back_addr":"172.21.15.43:6817/792572234","heartbeat_front_addr":"172.21.15.43:6815/792572234","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.156485+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.578126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.075440+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.639867+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.543602+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:41.919592+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:42.615557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T12:15:41.813337+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-31T12:29:37.258 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.257+0000 7f1d42bf8640 1 -- 172.21.15.19:0/3367633384 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1d28077a50 msgr2=0x7f1d28079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:37.258 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.257+0000 7f1d42bf8640 1 --2- 172.21.15.19:0/3367633384 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1d28077a50 0x7f1d28079f60 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f1d38000f60 tx=0x7f1d38009040 comp rx=0 tx=0).stop 2024-08-31T12:29:37.258 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.257+0000 7f1d42bf8640 1 -- 172.21.15.19:0/3367633384 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d3c0cdab0 msgr2=0x7f1d3c0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:37.259 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.257+0000 7f1d42bf8640 1 --2- 172.21.15.19:0/3367633384 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d3c0cdab0 0x7f1d3c0b4d90 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c009920 tx=0x7f1d2c0055e0 comp rx=0 tx=0).stop 2024-08-31T12:29:37.259 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.257+0000 7f1d42bf8640 1 -- 172.21.15.19:0/3367633384 shutdown_connections 2024-08-31T12:29:37.259 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.257+0000 7f1d42bf8640 1 --2- 172.21.15.19:0/3367633384 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1d28077a50 0x7f1d28079f60 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:37.259 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.257+0000 7f1d42bf8640 1 --2- 172.21.15.19:0/3367633384 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d3c0d1210 0x7f1d3c158020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:37.259 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.257+0000 7f1d42bf8640 1 --2- 172.21.15.19:0/3367633384 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d3c0cdab0 0x7f1d3c0b4d90 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:37.259 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.257+0000 7f1d42bf8640 1 --2- 172.21.15.19:0/3367633384 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d3c0cc620 0x7f1d3c1e6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:37.259 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.257+0000 7f1d42bf8640 1 -- 172.21.15.19:0/3367633384 >> 172.21.15.19:0/3367633384 conn(0x7f1d3c06e6e0 msgr2=0x7f1d3c0d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:29:37.260 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.257+0000 7f1d42bf8640 1 -- 172.21.15.19:0/3367633384 shutdown_connections 2024-08-31T12:29:37.260 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.257+0000 7f1d42bf8640 1 -- 172.21.15.19:0/3367633384 wait complete. 2024-08-31T12:29:37.270 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-08-31T12:29:37.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.384+0000 7f0e3e27e640 1 -- 172.21.15.19:0/3101834623 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0e3806e0e0 msgr2=0x7f0e380c9850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:37.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.384+0000 7f0e3e27e640 1 --2- 172.21.15.19:0/3101834623 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0e3806e0e0 0x7f0e380c9850 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c009830 tx=0x7f0e2c02a5a0 comp rx=0 tx=0).stop 2024-08-31T12:29:37.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.384+0000 7f0e3e27e640 1 -- 172.21.15.19:0/3101834623 shutdown_connections 2024-08-31T12:29:37.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.384+0000 7f0e3e27e640 1 --2- 172.21.15.19:0/3101834623 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0e3806e0e0 0x7f0e380c9850 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:37.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.384+0000 7f0e3e27e640 1 -- 172.21.15.19:0/3101834623 >> 172.21.15.19:0/3101834623 conn(0x7f0e380690d0 msgr2=0x7f0e380d0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:29:37.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.384+0000 7f0e3e27e640 1 -- 172.21.15.19:0/3101834623 shutdown_connections 2024-08-31T12:29:37.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.384+0000 7f0e3e27e640 1 -- 172.21.15.19:0/3101834623 wait complete. 2024-08-31T12:29:37.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.385+0000 7f0e3e27e640 1 Processor -- start 2024-08-31T12:29:37.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.386+0000 7f0e3e27e640 1 -- start start 2024-08-31T12:29:37.389 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.387+0000 7f0e3e27e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0e3806e0e0 0x7f0e381f1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:29:37.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.387+0000 7f0e3e27e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e380c8a40 0x7f0e3814e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:29:37.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.387+0000 7f0e3e27e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0e38155300 0x7f0e3814ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:29:37.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.387+0000 7f0e3e27e640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f0e3815b250 con 0x7f0e38155300 2024-08-31T12:29:37.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.387+0000 7f0e3e27e640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0e3815af70 con 0x7f0e380c8a40 2024-08-31T12:29:37.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.387+0000 7f0e3e27e640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f0e380c8610 con 0x7f0e3806e0e0 2024-08-31T12:29:37.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.387+0000 7f0e3c81b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0e38155300 0x7f0e3814ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:29:37.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.387+0000 7f0e3c81b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0e38155300 0x7f0e3814ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:53396/0 (socket says 172.21.15.19:53396) 2024-08-31T12:29:37.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.387+0000 7f0e3c81b640 1 -- 172.21.15.19:0/3474498123 learned_addr learned my addr 172.21.15.19:0/3474498123 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:29:37.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.387+0000 7f0e377fe640 1 --2- 172.21.15.19:0/3474498123 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e380c8a40 0x7f0e3814e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:29:37.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.387+0000 7f0e37fff640 1 --2- 172.21.15.19:0/3474498123 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0e3806e0e0 0x7f0e381f1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:29:37.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.387+0000 7f0e3c81b640 1 -- 172.21.15.19:0/3474498123 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0e3806e0e0 msgr2=0x7f0e381f1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:37.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.387+0000 7f0e3c81b640 1 --2- 172.21.15.19:0/3474498123 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0e3806e0e0 0x7f0e381f1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:37.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.387+0000 7f0e3c81b640 1 -- 172.21.15.19:0/3474498123 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e380c8a40 msgr2=0x7f0e3814e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:37.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.387+0000 7f0e3c81b640 1 --2- 172.21.15.19:0/3474498123 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e380c8a40 0x7f0e3814e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:37.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.387+0000 7f0e3c81b640 1 -- 172.21.15.19:0/3474498123 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e3814e390 con 0x7f0e38155300 2024-08-31T12:29:37.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.388+0000 7f0e3c81b640 1 --2- 172.21.15.19:0/3474498123 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0e38155300 0x7f0e3814ebb0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f0e3000b9a0 tx=0x7f0e3000be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:29:37.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.388+0000 7f0e34ff9640 1 -- 172.21.15.19:0/3474498123 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e30019070 con 0x7f0e38155300 2024-08-31T12:29:37.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.388+0000 7f0e34ff9640 1 -- 172.21.15.19:0/3474498123 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7f0e3000cba0 con 0x7f0e38155300 2024-08-31T12:29:37.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.388+0000 7f0e3e27e640 1 -- 172.21.15.19:0/3474498123 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e380c67e0 con 0x7f0e38155300 2024-08-31T12:29:37.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.388+0000 7f0e34ff9640 1 -- 172.21.15.19:0/3474498123 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e30014440 con 0x7f0e38155300 2024-08-31T12:29:37.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.388+0000 7f0e3e27e640 1 -- 172.21.15.19:0/3474498123 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e380c6da0 con 0x7f0e38155300 2024-08-31T12:29:37.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.389+0000 7f0e3e27e640 1 -- 172.21.15.19:0/3474498123 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e3815a030 con 0x7f0e38155300 2024-08-31T12:29:37.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.389+0000 7f0e34ff9640 1 -- 172.21.15.19:0/3474498123 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f0e3001c070 con 0x7f0e38155300 2024-08-31T12:29:37.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.390+0000 7f0e34ff9640 1 --2- 172.21.15.19:0/3474498123 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0e24077a50 0x7f0e24079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:29:37.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.390+0000 7f0e34ff9640 1 -- 172.21.15.19:0/3474498123 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f0e300993b0 con 0x7f0e38155300 2024-08-31T12:29:37.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.390+0000 7f0e37fff640 1 --2- 172.21.15.19:0/3474498123 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0e24077a50 0x7f0e24079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:29:37.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.391+0000 7f0e37fff640 1 --2- 172.21.15.19:0/3474498123 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0e24077a50 0x7f0e24079f60 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f0e28007fd0 tx=0x7f0e28009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:29:37.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.394+0000 7f0e34ff9640 1 -- 172.21.15.19:0/3474498123 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f0e3815a030 con 0x7f0e38155300 2024-08-31T12:29:37.615 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.614+0000 7f0e3e27e640 1 -- 172.21.15.19:0/3474498123 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f0e3806d860 con 0x7f0e38155300 2024-08-31T12:29:37.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.628+0000 7f0e34ff9640 1 -- 172.21.15.19:0/3474498123 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v13) ==== 161+0+0 (secure 0 0 0) 0x7f0e300619c0 con 0x7f0e38155300 2024-08-31T12:29:37.629 INFO:teuthology.orchestra.run.smithi019.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-08-31T12:29:37.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.631+0000 7f0e3e27e640 1 -- 172.21.15.19:0/3474498123 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0e24077a50 msgr2=0x7f0e24079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:37.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.631+0000 7f0e3e27e640 1 --2- 172.21.15.19:0/3474498123 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0e24077a50 0x7f0e24079f60 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f0e28007fd0 tx=0x7f0e28009040 comp rx=0 tx=0).stop 2024-08-31T12:29:37.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.631+0000 7f0e3e27e640 1 -- 172.21.15.19:0/3474498123 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0e38155300 msgr2=0x7f0e3814ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:37.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.631+0000 7f0e3e27e640 1 --2- 172.21.15.19:0/3474498123 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0e38155300 0x7f0e3814ebb0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f0e3000b9a0 tx=0x7f0e3000be60 comp rx=0 tx=0).stop 2024-08-31T12:29:37.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.631+0000 7f0e3e27e640 1 -- 172.21.15.19:0/3474498123 shutdown_connections 2024-08-31T12:29:37.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.631+0000 7f0e3e27e640 1 --2- 172.21.15.19:0/3474498123 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f0e24077a50 0x7f0e24079f60 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:37.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.631+0000 7f0e3e27e640 1 --2- 172.21.15.19:0/3474498123 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0e38155300 0x7f0e3814ebb0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:37.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.631+0000 7f0e3e27e640 1 --2- 172.21.15.19:0/3474498123 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e380c8a40 0x7f0e3814e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:37.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.631+0000 7f0e3e27e640 1 --2- 172.21.15.19:0/3474498123 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0e3806e0e0 0x7f0e381f1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:37.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.631+0000 7f0e3e27e640 1 -- 172.21.15.19:0/3474498123 >> 172.21.15.19:0/3474498123 conn(0x7f0e380690d0 msgr2=0x7f0e380cb2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:29:37.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.631+0000 7f0e3e27e640 1 -- 172.21.15.19:0/3474498123 shutdown_connections 2024-08-31T12:29:37.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.631+0000 7f0e3e27e640 1 -- 172.21.15.19:0/3474498123 wait complete. 2024-08-31T12:29:37.644 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-08-31T12:29:37.708 INFO:tasks.ceph.mds.b.smithi043.stdout:starting mds.b at 2024-08-31T12:29:37.714 INFO:tasks.ceph.mds.f.smithi043.stdout:starting mds.f at 2024-08-31T12:29:37.714 INFO:tasks.ceph.mds.d.smithi043.stdout:starting mds.d at 2024-08-31T12:29:37.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.761+0000 7f16feea9640 1 -- 172.21.15.19:0/347233033 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f16f815ec70 msgr2=0x7f16f815f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:37.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.761+0000 7f16feea9640 1 --2- 172.21.15.19:0/347233033 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f16f815ec70 0x7f16f815f080 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f16e4005fd0 tx=0x7f16e402a670 comp rx=0 tx=0).stop 2024-08-31T12:29:37.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.761+0000 7f16feea9640 1 -- 172.21.15.19:0/347233033 shutdown_connections 2024-08-31T12:29:37.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.761+0000 7f16feea9640 1 --2- 172.21.15.19:0/347233033 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f16f815ec70 0x7f16f815f080 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:37.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.761+0000 7f16feea9640 1 --2- 172.21.15.19:0/347233033 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f16f80d0b50 0x7f16f80d0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:37.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.761+0000 7f16feea9640 1 -- 172.21.15.19:0/347233033 >> 172.21.15.19:0/347233033 conn(0x7f16f8074210 msgr2=0x7f16f8074620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:29:37.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.761+0000 7f16feea9640 1 -- 172.21.15.19:0/347233033 shutdown_connections 2024-08-31T12:29:37.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.761+0000 7f16feea9640 1 -- 172.21.15.19:0/347233033 wait complete. 2024-08-31T12:29:37.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.763+0000 7f16feea9640 1 Processor -- start 2024-08-31T12:29:37.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.763+0000 7f16feea9640 1 -- start start 2024-08-31T12:29:37.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.764+0000 7f16feea9640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f16f80cd1b0 0x7f16f815cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:29:37.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.764+0000 7f16feea9640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f16f80d0b50 0x7f16f80c8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:29:37.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.764+0000 7f16feea9640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16f815ec70 0x7f16f8148f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:29:37.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.764+0000 7f16feea9640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f16f8152170 con 0x7f16f80cd1b0 2024-08-31T12:29:37.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.764+0000 7f16feea9640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f16f80cc130 con 0x7f16f815ec70 2024-08-31T12:29:37.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.765+0000 7f16feea9640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f16f8155680 con 0x7f16f80d0b50 2024-08-31T12:29:37.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.765+0000 7f16fcc45640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f16f80cd1b0 0x7f16f815cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:29:37.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.765+0000 7f16fcc45640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f16f80cd1b0 0x7f16f815cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:53424/0 (socket says 172.21.15.19:53424) 2024-08-31T12:29:37.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.765+0000 7f16fcc45640 1 -- 172.21.15.19:0/2573220343 learned_addr learned my addr 172.21.15.19:0/2573220343 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:29:37.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.765+0000 7f16fd446640 1 --2- 172.21.15.19:0/2573220343 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16f815ec70 0x7f16f8148f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:29:37.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.765+0000 7f16f7fff640 1 --2- 172.21.15.19:0/2573220343 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f16f80d0b50 0x7f16f80c8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:29:37.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.765+0000 7f16fcc45640 1 -- 172.21.15.19:0/2573220343 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f16f80d0b50 msgr2=0x7f16f80c8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:37.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.765+0000 7f16fcc45640 1 --2- 172.21.15.19:0/2573220343 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f16f80d0b50 0x7f16f80c8300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:37.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.765+0000 7f16fcc45640 1 -- 172.21.15.19:0/2573220343 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16f815ec70 msgr2=0x7f16f8148f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:37.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.765+0000 7f16fcc45640 1 --2- 172.21.15.19:0/2573220343 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16f815ec70 0x7f16f8148f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:37.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.765+0000 7f16fcc45640 1 -- 172.21.15.19:0/2573220343 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16f815d310 con 0x7f16f80cd1b0 2024-08-31T12:29:37.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.765+0000 7f16fd446640 1 --2- 172.21.15.19:0/2573220343 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16f815ec70 0x7f16f8148f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:29:37.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.765+0000 7f16fcc45640 1 --2- 172.21.15.19:0/2573220343 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f16f80cd1b0 0x7f16f815cdd0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f16ec00e790 tx=0x7f16ec00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:29:37.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.765+0000 7f16f57fa640 1 -- 172.21.15.19:0/2573220343 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ec00ca60 con 0x7f16f80cd1b0 2024-08-31T12:29:37.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.765+0000 7f16f57fa640 1 -- 172.21.15.19:0/2573220343 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7f16ec00cd10 con 0x7f16f80cd1b0 2024-08-31T12:29:37.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.765+0000 7f16f57fa640 1 -- 172.21.15.19:0/2573220343 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ec010690 con 0x7f16f80cd1b0 2024-08-31T12:29:37.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.765+0000 7f16feea9640 1 -- 172.21.15.19:0/2573220343 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16f8158db0 con 0x7f16f80cd1b0 2024-08-31T12:29:37.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.766+0000 7f16feea9640 1 -- 172.21.15.19:0/2573220343 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16f815e430 con 0x7f16f80cd1b0 2024-08-31T12:29:37.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.766+0000 7f16feea9640 1 -- 172.21.15.19:0/2573220343 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16f80c5b90 con 0x7f16f80cd1b0 2024-08-31T12:29:37.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.767+0000 7f16f57fa640 1 -- 172.21.15.19:0/2573220343 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f16ec010830 con 0x7f16f80cd1b0 2024-08-31T12:29:37.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.768+0000 7f16f57fa640 1 --2- 172.21.15.19:0/2573220343 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f16dc077a50 0x7f16dc079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:29:37.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.768+0000 7f16f57fa640 1 -- 172.21.15.19:0/2573220343 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6083+0+0 (secure 0 0 0) 0x7f16ec09a1e0 con 0x7f16f80cd1b0 2024-08-31T12:29:37.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.768+0000 7f16f7fff640 1 --2- 172.21.15.19:0/2573220343 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f16dc077a50 0x7f16dc079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:29:37.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.768+0000 7f16f7fff640 1 --2- 172.21.15.19:0/2573220343 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f16dc077a50 0x7f16dc079f60 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f16e4033000 tx=0x7f16e40046c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:29:37.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.772+0000 7f16f57fa640 1 -- 172.21.15.19:0/2573220343 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f16f80c5b90 con 0x7f16f80cd1b0 2024-08-31T12:29:37.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:37.992+0000 7f16feea9640 1 -- 172.21.15.19:0/2573220343 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f16f80b5d70 con 0x7f16f80cd1b0 2024-08-31T12:29:38.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.630+0000 7f16f57fa640 1 -- 172.21.15.19:0/2573220343 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v14) ==== 104+0+0 (secure 0 0 0) 0x7f16f80b5d70 con 0x7f16f80cd1b0 2024-08-31T12:29:38.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.632+0000 7f16feea9640 1 -- 172.21.15.19:0/2573220343 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f16dc077a50 msgr2=0x7f16dc079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:38.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.632+0000 7f16feea9640 1 --2- 172.21.15.19:0/2573220343 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f16dc077a50 0x7f16dc079f60 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f16e4033000 tx=0x7f16e40046c0 comp rx=0 tx=0).stop 2024-08-31T12:29:38.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.632+0000 7f16feea9640 1 -- 172.21.15.19:0/2573220343 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f16f80cd1b0 msgr2=0x7f16f815cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:38.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.632+0000 7f16feea9640 1 --2- 172.21.15.19:0/2573220343 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f16f80cd1b0 0x7f16f815cdd0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f16ec00e790 tx=0x7f16ec00ec50 comp rx=0 tx=0).stop 2024-08-31T12:29:38.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.633+0000 7f16feea9640 1 -- 172.21.15.19:0/2573220343 shutdown_connections 2024-08-31T12:29:38.635 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.633+0000 7f16feea9640 1 --2- 172.21.15.19:0/2573220343 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f16dc077a50 0x7f16dc079f60 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:38.635 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.633+0000 7f16feea9640 1 --2- 172.21.15.19:0/2573220343 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16f815ec70 0x7f16f8148f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:38.635 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.633+0000 7f16feea9640 1 --2- 172.21.15.19:0/2573220343 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f16f80d0b50 0x7f16f80c8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:38.635 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.633+0000 7f16feea9640 1 --2- 172.21.15.19:0/2573220343 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f16f80cd1b0 0x7f16f815cdd0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:38.635 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.633+0000 7f16feea9640 1 -- 172.21.15.19:0/2573220343 >> 172.21.15.19:0/2573220343 conn(0x7f16f8074210 msgr2=0x7f16f814b2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:29:38.635 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.633+0000 7f16feea9640 1 -- 172.21.15.19:0/2573220343 shutdown_connections 2024-08-31T12:29:38.635 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.633+0000 7f16feea9640 1 -- 172.21.15.19:0/2573220343 wait complete. 2024-08-31T12:29:38.648 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-08-31T12:29:38.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.774+0000 7fde21fa3640 1 -- 172.21.15.19:0/620115774 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fde1c15bcc0 msgr2=0x7fde1c0d0f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:38.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.774+0000 7fde21fa3640 1 --2- 172.21.15.19:0/620115774 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fde1c15bcc0 0x7fde1c0d0f50 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fde10005fd0 tx=0x7fde1002a5d0 comp rx=0 tx=0).stop 2024-08-31T12:29:38.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.774+0000 7fde21fa3640 1 -- 172.21.15.19:0/620115774 shutdown_connections 2024-08-31T12:29:38.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.774+0000 7fde21fa3640 1 --2- 172.21.15.19:0/620115774 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fde1c0ca4d0 0x7fde1c0ca8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:38.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.774+0000 7fde21fa3640 1 --2- 172.21.15.19:0/620115774 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fde1c15bcc0 0x7fde1c0d0f50 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:38.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.774+0000 7fde21fa3640 1 -- 172.21.15.19:0/620115774 >> 172.21.15.19:0/620115774 conn(0x7fde1c06da20 msgr2=0x7fde1c06de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:29:38.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.774+0000 7fde21fa3640 1 -- 172.21.15.19:0/620115774 shutdown_connections 2024-08-31T12:29:38.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.774+0000 7fde21fa3640 1 -- 172.21.15.19:0/620115774 wait complete. 2024-08-31T12:29:38.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.776+0000 7fde21fa3640 1 Processor -- start 2024-08-31T12:29:38.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.776+0000 7fde21fa3640 1 -- start start 2024-08-31T12:29:38.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.777+0000 7fde21fa3640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fde1c0ca4d0 0x7fde1c0c6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:29:38.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.777+0000 7fde21fa3640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde1c0cc990 0x7fde1c0b5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:29:38.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.777+0000 7fde21fa3640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fde1c15bcc0 0x7fde1c0c52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:29:38.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.777+0000 7fde21fa3640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fde1c15f130 con 0x7fde1c15bcc0 2024-08-31T12:29:38.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.777+0000 7fde21fa3640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fde1c15ee50 con 0x7fde1c0cc990 2024-08-31T12:29:38.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.777+0000 7fde21fa3640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fde1c15c0d0 con 0x7fde1c0ca4d0 2024-08-31T12:29:38.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.777+0000 7fde1bfff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fde1c15bcc0 0x7fde1c0c52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:29:38.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.777+0000 7fde1bfff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fde1c15bcc0 0x7fde1c0c52e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:59470/0 (socket says 172.21.15.19:59470) 2024-08-31T12:29:38.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.777+0000 7fde1bfff640 1 -- 172.21.15.19:0/2136282239 learned_addr learned my addr 172.21.15.19:0/2136282239 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:29:38.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.777+0000 7fde1b7fe640 1 --2- 172.21.15.19:0/2136282239 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fde1c0ca4d0 0x7fde1c0c6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:29:38.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.777+0000 7fde1affd640 1 --2- 172.21.15.19:0/2136282239 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde1c0cc990 0x7fde1c0b5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:29:38.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.777+0000 7fde1bfff640 1 -- 172.21.15.19:0/2136282239 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fde1c0ca4d0 msgr2=0x7fde1c0c6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:38.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.777+0000 7fde1bfff640 1 --2- 172.21.15.19:0/2136282239 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fde1c0ca4d0 0x7fde1c0c6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:38.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.777+0000 7fde1bfff640 1 -- 172.21.15.19:0/2136282239 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde1c0cc990 msgr2=0x7fde1c0b5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:38.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.777+0000 7fde1bfff640 1 --2- 172.21.15.19:0/2136282239 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde1c0cc990 0x7fde1c0b5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:38.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.777+0000 7fde1bfff640 1 -- 172.21.15.19:0/2136282239 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde1c1f2ce0 con 0x7fde1c15bcc0 2024-08-31T12:29:38.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.777+0000 7fde1b7fe640 1 --2- 172.21.15.19:0/2136282239 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fde1c0ca4d0 0x7fde1c0c6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:29:38.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.777+0000 7fde1affd640 1 --2- 172.21.15.19:0/2136282239 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde1c0cc990 0x7fde1c0b5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-08-31T12:29:38.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.778+0000 7fde1bfff640 1 --2- 172.21.15.19:0/2136282239 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fde1c15bcc0 0x7fde1c0c52e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fde1400d990 tx=0x7fde1400de50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:29:38.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.778+0000 7fde03fff640 1 -- 172.21.15.19:0/2136282239 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde14017070 con 0x7fde1c15bcc0 2024-08-31T12:29:38.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.778+0000 7fde03fff640 1 -- 172.21.15.19:0/2136282239 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7fde1400be20 con 0x7fde1c15bcc0 2024-08-31T12:29:38.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.778+0000 7fde03fff640 1 -- 172.21.15.19:0/2136282239 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde140106c0 con 0x7fde1c15bcc0 2024-08-31T12:29:38.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.778+0000 7fde21fa3640 1 -- 172.21.15.19:0/2136282239 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde1c1f2a00 con 0x7fde1c15bcc0 2024-08-31T12:29:38.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.778+0000 7fde21fa3640 1 -- 172.21.15.19:0/2136282239 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde1c1e6180 con 0x7fde1c15bcc0 2024-08-31T12:29:38.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.779+0000 7fde21fa3640 1 -- 172.21.15.19:0/2136282239 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde1c0cc380 con 0x7fde1c15bcc0 2024-08-31T12:29:38.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.780+0000 7fde03fff640 1 -- 172.21.15.19:0/2136282239 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7fde1401a070 con 0x7fde1c15bcc0 2024-08-31T12:29:38.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.780+0000 7fde03fff640 1 --2- 172.21.15.19:0/2136282239 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fde08077a50 0x7fde08079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:29:38.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.780+0000 7fde03fff640 1 -- 172.21.15.19:0/2136282239 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6083+0+0 (secure 0 0 0) 0x7fde14099c60 con 0x7fde1c15bcc0 2024-08-31T12:29:38.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.780+0000 7fde1b7fe640 1 --2- 172.21.15.19:0/2136282239 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fde08077a50 0x7fde08079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:29:38.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.781+0000 7fde1b7fe640 1 --2- 172.21.15.19:0/2136282239 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fde08077a50 0x7fde08079f60 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fde0c007fd0 tx=0x7fde0c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:29:38.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.785+0000 7fde03fff640 1 -- 172.21.15.19:0/2136282239 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fde140628e0 con 0x7fde1c15bcc0 2024-08-31T12:29:38.969 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:38.968+0000 7fde21fa3640 1 -- 172.21.15.19:0/2136282239 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fde1c158bf0 con 0x7fde1c15bcc0 2024-08-31T12:29:39.636 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.635+0000 7fde03fff640 1 -- 172.21.15.19:0/2136282239 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v25) ==== 181+0+0 (secure 0 0 0) 0x7fde140621e0 con 0x7fde1c15bcc0 2024-08-31T12:29:39.636 INFO:teuthology.orchestra.run.smithi019.stderr:pool 'cephfs_metadata' removed 2024-08-31T12:29:39.639 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.638+0000 7fde21fa3640 1 -- 172.21.15.19:0/2136282239 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fde08077a50 msgr2=0x7fde08079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:39.639 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.638+0000 7fde21fa3640 1 --2- 172.21.15.19:0/2136282239 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fde08077a50 0x7fde08079f60 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fde0c007fd0 tx=0x7fde0c009040 comp rx=0 tx=0).stop 2024-08-31T12:29:39.639 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.638+0000 7fde21fa3640 1 -- 172.21.15.19:0/2136282239 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fde1c15bcc0 msgr2=0x7fde1c0c52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:39.639 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.638+0000 7fde21fa3640 1 --2- 172.21.15.19:0/2136282239 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fde1c15bcc0 0x7fde1c0c52e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fde1400d990 tx=0x7fde1400de50 comp rx=0 tx=0).stop 2024-08-31T12:29:39.640 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.638+0000 7fde21fa3640 1 -- 172.21.15.19:0/2136282239 shutdown_connections 2024-08-31T12:29:39.640 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.638+0000 7fde21fa3640 1 --2- 172.21.15.19:0/2136282239 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7fde08077a50 0x7fde08079f60 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:39.640 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.638+0000 7fde21fa3640 1 --2- 172.21.15.19:0/2136282239 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fde1c15bcc0 0x7fde1c0c52e0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:39.640 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.638+0000 7fde21fa3640 1 --2- 172.21.15.19:0/2136282239 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde1c0cc990 0x7fde1c0b5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:39.640 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.638+0000 7fde21fa3640 1 --2- 172.21.15.19:0/2136282239 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fde1c0ca4d0 0x7fde1c0c6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:39.640 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.638+0000 7fde21fa3640 1 -- 172.21.15.19:0/2136282239 >> 172.21.15.19:0/2136282239 conn(0x7fde1c06da20 msgr2=0x7fde1c0d0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:29:39.640 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.638+0000 7fde21fa3640 1 -- 172.21.15.19:0/2136282239 shutdown_connections 2024-08-31T12:29:39.640 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.638+0000 7fde21fa3640 1 -- 172.21.15.19:0/2136282239 wait complete. 2024-08-31T12:29:39.653 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-08-31T12:29:39.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.775+0000 7f1c04c0a640 1 -- 172.21.15.19:0/78393205 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1bec004800 msgr2=0x7f1bec004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:39.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.775+0000 7f1c04c0a640 1 --2- 172.21.15.19:0/78393205 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1bec004800 0x7f1bec004c10 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f1be8009b20 tx=0x7f1bec005830 comp rx=0 tx=0).stop 2024-08-31T12:29:39.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.775+0000 7f1bfed9d640 1 -- 172.21.15.19:0/78393205 reap_dead start 2024-08-31T12:29:39.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.775+0000 7f1c04c0a640 1 -- 172.21.15.19:0/78393205 shutdown_connections 2024-08-31T12:29:39.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.775+0000 7f1c04c0a640 1 -- 172.21.15.19:0/78393205 >> 172.21.15.19:0/78393205 conn(0x7f1c0006da20 msgr2=0x7f1c0006de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:29:39.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.775+0000 7f1c04c0a640 1 -- 172.21.15.19:0/78393205 shutdown_connections 2024-08-31T12:29:39.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.775+0000 7f1c04c0a640 1 -- 172.21.15.19:0/78393205 wait complete. 2024-08-31T12:29:39.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.777+0000 7f1c04c0a640 1 Processor -- start 2024-08-31T12:29:39.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.777+0000 7f1c04c0a640 1 -- start start 2024-08-31T12:29:39.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.778+0000 7f1c04c0a640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1bec004800 0x7f1c000c9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:29:39.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.778+0000 7f1c04c0a640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1c000c6fa0 0x7f1c00152cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:29:39.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.778+0000 7f1c04c0a640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1c001531f0 0x7f1c000c5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:29:39.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.778+0000 7f1c04c0a640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f1c0015c0d0 con 0x7f1c000c6fa0 2024-08-31T12:29:39.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.778+0000 7f1c04c0a640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1c0015f130 con 0x7f1bec004800 2024-08-31T12:29:39.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.778+0000 7f1c04c0a640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f1c000cc6a0 con 0x7f1c001531f0 2024-08-31T12:29:39.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.778+0000 7f1bfdd9b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1c000c6fa0 0x7f1c00152cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:29:39.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.778+0000 7f1bfdd9b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1c000c6fa0 0x7f1c00152cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:59486/0 (socket says 172.21.15.19:59486) 2024-08-31T12:29:39.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.778+0000 7f1bfdd9b640 1 -- 172.21.15.19:0/2504655212 learned_addr learned my addr 172.21.15.19:0/2504655212 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-08-31T12:29:39.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.778+0000 7f1bfed9d640 1 --2- 172.21.15.19:0/2504655212 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1c001531f0 0x7f1c000c5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:29:39.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.778+0000 7f1bfdd9b640 1 -- 172.21.15.19:0/2504655212 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1c001531f0 msgr2=0x7f1c000c5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:39.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.778+0000 7f1bfdd9b640 1 --2- 172.21.15.19:0/2504655212 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1c001531f0 0x7f1c000c5070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:39.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.778+0000 7f1bfdd9b640 1 -- 172.21.15.19:0/2504655212 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1bec004800 msgr2=0x7f1c000c9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:39.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.778+0000 7f1bfdd9b640 1 --2- 172.21.15.19:0/2504655212 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1bec004800 0x7f1c000c9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:39.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.778+0000 7f1bfdd9b640 1 -- 172.21.15.19:0/2504655212 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c000c9d60 con 0x7f1c000c6fa0 2024-08-31T12:29:39.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.778+0000 7f1bfdd9b640 1 --2- 172.21.15.19:0/2504655212 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1c000c6fa0 0x7f1c00152cb0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f1be8009c40 tx=0x7f1be80336f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:29:39.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.779+0000 7f1bf6ffd640 1 -- 172.21.15.19:0/2504655212 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be8038040 con 0x7f1c000c6fa0 2024-08-31T12:29:39.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.779+0000 7f1bf6ffd640 1 -- 172.21.15.19:0/2504655212 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7f1be802b7b0 con 0x7f1c000c6fa0 2024-08-31T12:29:39.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.779+0000 7f1c04c0a640 1 -- 172.21.15.19:0/2504655212 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c00158980 con 0x7f1c000c6fa0 2024-08-31T12:29:39.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.779+0000 7f1bf6ffd640 1 -- 172.21.15.19:0/2504655212 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be8004260 con 0x7f1c000c6fa0 2024-08-31T12:29:39.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.779+0000 7f1c04c0a640 1 -- 172.21.15.19:0/2504655212 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c001586a0 con 0x7f1c000c6fa0 2024-08-31T12:29:39.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.780+0000 7f1c04c0a640 1 -- 172.21.15.19:0/2504655212 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c0015f650 con 0x7f1c000c6fa0 2024-08-31T12:29:39.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.780+0000 7f1bf6ffd640 1 -- 172.21.15.19:0/2504655212 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 100151+0+0 (secure 0 0 0) 0x7f1be8045020 con 0x7f1c000c6fa0 2024-08-31T12:29:39.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.781+0000 7f1bf6ffd640 1 --2- 172.21.15.19:0/2504655212 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1be0077a50 0x7f1be0079f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-31T12:29:39.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.781+0000 7f1bf6ffd640 1 -- 172.21.15.19:0/2504655212 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5645+0+0 (secure 0 0 0) 0x7f1be80b9d90 con 0x7f1c000c6fa0 2024-08-31T12:29:39.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.781+0000 7f1bfe59c640 1 --2- 172.21.15.19:0/2504655212 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1be0077a50 0x7f1be0079f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-31T12:29:39.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.782+0000 7f1bfe59c640 1 --2- 172.21.15.19:0/2504655212 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1be0077a50 0x7f1be0079f60 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f1bf0001be0 tx=0x7f1bf0001250 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-31T12:29:39.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.785+0000 7f1bf6ffd640 1 -- 172.21.15.19:0/2504655212 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f1be8082bd0 con 0x7f1c000c6fa0 2024-08-31T12:29:39.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:39.971+0000 7f1c04c0a640 1 -- 172.21.15.19:0/2504655212 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f1c000d12d0 con 0x7f1c000c6fa0 2024-08-31T12:29:40.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:40.645+0000 7f1bf6ffd640 1 -- 172.21.15.19:0/2504655212 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v26) ==== 169+0+0 (secure 0 0 0) 0x7f1be808b080 con 0x7f1c000c6fa0 2024-08-31T12:29:40.646 INFO:teuthology.orchestra.run.smithi019.stderr:pool 'cephfs_data' removed 2024-08-31T12:29:40.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:40.647+0000 7f1c04c0a640 1 -- 172.21.15.19:0/2504655212 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1be0077a50 msgr2=0x7f1be0079f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:40.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:40.647+0000 7f1c04c0a640 1 --2- 172.21.15.19:0/2504655212 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1be0077a50 0x7f1be0079f60 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f1bf0001be0 tx=0x7f1bf0001250 comp rx=0 tx=0).stop 2024-08-31T12:29:40.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:40.647+0000 7f1c04c0a640 1 -- 172.21.15.19:0/2504655212 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1c000c6fa0 msgr2=0x7f1c00152cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-31T12:29:40.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:40.647+0000 7f1c04c0a640 1 --2- 172.21.15.19:0/2504655212 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1c000c6fa0 0x7f1c00152cb0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f1be8009c40 tx=0x7f1be80336f0 comp rx=0 tx=0).stop 2024-08-31T12:29:40.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:40.647+0000 7f1c04c0a640 1 -- 172.21.15.19:0/2504655212 shutdown_connections 2024-08-31T12:29:40.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:40.647+0000 7f1c04c0a640 1 --2- 172.21.15.19:0/2504655212 >> [v2:172.21.15.43:6800/2824336655,v1:172.21.15.43:6801/2824336655] conn(0x7f1be0077a50 0x7f1be0079f60 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:40.650 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:40.647+0000 7f1c04c0a640 1 --2- 172.21.15.19:0/2504655212 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1c001531f0 0x7f1c000c5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:40.650 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:40.647+0000 7f1c04c0a640 1 --2- 172.21.15.19:0/2504655212 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1c000c6fa0 0x7f1c00152cb0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:40.650 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:40.647+0000 7f1c04c0a640 1 --2- 172.21.15.19:0/2504655212 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1bec004800 0x7f1c000c9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-31T12:29:40.650 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:40.647+0000 7f1c04c0a640 1 -- 172.21.15.19:0/2504655212 >> 172.21.15.19:0/2504655212 conn(0x7f1c0006da20 msgr2=0x7f1c00154c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-31T12:29:40.650 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:40.648+0000 7f1c04c0a640 1 -- 172.21.15.19:0/2504655212 shutdown_connections 2024-08-31T12:29:40.650 INFO:teuthology.orchestra.run.smithi019.stderr:2024-08-31T12:29:40.648+0000 7f1c04c0a640 1 -- 172.21.15.19:0/2504655212 wait complete. 2024-08-31T12:29:40.668 INFO:teuthology.misc:Shutting down mds daemons... 2024-08-31T12:29:40.669 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-08-31T12:29:40.669 INFO:teuthology.orchestra.run:waiting for 300 2024-08-31T12:29:40.683 INFO:tasks.ceph.mds.a:Stopped 2024-08-31T12:29:40.683 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-08-31T12:29:40.683 INFO:teuthology.orchestra.run:waiting for 300 2024-08-31T12:29:40.736 INFO:tasks.ceph.mds.c:Stopped 2024-08-31T12:29:40.736 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2024-08-31T12:29:40.736 INFO:teuthology.orchestra.run:waiting for 300 2024-08-31T12:29:40.786 INFO:tasks.ceph.mds.e:Stopped 2024-08-31T12:29:40.786 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-08-31T12:29:40.786 INFO:teuthology.orchestra.run:waiting for 300 2024-08-31T12:29:40.804 INFO:tasks.ceph.mds.b:Stopped 2024-08-31T12:29:40.805 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-08-31T12:29:40.805 INFO:teuthology.orchestra.run:waiting for 300 2024-08-31T12:29:40.860 INFO:tasks.ceph.mds.d:Stopped 2024-08-31T12:29:40.861 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2024-08-31T12:29:40.861 INFO:teuthology.orchestra.run:waiting for 300 2024-08-31T12:29:40.915 INFO:tasks.ceph.mds.f:Stopped 2024-08-31T12:29:40.915 INFO:teuthology.misc:Shutting down osd daemons... 2024-08-31T12:29:40.915 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-08-31T12:29:40.915 INFO:teuthology.orchestra.run:waiting for 300 2024-08-31T12:29:40.974 INFO:tasks.ceph.osd.0:Stopped 2024-08-31T12:29:40.974 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-08-31T12:29:40.974 INFO:teuthology.orchestra.run:waiting for 300 2024-08-31T12:29:41.025 INFO:tasks.ceph.osd.1:Stopped 2024-08-31T12:29:41.026 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-08-31T12:29:41.026 INFO:teuthology.orchestra.run:waiting for 300 2024-08-31T12:29:41.075 INFO:tasks.ceph.osd.2:Stopped 2024-08-31T12:29:41.076 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-08-31T12:29:41.076 INFO:teuthology.orchestra.run:waiting for 300 2024-08-31T12:29:41.115 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s 2024-08-31T12:29:41.115 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~0s 2024-08-31T12:29:41.116 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-08-31T12:29:41.116 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-08-31T12:29:41.116 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~0s 2024-08-31T12:29:41.136 INFO:tasks.ceph.osd.3:Stopped 2024-08-31T12:29:41.136 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-08-31T12:29:41.136 INFO:teuthology.orchestra.run:waiting for 300 2024-08-31T12:29:41.186 INFO:tasks.ceph.osd.4:Stopped 2024-08-31T12:29:41.187 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-08-31T12:29:41.187 INFO:teuthology.orchestra.run:waiting for 300 2024-08-31T12:29:41.246 INFO:tasks.ceph.osd.5:Stopped 2024-08-31T12:29:41.246 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-08-31T12:29:41.246 INFO:teuthology.orchestra.run:waiting for 300 2024-08-31T12:29:41.302 INFO:tasks.ceph.osd.6:Stopped 2024-08-31T12:29:41.303 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-08-31T12:29:41.303 INFO:teuthology.orchestra.run:waiting for 300 2024-08-31T12:29:41.353 INFO:tasks.ceph.osd.7:Stopped 2024-08-31T12:29:41.354 INFO:teuthology.misc:Shutting down mgr daemons... 2024-08-31T12:29:41.354 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-08-31T12:29:41.354 INFO:teuthology.orchestra.run:waiting for 300 2024-08-31T12:29:41.383 INFO:tasks.ceph.mgr.y:Stopped 2024-08-31T12:29:41.383 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-08-31T12:29:41.383 INFO:teuthology.orchestra.run:waiting for 300 2024-08-31T12:29:41.411 INFO:tasks.ceph.mgr.x:Stopped 2024-08-31T12:29:41.411 INFO:teuthology.misc:Shutting down mon daemons... 2024-08-31T12:29:41.411 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-08-31T12:29:41.411 INFO:teuthology.orchestra.run:waiting for 300 2024-08-31T12:29:41.434 INFO:tasks.ceph.mon.a:Stopped 2024-08-31T12:29:41.434 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-08-31T12:29:41.434 INFO:teuthology.orchestra.run:waiting for 300 2024-08-31T12:29:41.461 INFO:tasks.ceph.mon.b:Stopped 2024-08-31T12:29:41.461 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-08-31T12:29:41.461 INFO:teuthology.orchestra.run:waiting for 300 2024-08-31T12:29:41.511 INFO:tasks.ceph.mon.c:Stopped 2024-08-31T12:29:41.511 INFO:tasks.ceph:Checking cluster log for badness... 2024-08-31T12:29:41.511 DEBUG:teuthology.orchestra.run.smithi019:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v 'fs.*is degraded' | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v 'insufficient standby MDS daemons available' | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'online, but wants' | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v SLOW_OPS | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-08-31T12:29:41.606 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi019.front.sepia.ceph.com 2024-08-31T12:29:41.607 DEBUG:teuthology.orchestra.run.smithi019:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-08-31T12:29:41.892 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi019.front.sepia.ceph.com 2024-08-31T12:29:41.893 DEBUG:teuthology.orchestra.run.smithi019:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-08-31T12:29:42.010 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi019.front.sepia.ceph.com 2024-08-31T12:29:42.010 DEBUG:teuthology.orchestra.run.smithi019:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-08-31T12:29:42.075 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi019.front.sepia.ceph.com 2024-08-31T12:29:42.076 DEBUG:teuthology.orchestra.run.smithi019:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-08-31T12:29:42.141 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi043.front.sepia.ceph.com 2024-08-31T12:29:42.142 DEBUG:teuthology.orchestra.run.smithi043:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-08-31T12:29:42.408 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi043.front.sepia.ceph.com 2024-08-31T12:29:42.409 DEBUG:teuthology.orchestra.run.smithi043:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-08-31T12:29:42.487 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi043.front.sepia.ceph.com 2024-08-31T12:29:42.487 DEBUG:teuthology.orchestra.run.smithi043:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-08-31T12:29:42.555 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi043.front.sepia.ceph.com 2024-08-31T12:29:42.555 DEBUG:teuthology.orchestra.run.smithi043:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-08-31T12:29:42.629 INFO:tasks.ceph:Archiving mon data... 2024-08-31T12:29:42.630 DEBUG:teuthology.misc:Transferring archived files from smithi019:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/mchangir-2024-08-30_08:30:37-fs-wip-mchangir-testing-20240828.085847-main-debug-distro-default-smithi/7881510/data/mon.a.tgz 2024-08-31T12:29:42.630 DEBUG:teuthology.orchestra.run.smithi019:> mktemp 2024-08-31T12:29:42.664 INFO:teuthology.orchestra.run.smithi019.stdout:/tmp/tmp.qmx3JUfzar 2024-08-31T12:29:42.664 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.qmx3JUfzar 2024-08-31T12:29:42.811 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0666 /tmp/tmp.qmx3JUfzar 2024-08-31T12:29:42.915 DEBUG:teuthology.orchestra.remote:smithi019:/tmp/tmp.qmx3JUfzar is 379KB 2024-08-31T12:29:43.010 DEBUG:teuthology.orchestra.run.smithi019:> rm -fr /tmp/tmp.qmx3JUfzar 2024-08-31T12:29:43.025 DEBUG:teuthology.misc:Transferring archived files from smithi043:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/mchangir-2024-08-30_08:30:37-fs-wip-mchangir-testing-20240828.085847-main-debug-distro-default-smithi/7881510/data/mon.b.tgz 2024-08-31T12:29:43.026 DEBUG:teuthology.orchestra.run.smithi043:> mktemp 2024-08-31T12:29:43.043 INFO:teuthology.orchestra.run.smithi043.stdout:/tmp/tmp.ybPmyzp9o6 2024-08-31T12:29:43.043 DEBUG:teuthology.orchestra.run.smithi043:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.ybPmyzp9o6 2024-08-31T12:29:43.197 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 0666 /tmp/tmp.ybPmyzp9o6 2024-08-31T12:29:43.279 DEBUG:teuthology.orchestra.remote:smithi043:/tmp/tmp.ybPmyzp9o6 is 387KB 2024-08-31T12:29:43.376 DEBUG:teuthology.orchestra.run.smithi043:> rm -fr /tmp/tmp.ybPmyzp9o6 2024-08-31T12:29:43.391 DEBUG:teuthology.misc:Transferring archived files from smithi043:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/mchangir-2024-08-30_08:30:37-fs-wip-mchangir-testing-20240828.085847-main-debug-distro-default-smithi/7881510/data/mon.c.tgz 2024-08-31T12:29:43.392 DEBUG:teuthology.orchestra.run.smithi043:> mktemp 2024-08-31T12:29:43.445 INFO:teuthology.orchestra.run.smithi043.stdout:/tmp/tmp.QfuupKqOJF 2024-08-31T12:29:43.445 DEBUG:teuthology.orchestra.run.smithi043:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.QfuupKqOJF 2024-08-31T12:29:43.582 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 0666 /tmp/tmp.QfuupKqOJF 2024-08-31T12:29:43.665 DEBUG:teuthology.orchestra.remote:smithi043:/tmp/tmp.QfuupKqOJF is 387KB 2024-08-31T12:29:43.763 DEBUG:teuthology.orchestra.run.smithi043:> rm -fr /tmp/tmp.QfuupKqOJF 2024-08-31T12:29:43.777 INFO:tasks.ceph:Cleaning ceph cluster... 2024-08-31T12:29:43.778 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-08-31T12:29:43.781 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-08-31T12:29:43.863 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-08-31T12:29:43.897 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-08-31T12:29:43.897 INFO:tasks.ceph:Archiving crash dumps... 2024-08-31T12:29:43.899 DEBUG:teuthology.misc:Transferring archived files from smithi019:/var/lib/ceph/crash to /home/teuthworker/archive/mchangir-2024-08-30_08:30:37-fs-wip-mchangir-testing-20240828.085847-main-debug-distro-default-smithi/7881510/remote/smithi019/crash 2024-08-31T12:29:43.899 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-08-31T12:29:43.930 DEBUG:teuthology.misc:Transferring archived files from smithi043:/var/lib/ceph/crash to /home/teuthworker/archive/mchangir-2024-08-30_08:30:37-fs-wip-mchangir-testing-20240828.085847-main-debug-distro-default-smithi/7881510/remote/smithi043/crash 2024-08-31T12:29:43.931 DEBUG:teuthology.orchestra.run.smithi043:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-08-31T12:29:43.961 INFO:tasks.ceph:Shutting down logrotate 2024-08-31T12:29:43.961 DEBUG:teuthology.orchestra.run.smithi019:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:29:43.975 DEBUG:teuthology.orchestra.run.smithi043:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-08-31T12:29:44.047 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-08-31T12:29:44.088 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-08-31T12:29:44.112 INFO:tasks.ceph:Compressing logs... 2024-08-31T12:29:44.113 DEBUG:teuthology.orchestra.run.smithi019:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-08-31T12:29:44.131 DEBUG:teuthology.orchestra.run.smithi043:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-08-31T12:29:44.166 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-08-31T12:29:44.166 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64404.log 2024-08-31T12:29:44.169 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56295.log 2024-08-31T12:29:44.169 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.64404.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.17781.log 2024-08-31T12:29:44.169 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.56295.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.64404.log.gz 2024-08-31T12:29:44.170 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40404.log 2024-08-31T12:29:44.170 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.56295.log.gz 2024-08-31T12:29:44.170 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph.tmp-client.admin.17781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72374.log 2024-08-31T12:29:44.170 INFO:teuthology.orchestra.run.smithi019.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.17781.log.gz 2024-08-31T12:29:44.170 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41467.log 2024-08-31T12:29:44.170 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36091.log 2024-08-31T12:29:44.170 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.72374.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.72374.log.gz 2024-08-31T12:29:44.170 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42809.log 2024-08-31T12:29:44.171 INFO:teuthology.orchestra.run.smithi019.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41467.log.gz 2024-08-31T12:29:44.171 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.36091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23966.log 2024-08-31T12:29:44.171 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.42809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73546.log 2024-08-31T12:29:44.171 INFO:teuthology.orchestra.run.smithi019.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42809.log.gz 2024-08-31T12:29:44.171 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.23966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40572.log 2024-08-31T12:29:44.171 INFO:teuthology.orchestra.run.smithi019.stderr:gzip/var/log/ceph/ceph-client.admin.73546.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40758.log 2024-08-31T12:29:44.191 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23953.log 2024-08-31T12:29:44.192 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24033.log 2024-08-31T12:29:44.192 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.23953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24113.log 2024-08-31T12:29:44.192 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.24033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24153.log 2024-08-31T12:29:44.192 INFO:teuthology.orchestra.run.smithi043.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.23953.log.gz 2024-08-31T12:29:44.192 INFO:teuthology.orchestra.run.smithi043.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.24033.log.gz 2024-08-31T12:29:44.192 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.24113.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-08-31T12:29:44.192 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.24153.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.24113.log.gz 2024-08-31T12:29:44.193 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-08-31T12:29:44.193 INFO:teuthology.orchestra.run.smithi043.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.24153.log.gz 2024-08-31T12:29:44.193 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24193.log 2024-08-31T12:29:44.193 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-08-31T12:29:44.193 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.24193.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-08-31T12:29:44.193 INFO:teuthology.orchestra.run.smithi043.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.24193.log.gz 2024-08-31T12:29:44.193 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23993.log 2024-08-31T12:29:44.193 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24073.log 2024-08-31T12:29:44.194 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.23993.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-08-31T12:29:44.194 INFO:teuthology.orchestra.run.smithi043.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.23993.log.gz 2024-08-31T12:29:44.194 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.24073.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-08-31T12:29:44.194 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-mon.c.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.24073.log.gz 2024-08-31T12:29:44.194 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-08-31T12:29:44.194 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-08-31T12:29:44.194 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-08-31T12:29:44.195 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph.log: 93.9% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-08-31T12:29:44.195 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-08-31T12:29:44.195 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-mds.d.log: 92.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-08-31T12:29:44.195 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23913.log 2024-08-31T12:29:44.211 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-08-31T12:29:44.211 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.23913.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.23913.log.gz 2024-08-31T12:29:44.217 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.f.log 2024-08-31T12:29:44.228 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40572.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.73546.log.gz 2024-08-31T12:29:44.228 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34444.log 2024-08-31T12:29:44.229 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44409.log 2024-08-31T12:29:44.229 INFO:teuthology.orchestra.run.smithi019.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40758.log.gz 2024-08-31T12:29:44.229 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.34444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34810.log 2024-08-31T12:29:44.229 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.44409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31914.log 2024-08-31T12:29:44.229 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.34810.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44409.log.gz 2024-08-31T12:29:44.229 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31915.log 2024-08-31T12:29:44.229 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.31914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59250.log 2024-08-31T12:29:44.230 INFO:teuthology.orchestra.run.smithi019.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34444.log.gz 2024-08-31T12:29:44.230 INFO:teuthology.orchestra.run.smithi019.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.34810.log.gz 2024-08-31T12:29:44.230 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.31915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53387.log 2024-08-31T12:29:44.230 INFO:teuthology.orchestra.run.smithi019.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.31914.log.gz 2024-08-31T12:29:44.230 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.59250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32510.log 2024-08-31T12:29:44.230 INFO:teuthology.orchestra.run.smithi019.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.31915.log.gz 2024-08-31T12:29:44.230 INFO:teuthology.orchestra.run.smithi019.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59250.log.gz 2024-08-31T12:29:44.230 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.53387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42426.log 2024-08-31T12:29:44.231 INFO:teuthology.orchestra.run.smithi019.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53387.log.gz 2024-08-31T12:29:44.231 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.32510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40049.log 2024-08-31T12:29:44.231 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.42426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35227.log 2024-08-31T12:29:44.231 INFO:teuthology.orchestra.run.smithi019.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42426.log.gz 2024-08-31T12:29:44.231 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34093.log 2024-08-31T12:29:44.231 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.35227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72856.log 2024-08-31T12:29:44.231 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.34093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73276.log 2024-08-31T12:29:44.231 INFO:teuthology.orchestra.run.smithi019.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.32510.log.gz 2024-08-31T12:29:44.232 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.72856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39653.log 2024-08-31T12:29:44.232 INFO:teuthology.orchestra.run.smithi019.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.72856.log.gz 2024-08-31T12:29:44.232 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.73276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43457.log 2024-08-31T12:29:44.232 INFO:teuthology.orchestra.run.smithi019.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.73276.log.gz 2024-08-31T12:29:44.232 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36171.log 2024-08-31T12:29:44.232 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.43457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32997.log 2024-08-31T12:29:44.232 INFO:teuthology.orchestra.run.smithi019.stderr: 88.9%/var/log/ceph/ceph-client.admin.36171.log: -- replaced with /var/log/ceph/ceph-client.admin.43457.log.gz 2024-08-31T12:29:44.232 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24475.log 2024-08-31T12:29:44.233 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.32997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38903.log 2024-08-31T12:29:44.233 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.24475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39485.log 2024-08-31T12:29:44.233 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32955.log 2024-08-31T12:29:44.233 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38903.log: /var/log/ceph/ceph-client.admin.39485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57019.log 2024-08-31T12:29:44.233 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.32955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33081.log 2024-08-31T12:29:44.233 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.57019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44593.log 2024-08-31T12:29:44.233 INFO:teuthology.orchestra.run.smithi019.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.57019.log.gz 2024-08-31T12:29:44.234 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.33081.log: -- /var/log/ceph/ceph-client.admin.45025.log 2024-08-31T12:29:44.234 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.44593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39821.log 2024-08-31T12:29:44.234 INFO:teuthology.orchestra.run.smithi019.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40404.log.gz 2024-08-31T12:29:44.234 INFO:teuthology.orchestra.run.smithi019.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44593.log.gz 2024-08-31T12:29:44.234 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.45025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44635.log 2024-08-31T12:29:44.234 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39821.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45025.log.gz 2024-08-31T12:29:44.234 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-08-31T12:29:44.234 INFO:teuthology.orchestra.run.smithi019.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39485.log.gz 2024-08-31T12:29:44.235 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.44635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44493.log 2024-08-31T12:29:44.235 INFO:teuthology.orchestra.run.smithi019.stderr: 88.9%/var/log/ceph/ceph-osd.1.log: -- replaced with /var/log/ceph/ceph-client.admin.44635.log.gz 2024-08-31T12:29:44.235 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43792.log 2024-08-31T12:29:44.235 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.44493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52037.log 2024-08-31T12:29:44.235 INFO:teuthology.orchestra.run.smithi019.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44493.log.gz 2024-08-31T12:29:44.235 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.43792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40800.log 2024-08-31T12:29:44.235 INFO:teuthology.orchestra.run.smithi019.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43792.log.gz 2024-08-31T12:29:44.235 INFO:teuthology.orchestra.run.smithi019.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34093.log.gz 2024-08-31T12:29:44.236 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.52037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67104.log 2024-08-31T12:29:44.236 INFO:teuthology.orchestra.run.smithi019.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39821.log.gz 2024-08-31T12:29:44.236 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40800.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52037.log.gzgzip 2024-08-31T12:29:44.236 INFO:teuthology.orchestra.run.smithi019.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.23938.log 2024-08-31T12:29:44.236 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.67104.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40800.log.gz 2024-08-31T12:29:44.236 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72814.log 2024-08-31T12:29:44.236 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.67104.log.gz 2024-08-31T12:29:44.237 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.23938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38987.log 2024-08-31T12:29:44.237 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.72814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64482.log 2024-08-31T12:29:44.237 INFO:teuthology.orchestra.run.smithi019.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.72814.log.gz 2024-08-31T12:29:44.237 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56030.log 2024-08-31T12:29:44.237 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.64482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73772.log 2024-08-31T12:29:44.237 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.56030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44207.log 2024-08-31T12:29:44.237 INFO:teuthology.orchestra.run.smithi019.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.64482.log.gz 2024-08-31T12:29:44.237 INFO:teuthology.orchestra.run.smithi019.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56030.log.gz 2024-08-31T12:29:44.238 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.73772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53612.log 2024-08-31T12:29:44.238 INFO:teuthology.orchestra.run.smithi019.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.73772.log.gz 2024-08-31T12:29:44.238 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.44207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41052.log 2024-08-31T12:29:44.238 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4%/var/log/ceph/ceph-client.admin.53612.log: -- replaced with /var/log/ceph/ceph-client.admin.44207.log.gz 2024-08-31T12:29:44.238 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73192.log 2024-08-31T12:29:44.238 INFO:teuthology.orchestra.run.smithi019.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53612.log.gz 2024-08-31T12:29:44.238 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48324.log 2024-08-31T12:29:44.238 INFO:teuthology.orchestra.run.smithi019.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41052.log.gz 2024-08-31T12:29:44.239 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.73192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52718.log 2024-08-31T12:29:44.239 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.48324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31916.log 2024-08-31T12:29:44.239 INFO:teuthology.orchestra.run.smithi019.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.73192.log.gz 2024-08-31T12:29:44.239 INFO:teuthology.orchestra.run.smithi019.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48324.log.gz 2024-08-31T12:29:44.239 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.52718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39401.log 2024-08-31T12:29:44.239 INFO:teuthology.orchestra.run.smithi019.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52718.log.gz 2024-08-31T12:29:44.239 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.31916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73730.log 2024-08-31T12:29:44.239 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43271.log 2024-08-31T12:29:44.240 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.73730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73486.log 2024-08-31T12:29:44.240 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.73730.log.gz 2024-08-31T12:29:44.240 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.43271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24471.log 2024-08-31T12:29:44.240 INFO:teuthology.orchestra.run.smithi019.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.31916.log.gz 2024-08-31T12:29:44.240 INFO:teuthology.orchestra.run.smithi019.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.43271.log.gz 2024-08-31T12:29:44.240 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.73486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43145.log 2024-08-31T12:29:44.240 INFO:teuthology.orchestra.run.smithi019.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.73486.log.gz 2024-08-31T12:29:44.241 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.24471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31221.log 2024-08-31T12:29:44.241 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.43145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31263.log 2024-08-31T12:29:44.241 INFO:teuthology.orchestra.run.smithi019.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39653.log.gz 2024-08-31T12:29:44.241 INFO:teuthology.orchestra.run.smithi019.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43145.log.gz 2024-08-31T12:29:44.241 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.31221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40968.log 2024-08-31T12:29:44.241 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.31263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32254.log 2024-08-31T12:29:44.241 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41281.log 2024-08-31T12:29:44.241 INFO:teuthology.orchestra.run.smithi019.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40968.log.gz 2024-08-31T12:29:44.242 INFO:teuthology.orchestra.run.smithi019.stderr:gzip/var/log/ceph/ceph-client.admin.32254.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41720.log 2024-08-31T12:29:44.242 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45108.log 2024-08-31T12:29:44.242 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38693.log 2024-08-31T12:29:44.242 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41281.log.gz 2024-08-31T12:29:44.242 INFO:teuthology.orchestra.run.smithi019.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41720.log.gz 2024-08-31T12:29:44.242 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.31263.log.gz 2024-08-31T12:29:44.242 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.45108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24465.log 2024-08-31T12:29:44.242 INFO:teuthology.orchestra.run.smithi019.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.45108.log.gz 2024-08-31T12:29:44.243 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73898.log 2024-08-31T12:29:44.243 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.24465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64632.log 2024-08-31T12:29:44.243 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.73898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33399.log 2024-08-31T12:29:44.243 INFO:teuthology.orchestra.run.smithi019.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.73898.log.gz 2024-08-31T12:29:44.243 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.64632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43313.log 2024-08-31T12:29:44.243 INFO:teuthology.orchestra.run.smithi019.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38987.log.gz 2024-08-31T12:29:44.243 INFO:teuthology.orchestra.run.smithi019.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.38903.log.gz 2024-08-31T12:29:44.243 INFO:teuthology.orchestra.run.smithi019.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.64632.log.gz 2024-08-31T12:29:44.244 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.33399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34684.log 2024-08-31T12:29:44.244 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.43313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40007.log 2024-08-31T12:29:44.244 INFO:teuthology.orchestra.run.smithi019.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.32254.log.gz 2024-08-31T12:29:44.244 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.34684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56206.log 2024-08-31T12:29:44.244 INFO:teuthology.orchestra.run.smithi019.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43313.log.gz 2024-08-31T12:29:44.244 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24346.log 2024-08-31T12:29:44.244 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.56206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23910.log 2024-08-31T12:29:44.244 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.24346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32829.log 2024-08-31T12:29:44.245 INFO:teuthology.orchestra.run.smithi019.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56206.log.gz 2024-08-31T12:29:44.245 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64567.log/var/log/ceph/ceph-client.admin.23910.log: 2024-08-31T12:29:44.245 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.32829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43708.log 2024-08-31T12:29:44.245 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.64567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32451.log 2024-08-31T12:29:44.245 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.43708.log: gzip -5 --verbose -- 89.1% /var/log/ceph/ceph-client.admin.68406.log 2024-08-31T12:29:44.245 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/ceph-client.admin.64567.log.gz 2024-08-31T12:29:44.245 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.32451.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43708.log.gz 2024-08-31T12:29:44.246 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39257.log 2024-08-31T12:29:44.246 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.68406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31653.log 2024-08-31T12:29:44.246 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.68406.log.gz 2024-08-31T12:29:44.246 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67716.log 2024-08-31T12:29:44.246 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.31653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39527.log 2024-08-31T12:29:44.246 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.67716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31912.log 2024-08-31T12:29:44.246 INFO:teuthology.orchestra.run.smithi019.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.67716.log.gz 2024-08-31T12:29:44.247 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34177.log 2024-08-31T12:29:44.247 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.31912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72641.log 2024-08-31T12:29:44.247 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.34177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41635.log 2024-08-31T12:29:44.247 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.72641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41804.log 2024-08-31T12:29:44.247 INFO:teuthology.orchestra.run.smithi019.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.72641.log.gz 2024-08-31T12:29:44.247 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53536.log 2024-08-31T12:29:44.247 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41635.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.32829.log.gz 2024-08-31T12:29:44.247 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41804.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.32955.log.gz 2024-08-31T12:29:44.247 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72020.log 2024-08-31T12:29:44.248 INFO:teuthology.orchestra.run.smithi019.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41635.log.gz 2024-08-31T12:29:44.248 INFO:teuthology.orchestra.run.smithi019.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.32997.log.gz 2024-08-31T12:29:44.248 INFO:teuthology.orchestra.run.smithi019.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.41804.log.gz 2024-08-31T12:29:44.248 INFO:teuthology.orchestra.run.smithi019.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.33081.log.gz 2024-08-31T12:29:44.248 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.53536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31485.log 2024-08-31T12:29:44.248 INFO:teuthology.orchestra.run.smithi019.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53536.log.gz 2024-08-31T12:29:44.248 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.72020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41551.log 2024-08-31T12:29:44.248 INFO:teuthology.orchestra.run.smithi019.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34684.log.gz 2024-08-31T12:29:44.249 INFO:teuthology.orchestra.run.smithi019.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.72020.log.gz 2024-08-31T12:29:44.249 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.31485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31137.log 2024-08-31T12:29:44.249 INFO:teuthology.orchestra.run.smithi019.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.32451.log.gz 2024-08-31T12:29:44.249 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40175.log 2024-08-31T12:29:44.249 INFO:teuthology.orchestra.run.smithi019.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41551.log.gz 2024-08-31T12:29:44.249 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.31137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31400.log 2024-08-31T12:29:44.249 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44291.log 2024-08-31T12:29:44.249 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.31400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48250.log 2024-08-31T12:29:44.250 INFO:teuthology.orchestra.run.smithi019.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39257.log.gz 2024-08-31T12:29:44.250 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.44291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73318.log 2024-08-31T12:29:44.250 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31919.log 2024-08-31T12:29:44.250 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40716.log 2024-08-31T12:29:44.250 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56128.log 2024-08-31T12:29:44.250 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33967.log 2024-08-31T12:29:44.250 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69029.log 2024-08-31T12:29:44.250 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39695.log 2024-08-31T12:29:44.250 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-08-31T12:29:44.251 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73402.log 2024-08-31T12:29:44.251 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31179.log 2024-08-31T12:29:44.251 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42230.log 2024-08-31T12:29:44.251 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30613.log 2024-08-31T12:29:44.251 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34727.log 2024-08-31T12:29:44.251 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39215.log 2024-08-31T12:29:44.251 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39071.log 2024-08-31T12:29:44.251 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34051.log 2024-08-31T12:29:44.252 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58515.log 2024-08-31T12:29:44.252 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38819.log 2024-08-31T12:29:44.252 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40614.log 2024-08-31T12:29:44.252 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42749.log 2024-08-31T12:29:44.252 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44677.log 2024-08-31T12:29:44.252 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30947.log 2024-08-31T12:29:44.252 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30989.log 2024-08-31T12:29:44.253 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31918.log 2024-08-31T12:29:44.253 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31611.log 2024-08-31T12:29:44.253 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72940.log 2024-08-31T12:29:44.253 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24214.log 2024-08-31T12:29:44.253 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40446.log 2024-08-31T12:29:44.253 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64280.log 2024-08-31T12:29:44.254 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73856.log 2024-08-31T12:29:44.254 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40091.log 2024-08-31T12:29:44.254 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39863.log 2024-08-31T12:29:44.254 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42272.log 2024-08-31T12:29:44.254 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66858.log 2024-08-31T12:29:44.254 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32588.log 2024-08-31T12:29:44.254 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40362.log 2024-08-31T12:29:44.254 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43664.log 2024-08-31T12:29:44.255 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40926.log 2024-08-31T12:29:44.255 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24118.log 2024-08-31T12:29:44.255 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24325.log 2024-08-31T12:29:44.255 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24469.log 2024-08-31T12:29:44.255 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35311.log 2024-08-31T12:29:44.255 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39359.log 2024-08-31T12:29:44.255 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72506.log 2024-08-31T12:29:44.256 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39317.log 2024-08-31T12:29:44.256 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24222.log 2024-08-31T12:29:44.256 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73108.log 2024-08-31T12:29:44.256 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72290.log 2024-08-31T12:29:44.256 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31908.log 2024-08-31T12:29:44.256 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73940.log 2024-08-31T12:29:44.256 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40133.log 2024-08-31T12:29:44.257 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43523.log 2024-08-31T12:29:44.257 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42552.log 2024-08-31T12:29:44.257 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40884.log 2024-08-31T12:29:44.257 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35185.log 2024-08-31T12:29:44.257 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72688.log 2024-08-31T12:29:44.257 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73066.log 2024-08-31T12:29:44.257 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35688.log 2024-08-31T12:29:44.257 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32238.log 2024-08-31T12:29:44.258 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-08-31T12:29:44.258 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73444.log 2024-08-31T12:29:44.258 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43229.log 2024-08-31T12:29:44.258 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40656.log 2024-08-31T12:29:44.258 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42935.log 2024-08-31T12:29:44.258 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72982.log 2024-08-31T12:29:44.258 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41593.log 2024-08-31T12:29:44.259 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32253.log 2024-08-31T12:29:44.259 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32255.log 2024-08-31T12:29:44.259 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72206.log 2024-08-31T12:29:44.259 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34933.log 2024-08-31T12:29:44.259 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53681.log 2024-08-31T12:29:44.259 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34235.log 2024-08-31T12:29:44.259 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39905.log 2024-08-31T12:29:44.259 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34642.log 2024-08-31T12:29:44.260 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44164.log 2024-08-31T12:29:44.260 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31443.log 2024-08-31T12:29:44.260 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63570.log 2024-08-31T12:29:44.260 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42893.log 2024-08-31T12:29:44.260 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32252.log 2024-08-31T12:29:44.260 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42104.log 2024-08-31T12:29:44.260 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34009.log 2024-08-31T12:29:44.260 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43750.log 2024-08-31T12:29:44.261 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41010.log 2024-08-31T12:29:44.261 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.33817.log 2024-08-31T12:29:44.261 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39737.log 2024-08-31T12:29:44.261 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47391.log 2024-08-31T12:29:44.261 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40278.log 2024-08-31T12:29:44.262 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-08-31T12:29:44.262 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39569.log 2024-08-31T12:29:44.262 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32787.log 2024-08-31T12:29:44.262 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24337.log 2024-08-31T12:29:44.262 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62847.log 2024-08-31T12:29:44.262 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31917.log 2024-08-31T12:29:44.262 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34546.log 2024-08-31T12:29:44.262 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40320.log 2024-08-31T12:29:44.263 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41323.log 2024-08-31T12:29:44.263 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72122.log 2024-08-31T12:29:44.263 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43995.log 2024-08-31T12:29:44.263 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66938.log 2024-08-31T12:29:44.263 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41980.log 2024-08-31T12:29:44.263 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32590.log 2024-08-31T12:29:44.263 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25009.log 2024-08-31T12:29:44.264 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25097.log 2024-08-31T12:29:44.264 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43187.log 2024-08-31T12:29:44.264 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39965.log 2024-08-31T12:29:44.264 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44535.log 2024-08-31T12:29:44.264 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43911.log 2024-08-31T12:29:44.264 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73688.log 2024-08-31T12:29:44.264 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34402.log 2024-08-31T12:29:44.264 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33123.log 2024-08-31T12:29:44.265 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32242.log 2024-08-31T12:29:44.265 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31347.log 2024-08-31T12:29:44.265 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38945.log 2024-08-31T12:29:44.265 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31569.log 2024-08-31T12:29:44.265 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34975.log 2024-08-31T12:29:44.265 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-08-31T12:29:44.265 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38549.log 2024-08-31T12:29:44.265 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40217.log 2024-08-31T12:29:44.266 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24472.log 2024-08-31T12:29:44.266 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44122.log 2024-08-31T12:29:44.266 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23882.log 2024-08-31T12:29:44.266 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73150.log 2024-08-31T12:29:44.266 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44719.log 2024-08-31T12:29:44.266 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41856.log 2024-08-31T12:29:44.268 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-mds.f.log: 93.3% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-08-31T12:29:44.333 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38609.log 2024-08-31T12:29:44.334 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32246.log 2024-08-31T12:29:44.334 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43953.log 2024-08-31T12:29:44.334 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73360.log 2024-08-31T12:29:44.334 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35394.log 2024-08-31T12:29:44.334 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-08-31T12:29:44.334 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32871.log 2024-08-31T12:29:44.335 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34360.log 2024-08-31T12:29:44.335 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34135.log 2024-08-31T12:29:44.335 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24458.log 2024-08-31T12:29:44.335 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43019.log 2024-08-31T12:29:44.335 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39443.log 2024-08-31T12:29:44.335 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43061.log 2024-08-31T12:29:44.335 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-08-31T12:29:44.336 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31075.log 2024-08-31T12:29:44.336 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44451.log 2024-08-31T12:29:44.336 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38507.log 2024-08-31T12:29:44.336 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72080.log 2024-08-31T12:29:44.336 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41197.log 2024-08-31T12:29:44.336 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73024.log 2024-08-31T12:29:44.336 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31305.log 2024-08-31T12:29:44.336 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24142.log 2024-08-31T12:29:44.337 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38735.log 2024-08-31T12:29:44.337 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73814.log 2024-08-31T12:29:44.337 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42707.log 2024-08-31T12:29:44.337 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41239.log 2024-08-31T12:29:44.337 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32509.log 2024-08-31T12:29:44.337 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32511.log 2024-08-31T12:29:44.337 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38861.log 2024-08-31T12:29:44.338 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69936.log 2024-08-31T12:29:44.338 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35101.log 2024-08-31T12:29:44.338 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38777.log 2024-08-31T12:29:44.338 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45921.log 2024-08-31T12:29:44.338 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41136.log 2024-08-31T12:29:44.338 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44249.log 2024-08-31T12:29:44.338 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44957.log 2024-08-31T12:29:44.338 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57793.log 2024-08-31T12:29:44.339 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43834.log 2024-08-31T12:29:44.339 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40488.log 2024-08-31T12:29:44.339 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24347.log 2024-08-31T12:29:44.339 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30845.log 2024-08-31T12:29:44.339 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42188.log 2024-08-31T12:29:44.339 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24322.log 2024-08-31T12:29:44.339 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42977.log 2024-08-31T12:29:44.339 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53760.log 2024-08-31T12:29:44.340 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31031.log 2024-08-31T12:29:44.340 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39029.log 2024-08-31T12:29:44.340 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35143.log 2024-08-31T12:29:44.340 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41509.log 2024-08-31T12:29:44.340 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73234.log 2024-08-31T12:29:44.340 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66710.log 2024-08-31T12:29:44.340 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42594.log 2024-08-31T12:29:44.340 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42851.log 2024-08-31T12:29:44.341 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40842.log 2024-08-31T12:29:44.341 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32241.log 2024-08-31T12:29:44.341 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42324.log 2024-08-31T12:29:44.341 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41762.log 2024-08-31T12:29:44.341 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72164.log 2024-08-31T12:29:44.341 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33231.log 2024-08-31T12:29:44.341 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73646.log 2024-08-31T12:29:44.341 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43103.log 2024-08-31T12:29:44.342 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44079.log 2024-08-31T12:29:44.342 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72730.log 2024-08-31T12:29:44.342 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39611.log 2024-08-31T12:29:44.342 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34317.log 2024-08-31T12:29:44.342 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42468.log 2024-08-31T12:29:44.342 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42384.log 2024-08-31T12:29:44.342 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.48250.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44291.log.gz 2024-08-31T12:29:44.342 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41365.log 2024-08-31T12:29:44.343 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42146.log 2024-08-31T12:29:44.343 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.32241.log: /var/log/ceph/ceph-client.admin.40716.log: /var/log/ceph/ceph-client.admin.33231.log: /var/log/ceph/ceph-client.admin.73318.log: /var/log/ceph/ceph-client.admin.56128.log: /var/log/ceph/ceph-client.admin.69029.log: /var/log/ceph/ceph-client.admin.41365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32913.log 2024-08-31T12:29:44.343 INFO:teuthology.orchestra.run.smithi019.stderr: 87.9% 87.1% -- replaced with /var/log/ceph/ceph-client.admin.73318.log.gz 2024-08-31T12:29:44.343 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40716.log.gz 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56128.log.gz 2024-08-31T12:29:44.343 INFO:teuthology.orchestra.run.smithi019.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.69029.log.gz 2024-08-31T12:29:44.343 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-08-31T12:29:44.343 INFO:teuthology.orchestra.run.smithi019.stderr: 88.5%/var/log/ceph/ceph-client.admin.42146.log: /var/log/ceph/ceph-client.admin.34317.log: -- replaced with /var/log/ceph/ceph-client.admin.41365.log.gz 87.5% -- replaced with /var/log/ceph/ceph-client.admin.32241.log.gz 2024-08-31T12:29:44.343 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-08-31T12:29:44.344 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41677.log 2024-08-31T12:29:44.344 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.73402.log: /var/log/ceph/ceph-client.admin.42230.log: 87.4%/var/log/ceph/ceph-client.admin.39215.log: /var/log/ceph/ceph-client.admin.32913.log: -- replaced with /var/log/ceph/ceph-client.admin.31653.log.gz 89.1% 2024-08-31T12:29:44.344 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42146.log.gz 2024-08-31T12:29:44.344 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39071.log: /var/log/ceph/ceph-client.admin.58515.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.73402.log.gz 2024-08-31T12:29:44.344 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38819.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34317.log.gz 2024-08-31T12:29:44.344 INFO:teuthology.orchestra.run.smithi019.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42230.log.gz 2024-08-31T12:29:44.344 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40614.log: 86.6% 91.0%/var/log/ceph/ceph-client.admin.42749.log: -- replaced with /var/log/ceph/ceph-client.admin.33231.log.gz 2024-08-31T12:29:44.345 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67016.log 2024-08-31T12:29:44.345 INFO:teuthology.orchestra.run.smithi019.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.34177.log.gz 2024-08-31T12:29:44.345 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31912.log.gz 2024-08-31T12:29:44.345 INFO:teuthology.orchestra.run.smithi019.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.58515.log.gz 2024-08-31T12:29:44.345 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.44677.log: /var/log/ceph/ceph-client.admin.72940.log: /var/log/ceph/ceph-client.admin.40446.log: /var/log/ceph/ceph-client.admin.64280.log: /var/log/ceph/ceph-client.admin.41677.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42749.log.gz 2024-08-31T12:29:44.345 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.73856.log: gzip -5 --verbose 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44677.log.gz 2024-08-31T12:29:44.345 INFO:teuthology.orchestra.run.smithi019.stderr: -- 87.0% -- replaced with /var/log/ceph/ceph-client.admin.72940.log.gz 2024-08-31T12:29:44.345 INFO:teuthology.orchestra.run.smithi019.stderr: /var/log/ceph/ceph-client.admin.34588.log 2024-08-31T12:29:44.346 INFO:teuthology.orchestra.run.smithi019.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41677.log.gz 2024-08-31T12:29:44.346 INFO:teuthology.orchestra.run.smithi019.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.64280.log.gz 2024-08-31T12:29:44.346 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40091.log: /var/log/ceph/ceph-client.admin.67016.log: /var/log/ceph/ceph-client.admin.39863.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.73856.log.gz 2024-08-31T12:29:44.346 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.42272.log: /var/log/ceph/ceph-client.admin.66858.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36091.log.gz 2024-08-31T12:29:44.346 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73606.log 2024-08-31T12:29:44.346 INFO:teuthology.orchestra.run.smithi019.stderr: 88.8% 87.2% -- replaced with /var/log/ceph/ceph-client.admin.67016.log.gz -- replaced with /var/log/ceph/ceph-client.admin.39695.log.gz 2024-08-31T12:29:44.346 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-08-31T12:29:44.346 INFO:teuthology.orchestra.run.smithi019.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42272.log.gz 2024-08-31T12:29:44.347 INFO:teuthology.orchestra.run.smithi019.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.66858.log.gz 2024-08-31T12:29:44.347 INFO:teuthology.orchestra.run.smithi019.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36171.log.gz 2024-08-31T12:29:44.347 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.43664.log: /var/log/ceph/ceph-client.admin.40926.log: /var/log/ceph/ceph-client.admin.34588.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35227.log.gz 2024-08-31T12:29:44.347 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72898.log 2024-08-31T12:29:44.347 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.72506.log: /var/log/ceph/ceph-client.admin.39317.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40926.log.gz 2024-08-31T12:29:44.347 INFO:teuthology.orchestra.run.smithi019.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43664.log.gz 2024-08-31T12:29:44.347 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.73108.log: /var/log/ceph/ceph-client.admin.72290.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39401.log.gz 2024-08-31T12:29:44.348 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.73606.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.72506.log.gz 2024-08-31T12:29:44.348 INFO:teuthology.orchestra.run.smithi019.stderr: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72448.log 2024-08-31T12:29:44.348 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39527.log.gz 87.0% -- replaced with /var/log/ceph/ceph-client.admin.73108.log.gz 2024-08-31T12:29:44.348 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-08-31T12:29:44.348 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.73940.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.72290.log.gz 2024-08-31T12:29:44.348 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40133.log: 87.0%/var/log/ceph/ceph-client.admin.43523.log: /var/log/ceph/ceph-client.admin.42552.log: -- replaced with /var/log/ceph/ceph-client.admin.73606.log.gz 2024-08-31T12:29:44.348 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40884.log: /var/log/ceph/ceph-client.admin.72898.log: /var/log/ceph/ceph-client.admin.72688.log: /var/log/ceph/ceph-client.admin.73066.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.73940.log.gz 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43523.log.gz 2024-08-31T12:29:44.349 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-08-31T12:29:44.349 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42552.log.gz 2024-08-31T12:29:44.349 INFO:teuthology.orchestra.run.smithi019.stderr:gzip 87.2% -- replaced with /var/log/ceph/ceph-client.admin.72898.log.gz 2024-08-31T12:29:44.349 INFO:teuthology.orchestra.run.smithi019.stderr: -5 --verbose/var/log/ceph/ceph-client.admin.35688.log: -- 87.1% -- replaced with /var/log/ceph/ceph-client.admin.72688.log.gz 2024-08-31T12:29:44.349 INFO:teuthology.orchestra.run.smithi019.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.40884.log.gz 87.3% 2024-08-31T12:29:44.349 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/ceph-client.admin.73066.log.gz 2024-08-31T12:29:44.349 INFO:teuthology.orchestra.run.smithi019.stderr: /var/log/ceph/ceph-client.admin.24808.log 2024-08-31T12:29:44.349 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.72448.log: /var/log/ceph/ceph-mds.a.log: /var/log/ceph/ceph-client.admin.73444.log: /var/log/ceph/ceph-client.admin.43229.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39215.log.gz 2024-08-31T12:29:44.350 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40656.log: /var/log/ceph/ceph-client.admin.42935.log: /var/log/ceph/ceph-client.admin.72982.log: /var/log/ceph/ceph-client.admin.41593.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.32913.log.gz 2024-08-31T12:29:44.350 INFO:teuthology.orchestra.run.smithi019.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.38819.log.gz 2024-08-31T12:29:44.350 INFO:teuthology.orchestra.run.smithi019.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.73444.log.gz 2024-08-31T12:29:44.350 INFO:teuthology.orchestra.run.smithi019.stderr: 87.6%/var/log/ceph/ceph-client.admin.39905.log: -- replaced with /var/log/ceph/ceph-client.admin.43229.log.gz 2024-08-31T12:29:44.350 INFO:teuthology.orchestra.run.smithi019.stderr: 88.8%/var/log/ceph/ceph-client.admin.44164.log: -- replaced with /var/log/ceph/ceph-client.admin.39863.log.gz 87.1% 2024-08-31T12:29:44.350 INFO:teuthology.orchestra.run.smithi019.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42935.log.gz 2024-08-31T12:29:44.350 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/ceph-client.admin.72982.log.gz 2024-08-31T12:29:44.351 INFO:teuthology.orchestra.run.smithi019.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.40362.log.gz 2024-08-31T12:29:44.351 INFO:teuthology.orchestra.run.smithi019.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.72448.log.gz 2024-08-31T12:29:44.351 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.e.log 2024-08-31T12:29:44.351 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.63570.log: /var/log/ceph/ceph-client.admin.42893.log: 87.0%/var/log/ceph/ceph-client.admin.72206.log: /var/log/ceph/ceph-client.admin.42104.log: /var/log/ceph/ceph-client.admin.53681.log: /var/log/ceph/ceph-client.admin.24808.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41593.log.gz 2024-08-31T12:29:44.351 INFO:teuthology.orchestra.run.smithi019.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.63570.log.gz 2024-08-31T12:29:44.351 INFO:teuthology.orchestra.run.smithi019.stderr:gzip 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42893.log.gz 2024-08-31T12:29:44.351 INFO:teuthology.orchestra.run.smithi019.stderr: -5 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42104.log.gz 2024-08-31T12:29:44.352 INFO:teuthology.orchestra.run.smithi019.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.72206.log.gz 2024-08-31T12:29:44.352 INFO:teuthology.orchestra.run.smithi019.stderr: --verbose -- /var/log/ceph/ceph-client.admin.42022.log 2024-08-31T12:29:44.352 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44164.log.gz 2024-08-31T12:29:44.352 INFO:teuthology.orchestra.run.smithi019.stderr: 89.2%/var/log/ceph/ceph-client.admin.43750.log: -- replaced with /var/log/ceph/ceph-client.admin.53681.log.gz/var/log/ceph/ceph-client.admin.41010.log: 2024-08-31T12:29:44.352 INFO:teuthology.orchestra.run.smithi019.stderr: 85.8%/var/log/ceph/ceph-client.0.33817.log: /var/log/ceph/ceph-mds.e.log: -- replaced with /var/log/ceph/ceph-client.admin.24346.log.gz 86.4% 2024-08-31T12:29:44.352 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/ceph-client.admin.24471.log.gz 2024-08-31T12:29:44.352 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39737.log: 85.8% 85.6% -- replaced with /var/log/ceph/ceph-client.admin.24465.log.gz 2024-08-31T12:29:44.352 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/ceph-client.admin.24475.log.gz 2024-08-31T12:29:44.353 INFO:teuthology.orchestra.run.smithi019.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43750.log.gz 2024-08-31T12:29:44.353 INFO:teuthology.orchestra.run.smithi019.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.41010.log.gz 2024-08-31T12:29:44.353 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32589.log 2024-08-31T12:29:44.353 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40278.log: /var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.47391.log: /var/log/ceph/ceph-client.admin.39569.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.34588.log.gz 2024-08-31T12:29:44.353 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40320.log: /var/log/ceph/ceph-client.admin.62847.log: /var/log/ceph/ceph-client.admin.42022.log: /var/log/ceph/ceph-client.admin.41323.log: /var/log/ceph/ceph-client.admin.72122.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.47391.log.gz 2024-08-31T12:29:44.353 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.43995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43622.log 2024-08-31T12:29:44.353 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.62847.log.gz 2024-08-31T12:29:44.353 INFO:teuthology.orchestra.run.smithi019.stderr: 88.6% 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41323.log.gz 2024-08-31T12:29:44.354 INFO:teuthology.orchestra.run.smithi019.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.72122.log.gz 2024-08-31T12:29:44.354 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42022.log.gz/var/log/ceph/ceph-client.admin.66938.log: 2024-08-31T12:29:44.354 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41980.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43995.log.gz 2024-08-31T12:29:44.354 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.43187.log: /var/log/ceph/ceph-client.admin.39965.log: /var/log/ceph/ceph-client.admin.44535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34486.log 2024-08-31T12:29:44.354 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.43911.log: /var/log/ceph/ceph-client.admin.73688.log: /var/log/ceph/ceph-client.admin.38945.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.66938.log.gz 2024-08-31T12:29:44.354 INFO:teuthology.orchestra.run.smithi019.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41980.log.gz 2024-08-31T12:29:44.354 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-osd.2.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43187.log.gz/var/log/ceph/ceph-client.admin.43622.log: 2024-08-31T12:29:44.355 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44852.log 2024-08-31T12:29:44.355 INFO:teuthology.orchestra.run.smithi019.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.73688.log.gz 2024-08-31T12:29:44.355 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40217.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43911.log.gz 2024-08-31T12:29:44.355 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.44122.log: 87.7%/var/log/ceph/ceph-client.admin.73150.log: -- replaced with /var/log/ceph/ceph-client.admin.33399.log.gz 2024-08-31T12:29:44.355 INFO:teuthology.orchestra.run.smithi019.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43622.log.gz 2024-08-31T12:29:44.355 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24037.log 2024-08-31T12:29:44.355 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.44719.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.44122.log.gz 2024-08-31T12:29:44.355 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41856.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.73150.log.gz 2024-08-31T12:29:44.356 INFO:teuthology.orchestra.run.smithi019.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44535.log.gz 2024-08-31T12:29:44.356 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38609.log: /var/log/ceph/ceph-client.admin.34486.log: /var/log/ceph/ceph-client.admin.43953.log: /var/log/ceph/ceph-client.admin.73360.log: /var/log/ceph/ceph-client.admin.35394.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44719.log.gzgzip 2024-08-31T12:29:44.356 INFO:teuthology.orchestra.run.smithi019.stderr: -5 --verbose --/var/log/ceph/ceph-mgr.y.log: /var/log/ceph/ceph-client.admin.23994.log 2024-08-31T12:29:44.356 INFO:teuthology.orchestra.run.smithi019.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41856.log.gz 2024-08-31T12:29:44.356 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.44852.log: /var/log/ceph/ceph-client.admin.43019.log: /var/log/ceph/ceph-client.admin.39443.log: /var/log/ceph/ceph-client.admin.43061.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.73360.log.gz 2024-08-31T12:29:44.356 INFO:teuthology.orchestra.run.smithi019.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43953.log.gz 2024-08-31T12:29:44.356 INFO:teuthology.orchestra.run.smithi019.stderr:gzip 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44852.log.gz 2024-08-31T12:29:44.357 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph.log: -5/var/log/ceph/ceph-client.admin.44451.log: --verbose 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43019.log.gz 2024-08-31T12:29:44.357 INFO:teuthology.orchestra.run.smithi019.stderr: -- /var/log/ceph/ceph-client.admin.33165.log 2024-08-31T12:29:44.357 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.24037.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43061.log.gz 2024-08-31T12:29:44.357 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38507.log: /var/log/ceph/ceph-client.admin.72080.log: /var/log/ceph/ceph-client.admin.41197.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44451.log.gz 2024-08-31T12:29:44.357 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43373.log 2024-08-31T12:29:44.357 INFO:teuthology.orchestra.run.smithi019.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.72080.log.gz 2024-08-31T12:29:44.357 INFO:teuthology.orchestra.run.smithi019.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41197.log.gz 2024-08-31T12:29:44.358 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.73024.log: /var/log/ceph/ceph-client.admin.38735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33297.log 2024-08-31T12:29:44.358 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.73814.log: /var/log/ceph/ceph-client.admin.42707.log: /var/log/ceph/ceph-client.admin.41239.log: /var/log/ceph/ceph-client.admin.38861.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.73024.log.gz 2024-08-31T12:29:44.358 INFO:teuthology.orchestra.run.smithi019.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.73814.log.gz 2024-08-31T12:29:44.358 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.43373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31695.log 2024-08-31T12:29:44.358 INFO:teuthology.orchestra.run.smithi019.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42707.log.gz 2024-08-31T12:29:44.358 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.69936.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41239.log.gz 2024-08-31T12:29:44.358 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38777.log: /var/log/ceph/ceph-client.admin.45921.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.43373.log.gz 2024-08-31T12:29:44.358 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41136.log: /var/log/ceph/ceph-client.admin.33297.log: /var/log/ceph/ceph-client.admin.44249.log: /var/log/ceph/ceph-client.admin.44957.log: gzip -5 87.1% -- replaced with /var/log/ceph/ceph-client.admin.45921.log.gz 2024-08-31T12:29:44.359 INFO:teuthology.orchestra.run.smithi019.stderr: --verbose 88.6% -- /var/log/ceph/ceph-client.admin.33039.log 2024-08-31T12:29:44.359 INFO:teuthology.orchestra.run.smithi019.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.69936.log.gz 2024-08-31T12:29:44.359 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41136.log.gz 2024-08-31T12:29:44.359 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.57793.log: /var/log/ceph/ceph-client.admin.43834.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44957.log.gz 2024-08-31T12:29:44.359 INFO:teuthology.orchestra.run.smithi019.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44249.log.gz 2024-08-31T12:29:44.359 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31737.log 2024-08-31T12:29:44.359 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40488.log: 87.0% 76.5% -- replaced with /var/log/ceph/ceph-client.admin.23910.log.gz 2024-08-31T12:29:44.360 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57793.log.gz/var/log/ceph/ceph-client.admin.42188.log: 2024-08-31T12:29:44.360 INFO:teuthology.orchestra.run.smithi019.stderr: 87.6% 75.5% -- replaced with /var/log/ceph/ceph-client.admin.23938.log.gz 2024-08-31T12:29:44.360 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43834.log.gz 76.7% -- replaced with /var/log/ceph/ceph-client.admin.23966.log.gz 2024-08-31T12:29:44.360 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-08-31T12:29:44.360 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.42977.log: /var/log/ceph/ceph-client.admin.33039.log: /var/log/ceph/ceph-client.admin.53760.log: /var/log/ceph/ceph-client.admin.39029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38651.log 2024-08-31T12:29:44.360 INFO:teuthology.orchestra.run.smithi019.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42188.log.gz 2024-08-31T12:29:44.360 INFO:teuthology.orchestra.run.smithi019.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.31137.log.gz 2024-08-31T12:29:44.361 INFO:teuthology.orchestra.run.smithi019.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.31221.log.gz 2024-08-31T12:29:44.361 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41509.log: /var/log/ceph/ceph-client.admin.31737.log: /var/log/ceph/ceph-client.admin.73234.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53760.log.gz 2024-08-31T12:29:44.361 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.66710.log: 90.5%/var/log/ceph/ceph-client.admin.42594.log: -- replaced with /var/log/ceph/ceph-client.admin.42977.log.gz 2024-08-31T12:29:44.361 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35269.log 2024-08-31T12:29:44.361 INFO:teuthology.orchestra.run.smithi019.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41509.log.gz 2024-08-31T12:29:44.361 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.42851.log: /var/log/ceph/ceph-client.admin.40842.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.73234.log.gz 2024-08-31T12:29:44.361 INFO:teuthology.orchestra.run.smithi019.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.66710.log.gz 2024-08-31T12:29:44.362 INFO:teuthology.orchestra.run.smithi019.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42594.log.gz 2024-08-31T12:29:44.362 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38651.log: /var/log/ceph/ceph-client.admin.41762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34891.log 2024-08-31T12:29:44.362 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.72164.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42851.log.gz 2024-08-31T12:29:44.362 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.42324.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40842.log.gz 2024-08-31T12:29:44.362 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.35269.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.72164.log.gz 2024-08-31T12:29:44.362 INFO:teuthology.orchestra.run.smithi019.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.41762.log.gz 2024-08-31T12:29:44.362 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.73646.log: /var/log/ceph/ceph-client.admin.43103.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42324.log.gz 2024-08-31T12:29:44.363 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.44079.log: /var/log/ceph/ceph-client.admin.72730.log: gzip 87.1% -- replaced with /var/log/ceph/ceph-client.admin.73646.log.gz 2024-08-31T12:29:44.363 INFO:teuthology.orchestra.run.smithi019.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.59326.log 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43103.log.gz 2024-08-31T12:29:44.363 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-08-31T12:29:44.363 INFO:teuthology.orchestra.run.smithi019.stderr: 86.9%/var/log/ceph/ceph-client.admin.34891.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.72730.log.gz 2024-08-31T12:29:44.363 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44079.log.gz 2024-08-31T12:29:44.363 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39611.log: /var/log/ceph/ceph-client.admin.42468.log: /var/log/ceph/ceph-client.admin.33967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41938.log 2024-08-31T12:29:44.363 INFO:teuthology.orchestra.run.smithi019.stderr: 87.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-08-31T12:29:44.364 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.31179.log: /var/log/ceph/ceph-client.admin.31919.log: /var/log/ceph/ceph-client.admin.34051.log: /var/log/ceph/ceph-client.admin.30947.log: /var/log/ceph/ceph-client.admin.30613.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42468.log.gz 2024-08-31T12:29:44.364 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.34727.log: /var/log/ceph/ceph-client.admin.34975.log: /var/log/ceph/ceph-client.admin.30989.log: /var/log/ceph/ceph-client.admin.34360.log: gzip -5 --verbose --/var/log/ceph/ceph-client.admin.31611.log: /var/log/ceph/ceph-client.admin.35059.log 2024-08-31T12:29:44.364 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.24458.log: /var/log/ceph/ceph-client.admin.59326.log: /var/log/ceph/ceph-client.admin.31918.log: /var/log/ceph/ceph-client.admin.31031.log: /var/log/ceph/ceph-client.admin.32588.log: /var/log/ceph/ceph-client.admin.24214.log: /var/log/ceph/ceph-client.admin.24118.log: /var/log/ceph/ceph-client.admin.24469.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48250.log.gz 2024-08-31T12:29:44.364 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.24325.log: /var/log/ceph/ceph-client.admin.41938.log: /var/log/ceph/ceph-client.admin.35143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72332.log 88.5% 2024-08-31T12:29:44.364 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/ceph-client.admin.59326.log.gz 2024-08-31T12:29:44.364 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.35311.log: /var/log/ceph/ceph-client.admin.24222.log: /var/log/ceph/ceph-client.admin.32238.log: /var/log/ceph/ceph-client.admin.31908.log: /var/log/ceph/ceph-client.admin.35185.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41938.log.gz 2024-08-31T12:29:44.364 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.32253.log: /var/log/ceph/ceph-client.admin.32255.log: /var/log/ceph/ceph-client.admin.34235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72772.log 2024-08-31T12:29:44.365 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.34933.log: /var/log/ceph/ceph-client.admin.31443.log: /var/log/ceph/ceph-client.admin.34642.log: /var/log/ceph/ceph-client.admin.34009.log: /var/log/ceph/ceph-client.admin.32252.log: /var/log/ceph/ceph-client.admin.24337.log: /var/log/ceph/ceph-client.admin.32787.log: /var/log/ceph/ceph-client.admin.31917.log: /var/log/ceph/ceph-client.admin.34546.log: /var/log/ceph/ceph-client.admin.25009.log: /var/log/ceph/ceph-client.admin.72332.log: /var/log/ceph/ceph-client.admin.32590.log: /var/log/ceph/ceph-client.admin.25097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42636.log 2024-08-31T12:29:44.365 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.33123.log: /var/log/ceph/ceph-client.admin.34402.log: /var/log/ceph/ceph-client.admin.32242.log: /var/log/ceph/ceph-client.admin.42384.log: /var/log/ceph/ceph-client.admin.31569.log: /var/log/ceph/ceph-client.admin.31347.log: /var/log/ceph/ceph-client.admin.23882.log: /var/log/ceph/ceph-client.admin.24472.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23882.log.gz 2024-08-31T12:29:44.365 INFO:teuthology.orchestra.run.smithi019.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39173.log 2024-08-31T12:29:44.365 INFO:teuthology.orchestra.run.smithi019.stderr: 90.9% 87.4% -- replaced with /var/log/ceph/ceph-client.admin.72332.log.gz -- replaced with /var/log/ceph/ceph-client.admin.38693.log.gz 2024-08-31T12:29:44.365 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-08-31T12:29:44.365 INFO:teuthology.orchestra.run.smithi019.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42384.log.gz 2024-08-31T12:29:44.365 INFO:teuthology.orchestra.run.smithi019.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39071.log.gz 2024-08-31T12:29:44.365 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.32246.log: /var/log/ceph/ceph-client.admin.32871.log: /var/log/ceph/ceph-client.admin.72772.log: /var/log/ceph/ceph-client.admin.42636.log: /var/log/ceph/ceph-client.admin.34135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70006.log 2024-08-31T12:29:44.366 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40007.log.gz 2024-08-31T12:29:44.366 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.72772.log.gz 2024-08-31T12:29:44.366 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.31075.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42636.log.gz 2024-08-31T12:29:44.366 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.31305.log: /var/log/ceph/ceph-client.admin.39173.log: /var/log/ceph/ceph-client.admin.24142.log: /var/log/ceph/ceph-client.admin.32511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44037.log 2024-08-31T12:29:44.366 INFO:teuthology.orchestra.run.smithi019.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.40049.log.gz 2024-08-31T12:29:44.366 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.32509.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.40091.log.gz 2024-08-31T12:29:44.366 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.24347.log: /var/log/ceph/ceph-client.admin.35101.log: /var/log/ceph/ceph-client.admin.30845.log: /var/log/ceph/ceph-client.admin.24322.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.39359.log.gz 2024-08-31T12:29:44.367 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.31695.log: /var/log/ceph/ceph-client.admin.23994.log: /var/log/ceph/ceph-client.admin.70006.log: /var/log/ceph/ceph-client.admin.33165.log: /var/log/ceph/ceph-client.admin.35059.log: /var/log/ceph/ceph-client.admin.32589.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39317.log.gz 2024-08-31T12:29:44.367 INFO:teuthology.orchestra.run.smithi019.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40133.log.gz 2024-08-31T12:29:44.367 INFO:teuthology.orchestra.run.smithi019.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40175.log.gz 2024-08-31T12:29:44.367 INFO:teuthology.orchestra.run.smithi019.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.70006.log.gz 2024-08-31T12:29:44.367 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-08-31T12:29:44.367 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40572.log.gz 2024-08-31T12:29:44.367 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.44037.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40614.log.gz 2024-08-31T12:29:44.367 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40530.log 2024-08-31T12:29:44.368 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-osd.3.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.35688.log.gz 2024-08-31T12:29:44.368 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23825.log 2024-08-31T12:29:44.368 INFO:teuthology.orchestra.run.smithi019.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44037.log.gz 2024-08-31T12:29:44.368 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40530.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40656.log.gz 2024-08-31T12:29:44.368 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72248.log 2024-08-31T12:29:44.368 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.23825.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-08-31T12:29:44.368 INFO:teuthology.orchestra.run.smithi019.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23825.log.gz 2024-08-31T12:29:44.368 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.72248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43415.log 2024-08-31T12:29:44.369 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-mds.c.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39905.log.gz 2024-08-31T12:29:44.369 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46656.log 2024-08-31T12:29:44.369 INFO:teuthology.orchestra.run.smithi019.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.72248.log.gz 2024-08-31T12:29:44.369 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.43415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39779.log 2024-08-31T12:29:44.369 INFO:teuthology.orchestra.run.smithi019.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43415.log.gz 2024-08-31T12:29:44.369 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.46656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41094.log 2024-08-31T12:29:44.369 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39779.log: 87.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.46656.log.gz 2024-08-31T12:29:44.370 INFO:teuthology.orchestra.run.smithi019.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.55882.log 2024-08-31T12:29:44.370 INFO:teuthology.orchestra.run.smithi019.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34891.log.gz 2024-08-31T12:29:44.370 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42510.log 2024-08-31T12:29:44.370 INFO:teuthology.orchestra.run.smithi019.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38507.log.gz 2024-08-31T12:29:44.370 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.55882.log: 89.1% 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39443.log.gz 2024-08-31T12:29:44.370 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41094.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24464.log 2024-08-31T12:29:44.370 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-08-31T12:29:44.371 INFO:teuthology.orchestra.run.smithi019.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.39611.log.gz 2024-08-31T12:29:44.371 INFO:teuthology.orchestra.run.smithi019.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.39569.log.gz 2024-08-31T12:29:44.371 INFO:teuthology.orchestra.run.smithi019.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39737.log.gz 2024-08-31T12:29:44.371 INFO:teuthology.orchestra.run.smithi019.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55882.log.gz 2024-08-31T12:29:44.371 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.42510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41425.log 2024-08-31T12:29:44.371 INFO:teuthology.orchestra.run.smithi019.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.42510.log.gz 2024-08-31T12:29:44.371 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.24464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39131.log 2024-08-31T12:29:44.372 INFO:teuthology.orchestra.run.smithi019.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40278.log.gz 2024-08-31T12:29:44.372 INFO:teuthology.orchestra.run.smithi019.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.38861.log.gz 2024-08-31T12:29:44.372 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30730.log 2024-08-31T12:29:44.372 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39131.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41425.log.gz 2024-08-31T12:29:44.372 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32591.log 2024-08-31T12:29:44.372 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.30730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35017.log 2024-08-31T12:29:44.372 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.32591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31527.log 2024-08-31T12:29:44.372 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.35017.log: /var/log/ceph/ceph-client.admin.31527.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38945.log.gz 2024-08-31T12:29:44.373 INFO:teuthology.orchestra.run.smithi019.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39029.log.gz 2024-08-31T12:29:44.373 INFO:teuthology.orchestra.run.smithi019.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39779.log.gz 2024-08-31T12:29:44.373 INFO:teuthology.orchestra.run.smithi019.stderr: 77.6% -- replaced with /var/log/ceph/ceph-client.admin.23994.log.gz 2024-08-31T12:29:44.373 INFO:teuthology.orchestra.run.smithi019.stderr: 77.3% -- replaced with /var/log/ceph/ceph-client.admin.24037.log.gz 2024-08-31T12:29:44.373 INFO:teuthology.orchestra.run.smithi019.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.30947.log.gz 2024-08-31T12:29:44.373 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.31179.log.gz 2024-08-31T12:29:44.373 INFO:teuthology.orchestra.run.smithi019.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35394.log.gz 2024-08-31T12:29:44.374 INFO:teuthology.orchestra.run.smithi019.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38735.log.gz 2024-08-31T12:29:44.374 INFO:teuthology.orchestra.run.smithi019.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39131.log.gz 2024-08-31T12:29:44.374 INFO:teuthology.orchestra.run.smithi019.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39173.log.gz 2024-08-31T12:29:44.374 INFO:teuthology.orchestra.run.smithi019.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40217.log.gz 2024-08-31T12:29:44.374 INFO:teuthology.orchestra.run.smithi019.stderr: 87.5% 87.3% -- replaced with /var/log/ceph/ceph-client.admin.33967.log.gz -- replaced with /var/log/ceph/ceph-client.admin.34009.log.gz 2024-08-31T12:29:44.374 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-08-31T12:29:44.374 INFO:teuthology.orchestra.run.smithi019.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38651.log.gz 2024-08-31T12:29:44.375 INFO:teuthology.orchestra.run.smithi019.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39965.log.gz 2024-08-31T12:29:44.375 INFO:teuthology.orchestra.run.smithi019.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40320.log.gz 2024-08-31T12:29:44.375 INFO:teuthology.orchestra.run.smithi019.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.35017.log.gz 2024-08-31T12:29:44.375 INFO:teuthology.orchestra.run.smithi019.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35059.log.gz 2024-08-31T12:29:44.375 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.31611.log.gz 2024-08-31T12:29:44.375 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.31737.log.gz 2024-08-31T12:29:44.375 INFO:teuthology.orchestra.run.smithi019.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.31695.log.gz 2024-08-31T12:29:44.376 INFO:teuthology.orchestra.run.smithi019.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.31908.log.gz 2024-08-31T12:29:44.376 INFO:teuthology.orchestra.run.smithi019.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.31919.log.gz 2024-08-31T12:29:44.376 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.31917.log.gz 2024-08-31T12:29:44.376 INFO:teuthology.orchestra.run.smithi019.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.32242.log.gz 2024-08-31T12:29:44.376 INFO:teuthology.orchestra.run.smithi019.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.32238.log.gz 2024-08-31T12:29:44.376 INFO:teuthology.orchestra.run.smithi019.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.32255.log.gz 2024-08-31T12:29:44.376 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.32246.log.gz 87.5% 2024-08-31T12:29:44.376 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/ceph-client.admin.32589.log.gz 2024-08-31T12:29:44.377 INFO:teuthology.orchestra.run.smithi019.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.32511.log.gz 2024-08-31T12:29:44.377 INFO:teuthology.orchestra.run.smithi019.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.32509.log.gz 2024-08-31T12:29:44.377 INFO:teuthology.orchestra.run.smithi019.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.32590.log.gz 2024-08-31T12:29:44.377 INFO:teuthology.orchestra.run.smithi019.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.32591.log.gz 2024-08-31T12:29:44.377 INFO:teuthology.orchestra.run.smithi019.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.32787.log.gz 2024-08-31T12:29:44.377 INFO:teuthology.orchestra.run.smithi019.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.32871.log.gz 2024-08-31T12:29:44.377 INFO:teuthology.orchestra.run.smithi019.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.33039.log.gz 2024-08-31T12:29:44.377 INFO:teuthology.orchestra.run.smithi019.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.33165.log.gz 2024-08-31T12:29:44.378 INFO:teuthology.orchestra.run.smithi019.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.34642.log.gz 2024-08-31T12:29:44.378 INFO:teuthology.orchestra.run.smithi019.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34727.log.gz 2024-08-31T12:29:44.378 INFO:teuthology.orchestra.run.smithi019.stderr: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.24118.log.gz 2024-08-31T12:29:44.378 INFO:teuthology.orchestra.run.smithi019.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.24142.log.gz 2024-08-31T12:29:44.378 INFO:teuthology.orchestra.run.smithi019.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.31075.log.gz 2024-08-31T12:29:44.378 INFO:teuthology.orchestra.run.smithi019.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.30989.log.gz 2024-08-31T12:29:44.378 INFO:teuthology.orchestra.run.smithi019.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.31031.log.gz 2024-08-31T12:29:44.379 INFO:teuthology.orchestra.run.smithi019.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.34360.log.gz 2024-08-31T12:29:44.379 INFO:teuthology.orchestra.run.smithi019.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.34051.log.gz 2024-08-31T12:29:44.379 INFO:teuthology.orchestra.run.smithi019.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40446.log.gz 2024-08-31T12:29:44.379 INFO:teuthology.orchestra.run.smithi019.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40530.log.gz 2024-08-31T12:29:44.379 INFO:teuthology.orchestra.run.smithi019.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40488.log.gz 2024-08-31T12:29:44.379 INFO:teuthology.orchestra.run.smithi019.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.34135.log.gz 2024-08-31T12:29:44.379 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34235.log.gz 2024-08-31T12:29:44.379 INFO:teuthology.orchestra.run.smithi019.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34933.log.gz 2024-08-31T12:29:44.380 INFO:teuthology.orchestra.run.smithi019.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34975.log.gz 2024-08-31T12:29:44.380 INFO:teuthology.orchestra.run.smithi019.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35269.log.gz 2024-08-31T12:29:44.380 INFO:teuthology.orchestra.run.smithi019.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.31569.log.gz 2024-08-31T12:29:44.380 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.31918.log.gz 2024-08-31T12:29:44.380 INFO:teuthology.orchestra.run.smithi019.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.32253.log.gz 2024-08-31T12:29:44.380 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.32252.log.gz 2024-08-31T12:29:44.382 INFO:teuthology.orchestra.run.smithi043.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-08-31T12:29:44.382 INFO:teuthology.orchestra.run.smithi043.stderr: 93.4% -- replaced with /var/log/ceph/ceph-mds.f.log.gz 2024-08-31T12:29:44.384 INFO:teuthology.orchestra.run.smithi019.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.32588.log.gz 2024-08-31T12:29:44.384 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.33123.log.gz 2024-08-31T12:29:44.384 INFO:teuthology.orchestra.run.smithi019.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.24472.log.gz 2024-08-31T12:29:44.384 INFO:teuthology.orchestra.run.smithi019.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.24464.log.gz 2024-08-31T12:29:44.384 INFO:teuthology.orchestra.run.smithi019.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.24469.log.gz 2024-08-31T12:29:44.384 INFO:teuthology.orchestra.run.smithi019.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.24337.log.gz 2024-08-31T12:29:44.384 INFO:teuthology.orchestra.run.smithi019.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.24458.log.gz 2024-08-31T12:29:44.385 INFO:teuthology.orchestra.run.smithi019.stderr: 94.3% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-08-31T12:29:44.385 INFO:teuthology.orchestra.run.smithi019.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.24325.log.gz 2024-08-31T12:29:44.385 INFO:teuthology.orchestra.run.smithi019.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.24347.log.gz 2024-08-31T12:29:44.385 INFO:teuthology.orchestra.run.smithi019.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.24322.log.gz 2024-08-31T12:29:44.385 INFO:teuthology.orchestra.run.smithi019.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.24222.log.gz 2024-08-31T12:29:44.385 INFO:teuthology.orchestra.run.smithi019.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.25009.log.gz 2024-08-31T12:29:44.385 INFO:teuthology.orchestra.run.smithi019.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.25097.log.gz 2024-08-31T12:29:44.385 INFO:teuthology.orchestra.run.smithi019.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.24214.log.gz 2024-08-31T12:29:44.386 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.30730.log.gz 2024-08-31T12:29:44.386 INFO:teuthology.orchestra.run.smithi019.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.30845.log.gz 2024-08-31T12:29:44.386 INFO:teuthology.orchestra.run.smithi019.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.33297.log.gz 2024-08-31T12:29:44.386 INFO:teuthology.orchestra.run.smithi019.stderr: 94.1% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-08-31T12:29:44.386 INFO:teuthology.orchestra.run.smithi019.stderr: 94.2% -- replaced with /var/log/ceph/ceph-mds.e.log.gz 2024-08-31T12:29:44.395 INFO:teuthology.orchestra.run.smithi019.stderr: 89.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-08-31T12:29:44.396 INFO:teuthology.orchestra.run.smithi019.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.24808.log.gz 2024-08-31T12:29:44.399 INFO:teuthology.orchestra.run.smithi019.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.30613.log.gz 2024-08-31T12:29:44.439 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.31485.log.gz 2024-08-31T12:29:44.439 INFO:teuthology.orchestra.run.smithi019.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.31527.log.gz 2024-08-31T12:29:44.440 INFO:teuthology.orchestra.run.smithi019.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.35311.log.gz 2024-08-31T12:29:44.447 INFO:teuthology.orchestra.run.smithi019.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.31347.log.gz 2024-08-31T12:29:44.447 INFO:teuthology.orchestra.run.smithi019.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.31305.log.gz 2024-08-31T12:29:44.448 INFO:teuthology.orchestra.run.smithi019.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.31400.log.gz 2024-08-31T12:29:44.448 INFO:teuthology.orchestra.run.smithi019.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.31443.log.gz 2024-08-31T12:29:44.450 INFO:teuthology.orchestra.run.smithi019.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34402.log.gz 2024-08-31T12:29:44.450 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34546.log.gz 2024-08-31T12:29:44.450 INFO:teuthology.orchestra.run.smithi019.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.34486.log.gz 2024-08-31T12:29:44.450 INFO:teuthology.orchestra.run.smithi019.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38777.log.gz 2024-08-31T12:29:44.464 INFO:teuthology.orchestra.run.smithi043.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-08-31T12:29:44.483 INFO:teuthology.orchestra.run.smithi019.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38549.log.gz 2024-08-31T12:29:44.483 INFO:teuthology.orchestra.run.smithi019.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38609.log.gz 2024-08-31T12:29:44.491 INFO:teuthology.orchestra.run.smithi019.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35101.log.gz 2024-08-31T12:29:44.491 INFO:teuthology.orchestra.run.smithi019.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.35143.log.gz 2024-08-31T12:29:44.491 INFO:teuthology.orchestra.run.smithi019.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.35185.log.gz 2024-08-31T12:29:44.539 INFO:teuthology.orchestra.run.smithi043.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-08-31T12:29:44.627 INFO:teuthology.orchestra.run.smithi019.stderr: 94.2% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-08-31T12:29:46.117 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c has been restored 2024-08-31T12:29:46.117 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e has been restored 2024-08-31T12:29:46.117 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-08-31T12:29:46.117 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2024-08-31T12:29:46.117 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f has been restored 2024-08-31T12:29:46.385 INFO:teuthology.orchestra.run.smithi019.stderr: 90.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-08-31T12:29:48.153 INFO:teuthology.orchestra.run.smithi043.stderr: 93.2% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-08-31T12:29:50.930 INFO:teuthology.orchestra.run.smithi043.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-08-31T12:29:51.167 INFO:teuthology.orchestra.run.smithi019.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-08-31T12:29:52.689 INFO:teuthology.orchestra.run.smithi043.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-08-31T12:29:53.822 INFO:teuthology.orchestra.run.smithi019.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-08-31T12:29:55.139 INFO:teuthology.orchestra.run.smithi043.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-08-31T12:29:55.499 INFO:teuthology.orchestra.run.smithi043.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-08-31T12:29:55.501 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-08-31T12:29:55.501 INFO:teuthology.orchestra.run.smithi043.stderr:real 0m11.334s 2024-08-31T12:29:55.501 INFO:teuthology.orchestra.run.smithi043.stderr:user 0m41.037s 2024-08-31T12:29:55.501 INFO:teuthology.orchestra.run.smithi043.stderr:sys 0m1.244s 2024-08-31T12:29:57.346 INFO:teuthology.orchestra.run.smithi019.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-08-31T12:29:57.666 INFO:teuthology.orchestra.run.smithi019.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-08-31T12:32:11.604 INFO:teuthology.orchestra.run.smithi019.stderr: 96.1% -- replaced with /var/log/ceph/ceph-client.0.33817.log.gz 2024-08-31T12:32:11.606 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-08-31T12:32:11.606 INFO:teuthology.orchestra.run.smithi019.stderr:real 2m27.460s 2024-08-31T12:32:11.606 INFO:teuthology.orchestra.run.smithi019.stderr:user 2m47.083s 2024-08-31T12:32:11.606 INFO:teuthology.orchestra.run.smithi019.stderr:sys 0m6.748s 2024-08-31T12:32:11.607 INFO:tasks.ceph:Archiving logs... 2024-08-31T12:32:11.607 DEBUG:teuthology.misc:Transferring archived files from smithi019:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-08-30_08:30:37-fs-wip-mchangir-testing-20240828.085847-main-debug-distro-default-smithi/7881510/remote/smithi019/log 2024-08-31T12:32:11.608 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar c -f - -C /var/log/ceph -- . 2024-08-31T12:32:30.343 DEBUG:teuthology.misc:Transferring archived files from smithi043:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-08-30_08:30:37-fs-wip-mchangir-testing-20240828.085847-main-debug-distro-default-smithi/7881510/remote/smithi043/log 2024-08-31T12:32:30.344 DEBUG:teuthology.orchestra.run.smithi043:> sudo tar c -f - -C /var/log/ceph -- . 2024-08-31T12:32:35.810 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-08-31T12:32:35.822 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-08-31T12:32:35.822 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-31T12:32:35.825 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-31T12:32:35.906 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-08-31T12:32:35.919 INFO:teuthology.task.clock:Checking final clock skew... 2024-08-31T12:32:35.920 DEBUG:teuthology.orchestra.run.smithi019:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-31T12:32:35.922 DEBUG:teuthology.orchestra.run.smithi043:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-31T12:32:35.937 INFO:teuthology.orchestra.run.smithi019.stderr:bash: line 1: ntpq: command not found 2024-08-31T12:32:35.939 INFO:teuthology.orchestra.run.smithi043.stderr:bash: line 1: ntpq: command not found 2024-08-31T12:32:35.944 INFO:teuthology.orchestra.run.smithi043.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-31T12:32:35.944 INFO:teuthology.orchestra.run.smithi043.stdout:=============================================================================== 2024-08-31T12:32:35.944 INFO:teuthology.orchestra.run.smithi043.stdout:^* hv01.front.sepia.ceph.com 3 6 377 5 +238us[ +193us] +/- 61ms 2024-08-31T12:32:35.945 INFO:teuthology.orchestra.run.smithi043.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 69 -392us[ -432us] +/- 55ms 2024-08-31T12:32:35.945 INFO:teuthology.orchestra.run.smithi043.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 5 +153us[ +153us] +/- 71ms 2024-08-31T12:32:35.945 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-08-31T12:32:35.954 INFO:teuthology.orchestra.run.smithi019.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-31T12:32:35.954 INFO:teuthology.orchestra.run.smithi019.stdout:=============================================================================== 2024-08-31T12:32:35.954 INFO:teuthology.orchestra.run.smithi019.stdout:^* hv01.front.sepia.ceph.com 3 6 377 2 +272us[ +271us] +/- 61ms 2024-08-31T12:32:35.955 INFO:teuthology.orchestra.run.smithi019.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 5 -396us[ -397us] +/- 56ms 2024-08-31T12:32:35.955 INFO:teuthology.orchestra.run.smithi019.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 6 +178us[ +177us] +/- 71ms 2024-08-31T12:32:35.955 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-08-31T12:32:35.956 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-08-31T12:32:35.967 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-08-31T12:32:35.968 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-08-31T12:32:35.979 DEBUG:teuthology.orchestra.run.smithi019:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-08-31T12:32:36.072 DEBUG:teuthology.orchestra.run.smithi043:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-08-31T12:32:36.127 DEBUG:teuthology.orchestra.run.smithi019:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-08-31T12:32:36.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T12:32:36.156 DEBUG:teuthology.orchestra.run.smithi043:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-08-31T12:32:36.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T12:32:36.186 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-08-31T12:32:36.194 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-08-31T12:32:36.209 INFO:teuthology.task.internal:Duration was 1280.075770 seconds 2024-08-31T12:32:36.209 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-08-31T12:32:36.224 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-08-31T12:32:36.224 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-31T12:32:36.227 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-31T12:32:36.269 INFO:teuthology.orchestra.run.smithi043.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-31T12:32:36.310 INFO:teuthology.orchestra.run.smithi019.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-31T12:32:36.629 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-08-31T12:32:36.629 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi019.front.sepia.ceph.com 2024-08-31T12:32:36.630 DEBUG:teuthology.orchestra.run.smithi019:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-31T12:32:36.652 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi043.front.sepia.ceph.com 2024-08-31T12:32:36.652 DEBUG:teuthology.orchestra.run.smithi043:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-31T12:32:36.702 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-08-31T12:32:36.703 DEBUG:teuthology.orchestra.run.smithi019:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-31T12:32:36.705 DEBUG:teuthology.orchestra.run.smithi043:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-31T12:32:36.771 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-08-31T12:32:36.771 DEBUG:teuthology.orchestra.run.smithi019:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-31T12:32:36.775 DEBUG:teuthology.orchestra.run.smithi043:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-31T12:32:36.880 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-08-31T12:32:36.892 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-08-31T12:32:36.893 DEBUG:teuthology.orchestra.run.smithi019:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-31T12:32:36.925 DEBUG:teuthology.orchestra.run.smithi043:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-31T12:32:36.949 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-08-31T12:32:36.961 DEBUG:teuthology.orchestra.run.smithi019:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-31T12:32:36.967 DEBUG:teuthology.orchestra.run.smithi043:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-31T12:32:37.003 INFO:teuthology.orchestra.run.smithi019.stdout:kernel.core_pattern = core 2024-08-31T12:32:37.012 INFO:teuthology.orchestra.run.smithi043.stdout:kernel.core_pattern = core 2024-08-31T12:32:37.041 DEBUG:teuthology.orchestra.run.smithi019:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-31T12:32:37.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T12:32:37.087 DEBUG:teuthology.orchestra.run.smithi043:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-31T12:32:37.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T12:32:37.101 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-08-31T12:32:37.113 INFO:teuthology.task.internal:Transferring archived files... 2024-08-31T12:32:37.114 DEBUG:teuthology.misc:Transferring archived files from smithi019:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-08-30_08:30:37-fs-wip-mchangir-testing-20240828.085847-main-debug-distro-default-smithi/7881510/remote/smithi019 2024-08-31T12:32:37.114 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-08-31T12:32:37.157 DEBUG:teuthology.misc:Transferring archived files from smithi043:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-08-30_08:30:37-fs-wip-mchangir-testing-20240828.085847-main-debug-distro-default-smithi/7881510/remote/smithi043 2024-08-31T12:32:37.157 DEBUG:teuthology.orchestra.run.smithi043:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-08-31T12:32:37.187 INFO:teuthology.task.internal:Removing archive directory... 2024-08-31T12:32:37.187 DEBUG:teuthology.orchestra.run.smithi019:> rm -rf -- /home/ubuntu/cephtest/archive 2024-08-31T12:32:37.195 DEBUG:teuthology.orchestra.run.smithi043:> rm -rf -- /home/ubuntu/cephtest/archive 2024-08-31T12:32:37.237 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-08-31T12:32:37.250 INFO:teuthology.task.internal:Not uploading archives. 2024-08-31T12:32:37.250 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-08-31T12:32:37.265 INFO:teuthology.task.internal:Tidying up after the test... 2024-08-31T12:32:37.266 DEBUG:teuthology.orchestra.run.smithi019:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-31T12:32:37.268 DEBUG:teuthology.orchestra.run.smithi043:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-31T12:32:37.280 INFO:teuthology.orchestra.run.smithi019.stdout: 262217 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 31 12:32 /home/ubuntu/cephtest 2024-08-31T12:32:37.292 INFO:teuthology.orchestra.run.smithi043.stdout: 262217 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 31 12:32 /home/ubuntu/cephtest 2024-08-31T12:32:37.293 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-08-31T12:32:37.305 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-08-31T12:32:37.339 INFO:teuthology.run:Summary data: description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} duration: 1280.0757701396942 failure_reason: 'Command failed (workunit test suites/ffsb.sh) on smithi019 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d19e6118f7015f0047264006cb0c873528eb3ae6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh''' flavor: default owner: scheduled_mchangir@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=9c6858758b7441539ce94ced4a3719f4 status: fail success: false 2024-08-31T12:32:37.340 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-31T12:32:37.412 INFO:teuthology.run:FAIL